public/obsidian-livesync
1
0
Fork 0
mirror of https://github.com/vrtmrz/obsidian-livesync.git synced 2026-04-29 12:28:35 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add: mirror command

Browse Source
This commit is contained in:
vorotamoroz
2026-03-13 12:55:46 +09:00
parent 29ce9a5df4
commit 0c65b5add9
9 changed files with 288 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files

98
src/apps/cli/README.md
View File

@@ -63,43 +63,43 @@ As you know, the CLI is designed to be used in a headless environment. Hence all
```bash
# Sync local database with CouchDB (no files will be changed).
npm run cli -- /path/to/your-local-database --settings /path/to/settings.json sync
npm run --silent cli -- /path/to/your-local-database --settings /path/to/settings.json sync
# Push files to local database
npm run cli -- /path/to/your-local-database --settings /path/to/settings.json push /your/storage/file.md /vault/path/file.md
npm run --silent cli -- /path/to/your-local-database --settings /path/to/settings.json push /your/storage/file.md /vault/path/file.md
# Pull files from local database
npm run cli -- /path/to/your-local-database --settings /path/to/settings.json pull /vault/path/file.md /your/storage/file.md
npm run --silent cli -- /path/to/your-local-database --settings /path/to/settings.json pull /vault/path/file.md /your/storage/file.md
# Verbose logging
npm run cli -- /path/to/your-local-database --settings /path/to/settings.json --verbose
npm run --silent cli -- /path/to/your-local-database --settings /path/to/settings.json --verbose
# Apply setup URI to settings file (settings only; does not run synchronisation)
npm run cli -- /path/to/your-local-database --settings /path/to/settings.json setup "obsidian://setuplivesync?settings=..."
npm run --silent cli -- /path/to/your-local-database --settings /path/to/settings.json setup "obsidian://setuplivesync?settings=..."
# Put text from stdin into local database
echo "Hello from stdin" | npm run cli -- /path/to/your-local-database --settings /path/to/settings.json put /vault/path/file.md
echo "Hello from stdin" | npm run --silent cli -- /path/to/your-local-database --settings /path/to/settings.json put /vault/path/file.md
# Output a file from local database to stdout
npm run cli -- /path/to/your-local-database --settings /path/to/settings.json cat /vault/path/file.md
npm run --silent cli -- /path/to/your-local-database --settings /path/to/settings.json cat /vault/path/file.md
# Output a specific revision of a file from local database
npm run cli -- /path/to/your-local-database --settings /path/to/settings.json cat-rev /vault/path/file.md 3-abcdef
npm run --silent cli -- /path/to/your-local-database --settings /path/to/settings.json cat-rev /vault/path/file.md 3-abcdef
# Pull a specific revision of a file from local database to local storage
npm run cli -- /path/to/your-local-database --settings /path/to/settings.json pull-rev /vault/path/file.md /your/storage/file.old.md 3-abcdef
npm run --silent cli -- /path/to/your-local-database --settings /path/to/settings.json pull-rev /vault/path/file.md /your/storage/file.old.md 3-abcdef
# List files in local database
npm run cli -- /path/to/your-local-database --settings /path/to/settings.json ls /vault/path/
npm run --silent cli -- /path/to/your-local-database --settings /path/to/settings.json ls /vault/path/
# Show metadata for a file in local database
npm run cli -- /path/to/your-local-database --settings /path/to/settings.json info /vault/path/file.md
npm run --silent cli -- /path/to/your-local-database --settings /path/to/settings.json info /vault/path/file.md
# Mark a file as deleted in local database
npm run cli -- /path/to/your-local-database --settings /path/to/settings.json rm /vault/path/file.md
npm run --silent cli -- /path/to/your-local-database --settings /path/to/settings.json rm /vault/path/file.md
# Resolve conflict by keeping a specific revision
npm run cli -- /path/to/your-local-database --settings /path/to/settings.json resolve /vault/path/file.md 3-abcdef
npm run --silent cli -- /path/to/your-local-database --settings /path/to/settings.json resolve /vault/path/file.md 3-abcdef
```
### Configuration
@@ -159,14 +159,26 @@ Commands:
info <vaultPath> Show file metadata including current and past revisions, conflicts, and chunk list
rm <vaultPath> Mark file as deleted in local database
resolve <vaultPath> <revision> Resolve conflict by keeping the specified revision
mirror <storagePath> <vaultPath> Mirror local file into local database.
```
Run via npm script:
```bash
npm run cli -- [database-path] [options] [command] [command-args]
npm run --silent cli -- [database-path] [options] [command] [command-args]
```
#### Detailed Command Descriptions
##### ls
`ls` lists files in the local database with optional prefix filtering. Output format is:
```vault/path/file.md<TAB>size<TAB>mtime<TAB>revision[*]
```
Note: `*` indicates if the file has conflicts.
##### info
`info` output fields:
- `id`: Document ID
@@ -179,6 +191,38 @@ npm run cli -- [database-path] [options] [command] [command-args]
- `chunks`: Number of chunk IDs
- `children`: Chunk ID list
##### mirror
`mirror` is a command that synchronises your storage with your local vault. It is essentially a process that runs upon startup in Obsidian.
In other words, it performs the following actions:
1. **Precondition checks** — Aborts early if any of the following conditions are not met:
- Settings must be configured (`isConfigured: true`).
- File watching must not be suspended (`suspendFileWatching: false`).
- Remediation mode must be inactive (`maxMTimeForReflectEvents: 0`).
2. **State restoration** — On subsequent runs (after the first successful scan), restores the previous storage state before proceeding.
3. **Expired deletion cleanup** — If `automaticallyDeleteMetadataOfDeletedFiles` is set to a positive number of days, any document that is marked deleted and whose `mtime` is older than the retention period is permanently removed from the local database.
4. **File collection** — Enumerates files from two sources:
- **Storage**: all files under the vault path that pass `isTargetFile`.
- **Local database**: all normal documents (fetched with conflict information) whose paths are valid and pass `isTargetFile`.
- Both collections build case-insensitive ↔ case-sensitive path maps, controlled by `handleFilenameCaseSensitive`.
5. **Categorisation and synchronisation** — The union of both file sets is split into three groups and processed concurrently (up to 10 files at a time):
| Group | Condition | Action |
|---|---|---|
| **UPDATE DATABASE** | File exists in storage only | Store the file into the local database. |
| **UPDATE STORAGE** | File exists in database only | If the entry is active (not deleted) and not conflicted, restore the file from the database to storage. Deleted entries and conflicted entries are skipped. |
| **SYNC DATABASE AND STORAGE** | File exists in both | Compare `mtime` freshness. If storage is newer → write to database (`STORAGE → DB`). If database is newer → restore to storage (`STORAGE ← DB`). If equal → do nothing. Conflicted documents and files exceeding the size limit are always skipped. |
6. **Initialisation flag** — On the very first successful run, writes `initialized = true` to the key-value database so that subsequent runs can restore state in step 2.
Note: `mirror` does not respect file deletions. If a file is deleted in storage, it will be restored on the next `mirror` run. To delete a file, use the `rm` command instead. This is a little inconvenient, but it is intentional behaviour (if we handle this automatically in `mirror`, we should be against a ton of edge cases).
### Planned options:
- `--immediate`: Perform sync after the command (e.g. `push`, `pull`, `put`, `rm`).
@@ -192,9 +236,9 @@ npm run cli -- [database-path] [options] [command] [command-args]
Create default settings, apply a setup URI, then run one sync cycle.
```bash
npm run cli -- init-settings /data/livesync-settings.json
printf '%s\n' "$SETUP_PASSPHRASE" | npm run cli -- /data/vault --settings /data/livesync-settings.json setup "$SETUP_URI"
npm run cli -- /data/vault --settings /data/livesync-settings.json sync
npm run --silent cli -- init-settings /data/livesync-settings.json
printf '%s\n' "$SETUP_PASSPHRASE" | npm run --silent cli -- /data/vault --settings /data/livesync-settings.json setup "$SETUP_URI"
npm run --silent cli -- /data/vault --settings /data/livesync-settings.json sync
```
### 2. Scripted import and export
@@ -202,8 +246,8 @@ npm run cli -- /data/vault --settings /data/livesync-settings.json sync
Push local files into the database from automation, and pull them back for export or backup.
```bash
npm run cli -- /data/vault --settings /data/livesync-settings.json push ./note.md notes/note.md
npm run cli -- /data/vault --settings /data/livesync-settings.json pull notes/note.md ./exports/note.md
npm run --silent cli -- /data/vault --settings /data/livesync-settings.json push ./note.md notes/note.md
npm run --silent cli -- /data/vault --settings /data/livesync-settings.json pull notes/note.md ./exports/note.md
```
### 3. Revision inspection and restore
@@ -211,9 +255,9 @@ npm run cli -- /data/vault --settings /data/livesync-settings.json pull notes/no
List metadata, find an older revision, then restore it by content (`cat-rev`) or file output (`pull-rev`).
```bash
npm run cli -- /data/vault --settings /data/livesync-settings.json info notes/note.md
npm run cli -- /data/vault --settings /data/livesync-settings.json cat-rev notes/note.md 3-abcdef
npm run cli -- /data/vault --settings /data/livesync-settings.json pull-rev notes/note.md ./restore/note.old.md 3-abcdef
npm run --silent cli -- /data/vault --settings /data/livesync-settings.json info notes/note.md
npm run --silent cli -- /data/vault --settings /data/livesync-settings.json cat-rev notes/note.md 3-abcdef
npm run --silent cli -- /data/vault --settings /data/livesync-settings.json pull-rev notes/note.md ./restore/note.old.md 3-abcdef
```
### 4. Conflict and cleanup workflow
@@ -221,9 +265,9 @@ npm run cli -- /data/vault --settings /data/livesync-settings.json pull-rev note
Inspect conflicted revisions, resolve by keeping one revision, then delete obsolete files.
```bash
npm run cli -- /data/vault --settings /data/livesync-settings.json info notes/note.md
npm run cli -- /data/vault --settings /data/livesync-settings.json resolve notes/note.md 3-abcdef
npm run cli -- /data/vault --settings /data/livesync-settings.json rm notes/obsolete.md
npm run --silent cli -- /data/vault --settings /data/livesync-settings.json info notes/note.md
npm run --silent cli -- /data/vault --settings /data/livesync-settings.json resolve notes/note.md 3-abcdef
npm run --silent cli -- /data/vault --settings /data/livesync-settings.json rm notes/obsolete.md
```
### 5. CI smoke test for content round-trip
@@ -231,8 +275,8 @@ npm run cli -- /data/vault --settings /data/livesync-settings.json rm notes/obso
Validate that `put`/`cat` is behaving as expected in a pipeline.
```bash
echo "hello-ci" | npm run cli -- /data/vault --settings /data/livesync-settings.json put ci/test.md
npm run cli -- /data/vault --settings /data/livesync-settings.json cat ci/test.md
echo "hello-ci" | npm run --silent cli -- /data/vault --settings /data/livesync-settings.json put ci/test.md
npm run --silent cli -- /data/vault --settings /data/livesync-settings.json cat ci/test.md
```
## Development

9
src/apps/cli/commands/runCommand.ts
View File

@@ -6,6 +6,8 @@ import { DEFAULT_SETTINGS, type FilePathWithPrefix, type ObsidianLiveSyncSetting
import { stripAllPrefixes } from "@lib/string_and_binary/path";
import type { CLICommandContext, CLIOptions } from "./types";
import { promptForPassphrase, readStdinAsUtf8, toArrayBuffer, toVaultRelativePath } from "./utils";
import { performFullScan } from "@lib/serviceFeatures/offlineScanner";
import { UnresolvedErrorManager } from "@lib/services/base/UnresolvedErrorManager";
export async function runCommand(options: CLIOptions, context: CLICommandContext): Promise<boolean> {
const { vaultPath, core, settingsPath } = context;
@@ -309,5 +311,12 @@ export async function runCommand(options: CLIOptions, context: CLICommandContext
return true;
}
if (options.command === "mirror") {
console.error("[Command] mirror");
const log = (msg: unknown) => console.error(`[Mirror] ${msg}`);
const errorManager = new UnresolvedErrorManager(core.services.appLifecycle);
return await performFullScan(core as any, log, errorManager, false, true);
}
throw new Error(`Unsupported command: ${options.command}`);
}

2
src/apps/cli/commands/types.ts
View File

@@ -15,6 +15,7 @@ export type CLICommand =
| "info"
| "rm"
| "resolve"
| "mirror"
| "init-settings";
export interface CLIOptions {
@@ -45,5 +46,6 @@ export const VALID_COMMANDS = new Set([
"info",
"rm",
"resolve",
"mirror",
"init-settings",
] as const);

16
src/apps/cli/main.ts
View File

@@ -31,6 +31,8 @@ import { LOG_LEVEL_DEBUG, setGlobalLogFunction, defaultLoggerEnv } from "octagon
import { runCommand } from "./commands/runCommand";
import { VALID_COMMANDS } from "./commands/types";
import type { CLICommand, CLIOptions } from "./commands/types";
import { getPathFromUXFileInfo } from "@lib/common/typeUtils";
import { stripAllPrefixes } from "@lib/string_and_binary/path";
const SETTINGS_FILE = ".livesync/settings.json";
defaultLoggerEnv.minLogLevel = LOG_LEVEL_DEBUG;
@@ -254,6 +256,7 @@ export async function main() {
console.error(`[Info] Replication result received, but not processed automatically in CLI mode.`);
return await Promise.resolve(true);
}, -100);
// Setup settings handlers
const settingService = serviceHubInstance.setting;
@@ -298,7 +301,18 @@ export async function main() {
},
() => [], // No extra modules
() => [], // No add-ons
() => [] // No serviceFeatures
(core) => {
// Add target filter to prevent internal files are handled
core.services.vault.isTargetFile.addHandler(async (target) => {
const vaultPath = stripAllPrefixes(getPathFromUXFileInfo(target));
const parts = vaultPath.split(path.sep);
// if some part of the path starts with dot, treat it as internal file and ignore.
if (parts.some((part) => part.startsWith("."))) {
return await Promise.resolve(false);
}
return await Promise.resolve(true);
}, -1 /* highest priority */);
}
);
// Setup signal handlers for graceful shutdown

3
src/apps/cli/package.json
View File

@@ -18,7 +18,8 @@
"test:e2e:push-pull": "bash test/test-push-pull-linux.sh",
"test:e2e:setup-put-cat": "bash test/test-setup-put-cat-linux.sh",
"test:e2e:sync-two-local": "bash test/test-sync-two-local-databases-linux.sh",
"test:e2e:all": "npm run test:e2e:two-vaults && npm run test:e2e:push-pull && npm run test:e2e:setup-put-cat && npm run test:e2e:sync-two-local"
"test:e2e:mirror": "bash test/test-mirror-linux.sh",
"test:e2e:all": "npm run test:e2e:two-vaults && npm run test:e2e:push-pull && npm run test:e2e:setup-put-cat && npm run test:e2e:sync-two-local && npm run test:e2e:mirror"
},
"dependencies": {},
"devDependencies": {}

8
src/apps/cli/test/test-e2e-two-vaults-common.sh
View File

@@ -4,8 +4,12 @@ set -euo pipefail
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
CLI_DIR="$(cd -- "$SCRIPT_DIR/.." && pwd)"
cd "$CLI_DIR"
CLI_CMD=(npm --silent run cli -- -v)
VERBOSE_TEST_LOGGING="${VERBOSE_TEST_LOGGING:-0}"
if [[ "$VERBOSE_TEST_LOGGING" == "1" ]]; then
CLI_CMD=(npm --silent run cli -- -v)
else
CLI_CMD=(npm --silent run cli --)
fi
RUN_BUILD="${RUN_BUILD:-1}"
KEEP_TEST_DATA="${KEEP_TEST_DATA:-0}"
TEST_ENV_FILE="${TEST_ENV_FILE:-$CLI_DIR/.test.env}"

176
src/apps/cli/test/test-mirror-linux.sh Executable file
View File

@@ -0,0 +1,176 @@
#!/usr/bin/env bash
# Test: mirror command — storage <-> local database synchronisation
#
# Covered cases:
# 1. Storage-only file → synced into DB (UPDATE DATABASE)
# 2. DB-only file → restored to storage (UPDATE STORAGE)
# 3. DB-deleted file → NOT restored to storage (UPDATE STORAGE skip)
# 4. Both, storage newer → DB updated (SYNC: STORAGE → DB)
# 5. Both, DB newer → storage updated (SYNC: DB → STORAGE)
#
# Not covered (require precise mtime control or artificial conflict injection):
# - Both, equal mtime → no-op (EVEN)
# - Conflicted entry → skipped
#
set -euo pipefail
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
CLI_DIR="$(cd -- "$SCRIPT_DIR/.." && pwd)"
cd "$CLI_DIR"
CLI_CMD=(npm run cli --)
RUN_BUILD="${RUN_BUILD:-1}"
WORK_DIR="$(mktemp -d "${TMPDIR:-/tmp}/livesync-cli-test.XXXXXX")"
trap 'rm -rf "$WORK_DIR"' EXIT
SETTINGS_FILE="$WORK_DIR/data.json"
VAULT_DIR="$WORK_DIR/vault"
mkdir -p "$VAULT_DIR/test"
if [[ "$RUN_BUILD" == "1" ]]; then
echo "[INFO] building CLI..."
npm run build
fi
run_cli() {
"${CLI_CMD[@]}" "$@"
}
echo "[INFO] generating settings -> $SETTINGS_FILE"
run_cli init-settings --force "$SETTINGS_FILE"
# isConfigured=true is required for mirror (canProceedScan checks this)
SETTINGS_FILE="$SETTINGS_FILE" node -e "
const fs = require('node:fs');
const s = JSON.parse(fs.readFileSync(process.env.SETTINGS_FILE, 'utf-8'));
s.isConfigured = true;
fs.writeFileSync(process.env.SETTINGS_FILE, JSON.stringify(s, null, 2));
"
PASS=0
FAIL=0
assert_pass() { echo "[PASS] $1"; PASS=$((PASS + 1)); }
assert_fail() { echo "[FAIL] $1" >&2; FAIL=$((FAIL + 1)); }
# ─────────────────────────────────────────────────────────────────────────────
# Case 1: File exists only in storage → should be synced into DB after mirror
# ─────────────────────────────────────────────────────────────────────────────
echo ""
echo "=== Case 1: storage-only → DB ==="
printf 'storage-only content\n' > "$VAULT_DIR/test/storage-only.md"
run_cli "$VAULT_DIR" --settings "$SETTINGS_FILE" mirror
RESULT_FILE="$WORK_DIR/case1-cat.txt"
run_cli "$VAULT_DIR" --settings "$SETTINGS_FILE" pull test/storage-only.md "$RESULT_FILE"
if cmp -s "$VAULT_DIR/test/storage-only.md" "$RESULT_FILE"; then
assert_pass "storage-only file was synced into DB"
else
assert_fail "storage-only file NOT synced into DB"
echo "--- storage ---" >&2; cat "$VAULT_DIR/test/storage-only.md" >&2
echo "--- cat ---" >&2; cat "$RESULT_FILE" >&2
fi
# ─────────────────────────────────────────────────────────────────────────────
# Case 2: File exists only in DB → should be restored to storage after mirror
# ─────────────────────────────────────────────────────────────────────────────
echo ""
echo "=== Case 2: DB-only → storage ==="
printf 'db-only content\n' | run_cli "$VAULT_DIR" --settings "$SETTINGS_FILE" put test/db-only.md
if [[ -f "$VAULT_DIR/test/db-only.md" ]]; then
assert_fail "db-only.md unexpectedly exists in storage before mirror"
else
echo "[INFO] confirmed: test/db-only.md not in storage before mirror"
fi
run_cli "$VAULT_DIR" --settings "$SETTINGS_FILE" mirror
if [[ -f "$VAULT_DIR/test/db-only.md" ]]; then
STORAGE_CONTENT="$(cat "$VAULT_DIR/test/db-only.md")"
if [[ "$STORAGE_CONTENT" == "db-only content" ]]; then
assert_pass "DB-only file was restored to storage"
else
assert_fail "DB-only file restored but content mismatch (got: '${STORAGE_CONTENT}')"
fi
else
assert_fail "DB-only file was NOT restored to storage"
fi
# ─────────────────────────────────────────────────────────────────────────────
# Case 3: File deleted in DB → should NOT be created in storage
# ─────────────────────────────────────────────────────────────────────────────
echo ""
echo "=== Case 3: DB-deleted → storage untouched ==="
printf 'to-be-deleted\n' | run_cli "$VAULT_DIR" --settings "$SETTINGS_FILE" put test/deleted.md
run_cli "$VAULT_DIR" --settings "$SETTINGS_FILE" rm test/deleted.md
run_cli "$VAULT_DIR" --settings "$SETTINGS_FILE" mirror
if [[ ! -f "$VAULT_DIR/test/deleted.md" ]]; then
assert_pass "deleted DB entry was not restored to storage"
else
assert_fail "deleted DB entry was incorrectly restored to storage"
fi
# ─────────────────────────────────────────────────────────────────────────────
# Case 4: Both exist, storage is newer → DB should be updated
# ─────────────────────────────────────────────────────────────────────────────
echo ""
echo "=== Case 4: storage newer → DB updated ==="
# Seed DB with old content (mtime ≈ now)
printf 'old content\n' | run_cli "$VAULT_DIR" --settings "$SETTINGS_FILE" put test/sync-storage-newer.md
# Write new content to storage with a timestamp 1 hour in the future
printf 'new content\n' > "$VAULT_DIR/test/sync-storage-newer.md"
touch -t "$(date -d '+1 hour' +%Y%m%d%H%M)" "$VAULT_DIR/test/sync-storage-newer.md"
run_cli "$VAULT_DIR" --settings "$SETTINGS_FILE" mirror
DB_RESULT_FILE="$WORK_DIR/case4-pull.txt"
run_cli "$VAULT_DIR" --settings "$SETTINGS_FILE" pull test/sync-storage-newer.md "$DB_RESULT_FILE"
if cmp -s "$VAULT_DIR/test/sync-storage-newer.md" "$DB_RESULT_FILE"; then
assert_pass "DB updated to match newer storage file"
else
assert_fail "DB NOT updated to match newer storage file"
echo "--- expected(storage) ---" >&2; cat "$VAULT_DIR/test/sync-storage-newer.md" >&2
echo "--- pulled(from db) ---" >&2; cat "$DB_RESULT_FILE" >&2
fi
# ─────────────────────────────────────────────────────────────────────────────
# Case 5: Both exist, DB is newer → storage should be updated
# ─────────────────────────────────────────────────────────────────────────────
echo ""
echo "=== Case 5: DB newer → storage updated ==="
# Write old content to storage with a timestamp 1 hour in the past
printf 'old storage content\n' > "$VAULT_DIR/test/sync-db-newer.md"
touch -t "$(date -d '-1 hour' +%Y%m%d%H%M)" "$VAULT_DIR/test/sync-db-newer.md"
# Write new content to DB only (mtime ≈ now, newer than the storage file)
printf 'new db content\n' | run_cli "$VAULT_DIR" --settings "$SETTINGS_FILE" put test/sync-db-newer.md
run_cli "$VAULT_DIR" --settings "$SETTINGS_FILE" mirror
STORAGE_CONTENT="$(cat "$VAULT_DIR/test/sync-db-newer.md")"
if [[ "$STORAGE_CONTENT" == "new db content" ]]; then
assert_pass "storage updated to match newer DB entry"
else
assert_fail "storage NOT updated to match newer DB entry (got: '${STORAGE_CONTENT}')"
fi
# ─────────────────────────────────────────────────────────────────────────────
# Summary
# ─────────────────────────────────────────────────────────────────────────────
echo ""
echo "Results: PASS=$PASS FAIL=$FAIL"
if [[ "$FAIL" -gt 0 ]]; then
exit 1
fi

2
src/lib

Submodule src/lib updated: 35df9a1192...423f6ee3a6

6
updates.md
View File

@@ -3,6 +3,12 @@ Since 19th July, 2025 (beta1 in 0.25.0-beta1, 13th July, 2025)
The head note of 0.25 is now in [updates_old.md](https://github.com/vrtmrz/obsidian-livesync/blob/main/updates_old.md). Because 0.25 got a lot of updates, thankfully, compatibility is kept and we do not need breaking changes! In other words, when get enough stabled. The next version will be v1.0.0. Even though it my hope.
## -- unreleased --
### New features
- `mirror` command has been added to the CLI. This command is intended to mirror the storage to the local database.
## 0.25.52-patched-1
12th March, 2026