mirror of
https://github.com/vrtmrz/obsidian-livesync.git
synced 2026-11-02 22:38:18 +00:00
Preserve CLI database continuity after reconstruction
This commit is contained in:
1 parent
0e458e5e20
commit
d50ee9a83e
21 files changed
+1585
-60
No files matched your search
@@ -38,6 +38,7 @@ Use these references for the area being changed, together with its relevant ADR.
|
||||
| Encryption and sharing | [Metadata encryption](design_docs/internal_metadata_encryption.md) and [time-bound Setup URIs](design_docs/time_bound_setup_uri.md) |
|
||||
| P2P connection preparation | [TURN connection settings](design_docs/renewable_turn_credentials.md) |
|
||||
| Journal storage internals | [Journal Replicator 2nd Edition](design_docs_of_journalsync_2nd.md) |
|
||||
| CLI database selection and interrupted reconstruction | [CLI database continuity](design_docs/cli_database_continuity.md) |
|
||||
|
||||
The installed `@vrtmrz/livesync-commonlib` package is authoritative for shared runtime contracts. Use the exact dependency selected by [package.json](../package.json) and its lockfile when checking behaviour. The [Commonlib documentation](https://github.com/vrtmrz/livesync-commonlib/tree/main/docs) is maintained in that repository; this index does not replace its package contracts.
|
||||
|
||||
|
||||
@@ -0,0 +1,101 @@
|
||||
---
|
||||
date: 2026-11-01
|
||||
commonlib-version: "0.2.2"
|
||||
self-hosted-livesync-version: "1.0.43"
|
||||
status: accepted
|
||||
---
|
||||
|
||||
# CLI database continuity
|
||||
|
||||
Publication note: the correction for #1304 is implemented and locally validated,
|
||||
but has not been released. The reviewed baseline is Self-hosted LiveSync `0e458e5e` with
|
||||
Commonlib `0.2.2`. The [CLI guide](../../src/apps/cli/README.md) describes the
|
||||
command interface; the [test procedure](../../test/README.md#cli-fetch-database-continuity)
|
||||
describes the Object Storage reproduction.
|
||||
|
||||
## Required behaviour
|
||||
|
||||
A successful `fetch` or `rebuild` must leave subsequent CLI commands using the
|
||||
reconstructed local database. The Commonlib Rebuilder selects the host app ID
|
||||
as the database suffix. The CLI saves that selection on success even without
|
||||
`--write-settings`; the option continues to govern other lasting settings
|
||||
changes. Temporary suspension values remain runtime state.
|
||||
|
||||
Before reconstruction can mutate a database, the CLI records the operation,
|
||||
database directory, Vault path, and settings path. An interrupted operation
|
||||
blocks ordinary mutation and synchronisation. Retrying the same reconstruction
|
||||
with the same paths and `--force` is an explicit recovery action. It can repeat
|
||||
the destructive operation, so users must preserve unique edits and establish
|
||||
the authoritative copy before retrying.
|
||||
|
||||
Completion records the settings update before applying it. A later invocation
|
||||
can finish a pending settings commit; if the settings file has independently
|
||||
changed, it must stop for review rather than overwrite those changes. The
|
||||
pending record is removed only after completion and settings persistence.
|
||||
|
||||
## Ownership of persistent state
|
||||
|
||||
The selected local database name identifies the dataset. Its runtime state is
|
||||
further scoped by the canonical physical Vault path. This includes Journal
|
||||
checkpoints, scanner state, replication-result queues, file-reflection
|
||||
provenance, and storage-event snapshots. Paths identify local ownership; they
|
||||
are not sent to the remote as a new document identity.
|
||||
|
||||
A local database reset invalidates every Vault scope associated with that
|
||||
database, because each scope refers to the same discarded revision history.
|
||||
Other local database suffixes retain their state. PouchDB database names and
|
||||
Commonlib's reconstruction contract remain unchanged.
|
||||
|
||||
KV writes replace the file atomically. Invalid persisted data must fail
|
||||
initialisation instead of becoming an empty checkpoint history. Closing a KV
|
||||
handle must not overwrite the file from an old in-memory snapshot.
|
||||
|
||||
## Legacy state
|
||||
|
||||
The former shared `keyvalue-db.json` and Vault-local `.livesync-snapshot.json`
|
||||
do not record their complete owner. A single surviving database directory does
|
||||
not prove which physical Vault produced those records. Automatically importing
|
||||
them can replay events against the wrong database; dropping them and immediately
|
||||
scanning can misinterpret file absence before remote history is recovered.
|
||||
|
||||
Preserve unowned legacy state. When it contains work, stop ordinary mutating
|
||||
commands and require explicit reconstruction from a verified source. Inspection
|
||||
remains available. This deliberately requires more user action than automatic
|
||||
migration, and must not silently delete the old records.
|
||||
|
||||
Check the selected scope's completion flag before reading legacy files. For an
|
||||
unreviewed scope, inspect both legacy files after settings and scoped KV loading,
|
||||
but before readiness starts scanning or event replay. Distinguish non-empty
|
||||
unowned state, invalid JSON or shape, and read failures. Report affected paths
|
||||
once per invocation on stderr without logging file contents. Read commands may
|
||||
continue inspection; ordinary writes stop, and explicit reconstruction retains
|
||||
the files. Once reconstruction has succeeded for that scope, legacy parse or
|
||||
read failures must not block it again. The active scoped KV still fails
|
||||
initialisation on corruption.
|
||||
|
||||
## Evidence and limits
|
||||
|
||||
The original Object Storage scenario uses encryption, path obfuscation, and a
|
||||
second-generation file revision. Without settings writeback, A reopens an empty
|
||||
database while sharing the fetched database's Journal history, creates an
|
||||
independent root, and receives B's same-content resolution tombstone without
|
||||
the original surviving revision. The original assertion fails because A's
|
||||
physical file disappears. The explicit `--write-settings` control preserves
|
||||
the file and the original revision. The remote and B retain the content in this
|
||||
reproduction; it does not demonstrate irreversible loss.
|
||||
|
||||
Validation of the correction must check bytes and revision ancestry after
|
||||
restart, interrupted reconstruction admission, pending settings completion,
|
||||
legacy-state preservation, and reset isolation. A successful exit or an empty
|
||||
queue alone does not establish preservation.
|
||||
|
||||
Local validation passed the encrypted Object Storage CLI regression and the
|
||||
explicit-settings control on Node.js 26. Unit tests cover failed-operation
|
||||
admission, saved running/completed markers, settings-edit preservation, scoped
|
||||
stores, reset isolation, corrupt KV files, and legacy-state refusal. The marker
|
||||
tests inject interruption states; they do not emulate a machine power failure.
|
||||
|
||||
The shared device-value JSON file retains its existing concurrency limitations.
|
||||
This change does not establish concurrent CLI writers to one database directory
|
||||
as a supported workflow. Native Obsidian and other transport behaviour require
|
||||
their own runtime evidence.
|
||||
@@ -155,10 +155,11 @@ preflight failure. If a required setup decision cannot be made in a headless
|
||||
CLI invocation, the command must fail with a clear remediation message rather
|
||||
than report success after cancellation.
|
||||
|
||||
`--write-settings` retains the Rebuilder's selected local database name and
|
||||
lasting settings changes for subsequent invocations. Without this option, the
|
||||
existing CLI policy leaves the settings file unchanged, so a later command can
|
||||
reopen the previously selected database.
|
||||
The unreleased CLI correction for #1304 always retains the Rebuilder's selected
|
||||
local database suffix after successful `fetch` or `rebuild`. `--write-settings`
|
||||
still controls other lasting settings changes. Earlier versions required that
|
||||
option to retain the database selection; omitting it could reopen the previous
|
||||
database on the next invocation. The examples retain the option for compatibility.
|
||||
|
||||
## State which follows the selected identity
|
||||
|
||||
|
||||
Generated
+1
@@ -11765,6 +11765,7 @@
|
||||
"name": "self-hosted-livesync-cli",
|
||||
"version": "1.0.43-cli",
|
||||
"dependencies": {
|
||||
"@aws-crypto/sha256-js": "5.2.0",
|
||||
"chokidar": "^4.0.0",
|
||||
"minimatch": "^10.2.5",
|
||||
"octagonal-wheels": "^0.1.55",
|
||||
|
||||
+36
-1
@@ -70,6 +70,7 @@ livesync-cli [database-path] [command] [args...]
|
||||
- `--force` / `-f`: Required by `rebuild` and `fetch` because they replace a database. It also retains its existing `init-settings` overwrite behaviour.
|
||||
- `--write-settings`: Write setting migrations and other lasting changes after the command succeeds.
|
||||
- `init-settings` writes its target file. `setup`, `remote-add`, `remote-rm`, `remote-set`, and `remote-activate` write their settings changes without this option.
|
||||
- `fetch` and `rebuild` always retain the selected local database suffix after success. Other lasting changes from these commands still require this option.
|
||||
- All remaining commands leave the settings file unchanged by default.
|
||||
- Temporary values used to suspend synchronisation or select a remote for one command are never written.
|
||||
- `--compat-remote-admin-exit-zero`: Preserve the former zero exit code when `mark-resolved`, `lock-remote`, or `unlock-remote` returns a provider verification failure.
|
||||
@@ -84,8 +85,9 @@ livesync-cli [database-path] [command] [args...]
|
||||
- If `vault-path` is omitted, it defaults to `database-path` (compatibility mode).
|
||||
- Use this command to keep your local `.md` files in sync with the database.
|
||||
- `rebuild`: Rebuild the local and remote databases from the selected Vault. This replaces existing local and remote database state and requires `--force`.
|
||||
- This command may select a new local database suffix. Add `--write-settings` if later CLI commands should use the resulting database selection; otherwise the existing settings file remains unchanged.
|
||||
- The selected local database suffix is saved automatically, so later commands use the reconstructed database.
|
||||
- `fetch`: Replace the local database with the configured remote state, then reflect fetched files to the selected Vault. Existing local database state is discarded, so this command requires `--force`.
|
||||
- The selected local database suffix is saved automatically, including when `--write-settings` is omitted.
|
||||
- `ls [prefix]`: List files currently stored in the local database.
|
||||
- `push <src> <dst>`: Push a local file `<src>` into the database at path `<dst>`.
|
||||
- `pull <src> <dst>`: Pull a file `<src>` from the database into local file `<dst>`.
|
||||
@@ -105,6 +107,39 @@ livesync-cli [database-path] [command] [args...]
|
||||
|
||||
Remote-administration commands verify the resulting milestone state through the selected provider. The existing `[Verification]` lines remain suitable for scripts which inspect command output, while the default exit code now reflects whether that verification succeeded.
|
||||
|
||||
### Interrupted reconstruction and older runtime state
|
||||
|
||||
An interrupted `fetch` or `rebuild` leaves a record in
|
||||
`<database-path>/.livesync/reconstruction.json`. Ordinary synchronisation and
|
||||
mutating commands stop while reconstruction is incomplete. Preserve the Vault,
|
||||
database directory, and settings, check the reported failure, then retry the
|
||||
same command with `--force` and the same database, Vault, and settings paths.
|
||||
`rebuild` replaces remote state as well: confirm that the Vault remains the
|
||||
authoritative copy before retrying it. Do not remove the record merely to
|
||||
restart `daemon`.
|
||||
|
||||
If reconstruction finished but its settings update was interrupted, the next
|
||||
invocation completes that update before starting the core. If the settings
|
||||
file was independently edited, the CLI stops for review. The pending record
|
||||
can contain settings and credentials; protect it like the settings file.
|
||||
|
||||
Older CLI versions stored runtime state without recording its database and
|
||||
Vault owner. If unowned state is present, the CLI preserves it and stops
|
||||
ordinary mutating commands. Read commands remain available for inspection.
|
||||
Back up unique edits and establish the authoritative copy before explicitly
|
||||
using `fetch --force` or `rebuild --force`. The old runtime files remain
|
||||
available for diagnosis; they are not automatically imported into a new scope.
|
||||
|
||||
This check runs before scanning, event replay, and synchronisation. Its message
|
||||
identifies each affected file and distinguishes an unknown owner from a file
|
||||
which cannot be read or parsed. Read commands display the warning and continue
|
||||
inspection. After successful reconstruction for the selected database and Vault,
|
||||
retained legacy files are no longer inspected, even if they are malformed.
|
||||
Corruption in the currently selected runtime KV remains an initialisation error.
|
||||
|
||||
Use one CLI process at a time for each database directory. Separate runtime
|
||||
scopes do not make concurrent writers to shared device settings supported.
|
||||
|
||||
### Examples
|
||||
|
||||
```bash
|
||||
|
||||
@@ -78,12 +78,20 @@ describe("CLI database preparation", () => {
|
||||
await rm(directory, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
async function start(command: CLICommand, runner: CliCommandRunner, exitCode = 1, force = false) {
|
||||
async function start(
|
||||
command: CLICommand,
|
||||
runner: CliCommandRunner,
|
||||
exitCode = 1,
|
||||
force = false,
|
||||
writeSettings = false
|
||||
) {
|
||||
process.exitCode = undefined;
|
||||
process.argv = [
|
||||
"node",
|
||||
"livesync-cli",
|
||||
directory,
|
||||
...(force ? ["--force"] : []),
|
||||
...(writeSettings ? ["--write-settings"] : []),
|
||||
"--vault",
|
||||
vaultPath,
|
||||
"--settings",
|
||||
@@ -386,6 +394,194 @@ describe("CLI database preparation", () => {
|
||||
expect(await readFile(join(vaultPath, "notes/local.md"), "utf-8")).toBe("local content");
|
||||
});
|
||||
|
||||
it.each(["fetch", "rebuild"] as const)(
|
||||
"persists the selected database after %s without other command settings",
|
||||
async (command) => {
|
||||
const before = JSON.parse(await readFile(settingsPath, "utf-8"));
|
||||
const runner = vi.fn<CliCommandRunner>(async (_options, { core }) => {
|
||||
await core.services.setting.applyPartial({
|
||||
additionalSuffixOfDatabaseName: "headless-app",
|
||||
customChunkSize: before.customChunkSize + 10,
|
||||
});
|
||||
return true;
|
||||
});
|
||||
|
||||
await start(command, runner, 0, true);
|
||||
|
||||
const after = JSON.parse(await readFile(settingsPath, "utf-8"));
|
||||
expect(after.additionalSuffixOfDatabaseName).toBe("headless-app");
|
||||
expect(after.customChunkSize).toBe(before.customChunkSize);
|
||||
}
|
||||
);
|
||||
|
||||
it.each(["suffix", "vault"] as const)("isolates persistent state by selected %s", async (selection) => {
|
||||
await start("ls", async (_options, { core }) => {
|
||||
await core.services.keyValueDB.kvDB.set("scope-probe", "original");
|
||||
await core.services.config.setValue("scope-probe", "original");
|
||||
return false;
|
||||
});
|
||||
if (selection === "suffix") {
|
||||
const settings = JSON.parse(await readFile(settingsPath, "utf-8"));
|
||||
settings.additionalSuffixOfDatabaseName = "other";
|
||||
await writeFile(settingsPath, JSON.stringify(settings));
|
||||
} else {
|
||||
vaultPath = join(directory, "other-vault");
|
||||
await mkdir(vaultPath);
|
||||
}
|
||||
const observed: unknown[] = [];
|
||||
await start("ls", async (_options, { core }) => {
|
||||
observed.push(await core.services.keyValueDB.kvDB.get("scope-probe"));
|
||||
observed.push(await core.services.config.getValue("scope-probe"));
|
||||
return false;
|
||||
});
|
||||
expect(observed).toEqual([undefined, null]);
|
||||
});
|
||||
|
||||
it.each(["sync", "daemon", "mirror", "put"] as const)(
|
||||
"blocks %s after failed reconstruction before command preparation",
|
||||
async (command) => {
|
||||
await start("fetch", async () => false, 1, true);
|
||||
const runner = vi.fn<CliCommandRunner>(async () => false);
|
||||
const ready = vi.spyOn(ControlService.prototype, "onReady");
|
||||
|
||||
await start(command, runner);
|
||||
|
||||
expect(runner).not.toHaveBeenCalled();
|
||||
expect(ready).not.toHaveBeenCalled();
|
||||
}
|
||||
);
|
||||
|
||||
it("retains legacy queued work and blocks a scanner until explicit reconstruction succeeds", async () => {
|
||||
const legacy = join(directory, ".livesync/runtime/keyvalue-db.json");
|
||||
const legacyContents = JSON.stringify({
|
||||
"queued-files": ["notes/unique.md"],
|
||||
fileStatusMap: { "notes/local.md": 1 },
|
||||
});
|
||||
await mkdir(join(directory, ".livesync/runtime"), { recursive: true });
|
||||
await writeFile(legacy, legacyContents);
|
||||
const runner = vi.fn<CliCommandRunner>(async () => false);
|
||||
const scan = vi.spyOn(ServiceFileAccessCLI.prototype, "getFiles");
|
||||
|
||||
await start("mirror", runner);
|
||||
expect(runner).not.toHaveBeenCalled();
|
||||
expect(scan).not.toHaveBeenCalled();
|
||||
expect(await readFile(legacy, "utf-8")).toBe(legacyContents);
|
||||
|
||||
await start("fetch", async () => true, 0, true);
|
||||
const later = vi.fn<CliCommandRunner>(async () => false);
|
||||
await start("sync", later);
|
||||
expect(later).toHaveBeenCalledOnce();
|
||||
expect(await readFile(legacy, "utf-8")).toBe(legacyContents);
|
||||
});
|
||||
|
||||
it("permits normal commands after reconstruction while preserving malformed legacy JSON", async () => {
|
||||
const legacy = join(directory, ".livesync/runtime/keyvalue-db.json");
|
||||
await mkdir(join(directory, ".livesync/runtime"), { recursive: true });
|
||||
await writeFile(legacy, '{"unfinished":');
|
||||
await start("fetch", async () => true, 0, true);
|
||||
const runner = vi.fn<CliCommandRunner>(async () => false);
|
||||
await start("sync", runner);
|
||||
expect(runner).toHaveBeenCalledOnce();
|
||||
expect(await readFile(legacy, "utf-8")).toBe('{"unfinished":');
|
||||
});
|
||||
|
||||
it.each([
|
||||
["kv", '{"unfinished":', "cannot be parsed"],
|
||||
["snapshot", "[", "cannot be parsed"],
|
||||
["kv", '{"queued-files":["unique.md"]}', "owner is unknown"],
|
||||
["snapshot", '[{"path":"unique.md"}]', "owner is unknown"],
|
||||
])("diagnoses unowned %s (%s) before readiness and ignores it after recovery", async (kind, contents, reason) => {
|
||||
const legacy =
|
||||
kind === "kv"
|
||||
? join(directory, ".livesync/runtime/keyvalue-db.json")
|
||||
: join(vaultPath, ".livesync-snapshot.json");
|
||||
await mkdir(join(directory, ".livesync/runtime"), { recursive: true });
|
||||
await writeFile(legacy, contents);
|
||||
const messages = () => standardIo.writeStderr.mock.calls.flat().join("");
|
||||
const reader = vi.fn<CliCommandRunner>(async () => false);
|
||||
await start("ls", reader);
|
||||
expect(reader).toHaveBeenCalledOnce();
|
||||
expect(messages()).toContain(reason);
|
||||
expect(messages()).toContain(legacy);
|
||||
expect(messages()).toContain("inspection may continue");
|
||||
expect(messages().split("[Legacy state]")).toHaveLength(2);
|
||||
expect(standardIo.writeStdout.mock.calls.flat().join("")).not.toContain("[Legacy state]");
|
||||
|
||||
standardIo.writeStderr.mockClear();
|
||||
const writer = vi.fn<CliCommandRunner>(async () => false);
|
||||
const ready = vi.spyOn(ControlService.prototype, "onReady");
|
||||
await start("mirror", writer);
|
||||
expect(writer).not.toHaveBeenCalled();
|
||||
expect(ready).not.toHaveBeenCalled();
|
||||
expect(messages()).toContain("stopped before scanning");
|
||||
expect(messages()).toContain(reason);
|
||||
expect(messages().split("[Legacy state]")).toHaveLength(2);
|
||||
ready.mockRestore();
|
||||
|
||||
await start("fetch", async () => true, 0, true);
|
||||
standardIo.writeStderr.mockClear();
|
||||
const later = vi.fn<CliCommandRunner>(async () => false);
|
||||
await start("sync", later);
|
||||
expect(later).toHaveBeenCalledOnce();
|
||||
expect(messages()).not.toContain("[Legacy state]");
|
||||
expect(await readFile(legacy, "utf-8")).toBe(contents);
|
||||
});
|
||||
|
||||
it("blocks read commands which request settings writes while reconstruction is interrupted", async () => {
|
||||
await start("fetch", async () => false, 1, true);
|
||||
const runner = vi.fn<CliCommandRunner>(async () => false);
|
||||
const load = vi.spyOn(ControlService.prototype, "onLoad");
|
||||
|
||||
await start("ls", runner, 1, false, true);
|
||||
|
||||
expect(runner).not.toHaveBeenCalled();
|
||||
expect(load).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("invalidates every Vault scope for the reset database and retains other database state", async () => {
|
||||
const originalVault = vaultPath;
|
||||
const otherVault = join(directory, "other-vault");
|
||||
await mkdir(otherVault);
|
||||
const select = async (suffix: string, selectedVault: string) => {
|
||||
vaultPath = selectedVault;
|
||||
const settings = JSON.parse(await readFile(settingsPath, "utf-8"));
|
||||
settings.additionalSuffixOfDatabaseName = suffix;
|
||||
await writeFile(settingsPath, JSON.stringify(settings));
|
||||
};
|
||||
for (const [suffix, selectedVault] of [
|
||||
["", originalVault],
|
||||
["", otherVault],
|
||||
["other", originalVault],
|
||||
]) {
|
||||
await select(suffix, selectedVault);
|
||||
await start("ls", async (_options, { core }) => {
|
||||
await core.services.keyValueDB.kvDB.set("scope-probe", "stored");
|
||||
return false;
|
||||
});
|
||||
}
|
||||
await select("", originalVault);
|
||||
let reset: boolean | undefined;
|
||||
await start("ls", async (_options, { core }) => {
|
||||
reset = await core.services.database.resetDatabase();
|
||||
return false;
|
||||
});
|
||||
expect(reset).toBe(true);
|
||||
|
||||
const observed: unknown[] = [];
|
||||
for (const [suffix, selectedVault] of [
|
||||
["", originalVault],
|
||||
["", otherVault],
|
||||
["other", originalVault],
|
||||
]) {
|
||||
await select(suffix, selectedVault);
|
||||
await start("ls", async (_options, { core }) => {
|
||||
observed.push(await core.services.keyValueDB.kvDB.get("scope-probe"));
|
||||
return false;
|
||||
});
|
||||
}
|
||||
expect(observed).toEqual([undefined, undefined, "stored"]);
|
||||
});
|
||||
|
||||
it("does not import vault files for standalone database commands", async () => {
|
||||
const storedPaths: string[] = [];
|
||||
const runner = vi.fn<CliCommandRunner>(async (_options, { core }) => {
|
||||
|
||||
+83
-19
@@ -41,6 +41,14 @@ import {
|
||||
reconcileDurableSettings,
|
||||
settingsEqual,
|
||||
} from "./settingsPersistence";
|
||||
import {
|
||||
inspectLegacyState,
|
||||
isMutatingCommand,
|
||||
isReconstructionCommand,
|
||||
readOptionalFile,
|
||||
ReconstructionState,
|
||||
writeFileAtomically,
|
||||
} from "./runtimeState";
|
||||
|
||||
const SETTINGS_FILE = ".livesync/settings.json";
|
||||
|
||||
@@ -381,7 +389,7 @@ export async function main(
|
||||
}
|
||||
|
||||
// Resolve database path
|
||||
const databasePath = path.resolve(options.databasePath!);
|
||||
let databasePath = path.resolve(options.databasePath!);
|
||||
// Check if database directory exists
|
||||
try {
|
||||
const stat = await fs.stat(databasePath);
|
||||
@@ -389,18 +397,19 @@ export async function main(
|
||||
writeStderrLine(standardIo, `Error: ${databasePath} is not a directory`);
|
||||
process.exit(1);
|
||||
}
|
||||
databasePath = await fs.realpath(databasePath);
|
||||
} catch {
|
||||
writeStderrLine(standardIo, `Error: Database directory ${databasePath} does not exist`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Resolve settings path
|
||||
const settingsPath = options.settingsPath
|
||||
let settingsPath = options.settingsPath
|
||||
? path.resolve(options.settingsPath)
|
||||
: path.join(databasePath, SETTINGS_FILE);
|
||||
// Resolve vault path: mirror positional argument takes priority,
|
||||
// then --vault flag, otherwise fall back to databasePath.
|
||||
const vaultPath =
|
||||
let vaultPath =
|
||||
options.command === "mirror" && options.commandArgs[0]
|
||||
? path.resolve(options.commandArgs[0])
|
||||
: options.vaultPath
|
||||
@@ -414,11 +423,22 @@ export async function main(
|
||||
writeStderrLine(standardIo, `Error: Vault path ${vaultPath} is not a directory`);
|
||||
process.exit(1);
|
||||
}
|
||||
vaultPath = await fs.realpath(vaultPath);
|
||||
} catch {
|
||||
writeStderrLine(standardIo, `Error: Vault directory ${vaultPath} does not exist`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
settingsPath = await fs.realpath(settingsPath).catch(() => settingsPath);
|
||||
const reconstruction = new ReconstructionState(databasePath, vaultPath, settingsPath);
|
||||
try {
|
||||
reconstruction.prepare(options.command, options.force ?? false, options.writeSettings ?? false);
|
||||
} catch (error) {
|
||||
writeStderrLine(standardIo, "[Error] Reconstruction recovery is required:", error);
|
||||
process.exit(1);
|
||||
return;
|
||||
}
|
||||
|
||||
infoLog(`Self-hosted LiveSync CLI`);
|
||||
infoLog(`Database Path: ${databasePath}`);
|
||||
infoLog(`Vault Path: ${vaultPath}`);
|
||||
@@ -437,7 +457,7 @@ export async function main(
|
||||
}
|
||||
|
||||
// Create service context and hub
|
||||
const context = new NodeServiceContext(databasePath, standardIo);
|
||||
const context = new NodeServiceContext(databasePath, standardIo, vaultPath);
|
||||
const serviceHubInstance = new NodeServiceHub<NodeServiceContext>(databasePath, context);
|
||||
serviceHubInstance.API.addLog.setHandler((message: unknown, level: LOG_LEVEL) => {
|
||||
let levelStr = "";
|
||||
@@ -479,7 +499,7 @@ export async function main(
|
||||
|
||||
// Setup settings handlers
|
||||
const settingService = serviceHubInstance.setting;
|
||||
const originalSettingsText = await fs.readFile(settingsPath, "utf-8").catch(() => undefined);
|
||||
const originalSettingsText = readOptionalFile(settingsPath);
|
||||
let latestPreparedSettingsText: string | undefined;
|
||||
let preparedSettingsRevision = 0;
|
||||
let commandIsRunning = false;
|
||||
@@ -600,21 +620,19 @@ export async function main(
|
||||
|
||||
const writeSettingsAtomically = async (content: string | undefined): Promise<void> => {
|
||||
if (content === undefined || content === originalSettingsText) return;
|
||||
const temporaryPath = `${settingsPath}.${process.pid}.tmp`;
|
||||
await fs.mkdir(path.dirname(settingsPath), { recursive: true });
|
||||
try {
|
||||
await fs.writeFile(temporaryPath, content, "utf-8");
|
||||
await fs.rename(temporaryPath, settingsPath);
|
||||
if (options.verbose) {
|
||||
writeStderrLine(standardIo, `[Settings] Saved to ${settingsPath}`);
|
||||
}
|
||||
} catch (error) {
|
||||
await fs.unlink(temporaryPath).catch(() => {});
|
||||
throw error;
|
||||
writeFileAtomically(settingsPath, content);
|
||||
if (options.verbose) {
|
||||
writeStderrLine(standardIo, `[Settings] Saved to ${settingsPath}`);
|
||||
}
|
||||
};
|
||||
|
||||
// Start the core
|
||||
let unloaded = false;
|
||||
const unload = async () => {
|
||||
if (unloaded) return;
|
||||
unloaded = true;
|
||||
await core.services.control.onUnload();
|
||||
};
|
||||
try {
|
||||
infoLog(`[Starting] Initializing LiveSync...`);
|
||||
|
||||
@@ -623,6 +641,31 @@ export async function main(
|
||||
writeStderrLine(standardIo, `[Error] Failed to initialize LiveSync`);
|
||||
process.exit(1);
|
||||
}
|
||||
if ((await core.services.keyValueDB.kvDB.get("cli-legacy-state-reviewed")) !== true) {
|
||||
const legacyIssues = inspectLegacyState(databasePath, vaultPath);
|
||||
if (legacyIssues.length > 0) {
|
||||
const reasons = {
|
||||
unowned: "database and Vault owner is unknown",
|
||||
invalid: "state cannot be parsed in its expected format; owner and contents cannot be established",
|
||||
unreadable: "state cannot be read; owner and contents cannot be established",
|
||||
};
|
||||
const details = legacyIssues.map((issue) => `${issue.path}: ${reasons[issue.reason]}`).join("; ");
|
||||
const advice =
|
||||
"Preserve these files, back up unique edits, and verify the authoritative source before choosing fetch --force or rebuild --force.";
|
||||
const blocked =
|
||||
!isReconstructionCommand(options.command) &&
|
||||
(isMutatingCommand(options.command) || options.writeSettings);
|
||||
if (blocked) {
|
||||
throw new Error(
|
||||
`[Legacy state] ${details}. Command '${options.command}' stopped before scanning, event replay, or synchronisation. ${advice}`
|
||||
);
|
||||
}
|
||||
const action = isReconstructionCommand(options.command)
|
||||
? "Explicit reconstruction will proceed; the legacy files will be retained."
|
||||
: "Read-only inspection may continue; the legacy files will be retained.";
|
||||
writeStderrLine(standardIo, `[Legacy state] ${details}. ${action} ${advice}`);
|
||||
}
|
||||
}
|
||||
const settingsAfterLoadText = latestPreparedSettingsText
|
||||
? preserveStoredSetting(latestPreparedSettingsText, originalSettingsText, "useIndexedDBAdapter")
|
||||
: originalSettingsText;
|
||||
@@ -655,7 +698,7 @@ export async function main(
|
||||
try {
|
||||
readyResult = await core.services.control.onReady();
|
||||
} finally {
|
||||
if (!readyResult) await core.services.control.onUnload();
|
||||
if (!readyResult) await unload();
|
||||
}
|
||||
if (!readyResult) {
|
||||
throw new Error("Failed to initialise LiveSync.");
|
||||
@@ -751,6 +794,21 @@ export async function main(
|
||||
settingsTextToCommit = settingsAfterLoadText;
|
||||
}
|
||||
}
|
||||
if (result && isReconstructionCommand(options.command)) {
|
||||
if (!options.writeSettings) {
|
||||
const durable =
|
||||
originalSettingsText === undefined
|
||||
? cloneSettings(durableSettingsBeforeSuspend)
|
||||
: (JSON.parse(originalSettingsText) as ObsidianLiveSyncSettings);
|
||||
durable.additionalSuffixOfDatabaseName =
|
||||
core.services.setting.currentSettings().additionalSuffixOfDatabaseName;
|
||||
settingsTextToCommit = JSON.stringify(durable, null, 2);
|
||||
}
|
||||
settingsTextToCommit ??= originalSettingsText;
|
||||
if (settingsTextToCommit === undefined)
|
||||
throw new Error("Reconstruction completed without database selection settings.");
|
||||
await core.services.keyValueDB.kvDB.set("cli-legacy-state-reviewed", true);
|
||||
}
|
||||
|
||||
if (!result) {
|
||||
writeStderrLine(standardIo, `[Error] Command '${options.command}' failed`);
|
||||
@@ -764,10 +822,16 @@ export async function main(
|
||||
// Keep the process running
|
||||
await new Promise(() => {});
|
||||
} else {
|
||||
await core.services.control.onUnload();
|
||||
await writeSettingsAtomically(settingsTextToCommit);
|
||||
await unload();
|
||||
if (result && isReconstructionCommand(options.command)) {
|
||||
reconstruction.complete(settingsTextToCommit!);
|
||||
reconstruction.commit();
|
||||
} else {
|
||||
await writeSettingsAtomically(settingsTextToCommit);
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
await unload().catch(() => {});
|
||||
writeStderrLine(standardIo, `[Error] Failed to start:`, error);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
@@ -15,6 +15,7 @@ import { watch as chokidarWatch, type FSWatcher } from "chokidar";
|
||||
import type { IgnoreRules } from "@/apps/cli/serviceModules/IgnoreRules";
|
||||
import { fsPromises as fs, getNodeFileCreationTime, path, type Stats } from "@vrtmrz/livesync-commonlib/node";
|
||||
import type { CliDiagnosticReporter } from "@/apps/cli/cliOutput";
|
||||
import { writeFileAtomically } from "@/apps/cli/runtimeState";
|
||||
|
||||
/**
|
||||
* CLI-specific type guard adapter
|
||||
@@ -44,18 +45,19 @@ class CLITypeGuardAdapter implements IStorageEventTypeGuardAdapter<NodeFile, Nod
|
||||
* CLI-specific persistence adapter (file-based snapshot)
|
||||
*/
|
||||
class CLIPersistenceAdapter implements IStorageEventPersistenceAdapter {
|
||||
private snapshotPath: string;
|
||||
private snapshotPath: () => string;
|
||||
|
||||
constructor(
|
||||
basePath: string,
|
||||
private reportDiagnostic: CliDiagnosticReporter = () => undefined
|
||||
private reportDiagnostic: CliDiagnosticReporter = () => undefined,
|
||||
snapshotPath?: () => string
|
||||
) {
|
||||
this.snapshotPath = path.join(basePath, ".livesync-snapshot.json");
|
||||
this.snapshotPath = snapshotPath ?? (() => path.join(basePath, ".livesync-snapshot.json"));
|
||||
}
|
||||
|
||||
async saveSnapshot(snapshot: (FileEventItem | FileEventItemSentinel)[]): Promise<void> {
|
||||
try {
|
||||
await fs.writeFile(this.snapshotPath, JSON.stringify(snapshot, null, 2), "utf-8");
|
||||
writeFileAtomically(this.snapshotPath(), JSON.stringify(snapshot, null, 2));
|
||||
} catch (error) {
|
||||
this.reportDiagnostic("Failed to save snapshot:", error);
|
||||
}
|
||||
@@ -63,7 +65,7 @@ class CLIPersistenceAdapter implements IStorageEventPersistenceAdapter {
|
||||
|
||||
async loadSnapshot(): Promise<(FileEventItem | FileEventItemSentinel)[] | null> {
|
||||
try {
|
||||
const content = await fs.readFile(this.snapshotPath, "utf-8");
|
||||
const content = await fs.readFile(this.snapshotPath(), "utf-8");
|
||||
return JSON.parse(content) as (FileEventItem | FileEventItemSentinel)[];
|
||||
} catch {
|
||||
return null;
|
||||
@@ -221,10 +223,11 @@ export class CLIStorageEventManagerAdapter implements IStorageEventManagerAdapte
|
||||
basePath: string,
|
||||
ignoreRules?: IgnoreRules,
|
||||
watchEnabled: boolean = false,
|
||||
reportDiagnostic: CliDiagnosticReporter = () => undefined
|
||||
reportDiagnostic: CliDiagnosticReporter = () => undefined,
|
||||
snapshotPath?: () => string
|
||||
) {
|
||||
this.typeGuard = new CLITypeGuardAdapter();
|
||||
this.persistence = new CLIPersistenceAdapter(basePath, reportDiagnostic);
|
||||
this.persistence = new CLIPersistenceAdapter(basePath, reportDiagnostic, snapshotPath);
|
||||
this.watch = new CLIWatchAdapter(basePath, ignoreRules, watchEnabled, reportDiagnostic);
|
||||
this.status = new CLIStatusAdapter();
|
||||
this.converter = new CLIConverterAdapter();
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
import { describe, expect, it, vi, beforeEach } from "vitest";
|
||||
import type { IStorageEventWatchHandlers } from "@vrtmrz/livesync-commonlib/compat/managers/adapters";
|
||||
import type { NodeFile } from "@/apps/cli/adapters/NodeTypes";
|
||||
import { fs, os, path } from "@vrtmrz/livesync-commonlib/node";
|
||||
import { runtimeScopeDirectory } from "../runtimeState";
|
||||
|
||||
// ── chokidar mock ──────────────────────────────────────────────────────────────
|
||||
// Must be hoisted before imports that pull in chokidar.
|
||||
@@ -44,6 +46,27 @@ describe("CLIStorageEventManagerAdapter", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it("keeps pending snapshots with the selected database and Vault pair", async () => {
|
||||
const directory = fs.mkdtempSync(path.join(os.tmpdir(), "livesync-snapshot-"));
|
||||
let database = "first";
|
||||
let vault = "vault-a";
|
||||
const adapter = new CLIStorageEventManagerAdapter(directory, undefined, false, undefined, () =>
|
||||
path.join(runtimeScopeDirectory(directory, database, vault), "storage-events.json")
|
||||
);
|
||||
try {
|
||||
await adapter.persistence.saveSnapshot([{ type: "SENTINEL_FLUSH" }]);
|
||||
database = "second";
|
||||
expect(await adapter.persistence.loadSnapshot()).toBeNull();
|
||||
database = "first";
|
||||
vault = "vault-b";
|
||||
expect(await adapter.persistence.loadSnapshot()).toBeNull();
|
||||
vault = "vault-a";
|
||||
expect(await adapter.persistence.loadSnapshot()).toEqual([{ type: "SENTINEL_FLUSH" }]);
|
||||
} finally {
|
||||
fs.rmSync(directory, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("beginWatch is no-op when watchEnabled=false", async () => {
|
||||
const adapter = new CLIStorageEventManagerAdapter("/base", undefined, false);
|
||||
const handlers = makeHandlers();
|
||||
|
||||
@@ -1,9 +1,15 @@
|
||||
import { StorageEventManagerBase, type StorageEventManagerBaseDependencies } from "@vrtmrz/livesync-commonlib/compat/managers/StorageEventManager";
|
||||
import {
|
||||
StorageEventManagerBase,
|
||||
type StorageEventManagerBaseDependencies,
|
||||
} from "@vrtmrz/livesync-commonlib/compat/managers/StorageEventManager";
|
||||
import { CLIStorageEventManagerAdapter } from "./CLIStorageEventManagerAdapter";
|
||||
import type { IMinimumLiveSyncCommands, LiveSyncBaseCore } from "@/LiveSyncBaseCore";
|
||||
import type { ServiceContext } from "@vrtmrz/livesync-commonlib/context";
|
||||
import type { IgnoreRules } from "@/apps/cli/serviceModules/IgnoreRules";
|
||||
import { LOG_LEVEL_NOTICE } from "octagonal-wheels/common/logger";
|
||||
import { runtimeScopeDirectory } from "@/apps/cli/runtimeState";
|
||||
import type { NodeServiceContext } from "@/apps/cli/services/NodeServiceContext";
|
||||
import { path as nodePath } from "@vrtmrz/livesync-commonlib/node";
|
||||
// import type { IMinimumLiveSyncCommands } from "@vrtmrz/livesync-commonlib/compat/services/base/IService";
|
||||
|
||||
export class StorageEventManagerCLI extends StorageEventManagerBase<CLIStorageEventManagerAdapter> {
|
||||
@@ -16,12 +22,23 @@ export class StorageEventManagerCLI extends StorageEventManagerBase<CLIStorageEv
|
||||
ignoreRules?: IgnoreRules,
|
||||
watchEnabled?: boolean
|
||||
) {
|
||||
const adapter = new CLIStorageEventManagerAdapter(basePath, ignoreRules, watchEnabled, (message, detail) => {
|
||||
dependencies.APIService.addLog(message, LOG_LEVEL_NOTICE);
|
||||
if (detail !== undefined) {
|
||||
dependencies.APIService.addLog(detail, LOG_LEVEL_NOTICE);
|
||||
}
|
||||
});
|
||||
const context = core.services.context as NodeServiceContext;
|
||||
const adapter = new CLIStorageEventManagerAdapter(
|
||||
basePath,
|
||||
ignoreRules,
|
||||
watchEnabled,
|
||||
(message, detail) => {
|
||||
dependencies.APIService.addLog(message, LOG_LEVEL_NOTICE);
|
||||
if (detail !== undefined) {
|
||||
dependencies.APIService.addLog(detail, LOG_LEVEL_NOTICE);
|
||||
}
|
||||
},
|
||||
() =>
|
||||
nodePath.join(
|
||||
runtimeScopeDirectory(context.databasePath, dependencies.vaultService.getVaultName(), basePath),
|
||||
"storage-events.json"
|
||||
)
|
||||
);
|
||||
super(adapter, dependencies);
|
||||
this.core = core;
|
||||
}
|
||||
|
||||
@@ -12,7 +12,7 @@
|
||||
"buildRun": "npm run build && npm run cli --",
|
||||
"build:docker": "docker build -f Dockerfile -t livesync-cli ../../..",
|
||||
"check": "tsc -p tsconfig.json",
|
||||
"test:unit": "cd ../../.. && npx vitest run --config vitest.config.unit.ts src/apps/cli/main.unit.spec.ts src/apps/cli/main.bootstrap.unit.spec.ts src/apps/cli/settingsPersistence.unit.spec.ts src/apps/cli/services/NodeLocalStorage.unit.spec.ts src/apps/cli/commands/utils.unit.spec.ts src/apps/cli/commands/runCommand.unit.spec.ts src/apps/cli/commands/daemonCommand.unit.spec.ts src/apps/cli/commands/p2p.unit.spec.ts src/apps/cli/deploy/install.unit.spec.ts src/apps/cli/adapters/NodeFileSystemAdapter.unit.spec.ts",
|
||||
"test:unit": "cd ../../.. && npx vitest run --config vitest.config.unit.ts src/apps/cli/main.unit.spec.ts src/apps/cli/main.bootstrap.unit.spec.ts src/apps/cli/settingsPersistence.unit.spec.ts src/apps/cli/runtimeState.unit.spec.ts src/apps/cli/services/NodeKeyValueDBService.unit.spec.ts src/apps/cli/services/NodeLocalStorage.unit.spec.ts src/apps/cli/managers/CLIStorageEventManagerAdapter.unit.spec.ts src/apps/cli/commands/utils.unit.spec.ts src/apps/cli/commands/runCommand.unit.spec.ts src/apps/cli/commands/daemonCommand.unit.spec.ts src/apps/cli/commands/p2p.unit.spec.ts src/apps/cli/deploy/install.unit.spec.ts src/apps/cli/adapters/NodeFileSystemAdapter.unit.spec.ts",
|
||||
"test:e2e:two-vaults": "bash test/test-e2e-two-vaults-with-docker-linux.sh",
|
||||
"test:e2e:two-vaults:common": "bash test/test-e2e-two-vaults-common.sh",
|
||||
"test:e2e:two-vaults:matrix": "bash test/test-e2e-two-vaults-matrix.sh",
|
||||
@@ -37,6 +37,7 @@
|
||||
"test:e2e:coverage": "node ../../../test/e2e-cli/run-coverage.ts"
|
||||
},
|
||||
"dependencies": {
|
||||
"@aws-crypto/sha256-js": "5.2.0",
|
||||
"chokidar": "^4.0.0",
|
||||
"minimatch": "^10.2.5",
|
||||
"octagonal-wheels": "^0.1.55",
|
||||
|
||||
@@ -0,0 +1,222 @@
|
||||
// This module belongs exclusively to the Node.js CLI host.
|
||||
import { Sha256 } from "@aws-crypto/sha256-js";
|
||||
import { fs, path } from "@vrtmrz/livesync-commonlib/node";
|
||||
import type { CLICommand } from "./commands/types";
|
||||
|
||||
function sha256Hex(value: string): string {
|
||||
const hash = new Sha256();
|
||||
hash.update(new TextEncoder().encode(value));
|
||||
return Array.from(hash.digestSync(), (byte) => byte.toString(16).padStart(2, "0")).join("");
|
||||
}
|
||||
|
||||
export function runtimeScopeDirectory(databasePath: string, databaseName: string, vaultPath: string): string {
|
||||
const scope = sha256Hex(vaultPath);
|
||||
return path.join(runtimeDatabaseDirectory(databasePath, databaseName), scope);
|
||||
}
|
||||
|
||||
export function runtimeDatabaseDirectory(databasePath: string, databaseName: string): string {
|
||||
const database = sha256Hex(databaseName);
|
||||
return path.join(databasePath, ".livesync", "runtime", "scopes", database);
|
||||
}
|
||||
|
||||
export function writeFileAtomically(filePath: string, content: string): void {
|
||||
fs.mkdirSync(path.dirname(filePath), { recursive: true });
|
||||
const temporaryPath = `${filePath}.${process.pid}.${crypto.randomUUID()}.tmp`;
|
||||
try {
|
||||
const descriptor = fs.openSync(temporaryPath, "wx", 0o600);
|
||||
try {
|
||||
fs.writeFileSync(descriptor, content, "utf-8");
|
||||
fs.fsyncSync(descriptor);
|
||||
} finally {
|
||||
fs.closeSync(descriptor);
|
||||
}
|
||||
fs.renameSync(temporaryPath, filePath);
|
||||
syncDirectory(path.dirname(filePath));
|
||||
} finally {
|
||||
fs.rmSync(temporaryPath, { force: true });
|
||||
}
|
||||
}
|
||||
|
||||
function syncDirectory(directory: string): void {
|
||||
// Windows does not support opening a directory for fsync.
|
||||
if (process.platform === "win32") return;
|
||||
const descriptor = fs.openSync(directory, "r");
|
||||
try {
|
||||
fs.fsyncSync(descriptor);
|
||||
} finally {
|
||||
fs.closeSync(descriptor);
|
||||
}
|
||||
}
|
||||
|
||||
export function readOptionalFile(filePath: string): string | undefined {
|
||||
try {
|
||||
return fs.readFileSync(filePath, "utf-8");
|
||||
} catch (error) {
|
||||
if ((error as { code?: string }).code === "ENOENT") return undefined;
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
export function isReconstructionCommand(command: CLICommand): command is "fetch" | "rebuild" {
|
||||
return command === "fetch" || command === "rebuild";
|
||||
}
|
||||
|
||||
const READ_COMMANDS = new Set<CLICommand>([
|
||||
"cat",
|
||||
"cat-rev",
|
||||
"ls",
|
||||
"info",
|
||||
"remote-ls",
|
||||
"remote-export",
|
||||
"remote-status",
|
||||
"p2p-peers",
|
||||
]);
|
||||
|
||||
export function isMutatingCommand(command: CLICommand): boolean {
|
||||
return !READ_COMMANDS.has(command);
|
||||
}
|
||||
|
||||
interface ReconstructionMarker {
|
||||
version: 1;
|
||||
command: "fetch" | "rebuild";
|
||||
databasePath: string;
|
||||
vaultPath: string;
|
||||
settingsPath: string;
|
||||
settingsBeforeHash: string;
|
||||
phase: "running" | "completed";
|
||||
settingsText?: string;
|
||||
}
|
||||
|
||||
function hashSettings(content: string | undefined): string {
|
||||
return sha256Hex(JSON.stringify(content ?? null));
|
||||
}
|
||||
|
||||
export class ReconstructionState {
|
||||
readonly markerPath: string;
|
||||
private marker: ReconstructionMarker | undefined;
|
||||
|
||||
constructor(
|
||||
private readonly databasePath: string,
|
||||
private readonly vaultPath: string,
|
||||
private readonly settingsPath: string
|
||||
) {
|
||||
this.markerPath = path.join(databasePath, ".livesync", "reconstruction.json");
|
||||
}
|
||||
|
||||
prepare(command: CLICommand, force: boolean, writeSettings = false): void {
|
||||
const contents = readOptionalFile(this.markerPath);
|
||||
if (contents !== undefined) {
|
||||
const parsed: unknown = JSON.parse(contents);
|
||||
if (typeof parsed !== "object" || parsed === null || Array.isArray(parsed)) {
|
||||
throw new Error(`Invalid reconstruction state: ${this.markerPath}. Preserve it for recovery.`);
|
||||
}
|
||||
const marker = parsed as ReconstructionMarker;
|
||||
if (
|
||||
marker.version !== 1 ||
|
||||
!isReconstructionCommand(marker.command) ||
|
||||
(marker.phase !== "running" && marker.phase !== "completed") ||
|
||||
typeof marker.settingsPath !== "string" ||
|
||||
typeof marker.vaultPath !== "string" ||
|
||||
typeof marker.settingsBeforeHash !== "string" ||
|
||||
marker.databasePath !== this.databasePath ||
|
||||
(marker.phase === "completed" && typeof marker.settingsText !== "string")
|
||||
) {
|
||||
throw new Error(`Invalid reconstruction state: ${this.markerPath}. Preserve it for recovery.`);
|
||||
}
|
||||
this.marker = marker;
|
||||
if (marker.phase === "completed") {
|
||||
this.commit();
|
||||
} else if (
|
||||
(isMutatingCommand(command) || writeSettings) &&
|
||||
!(
|
||||
command === marker.command &&
|
||||
force &&
|
||||
marker.vaultPath === this.vaultPath &&
|
||||
marker.settingsPath === this.settingsPath
|
||||
)
|
||||
) {
|
||||
throw new Error(
|
||||
`An interrupted ${marker.command} requires recovery. Retry ${marker.command} --force with the same database, Vault, and settings paths. State: ${this.markerPath}`
|
||||
);
|
||||
}
|
||||
}
|
||||
if (isReconstructionCommand(command)) {
|
||||
this.marker = {
|
||||
version: 1,
|
||||
command,
|
||||
databasePath: this.databasePath,
|
||||
vaultPath: this.vaultPath,
|
||||
settingsPath: this.settingsPath,
|
||||
phase: "running",
|
||||
settingsBeforeHash: hashSettings(readOptionalFile(this.settingsPath)),
|
||||
};
|
||||
this.save();
|
||||
}
|
||||
}
|
||||
|
||||
complete(settingsText: string): void {
|
||||
if (!this.marker || this.marker.phase !== "running") throw new Error("Reconstruction was not started.");
|
||||
this.marker = { ...this.marker, phase: "completed", settingsText };
|
||||
this.save();
|
||||
}
|
||||
|
||||
commit(): void {
|
||||
const marker = this.marker;
|
||||
if (!marker || marker.phase !== "completed" || marker.settingsText === undefined) {
|
||||
throw new Error("Reconstruction has no completed settings update.");
|
||||
}
|
||||
const current = readOptionalFile(marker.settingsPath);
|
||||
if (current !== marker.settingsText && hashSettings(current) !== marker.settingsBeforeHash) {
|
||||
throw new Error(
|
||||
`The settings file changed after reconstruction: ${marker.settingsPath}. Preserve the reconstruction state and review both settings versions before continuing.`
|
||||
);
|
||||
}
|
||||
if (current !== marker.settingsText) writeFileAtomically(marker.settingsPath, marker.settingsText);
|
||||
this.clear();
|
||||
}
|
||||
|
||||
clear(): void {
|
||||
fs.rmSync(this.markerPath, { force: true });
|
||||
syncDirectory(path.dirname(this.markerPath));
|
||||
this.marker = undefined;
|
||||
}
|
||||
|
||||
private save(): void {
|
||||
writeFileAtomically(this.markerPath, JSON.stringify(this.marker, null, 2));
|
||||
}
|
||||
}
|
||||
|
||||
export interface LegacyStateIssue {
|
||||
path: string;
|
||||
reason: "unowned" | "invalid" | "unreadable";
|
||||
}
|
||||
|
||||
export function inspectLegacyState(databasePath: string, vaultPath: string): LegacyStateIssue[] {
|
||||
const issues: LegacyStateIssue[] = [];
|
||||
for (const [filePath, expectArray] of [
|
||||
[path.join(databasePath, ".livesync", "runtime", "keyvalue-db.json"), false],
|
||||
[path.join(vaultPath, ".livesync-snapshot.json"), true],
|
||||
] as const) {
|
||||
let contents: string | undefined;
|
||||
try {
|
||||
contents = readOptionalFile(filePath);
|
||||
} catch {
|
||||
issues.push({ path: filePath, reason: "unreadable" });
|
||||
continue;
|
||||
}
|
||||
if (contents === undefined) continue;
|
||||
let value: unknown;
|
||||
try {
|
||||
value = JSON.parse(contents);
|
||||
} catch {
|
||||
issues.push({ path: filePath, reason: "invalid" });
|
||||
continue;
|
||||
}
|
||||
if (value === null || typeof value !== "object" || Array.isArray(value) !== expectArray) {
|
||||
issues.push({ path: filePath, reason: "invalid" });
|
||||
} else if (Object.keys(value).length > 0) {
|
||||
issues.push({ path: filePath, reason: "unowned" });
|
||||
}
|
||||
}
|
||||
return issues;
|
||||
}
|
||||
@@ -0,0 +1,160 @@
|
||||
import { fs, os, path } from "@vrtmrz/livesync-commonlib/node";
|
||||
import { afterEach, beforeEach, describe, expect, it } from "vitest";
|
||||
import {
|
||||
ReconstructionState,
|
||||
inspectLegacyState,
|
||||
readOptionalFile,
|
||||
runtimeDatabaseDirectory,
|
||||
runtimeScopeDirectory,
|
||||
} from "./runtimeState";
|
||||
|
||||
describe("CLI runtime scope compatibility", () => {
|
||||
it("retains SHA-256 directory names for database and Unicode Vault paths", () => {
|
||||
expect(runtimeScopeDirectory("state", "headless-app", "/vault/日本語😀")).toBe(
|
||||
path.join(
|
||||
"state",
|
||||
".livesync",
|
||||
"runtime",
|
||||
"scopes",
|
||||
"511afda5613706862fa4add801b3f9b19e0db2acacc07b9218f2d01ff41ca61d",
|
||||
"542d31d6918ad8bf915e204e41e479fdbd7251690f182fd802378323b4c6de23"
|
||||
)
|
||||
);
|
||||
expect(path.basename(runtimeDatabaseDirectory("state", ""))).toBe(
|
||||
"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe("CLI reconstruction state", () => {
|
||||
let directory: string;
|
||||
let settingsPath: string;
|
||||
let state: ReconstructionState;
|
||||
|
||||
beforeEach(() => {
|
||||
directory = fs.mkdtempSync(path.join(os.tmpdir(), "livesync-reconstruction-"));
|
||||
settingsPath = path.join(directory, "settings.json");
|
||||
fs.writeFileSync(settingsPath, JSON.stringify({ additionalSuffixOfDatabaseName: "original" }));
|
||||
state = new ReconstructionState(directory, path.join(directory, "vault"), settingsPath);
|
||||
});
|
||||
|
||||
afterEach(() => fs.rmSync(directory, { recursive: true, force: true }));
|
||||
|
||||
it("retains the persisted settings fingerprint across hash implementations", () => {
|
||||
state.prepare("fetch", true);
|
||||
expect(JSON.parse(fs.readFileSync(state.markerPath, "utf-8"))).toMatchObject({
|
||||
settingsBeforeHash: "29e04e3dd1d10b352678a922ff5a03fbe529d3505e9cf46c2ecc860f44478ae8",
|
||||
});
|
||||
});
|
||||
|
||||
it("persists progress before returning and permits only the matching forced retry", () => {
|
||||
state.prepare("fetch", true);
|
||||
expect(JSON.parse(fs.readFileSync(state.markerPath, "utf-8"))).toMatchObject({
|
||||
phase: "running",
|
||||
command: "fetch",
|
||||
});
|
||||
if (process.platform !== "win32") expect(fs.statSync(state.markerPath).mode & 0o777).toBe(0o600);
|
||||
expect(() => state.prepare("sync", false)).toThrow("interrupted fetch");
|
||||
expect(() => state.prepare("ls", false, true)).toThrow("interrupted fetch");
|
||||
expect(() => state.prepare("fetch", false)).toThrow("interrupted fetch");
|
||||
expect(() => state.prepare("rebuild", true)).toThrow("interrupted fetch");
|
||||
expect(() =>
|
||||
new ReconstructionState(directory, path.join(directory, "other"), settingsPath).prepare("fetch", true)
|
||||
).toThrow("interrupted fetch");
|
||||
expect(() =>
|
||||
new ReconstructionState(
|
||||
directory,
|
||||
path.join(directory, "vault"),
|
||||
path.join(directory, "other.json")
|
||||
).prepare("fetch", true)
|
||||
).toThrow("interrupted fetch");
|
||||
expect(() => state.prepare("ls", false)).not.toThrow();
|
||||
expect(() => state.prepare("fetch", true)).not.toThrow();
|
||||
});
|
||||
|
||||
it("recovers completed settings before permitting another command", () => {
|
||||
state.prepare("fetch", true);
|
||||
const completed = JSON.stringify({ additionalSuffixOfDatabaseName: "headless-app", retained: true });
|
||||
state.complete(completed);
|
||||
|
||||
new ReconstructionState(directory, path.join(directory, "vault"), settingsPath).prepare("sync", false);
|
||||
|
||||
expect(fs.readFileSync(settingsPath, "utf-8")).toBe(completed);
|
||||
expect(readOptionalFile(state.markerPath)).toBeUndefined();
|
||||
});
|
||||
|
||||
it.each(["normal completion", "restart"] as const)(
|
||||
"preserves user settings changed during reconstruction at %s",
|
||||
(phase) => {
|
||||
state.prepare("rebuild", true);
|
||||
state.complete(JSON.stringify({ additionalSuffixOfDatabaseName: "headless-app" }));
|
||||
const edited = JSON.stringify({ additionalSuffixOfDatabaseName: "user-selection", unique: "edit" });
|
||||
fs.writeFileSync(settingsPath, edited);
|
||||
|
||||
const commit =
|
||||
phase === "restart"
|
||||
? () =>
|
||||
new ReconstructionState(directory, path.join(directory, "vault"), settingsPath).prepare(
|
||||
"sync",
|
||||
false
|
||||
)
|
||||
: () => state.commit();
|
||||
|
||||
expect(commit).toThrow("settings file changed");
|
||||
expect(fs.readFileSync(settingsPath, "utf-8")).toBe(edited);
|
||||
expect(JSON.parse(fs.readFileSync(state.markerPath, "utf-8"))).toMatchObject({ phase: "completed" });
|
||||
}
|
||||
);
|
||||
|
||||
it("clears a completed marker after an interruption following the settings rename", () => {
|
||||
state.prepare("fetch", true);
|
||||
const completed = JSON.stringify({ additionalSuffixOfDatabaseName: "headless-app" });
|
||||
state.complete(completed);
|
||||
fs.writeFileSync(settingsPath, completed);
|
||||
state.prepare("ls", false);
|
||||
expect(readOptionalFile(state.markerPath)).toBeUndefined();
|
||||
});
|
||||
|
||||
it.each(["null", "[]", "{}", '{"version":2}', '{"version":1,"phase":"completed"}'])(
|
||||
"refuses malformed state %s without clearing it",
|
||||
(contents) => {
|
||||
fs.mkdirSync(path.dirname(state.markerPath), { recursive: true });
|
||||
fs.writeFileSync(state.markerPath, contents);
|
||||
expect(() => state.prepare("sync", false)).toThrow("Invalid reconstruction state");
|
||||
expect(fs.readFileSync(state.markerPath, "utf-8")).toBe(contents);
|
||||
}
|
||||
);
|
||||
|
||||
it.each(["null", "[]", '"text"', "{", ""])("reports invalid legacy KV %s without changing it", (contents) => {
|
||||
const legacy = path.join(directory, ".livesync/runtime/keyvalue-db.json");
|
||||
fs.mkdirSync(path.dirname(legacy), { recursive: true });
|
||||
fs.writeFileSync(legacy, contents);
|
||||
expect(inspectLegacyState(directory, directory)).toEqual([{ path: legacy, reason: "invalid" }]);
|
||||
expect(fs.readFileSync(legacy, "utf-8")).toBe(contents);
|
||||
});
|
||||
|
||||
it("reports both unreadable KV and invalid snapshot instead of hiding the second problem", () => {
|
||||
const legacy = path.join(directory, ".livesync/runtime/keyvalue-db.json");
|
||||
const snapshot = path.join(directory, ".livesync-snapshot.json");
|
||||
fs.mkdirSync(legacy, { recursive: true });
|
||||
fs.writeFileSync(snapshot, "{}");
|
||||
expect(inspectLegacyState(directory, directory)).toEqual([
|
||||
{ path: legacy, reason: "unreadable" },
|
||||
{ path: snapshot, reason: "invalid" },
|
||||
]);
|
||||
expect(fs.statSync(legacy).isDirectory()).toBe(true);
|
||||
expect(fs.readFileSync(snapshot, "utf-8")).toBe("{}");
|
||||
});
|
||||
|
||||
it("preserves non-empty unowned legacy state and ignores empty legacy containers", () => {
|
||||
const legacy = path.join(directory, ".livesync", "runtime", "keyvalue-db.json");
|
||||
fs.mkdirSync(path.dirname(legacy), { recursive: true });
|
||||
fs.writeFileSync(legacy, "{}");
|
||||
fs.writeFileSync(path.join(directory, ".livesync-snapshot.json"), "[]");
|
||||
expect(inspectLegacyState(directory, directory)).toEqual([]);
|
||||
const queued = JSON.stringify({ "queued-files": ["unique-edit.md"] });
|
||||
fs.writeFileSync(legacy, queued);
|
||||
expect(inspectLegacyState(directory, directory)).toEqual([{ path: legacy, reason: "unowned" }]);
|
||||
expect(fs.readFileSync(legacy, "utf-8")).toBe(queued);
|
||||
});
|
||||
});
|
||||
@@ -9,6 +9,7 @@ import type { IVaultService } from "@vrtmrz/livesync-commonlib/compat/services/b
|
||||
import type { SimpleStore } from "octagonal-wheels/databases/SimpleStoreBase";
|
||||
import { createInstanceLogFunction } from "@vrtmrz/livesync-commonlib/compat/services/lib/logUtils";
|
||||
import { fs as nodeFs, path as nodePath } from "@vrtmrz/livesync-commonlib/node";
|
||||
import { readOptionalFile, writeFileAtomically } from "@/apps/cli/runtimeState";
|
||||
|
||||
const NODE_KV_TYPED_KEY = "__nodeKvType";
|
||||
const NODE_KV_VALUES_KEY = "values";
|
||||
@@ -103,12 +104,11 @@ class NodeFileKeyValueDatabase implements KeyValueDatabase {
|
||||
}
|
||||
|
||||
private load() {
|
||||
try {
|
||||
const loaded = JSON.parse(nodeFs.readFileSync(this.filePath, "utf-8")) as Record<string, unknown>;
|
||||
this.data = new Map(Object.entries(loaded).map(([key, value]) => [key, deserializeFromNodeKV(value)]));
|
||||
} catch {
|
||||
this.data = new Map();
|
||||
}
|
||||
const contents = readOptionalFile(this.filePath);
|
||||
if (contents === undefined) return;
|
||||
const loaded: unknown = JSON.parse(contents);
|
||||
if (!isRecord(loaded) || Array.isArray(loaded)) throw new Error("Invalid Node key-value database");
|
||||
this.data = new Map(Object.entries(loaded).map(([key, value]) => [key, deserializeFromNodeKV(value)]));
|
||||
}
|
||||
|
||||
private flush() {
|
||||
@@ -116,7 +116,7 @@ class NodeFileKeyValueDatabase implements KeyValueDatabase {
|
||||
const serializable = Object.fromEntries(
|
||||
[...this.data.entries()].map(([key, value]) => [key, serializeForNodeKV(value)])
|
||||
);
|
||||
nodeFs.writeFileSync(this.filePath, JSON.stringify(serializable, null, 2), "utf-8");
|
||||
writeFileAtomically(this.filePath, JSON.stringify(serializable, null, 2));
|
||||
}
|
||||
|
||||
async get<T>(key: IDBValidKey): Promise<T> {
|
||||
@@ -163,9 +163,7 @@ class NodeFileKeyValueDatabase implements KeyValueDatabase {
|
||||
return filtered;
|
||||
}
|
||||
|
||||
async close(): Promise<void> {
|
||||
this.flush();
|
||||
}
|
||||
async close(): Promise<void> {}
|
||||
|
||||
async destroy(): Promise<void> {
|
||||
this.data.clear();
|
||||
@@ -185,7 +183,8 @@ export class NodeKeyValueDBService<T extends ServiceContext = ServiceContext>
|
||||
{
|
||||
private _kvDB: KeyValueDatabase | undefined;
|
||||
private _simpleStore: SimpleStore<unknown> | undefined;
|
||||
private filePath: string;
|
||||
private filePath: string | (() => string);
|
||||
private openedFilePath: string | undefined;
|
||||
private _log = createInstanceLogFunction("NodeKeyValueDBService");
|
||||
|
||||
get simpleStore() {
|
||||
@@ -202,7 +201,7 @@ export class NodeKeyValueDBService<T extends ServiceContext = ServiceContext>
|
||||
return this._kvDB;
|
||||
}
|
||||
|
||||
constructor(context: T, dependencies: NodeKeyValueDBDependencies<T>, filePath: string) {
|
||||
constructor(context: T, dependencies: NodeKeyValueDBDependencies<T>, filePath: string | (() => string)) {
|
||||
super(context);
|
||||
this.filePath = filePath;
|
||||
|
||||
@@ -215,7 +214,13 @@ export class NodeKeyValueDBService<T extends ServiceContext = ServiceContext>
|
||||
|
||||
private async openKeyValueDB(): Promise<boolean> {
|
||||
try {
|
||||
this._kvDB = new NodeFileKeyValueDatabase(this.filePath);
|
||||
const filePath = typeof this.filePath === "string" ? this.filePath : this.filePath();
|
||||
if (this._kvDB && this.openedFilePath === filePath) return true;
|
||||
await this._kvDB?.close();
|
||||
this._kvDB = undefined;
|
||||
this.openedFilePath = undefined;
|
||||
this._kvDB = new NodeFileKeyValueDatabase(filePath);
|
||||
this.openedFilePath = filePath;
|
||||
return true;
|
||||
} catch (ex) {
|
||||
this._log("Failed to open Node key-value database", LOG_LEVEL_NOTICE);
|
||||
@@ -226,8 +231,10 @@ export class NodeKeyValueDBService<T extends ServiceContext = ServiceContext>
|
||||
|
||||
private async _everyOnResetDatabase(): Promise<boolean> {
|
||||
try {
|
||||
await this._kvDB?.del("queued-files");
|
||||
if (!(await this.openKeyValueDB())) return false;
|
||||
await this._kvDB?.destroy();
|
||||
this._kvDB = undefined;
|
||||
this.openedFilePath = undefined;
|
||||
return await this.openKeyValueDB();
|
||||
} catch (ex) {
|
||||
this._log("Failed to reset Node key-value database", LOG_LEVEL_NOTICE);
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { fs, os, path } from "@vrtmrz/livesync-commonlib/node";
|
||||
import { createServiceContext } from "@vrtmrz/livesync-commonlib/compat/services/base/ServiceBase";
|
||||
import type { NodeKeyValueDBDependencies } from "./NodeKeyValueDBService";
|
||||
import { NodeKeyValueDBService } from "./NodeKeyValueDBService";
|
||||
@@ -45,3 +46,70 @@ describe("NodeKeyValueDBService.openSimpleStore", () => {
|
||||
await expect(store.get("key")).rejects.toThrow("KeyValueDB is not initialized yet");
|
||||
});
|
||||
});
|
||||
|
||||
describe("NodeKeyValueDBService persistence", () => {
|
||||
let directory: string;
|
||||
|
||||
beforeEach(() => {
|
||||
directory = fs.mkdtempSync(path.join(os.tmpdir(), "livesync-node-kv-"));
|
||||
});
|
||||
afterEach(() => fs.rmSync(directory, { recursive: true, force: true }));
|
||||
|
||||
function compose(filePath: string | (() => string)) {
|
||||
const handlers: Record<string, () => Promise<boolean>> = {};
|
||||
const event = (name: string) => ({
|
||||
addHandler: (handler: () => Promise<boolean>) => {
|
||||
handlers[name] = handler;
|
||||
},
|
||||
});
|
||||
const dependencies = {
|
||||
appLifecycle: { onSettingLoaded: event("load") },
|
||||
databaseEvents: {
|
||||
onResetDatabase: event("reset"),
|
||||
onDatabaseInitialisation: event("initialise"),
|
||||
onUnloadDatabase: event("unload"),
|
||||
onCloseDatabase: event("close"),
|
||||
},
|
||||
vault: {},
|
||||
} as unknown as NodeKeyValueDBDependencies;
|
||||
return { service: new NodeKeyValueDBService(createServiceContext(), dependencies, filePath), handlers };
|
||||
}
|
||||
|
||||
it("retargets existing SimpleStore handles after database selection and resets only the selected file", async () => {
|
||||
let selected = path.join(directory, "first.json");
|
||||
const { service, handlers } = compose(() => selected);
|
||||
const store = service.openSimpleStore("journal");
|
||||
expect(await handlers.load()).toBe(true);
|
||||
await store.set("checkpoint", new Set(["first"]));
|
||||
selected = path.join(directory, "second.json");
|
||||
expect(await handlers.initialise()).toBe(true);
|
||||
expect(await store.get("checkpoint")).toBeUndefined();
|
||||
await store.set("checkpoint", "second");
|
||||
expect(await handlers.reset()).toBe(true);
|
||||
expect(await store.get("checkpoint")).toBeUndefined();
|
||||
selected = path.join(directory, "first.json");
|
||||
expect(await handlers.initialise()).toBe(true);
|
||||
expect(await store.get("checkpoint")).toEqual(new Set(["first"]));
|
||||
});
|
||||
|
||||
it("does not overwrite newer data when a stale handle closes", async () => {
|
||||
const filePath = path.join(directory, "state.json");
|
||||
const first = compose(filePath);
|
||||
await first.handlers.load();
|
||||
await first.service.kvDB.set("queued-files", ["original.md"]);
|
||||
const second = compose(filePath);
|
||||
await second.handlers.load();
|
||||
await first.service.kvDB.set("queued-files", ["unique-edit.md"]);
|
||||
await second.handlers.close();
|
||||
expect(JSON.parse(fs.readFileSync(filePath, "utf-8"))["queued-files"]).toEqual(["unique-edit.md"]);
|
||||
});
|
||||
|
||||
it.each(["{", "null", "[]"])("refuses corrupt state %s without replacing it", async (contents) => {
|
||||
const filePath = path.join(directory, "state.json");
|
||||
fs.writeFileSync(filePath, contents);
|
||||
const { service, handlers } = compose(filePath);
|
||||
expect(await handlers.load()).toBe(false);
|
||||
expect(() => service.kvDB).toThrow("not initialized");
|
||||
expect(fs.readFileSync(filePath, "utf-8")).toBe(contents);
|
||||
});
|
||||
});
|
||||
@@ -6,7 +6,8 @@ import { ServiceContext, type StandardIo } from "@vrtmrz/livesync-commonlib/cont
|
||||
export class NodeServiceContext extends ServiceContext {
|
||||
constructor(
|
||||
readonly databasePath: string,
|
||||
readonly standardIo: StandardIo
|
||||
readonly standardIo: StandardIo,
|
||||
readonly runtimeVaultPath: string = databasePath
|
||||
) {
|
||||
super({ events: eventHub, translate: translateLiveSyncMessage });
|
||||
}
|
||||
|
||||
@@ -24,11 +24,14 @@ import { NodeKeyValueDBService } from "./NodeKeyValueDBService";
|
||||
import { createNodeLocalStorage } from "./NodeLocalStorage";
|
||||
import { DatabaseService } from "@vrtmrz/livesync-commonlib/compat/services/base/DatabaseService";
|
||||
import type { ObsidianLiveSyncSettings } from "@vrtmrz/livesync-commonlib/compat/common/types";
|
||||
import { path as nodePath } from "@vrtmrz/livesync-commonlib/node";
|
||||
import { fs, path as nodePath } from "@vrtmrz/livesync-commonlib/node";
|
||||
import type { KeyValueDBService } from "@vrtmrz/livesync-commonlib/compat/services/base/KeyValueDBService";
|
||||
import { PouchDB } from "@/apps/cli/lib/pouchdb-node";
|
||||
import { NodeServiceContext } from "./NodeServiceContext";
|
||||
import { setLang } from "@/common/translation";
|
||||
import { runtimeDatabaseDirectory, runtimeScopeDirectory } from "@/apps/cli/runtimeState";
|
||||
import { createDeletableValueStore } from "@vrtmrz/value-store";
|
||||
import type { SettingServiceDependencies } from "@vrtmrz/livesync-commonlib/compat/services/base/SettingService";
|
||||
|
||||
export { NodeServiceContext } from "./NodeServiceContext";
|
||||
|
||||
@@ -86,17 +89,31 @@ class NodeUIService<T extends ServiceContext> extends UIService<T> {
|
||||
}
|
||||
}
|
||||
|
||||
class NodeSettingService<T extends NodeServiceContext> extends InjectableSettingService<T> {
|
||||
declare readonly config: InjectableSettingService<T>["config"];
|
||||
|
||||
constructor(context: T, dependencies: SettingServiceDependencies) {
|
||||
super(context, dependencies);
|
||||
const databaseConfig = this.config;
|
||||
const vaultScope = nodePath.basename(runtimeScopeDirectory(context.databasePath, "", context.runtimeVaultPath));
|
||||
this.config = createDeletableValueStore({
|
||||
get: (key) => databaseConfig.getValue(`${key}-${vaultScope}`),
|
||||
set: (key, value) => databaseConfig.setValue(`${key}-${vaultScope}`, value),
|
||||
delete: (key) => databaseConfig.delete(`${key}-${vaultScope}`),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
export class NodeServiceHub<T extends NodeServiceContext> extends InjectableServiceHub<T> {
|
||||
constructor(basePath: string, context: T) {
|
||||
const runtimeDir = nodePath.join(basePath, ".livesync", "runtime");
|
||||
const localStoragePath = nodePath.join(runtimeDir, "local-storage.json");
|
||||
const keyValueDBPath = nodePath.join(runtimeDir, "keyvalue-db.json");
|
||||
|
||||
const API = new HeadlessAPIService<T>(context);
|
||||
const conflict = new InjectableConflictService(context);
|
||||
const fileProcessing = new InjectableFileProcessingService(context);
|
||||
|
||||
const setting = new InjectableSettingService<T>(context, {
|
||||
const setting = new NodeSettingService<T>(context, {
|
||||
APIService: API,
|
||||
onDisplayLanguageChanged: setLang,
|
||||
deviceValues: createNodeLocalStorage(localStoragePath),
|
||||
@@ -118,8 +135,13 @@ export class NodeServiceHub<T extends NodeServiceContext> extends InjectableServ
|
||||
settingService: setting,
|
||||
APIService: API,
|
||||
});
|
||||
const scopeDirectory = () => runtimeScopeDirectory(basePath, vault.getVaultName(), context.runtimeVaultPath);
|
||||
const test = new InjectableTestService(context);
|
||||
const databaseEvents = new InjectableDatabaseEventService(context);
|
||||
databaseEvents.onResetDatabase.addHandler(async () => {
|
||||
fs.rmSync(runtimeDatabaseDirectory(basePath, vault.getVaultName()), { force: true, recursive: true });
|
||||
return true;
|
||||
});
|
||||
const path = new PathServiceCompat(context, {
|
||||
settingService: setting,
|
||||
});
|
||||
@@ -156,7 +178,7 @@ export class NodeServiceHub<T extends NodeServiceContext> extends InjectableServ
|
||||
databaseEvents,
|
||||
vault,
|
||||
},
|
||||
keyValueDBPath
|
||||
() => nodePath.join(scopeDirectory(), "keyvalue-db.json")
|
||||
);
|
||||
|
||||
const control = new ControlService(context, {
|
||||
|
||||
@@ -77,6 +77,32 @@ npm run test:e2e:docker:mirror --workspace self-hosted-livesync-cli
|
||||
|
||||
The expected result is `PASS=7 FAIL=0`. These mirror suites use temporary local databases and need no CouchDB service. The complete `test:e2e:docker:all` command also runs the other Docker CLI suites and manages a disposable CouchDB fixture; use it to check the full Docker CI gate. Keep the ordinary-edit and unknown-provenance scenarios aligned between the Node.js and Bash suites.
|
||||
|
||||
## CLI Fetch database continuity
|
||||
|
||||
After building the CLI, run `node test/e2e-cli/run-task.ts test:fetch-database-continuity`
|
||||
with Node.js 26 and a disposable S3-compatible service. Set
|
||||
`LIVESYNC_ISSUE1304_MINIO_ENDPOINT`, `LIVESYNC_ISSUE1304_MINIO_ACCESS_KEY`,
|
||||
`LIVESYNC_ISSUE1304_MINIO_SECRET_KEY`, and `LIVESYNC_ISSUE1304_MINIO_BUCKET`.
|
||||
The bucket must be fresh and empty for every run. Set
|
||||
`LIVESYNC_ISSUE1304_EVIDENCE_DIR` to an evidence directory outside the worktree.
|
||||
This test writes remote data; never point it at a production bucket.
|
||||
|
||||
The test seeds an encrypted, path-obfuscated file with a second-generation
|
||||
revision, imports one Setup URI on two clients, fetches on A without
|
||||
`--write-settings`, and restarts A while B remains online. It checks the saved
|
||||
database suffix, file bytes, original revision, and absence of a duplicate root.
|
||||
Set `LIVESYNC_ISSUE1304_POSITIVE_CONTROL=1` and
|
||||
`LIVESYNC_ISSUE1304_POSITIVE_CONTROL_BUCKET` to a second fresh bucket to exercise
|
||||
the explicit `--write-settings` variant as well.
|
||||
|
||||
For historical reproduction against the unmodified implementation, set
|
||||
`LIVESYNC_ISSUE1304_VERIFY_LEGACY_FAILURE=1`. This preserves the original
|
||||
same-content conflict-resolution sequence and its file-preservation assertion;
|
||||
the affected implementation fails because A's file disappears. Leave this
|
||||
variable unset when validating the correction. The test saves redacted process
|
||||
logs, file hashes, and database evidence. It does not cover Obsidian runtime or
|
||||
CouchDB behaviour.
|
||||
|
||||
## Multiple-device conflict regression tests
|
||||
|
||||
### Preparation and execution
|
||||
|
||||
@@ -47,6 +47,11 @@
|
||||
"test-settings-writeback.ts"
|
||||
]
|
||||
},
|
||||
"test:fetch-database-continuity": {
|
||||
"test": [
|
||||
"test-fetch-database-continuity.ts"
|
||||
]
|
||||
},
|
||||
"test:remote-administration-exit-codes": {
|
||||
"test": [
|
||||
"test-remote-administration-exit-codes.ts"
|
||||
|
||||
@@ -0,0 +1,568 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { createHash } from "node:crypto";
|
||||
import { cp, mkdir, readFile, readdir, writeFile } from "node:fs/promises";
|
||||
import { join, resolve } from "node:path";
|
||||
import { test } from "node:test";
|
||||
import { CLI_DIR, CLI_DIST } from "./helpers/cli.ts";
|
||||
import { applyRemoteSyncSettings, generateSetupUriFromSettings, initSettingsFile } from "./helpers/settings.ts";
|
||||
import { TempDir } from "./helpers/temp.ts";
|
||||
import { spawnProcess, type RunningProcess } from "./helpers/process.ts";
|
||||
|
||||
type RemoteSettings = {
|
||||
endpoint: string;
|
||||
accessKey: string;
|
||||
secretKey: string;
|
||||
bucket: string;
|
||||
evidenceDir: string;
|
||||
};
|
||||
|
||||
type CliResult = { code: number; stdout: string; stderr: string; combined: string };
|
||||
type ScenarioMode = "reproduction" | "write-settings-control";
|
||||
|
||||
const SETUP_PASSPHRASE = "issue-1304-setup-passphrase";
|
||||
const RESOLUTION_MARKER = "Resolving [same]: Conflicted revision has been deleted";
|
||||
const INITIAL_REPLICATION_COMPLETE = "[Daemon] Initial replication complete";
|
||||
const INITIAL_REPLICATION_FAILED = "[Daemon] Initial replication failed";
|
||||
const VERIFY_LEGACY_FAILURE = process.env["LIVESYNC_ISSUE1304_VERIFY_LEGACY_FAILURE"] === "1";
|
||||
|
||||
function requiredEnv(name: string): string {
|
||||
const value = process.env[name]?.trim();
|
||||
if (!value) throw new Error(`Required environment variable is missing: ${name}`);
|
||||
return value;
|
||||
}
|
||||
|
||||
function remoteSettings(bucketOverride?: string): RemoteSettings {
|
||||
return {
|
||||
endpoint: requiredEnv("LIVESYNC_ISSUE1304_MINIO_ENDPOINT").replace(/\/$/, ""),
|
||||
accessKey: requiredEnv("LIVESYNC_ISSUE1304_MINIO_ACCESS_KEY"),
|
||||
secretKey: requiredEnv("LIVESYNC_ISSUE1304_MINIO_SECRET_KEY"),
|
||||
bucket: bucketOverride ?? requiredEnv("LIVESYNC_ISSUE1304_MINIO_BUCKET"),
|
||||
evidenceDir: resolve(requiredEnv("LIVESYNC_ISSUE1304_EVIDENCE_DIR")),
|
||||
};
|
||||
}
|
||||
|
||||
function redactText(value: string, secrets: readonly string[]): string {
|
||||
let redacted = value;
|
||||
for (const secret of secrets) {
|
||||
if (!secret) continue;
|
||||
redacted = redacted.split(secret).join("[redacted]");
|
||||
const encoded = encodeURIComponent(secret);
|
||||
if (encoded !== secret) redacted = redacted.split(encoded).join("[redacted]");
|
||||
}
|
||||
return redacted;
|
||||
}
|
||||
|
||||
async function streamText(stream: ReadableStream<Uint8Array>, onText: (text: string) => void): Promise<void> {
|
||||
const reader = stream.getReader();
|
||||
const decoder = new TextDecoder();
|
||||
try {
|
||||
while (true) {
|
||||
const { done, value } = await reader.read();
|
||||
if (done) break;
|
||||
if (value) onText(decoder.decode(value, { stream: true }));
|
||||
}
|
||||
const tail = decoder.decode();
|
||||
if (tail) onText(tail);
|
||||
} finally {
|
||||
reader.releaseLock();
|
||||
}
|
||||
}
|
||||
|
||||
class CapturedCliProcess {
|
||||
readonly child: RunningProcess;
|
||||
#stdout = "";
|
||||
#stderr = "";
|
||||
#secrets = [
|
||||
process.env["LIVESYNC_ISSUE1304_MINIO_ENDPOINT"] ?? "",
|
||||
process.env["LIVESYNC_ISSUE1304_MINIO_ACCESS_KEY"] ?? "",
|
||||
process.env["LIVESYNC_ISSUE1304_MINIO_SECRET_KEY"] ?? "",
|
||||
SETUP_PASSPHRASE,
|
||||
"issue-1304-content-passphrase",
|
||||
];
|
||||
#streamsDone: Promise<void>;
|
||||
|
||||
constructor(args: string[]) {
|
||||
this.child = spawnProcess(process.execPath, {
|
||||
args: [CLI_DIST, ...args],
|
||||
cwd: CLI_DIR,
|
||||
stdin: "null",
|
||||
stdout: "piped",
|
||||
stderr: "piped",
|
||||
});
|
||||
this.#streamsDone = Promise.all([
|
||||
streamText(this.child.stdout, (text) => (this.#stdout += text)),
|
||||
streamText(this.child.stderr, (text) => (this.#stderr += text)),
|
||||
]).then(() => undefined);
|
||||
}
|
||||
|
||||
get combined(): string {
|
||||
return this.#stdout + this.#stderr;
|
||||
}
|
||||
|
||||
async waitUntil(predicate: (output: string) => boolean, description: string, timeoutMs = 60_000): Promise<void> {
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
while (Date.now() < deadline) {
|
||||
if (predicate(this.combined)) return;
|
||||
const tick = Math.min(100, deadline - Date.now());
|
||||
const event = await Promise.race([
|
||||
this.child.status.then((status) => ({ kind: "exit" as const, status })),
|
||||
new Promise<{ kind: "tick" }>((resolveTick) => setTimeout(() => resolveTick({ kind: "tick" }), tick)),
|
||||
]);
|
||||
if (event.kind === "exit") {
|
||||
await this.#streamsDone;
|
||||
if (predicate(this.combined)) return;
|
||||
throw new Error(
|
||||
`${description}; process exited with ${event.status.code}\n${redactText(this.combined, this.#secrets).slice(-5000)}`
|
||||
);
|
||||
}
|
||||
}
|
||||
throw new Error(`Timed out waiting for ${description}\n${redactText(this.combined, this.#secrets).slice(-5000)}`);
|
||||
}
|
||||
|
||||
async stop(): Promise<number> {
|
||||
try {
|
||||
this.child.kill("SIGTERM");
|
||||
} catch {
|
||||
// A completed process needs no further signal.
|
||||
}
|
||||
const status = await this.child.status;
|
||||
await this.#streamsDone;
|
||||
return status.code;
|
||||
}
|
||||
}
|
||||
|
||||
async function runCliQuietly(args: string[], input?: string): Promise<CliResult> {
|
||||
const child = spawnProcess(process.execPath, {
|
||||
args: [CLI_DIST, ...args],
|
||||
cwd: CLI_DIR,
|
||||
stdin: input === undefined ? "null" : "piped",
|
||||
stdout: "piped",
|
||||
stderr: "piped",
|
||||
});
|
||||
let stdout = "";
|
||||
let stderr = "";
|
||||
const streamsDone = Promise.all([
|
||||
streamText(child.stdout, (text) => (stdout += text)),
|
||||
streamText(child.stderr, (text) => (stderr += text)),
|
||||
]);
|
||||
if (input !== undefined) {
|
||||
const writer = child.stdin.getWriter();
|
||||
await writer.write(new TextEncoder().encode(input));
|
||||
await writer.close();
|
||||
}
|
||||
const [status] = await Promise.all([child.status, streamsDone]);
|
||||
return { code: status.code, stdout, stderr, combined: stdout + stderr };
|
||||
}
|
||||
|
||||
function countOccurrences(value: string, needle: string): number {
|
||||
return value.split(needle).length - 1;
|
||||
}
|
||||
|
||||
function safeCommand(
|
||||
label: string,
|
||||
result: CliResult,
|
||||
commandLog: Array<{ label: string; code: number; output: string }>,
|
||||
secrets: readonly string[]
|
||||
): void {
|
||||
const output = redactText(result.combined, secrets);
|
||||
commandLog.push({ label, code: result.code, output });
|
||||
assert.equal(result.code, 0, `${label} failed with exit code ${result.code}\n${output.slice(-5000)}`);
|
||||
}
|
||||
|
||||
async function fileState(filePath: string): Promise<{ exists: boolean; bytes?: number; sha256?: string }> {
|
||||
try {
|
||||
const contents = await readFile(filePath);
|
||||
return {
|
||||
exists: true,
|
||||
bytes: contents.byteLength,
|
||||
sha256: createHash("sha256").update(contents).digest("hex"),
|
||||
};
|
||||
} catch (error) {
|
||||
if ((error as NodeJS.ErrnoException).code === "ENOENT") return { exists: false };
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
async function collectDatabaseTree(root: string, relative = ""): Promise<Array<{ path: string; bytes: number; sha256: string }>> {
|
||||
const result: Array<{ path: string; bytes: number; sha256: string }> = [];
|
||||
let entries;
|
||||
try {
|
||||
entries = await readdir(join(root, relative), { withFileTypes: true });
|
||||
} catch (error) {
|
||||
if ((error as NodeJS.ErrnoException).code === "ENOENT") return result;
|
||||
throw error;
|
||||
}
|
||||
entries.sort((left, right) => left.name.localeCompare(right.name));
|
||||
for (const entry of entries) {
|
||||
if (!relative && entry.name === ".livesync") continue;
|
||||
const child = relative ? join(relative, entry.name) : entry.name;
|
||||
const absolute = join(root, child);
|
||||
if (entry.isDirectory()) {
|
||||
result.push(...(await collectDatabaseTree(root, child)));
|
||||
} else if (entry.isFile()) {
|
||||
const contents = await readFile(absolute);
|
||||
result.push({
|
||||
path: child,
|
||||
bytes: contents.byteLength,
|
||||
sha256: createHash("sha256").update(contents).digest("hex"),
|
||||
});
|
||||
}
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
async function copyDatabaseTree(source: string, destination: string): Promise<void> {
|
||||
await mkdir(destination, { recursive: true });
|
||||
let entries;
|
||||
try {
|
||||
entries = await readdir(source, { withFileTypes: true });
|
||||
} catch (error) {
|
||||
if ((error as NodeJS.ErrnoException).code === "ENOENT") return;
|
||||
throw error;
|
||||
}
|
||||
for (const entry of entries) {
|
||||
if (entry.name === ".livesync") continue;
|
||||
await cp(join(source, entry.name), join(destination, entry.name), { recursive: true, errorOnExist: false });
|
||||
}
|
||||
}
|
||||
|
||||
async function sanitisedRuntimeState(root: string, secrets: readonly string[]): Promise<Record<string, string>> {
|
||||
const runtimeRoot = join(root, ".livesync", "runtime");
|
||||
const files: Record<string, string> = {};
|
||||
let entries;
|
||||
try {
|
||||
entries = await readdir(runtimeRoot, { withFileTypes: true });
|
||||
} catch (error) {
|
||||
if ((error as NodeJS.ErrnoException).code === "ENOENT") return files;
|
||||
throw error;
|
||||
}
|
||||
for (const entry of entries) {
|
||||
if (!entry.isFile() || !entry.name.endsWith(".json")) continue;
|
||||
const text = await readFile(join(runtimeRoot, entry.name), "utf8");
|
||||
try {
|
||||
files[entry.name] = JSON.stringify(JSON.parse(text), (_key, value: unknown) => {
|
||||
return typeof value === "string" ? redactText(value, secrets) : value;
|
||||
}, 2);
|
||||
} catch {
|
||||
files[entry.name] = redactText(text, secrets);
|
||||
}
|
||||
}
|
||||
return files;
|
||||
}
|
||||
|
||||
async function settingsSummary(settingsPath: string): Promise<Record<string, unknown> | null> {
|
||||
try {
|
||||
const settings = JSON.parse(await readFile(settingsPath, "utf8")) as Record<string, unknown>;
|
||||
return {
|
||||
additionalSuffixOfDatabaseName: settings.additionalSuffixOfDatabaseName ?? null,
|
||||
remoteType: settings.remoteType ?? null,
|
||||
liveSync: settings.liveSync ?? null,
|
||||
syncOnStart: settings.syncOnStart ?? null,
|
||||
};
|
||||
} catch (error) {
|
||||
if ((error as NodeJS.ErrnoException).code === "ENOENT") return null;
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
async function runScenario(mode: ScenarioMode, bucket: string): Promise<void> {
|
||||
const remote = remoteSettings(bucket);
|
||||
const evidenceRoot = remote.evidenceDir;
|
||||
const evidenceName = `${mode}-${Date.now()}-${process.pid}`;
|
||||
const evidenceDir = join(evidenceRoot, evidenceName);
|
||||
const workDir = await TempDir.create(`livesync-cli-issue1304-${mode}`);
|
||||
const scenarioId = `${Date.now()}-${process.pid}`;
|
||||
const suffix = `issue1304-${scenarioId}`;
|
||||
const targetPath = `notes/issue1304-${scenarioId}/continuity.md`;
|
||||
const initialContent = `issue-1304 initial ${scenarioId}\n`;
|
||||
const finalContent = `issue-1304 final ${scenarioId}\n`;
|
||||
const finalBytes = new TextEncoder().encode(finalContent);
|
||||
const secrets = [remote.endpoint, remote.accessKey, remote.secretKey, SETUP_PASSPHRASE, "issue-1304-content-passphrase"];
|
||||
const commandLog: Array<{ label: string; code: number; output: string }> = [];
|
||||
const processes: Record<string, CapturedCliProcess | undefined> = {};
|
||||
const settingsTimeline: Record<string, unknown> = {};
|
||||
const postResolutionInfo: Record<string, Record<string, unknown> | null> = {};
|
||||
let observedFinalFileA: { exists: boolean; bytes?: number; sha256?: string } | undefined;
|
||||
let observedFinalContentA: string | undefined;
|
||||
let setupUri: string | undefined;
|
||||
let failure: unknown;
|
||||
|
||||
const databaseSeed = workDir.join("database-seed");
|
||||
const databaseA = workDir.join("database-a");
|
||||
const databaseB = workDir.join("database-b");
|
||||
const vaultSeed = workDir.join("vault-seed");
|
||||
const vaultA = workDir.join("vault-a");
|
||||
const vaultB = workDir.join("vault-b");
|
||||
const settingsSeed = workDir.join("settings-seed.json");
|
||||
const settingsA = workDir.join("settings-a.json");
|
||||
const settingsB = workDir.join("settings-b.json");
|
||||
const finalFileA = join(vaultA, targetPath);
|
||||
const finalFileB = join(vaultB, targetPath);
|
||||
|
||||
try {
|
||||
await Promise.all([
|
||||
mkdir(databaseSeed, { recursive: true }),
|
||||
mkdir(databaseA, { recursive: true }),
|
||||
mkdir(databaseB, { recursive: true }),
|
||||
mkdir(vaultSeed, { recursive: true }),
|
||||
mkdir(vaultA, { recursive: true }),
|
||||
mkdir(vaultB, { recursive: true }),
|
||||
]);
|
||||
await initSettingsFile(settingsSeed);
|
||||
await applyRemoteSyncSettings(settingsSeed, {
|
||||
remoteType: "MINIO",
|
||||
minioBucket: remote.bucket,
|
||||
minioEndpoint: remote.endpoint,
|
||||
minioAccessKey: remote.accessKey,
|
||||
minioSecretKey: remote.secretKey,
|
||||
encrypt: true,
|
||||
passphrase: "issue-1304-content-passphrase",
|
||||
usePathObfuscation: true,
|
||||
});
|
||||
const seedSettings = JSON.parse(await readFile(settingsSeed, "utf8")) as Record<string, unknown>;
|
||||
seedSettings.additionalSuffixOfDatabaseName = suffix;
|
||||
await writeFile(settingsSeed, JSON.stringify(seedSettings, null, 2));
|
||||
settingsTimeline.setupSuffix = suffix;
|
||||
setupUri = await generateSetupUriFromSettings(settingsSeed, SETUP_PASSPHRASE, true);
|
||||
secrets.push(setupUri);
|
||||
|
||||
const seedRun = async (label: string, args: string[], input?: string) => {
|
||||
const result = await runCliQuietly(args, input);
|
||||
safeCommand(label, result, commandLog, secrets);
|
||||
return result;
|
||||
};
|
||||
|
||||
await seedRun("seed-put-initial", [databaseSeed, "--vault", vaultSeed, "--settings", settingsSeed, "put", targetPath], initialContent);
|
||||
await seedRun("seed-sync-initial", [databaseSeed, "--vault", vaultSeed, "--settings", settingsSeed, "sync"]);
|
||||
await seedRun("seed-put-final", [databaseSeed, "--vault", vaultSeed, "--settings", settingsSeed, "put", targetPath], finalContent);
|
||||
await seedRun("seed-sync-final", [databaseSeed, "--vault", vaultSeed, "--settings", settingsSeed, "sync"]);
|
||||
const seedInfoResult = await seedRun("seed-info", [databaseSeed, "--vault", vaultSeed, "--settings", settingsSeed, "info", targetPath]);
|
||||
const seedInfo = JSON.parse(seedInfoResult.stdout) as { revision?: string; revisions?: string[] };
|
||||
const seedRevisionGeneration = Number(seedInfo.revision?.split("-")[0] ?? 0);
|
||||
assert(
|
||||
seedRevisionGeneration >= 2 && (seedInfo.revisions?.length ?? 0) >= 1,
|
||||
`Expected a remote seed revision with generation >= 2; got ${JSON.stringify({ revision: seedInfo.revision, revisions: seedInfo.revisions })}`
|
||||
);
|
||||
settingsTimeline.seedRevision = seedInfo.revision;
|
||||
|
||||
await initSettingsFile(settingsA);
|
||||
await initSettingsFile(settingsB);
|
||||
const setupA = await runCliQuietly(
|
||||
[databaseA, "--vault", vaultA, "--settings", settingsA, "setup", setupUri],
|
||||
`${SETUP_PASSPHRASE}\n`
|
||||
);
|
||||
safeCommand("setup-client-a", setupA, commandLog, secrets);
|
||||
const setupB = await runCliQuietly(
|
||||
[databaseB, "--vault", vaultB, "--settings", settingsB, "setup", setupUri],
|
||||
`${SETUP_PASSPHRASE}\n`
|
||||
);
|
||||
safeCommand("setup-client-b", setupB, commandLog, secrets);
|
||||
|
||||
const bFetchArgs = [databaseB, "--vault", vaultB, "--settings", settingsB, "--force", "fetch", "--write-settings"];
|
||||
const bFetch = await runCliQuietly(bFetchArgs);
|
||||
safeCommand("fetch-client-b", bFetch, commandLog, secrets);
|
||||
assert.equal(await readFile(finalFileB, "utf8"), finalContent, "Client B did not fetch the final file bytes");
|
||||
settingsTimeline.clientBSuffixAfterFetch = (await settingsSummary(settingsB))?.additionalSuffixOfDatabaseName;
|
||||
assert.equal(settingsTimeline.clientBSuffixAfterFetch, "headless-app");
|
||||
|
||||
// Client B remains live so it can resolve the equal-content root after A
|
||||
// scans its old database and uploads a new root.
|
||||
processes.clientB = new CapturedCliProcess([
|
||||
"-v",
|
||||
databaseB,
|
||||
"--vault",
|
||||
vaultB,
|
||||
"--settings",
|
||||
settingsB,
|
||||
"--interval",
|
||||
"1",
|
||||
"daemon",
|
||||
]);
|
||||
await processes.clientB.waitUntil((output) => output.includes(INITIAL_REPLICATION_COMPLETE), "Client B initial replication");
|
||||
|
||||
settingsTimeline.clientASuffixBeforeFetch = (await settingsSummary(settingsA))?.additionalSuffixOfDatabaseName;
|
||||
assert.equal(settingsTimeline.clientASuffixBeforeFetch, suffix);
|
||||
const aFetchArgs = [databaseA, "--vault", vaultA, "--settings", settingsA, "--force", "fetch"];
|
||||
if (mode === "write-settings-control") aFetchArgs.push("--write-settings");
|
||||
const aFetch = await runCliQuietly(aFetchArgs);
|
||||
safeCommand("fetch-client-a", aFetch, commandLog, secrets);
|
||||
assert.equal(await readFile(finalFileA, "utf8"), finalContent, "Client A did not fetch the final file bytes");
|
||||
settingsTimeline.clientASuffixAfterFetch = (await settingsSummary(settingsA))?.additionalSuffixOfDatabaseName;
|
||||
assert.equal(
|
||||
settingsTimeline.clientASuffixAfterFetch,
|
||||
mode === "reproduction" && VERIFY_LEGACY_FAILURE ? suffix : "headless-app",
|
||||
"Client A settings suffix did not match the selected fetch mode"
|
||||
);
|
||||
|
||||
// This startup selects A's persisted suffix. In the reproduction mode it
|
||||
// scans the old, empty database while the fetched bytes are in the vault.
|
||||
processes.clientAStartup = new CapturedCliProcess([
|
||||
"-v",
|
||||
databaseA,
|
||||
"--vault",
|
||||
vaultA,
|
||||
"--settings",
|
||||
settingsA,
|
||||
"daemon",
|
||||
]);
|
||||
await processes.clientAStartup.waitUntil(
|
||||
(output) => output.includes(INITIAL_REPLICATION_COMPLETE) || output.includes(INITIAL_REPLICATION_FAILED),
|
||||
"Client A startup replication result",
|
||||
45_000
|
||||
);
|
||||
await processes.clientAStartup.stop();
|
||||
|
||||
const markResolved = await runCliQuietly([databaseA, "--vault", vaultA, "--settings", settingsA, "mark-resolved"]);
|
||||
safeCommand("mark-resolved-client-a", markResolved, commandLog, secrets);
|
||||
|
||||
if (mode === "reproduction" && VERIFY_LEGACY_FAILURE) {
|
||||
processes.clientA = new CapturedCliProcess([
|
||||
"-v",
|
||||
databaseA,
|
||||
"--vault",
|
||||
vaultA,
|
||||
"--settings",
|
||||
settingsA,
|
||||
"--interval",
|
||||
"1",
|
||||
"daemon",
|
||||
]);
|
||||
await processes.clientA.waitUntil((output) => output.includes(INITIAL_REPLICATION_COMPLETE), "Client A interval daemon initial replication");
|
||||
|
||||
const processingMarker = `Processing ${targetPath}`;
|
||||
const aProcessingBeforeResolution = countOccurrences(processes.clientA.combined, processingMarker);
|
||||
await processes.clientB.waitUntil((output) => output.includes(RESOLUTION_MARKER), "Client B automatic same-content resolution");
|
||||
await processes.clientA.waitUntil(
|
||||
(output) => countOccurrences(output, processingMarker) > aProcessingBeforeResolution,
|
||||
"Client A to reflect the post-resolution revision",
|
||||
45_000
|
||||
);
|
||||
} else {
|
||||
processes.clientA = new CapturedCliProcess([
|
||||
"-v",
|
||||
databaseA,
|
||||
"--vault",
|
||||
vaultA,
|
||||
"--settings",
|
||||
settingsA,
|
||||
"--interval",
|
||||
"1",
|
||||
"daemon",
|
||||
]);
|
||||
await processes.clientA.waitUntil((output) => output.includes(INITIAL_REPLICATION_COMPLETE), "Client A control daemon initial replication");
|
||||
await new Promise((resolveDelay) => setTimeout(resolveDelay, 2_500));
|
||||
assert.equal(await readFile(finalFileA, "utf8"), finalContent, "Client A control lost the fetched file");
|
||||
}
|
||||
} catch (error) {
|
||||
failure = error;
|
||||
}
|
||||
|
||||
for (const processCapture of Object.values(processes)) {
|
||||
if (!processCapture) continue;
|
||||
try {
|
||||
await processCapture.stop();
|
||||
} catch (error) {
|
||||
if (failure === undefined) failure = error;
|
||||
}
|
||||
}
|
||||
|
||||
try {
|
||||
await mkdir(evidenceDir, { recursive: true });
|
||||
for (const [name, database, vault, settingsPath] of [
|
||||
["clientA", databaseA, vaultA, settingsA],
|
||||
["clientB", databaseB, vaultB, settingsB],
|
||||
] as const) {
|
||||
const result = await runCliQuietly([database, "--vault", vault, "--settings", settingsPath, "info", targetPath]);
|
||||
const output = redactText(result.combined, secrets);
|
||||
commandLog.push({ label: `post-resolution-info-${name}`, code: result.code, output });
|
||||
if (result.code === 0) {
|
||||
try {
|
||||
postResolutionInfo[name] = JSON.parse(result.stdout) as Record<string, unknown>;
|
||||
} catch {
|
||||
postResolutionInfo[name] = null;
|
||||
}
|
||||
} else {
|
||||
postResolutionInfo[name] = null;
|
||||
}
|
||||
}
|
||||
observedFinalFileA = await fileState(finalFileA);
|
||||
if (observedFinalFileA.exists) observedFinalContentA = await readFile(finalFileA, "utf8");
|
||||
const databaseTrees: Record<string, Array<{ path: string; bytes: number; sha256: string }>> = {};
|
||||
const runtimeState: Record<string, Record<string, string>> = {};
|
||||
const databasePaths = { seed: databaseSeed, clientA: databaseA, clientB: databaseB };
|
||||
for (const [name, path] of Object.entries(databasePaths)) {
|
||||
databaseTrees[name] = await collectDatabaseTree(path);
|
||||
await copyDatabaseTree(path, join(evidenceDir, `database-${name}`));
|
||||
runtimeState[name] = await sanitisedRuntimeState(path, secrets);
|
||||
if (Object.keys(runtimeState[name]).length > 0) {
|
||||
const runtimeDir = join(evidenceDir, `runtime-${name}`);
|
||||
await mkdir(runtimeDir, { recursive: true });
|
||||
for (const [fileName, contents] of Object.entries(runtimeState[name])) {
|
||||
await writeFile(join(runtimeDir, fileName), contents);
|
||||
}
|
||||
}
|
||||
}
|
||||
const logs = Object.fromEntries(
|
||||
Object.entries(processes).map(([name, processCapture]) => [
|
||||
name,
|
||||
processCapture ? redactText(processCapture.combined, secrets) : "",
|
||||
])
|
||||
);
|
||||
const fileStates = {
|
||||
clientA: await fileState(finalFileA),
|
||||
clientB: await fileState(finalFileB),
|
||||
};
|
||||
const evidence = {
|
||||
scenario: mode,
|
||||
targetPath,
|
||||
expectedContentBytes: finalBytes.byteLength,
|
||||
expectedContentSha256: createHash("sha256").update(finalBytes).digest("hex"),
|
||||
settingsTimeline,
|
||||
settings: {
|
||||
seed: await settingsSummary(settingsSeed),
|
||||
clientA: await settingsSummary(settingsA),
|
||||
clientB: await settingsSummary(settingsB),
|
||||
},
|
||||
fileStates,
|
||||
postResolutionInfo,
|
||||
databaseTrees,
|
||||
runtimeState,
|
||||
commands: commandLog,
|
||||
processLogs: logs,
|
||||
remote: { endpoint: "[configured]", bucket: remote.bucket },
|
||||
failure: failure instanceof Error ? redactText(failure.stack ?? failure.message, secrets) : String(failure ?? ""),
|
||||
};
|
||||
await writeFile(join(evidenceDir, "evidence.json"), JSON.stringify(evidence, null, 2));
|
||||
} catch (error) {
|
||||
if (failure === undefined) failure = error;
|
||||
else if (failure instanceof Error) {
|
||||
failure = new Error(`${failure.message}\nEvidence capture failed: ${error instanceof Error ? error.message : String(error)}`);
|
||||
}
|
||||
}
|
||||
|
||||
await workDir[Symbol.asyncDispose]();
|
||||
if (failure !== undefined) throw failure;
|
||||
assert.equal(observedFinalFileA?.exists, true, "Client A file is absent after the resolved journal was applied");
|
||||
assert.equal(observedFinalContentA, finalContent, "Client A lost the fetched file after remote conflict resolution");
|
||||
if (!VERIFY_LEGACY_FAILURE) {
|
||||
const infoA = postResolutionInfo.clientA;
|
||||
const infoB = postResolutionInfo.clientB;
|
||||
assert(infoA && infoB, "Could not inspect both local databases after the remote resolution");
|
||||
assert.equal(infoA.conflicts, "N/A", "Client A still has a duplicate root revision after the resolution");
|
||||
assert.equal(infoA.revision, infoB.revision, "Client A did not apply the revision selected by Client B after the resolved journal");
|
||||
assert.equal(infoA.revision, settingsTimeline.seedRevision, "Daemon startup replaced the fetched revision ancestry");
|
||||
assert.equal(processes.clientB?.combined.includes(RESOLUTION_MARKER), false, "Client A created an unnecessary competing root");
|
||||
}
|
||||
}
|
||||
|
||||
test("CLI fetch: files remain after same-content duplicate root resolution", async () => {
|
||||
const remote = remoteSettings();
|
||||
await runScenario("reproduction", remote.bucket);
|
||||
});
|
||||
|
||||
if (process.env["LIVESYNC_ISSUE1304_POSITIVE_CONTROL"] === "1") {
|
||||
test("CLI fetch: --write-settings keeps the selected database across daemon startup", async () => {
|
||||
const controlBucket = requiredEnv("LIVESYNC_ISSUE1304_POSITIVE_CONTROL_BUCKET");
|
||||
await runScenario("write-settings-control", controlBucket);
|
||||
});
|
||||
}
|
||||
@@ -12,6 +12,9 @@ Earlier releases remain available in the 1.0 release history, the 1.0 preview hi
|
||||
|
||||
## Unreleased
|
||||
|
||||
- CLI `fetch` and `rebuild` retain their selected local database for subsequent commands even without `--write-settings`, preventing the empty-database restart and file deletion reported in #1304.
|
||||
- Runtime state is separated by database and Vault. Interrupted reconstruction blocks ordinary synchronisation until recovery, and a pending settings commit can be completed on restart.
|
||||
- Older runtime state with no recorded owner is preserved and requires explicit reconstruction from a verified source before ordinary writes resume. Back up unique edits before choosing Fetch or Rebuild.
|
||||
- The CLI mirror now starts correctly on Windows when the Vault path is omitted and the database directory is also used as the Vault. It checks logical Vault paths using `/` separators, correctly excluding internal database files and nested hidden directories even when the host uses Windows path separators.
|
||||
|
||||
## 1.0.43
|
||||
|
||||
Reference in new issue
Block a user