mirror of
https://github.com/vrtmrz/obsidian-livesync.git
synced 2026-11-05 15:58:19 +00:00
Add explicit stopped-Journal recovery to Obsidian and CLI
Offer a confirmation in Maintenance and the journal-skip command to skip only the current stopped Journal and resume receipt. Keep Commonlib notices visible on stderr during ordinary CLI use. Document the missing-content risk and replay procedure, and cover cancellation, stale failures, subsequent failures, and receipt with unit and plain and encrypted CLI and Obsidian end-to-end tests. Requires the Commonlib stopped-Journal recovery API. Adopt its exact published package version before integrating this consumer change.
This commit is contained in:
1 parent
bf2241e5ee
commit
a94c3fef90
18 files changed
+1210
-123
No files matched your search
@@ -1,5 +1,15 @@
|
|||||||
## The design document of the Journal Replicator 2nd Edition
|
## The design document of the Journal Replicator 2nd Edition
|
||||||
|
|
||||||
|
Implementation note: the receive-failure correction and single-Journal skip
|
||||||
|
action are unreleased. Their owning contract is Commonlib's
|
||||||
|
[Journal receive failures and recovery](https://github.com/vrtmrz/livesync-commonlib/blob/main/docs/journal-receive-recovery.md).
|
||||||
|
The receiver acknowledges successful processing, stops on failure by default,
|
||||||
|
and retries from that Journal with the same database. Explicit skipping does
|
||||||
|
not establish Vault reflection success. The host action names one stopped
|
||||||
|
Journal and its failure stage, requires confirmation, and resumes receipt after
|
||||||
|
skipping that Journal; the next failure still stops. The [settings reference](settings.md#skip-stopped-journal-and-resume)
|
||||||
|
describes the host control and replay operation.
|
||||||
|
|
||||||
### Goal
|
### Goal
|
||||||
- Build a robust and memory-efficient replication foundation that decouples the physical storage layer by leveraging the Web Streams API.
|
- Build a robust and memory-efficient replication foundation that decouples the physical storage layer by leveraging the Web Streams API.
|
||||||
- Maintain strict compliance with the data consistency and replication protocols of CouchDB/PouchDB.
|
- Maintain strict compliance with the data consistency and replication protocols of CouchDB/PouchDB.
|
||||||
|
|||||||
+19
-1
@@ -1273,9 +1273,27 @@ Disables all synchronisation and restart.
|
|||||||
|
|
||||||
Explicitly resend all locally available chunks to the remote. This is a recovery and maintenance operation; ordinary replication does not pre-send every chunk.
|
Explicitly resend all locally available chunks to the remote. This is a recovery and maintenance operation; ordinary replication does not pre-send every chunk.
|
||||||
|
|
||||||
|
#### Skip stopped Journal and resume
|
||||||
|
|
||||||
|
Object Storage only. If a stopped Journal is not available, restart replication
|
||||||
|
normally with the same local database so the receive failure identifies its key
|
||||||
|
and stage. Confirming this action skips only that Journal and resumes receipt;
|
||||||
|
it does not skip the rest of the remote history. The next Journal failure still
|
||||||
|
stops replication.
|
||||||
|
|
||||||
|
Skipping may leave content or history missing. Normal replication does not
|
||||||
|
retry a Journal which has been explicitly skipped. After correcting the cause,
|
||||||
|
use **Reset journal received history** to replay retained remote Journals,
|
||||||
|
except those sent by this device. It cannot restore Journals which are no
|
||||||
|
longer available remotely. This action does not add a persistent ignore setting.
|
||||||
|
|
||||||
#### Reset journal received history
|
#### Reset journal received history
|
||||||
|
|
||||||
Initialise journal received history. On the next sync, every item except this device sent will be downloaded again.
|
Clear this device's received Journal history and known-document transfer
|
||||||
|
records. The next sync downloads the retained remote Journals again, except
|
||||||
|
those sent by this device. The local database, Vault files, and sent history
|
||||||
|
are retained. Recovery requires intact remote data and a working local
|
||||||
|
database; resetting history does not recreate missing remote content.
|
||||||
|
|
||||||
#### Reset journal sent history
|
#### Reset journal sent history
|
||||||
|
|
||||||
|
|||||||
@@ -47,6 +47,29 @@ Do not switch to P2P or reset the database as the first response. Check:
|
|||||||
|
|
||||||
If the remote is healthy but one device's local database is not, use [Reset Synchronisation on This Device](recovery.md#reset-synchronisation-on-this-device) only after backing up unsynchronised local files.
|
If the remote is healthy but one device's local database is not, use [Reset Synchronisation on This Device](recovery.md#reset-synchronisation-on-this-device) only after backing up unsynchronised local files.
|
||||||
|
|
||||||
|
## Object Storage stops at a Journal receive failure
|
||||||
|
|
||||||
|
The receiver stops when a Journal cannot be processed or its records cannot be
|
||||||
|
saved. Correct the reported cause, then retry replication with the same local
|
||||||
|
database; the failed Journal is retried. If the stopped Journal is no longer
|
||||||
|
available to the active replicator, run normal replication again to identify
|
||||||
|
the current failure.
|
||||||
|
|
||||||
|
If the Journal cannot be recovered and missing content or history is
|
||||||
|
acceptable, open **Maintenance → Syncing → Skip stopped Journal and resume**.
|
||||||
|
The confirmation names the exact Journal key and failure stage. Confirming
|
||||||
|
skips only that Journal and resumes receipt; the next Journal failure stops
|
||||||
|
replication again. Use this only when the missing content or history is
|
||||||
|
acceptable. See the [setting reference](settings.md#skip-stopped-journal-and-resume).
|
||||||
|
|
||||||
|
**Reset journal received history** replays remote Journals that are still
|
||||||
|
available, except those sent by this device. It retains local data and this
|
||||||
|
device's sent history, and cannot restore remote Journals which are missing.
|
||||||
|
Check the resulting file contents and retained history; a completed receive
|
||||||
|
after skipping does not prove that every Vault file is readable. CLI Fetch
|
||||||
|
replaces its local database and has separate [interrupted reconstruction
|
||||||
|
rules](../src/apps/cli/README.md#interrupted-reconstruction-and-older-runtime-state).
|
||||||
|
|
||||||
## An unchanged file appears as a conflict after restart
|
## An unchanged file appears as a conflict after restart
|
||||||
|
|
||||||
If the log says `Preserved unsynchronised local changes as a conflict`, keep both versions available until you have checked their contents. The message means that LiveSync could not prove that the Vault file was unchanged; it does not establish that you edited the file. Use **Inspect conflicts and file/database differences** in Hatch to review the current branches before selecting a version.
|
If the log says `Preserved unsynchronised local changes as a conflict`, keep both versions available until you have checked their contents. The message means that LiveSync could not prove that the Vault file was unchanged; it does not establish that you edited the file. Use **Inspect conflicts and file/database differences** in Hatch to review the current branches before selecting a version.
|
||||||
|
|||||||
@@ -80,6 +80,7 @@ livesync-cli [database-path] [command] [args...]
|
|||||||
### Commands
|
### Commands
|
||||||
|
|
||||||
- `sync`: Run one replication cycle with the remote CouchDB.
|
- `sync`: Run one replication cycle with the remote CouchDB.
|
||||||
|
- `journal-skip <journal-key>`: Retry replication, skip only the exact currently stopped Object Storage Journal, and resume.
|
||||||
- `mirror [vault-path]`: Bidirectional sync between the local database and a local directory (**the actual vault**).
|
- `mirror [vault-path]`: Bidirectional sync between the local database and a local directory (**the actual vault**).
|
||||||
- If `vault-path` is provided, the CLI will synchronise the database with files in the vault directory.
|
- If `vault-path` is provided, the CLI will synchronise the database with files in the vault directory.
|
||||||
- If `vault-path` is omitted, it defaults to `database-path` (compatibility mode).
|
- If `vault-path` is omitted, it defaults to `database-path` (compatibility mode).
|
||||||
@@ -108,6 +109,27 @@ livesync-cli [database-path] [command] [args...]
|
|||||||
|
|
||||||
Remote-administration commands verify the resulting milestone state through the selected provider. The existing `[Verification]` lines remain suitable for scripts which inspect command output, while the default exit code now reflects whether that verification succeeded.
|
Remote-administration commands verify the resulting milestone state through the selected provider. The existing `[Verification]` lines remain suitable for scripts which inspect command output, while the default exit code now reflects whether that verification succeeded.
|
||||||
|
|
||||||
|
### Object Storage receive failures
|
||||||
|
|
||||||
|
`sync` stops at a Journal which cannot be processed or whose records cannot be
|
||||||
|
saved. After correcting the cause, run `sync` again with the same database and
|
||||||
|
settings to retry it. Existing partial writes are retained. `fetch --force`
|
||||||
|
replaces the local database instead and follows the reconstruction rules below.
|
||||||
|
|
||||||
|
If the Journal cannot be recovered and missing content or history is
|
||||||
|
acceptable, run `journal-skip <journal-key>`. The CLI first performs normal
|
||||||
|
replication with the retained local database. It skips only when that attempt
|
||||||
|
stops at the exact supplied key; if replication succeeds or stops at another
|
||||||
|
Journal, the command refuses the explicit skip. A successful normal attempt
|
||||||
|
can still acknowledge Journals which it processed successfully.
|
||||||
|
After the matching Journal is skipped, replication resumes. Any later failed
|
||||||
|
Journal still stops, so each additional skip requires a separate command with
|
||||||
|
that Journal's key. Normal identity, connectivity, authentication, and
|
||||||
|
readiness checks still apply. This action does not persist a setting for future
|
||||||
|
replication. Review the resulting file contents and retained history; a
|
||||||
|
completed receive does not guarantee that later file reflection or reads will
|
||||||
|
succeed.
|
||||||
|
|
||||||
### Interrupted reconstruction and older runtime state
|
### Interrupted reconstruction and older runtime state
|
||||||
|
|
||||||
An interrupted `fetch` or `rebuild` leaves a record in
|
An interrupted `fetch` or `rebuild` leaves a record in
|
||||||
@@ -146,6 +168,7 @@ scopes do not make concurrent writers to shared device settings supported.
|
|||||||
```bash
|
```bash
|
||||||
# Basic sync with remote
|
# Basic sync with remote
|
||||||
livesync-cli ./my-db sync
|
livesync-cli ./my-db sync
|
||||||
|
livesync-cli ./my-db journal-skip 7-docs.jsonl.gz
|
||||||
|
|
||||||
# Mirroring to your actual Obsidian vault
|
# Mirroring to your actual Obsidian vault
|
||||||
livesync-cli ./my-db mirror /path/to/obsidian-vault
|
livesync-cli ./my-db mirror /path/to/obsidian-vault
|
||||||
@@ -475,6 +498,8 @@ Commands:
|
|||||||
daemon (default) Run mirror scan then continuously sync CouchDB <-> local filesystem
|
daemon (default) Run mirror scan then continuously sync CouchDB <-> local filesystem
|
||||||
init-settings [path] Create unconfigured settings JSON with the new-Vault recommendations
|
init-settings [path] Create unconfigured settings JSON with the new-Vault recommendations
|
||||||
sync Run one replication cycle and exit
|
sync Run one replication cycle and exit
|
||||||
|
journal-skip <journal-key>
|
||||||
|
Retry replication, skip only the matching stopped Journal, and resume
|
||||||
p2p-peers <timeout> Show discovered peers as [peer]<TAB><peer-id><TAB><peer-name>
|
p2p-peers <timeout> Show discovered peers as [peer]<TAB><peer-id><TAB><peer-name>
|
||||||
p2p-sync <peer> <timeout> Synchronise with specified peer-id or peer-name
|
p2p-sync <peer> <timeout> Synchronise with specified peer-id or peer-name
|
||||||
p2p-host Start P2P host mode and wait until interrupted (Ctrl+C)
|
p2p-host Start P2P host mode and wait until interrupted (Ctrl+C)
|
||||||
|
|||||||
@@ -28,6 +28,7 @@ import {
|
|||||||
USER_INITIATED_REPLICATION_AUTHORITY,
|
USER_INITIATED_REPLICATION_AUTHORITY,
|
||||||
} from "@vrtmrz/livesync-commonlib/replication";
|
} from "@vrtmrz/livesync-commonlib/replication";
|
||||||
import { withOwnedRemoteResource } from "@/common/ownedRemoteResource";
|
import { withOwnedRemoteResource } from "@/common/ownedRemoteResource";
|
||||||
|
import { LiveSyncJournalReplicator } from "@vrtmrz/livesync-commonlib/compat/replication/journal/LiveSyncJournalReplicator";
|
||||||
import {
|
import {
|
||||||
isCentralRemoteAdministrationCommand,
|
isCentralRemoteAdministrationCommand,
|
||||||
runCentralRemoteAdministrationCommand,
|
runCentralRemoteAdministrationCommand,
|
||||||
@@ -45,10 +46,7 @@ export async function runCommand(options: CLIOptions, context: CLICommandContext
|
|||||||
writeStderrLine(standardIo, `[Error] ${forceRequirementError}`);
|
writeStderrLine(standardIo, `[Error] ${forceRequirementError}`);
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
if (
|
if (options.command === "fetch" && isRemediationModeActive(core.services.setting.currentSettings())) {
|
||||||
options.command === "fetch" &&
|
|
||||||
isRemediationModeActive(core.services.setting.currentSettings())
|
|
||||||
) {
|
|
||||||
writeStderrLine(
|
writeStderrLine(
|
||||||
standardIo,
|
standardIo,
|
||||||
`[Error] The fetch command cannot run while remediation mode is active. Clear the modification-time limit before fetching.`
|
`[Error] The fetch command cannot run while remediation mode is active. Clear the modification-time limit before fetching.`
|
||||||
@@ -202,6 +200,64 @@ export async function runCommand(options: CLIOptions, context: CLICommandContext
|
|||||||
return isReplicationCompleted(result);
|
return isReplicationCompleted(result);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (options.command === "journal-skip") {
|
||||||
|
const [journalKey] = options.commandArgs;
|
||||||
|
if (options.commandArgs.length !== 1 || journalKey === undefined || journalKey.length === 0) {
|
||||||
|
writeStderrLine(standardIo, `[Error] journal-skip requires one argument: <journal-key>`);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
writeStdoutLine(standardIo, `[Command] journal-skip ${journalKey}`);
|
||||||
|
const replicatorBeforeReplication = core.services.replicator.getActiveReplicator();
|
||||||
|
const result = await core.services.replication.replicateUserInitiated({
|
||||||
|
trigger: "manual",
|
||||||
|
progressPresentation: REPLICATION_PROGRESS_PRESENTATIONS.NOTICE,
|
||||||
|
interaction: USER_INITIATED_REPLICATION_AUTHORITY,
|
||||||
|
});
|
||||||
|
const activeReplicator = core.services.replicator.getActiveReplicator();
|
||||||
|
|
||||||
|
if (activeReplicator !== replicatorBeforeReplication) {
|
||||||
|
writeStderrLine(
|
||||||
|
standardIo,
|
||||||
|
`[Error] The active replicator changed during replication; no Journal was skipped.`
|
||||||
|
);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
if (isReplicationCompleted(result)) {
|
||||||
|
writeStderrLine(
|
||||||
|
standardIo,
|
||||||
|
`[Error] Replication completed without a stopped Journal; no Journal was skipped.`
|
||||||
|
);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
if (!(activeReplicator instanceof LiveSyncJournalReplicator)) {
|
||||||
|
writeStderrLine(
|
||||||
|
standardIo,
|
||||||
|
`[Error] The active replicator is not a Journal replicator; no Journal was skipped.`
|
||||||
|
);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
const stoppedJournal = activeReplicator.getStoppedJournal();
|
||||||
|
if (!stoppedJournal) {
|
||||||
|
writeStderrLine(standardIo, `[Error] Replication did not stop at a Journal; no Journal was skipped.`);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
if (stoppedJournal.key !== journalKey) {
|
||||||
|
writeStderrLine(
|
||||||
|
standardIo,
|
||||||
|
`[Error] Replication stopped at Journal '${stoppedJournal.key}' (${stoppedJournal.stage}), not '${journalKey}'; no Journal was skipped.`
|
||||||
|
);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
return await activeReplicator.skipStoppedJournalAndResume(
|
||||||
|
core.services.setting.currentSettings(),
|
||||||
|
stoppedJournal,
|
||||||
|
true
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
if (options.command === "p2p-peers") {
|
if (options.command === "p2p-peers") {
|
||||||
if (options.commandArgs.length < 1) {
|
if (options.commandArgs.length < 1) {
|
||||||
throw new Error("p2p-peers requires one argument: <timeout>");
|
throw new Error("p2p-peers requires one argument: <timeout>");
|
||||||
|
|||||||
@@ -11,6 +11,7 @@ import {
|
|||||||
import { describe, expect, it, vi, beforeEach, afterEach } from "vitest";
|
import { describe, expect, it, vi, beforeEach, afterEach } from "vitest";
|
||||||
import { runCommand } from "./runCommand";
|
import { runCommand } from "./runCommand";
|
||||||
import type { CLIOptions } from "./types";
|
import type { CLIOptions } from "./types";
|
||||||
|
import { LiveSyncJournalReplicator } from "@vrtmrz/livesync-commonlib/compat/replication/journal/LiveSyncJournalReplicator";
|
||||||
import {
|
import {
|
||||||
CENTRAL_REMOTE_ADMINISTRATION_ACTIONS,
|
CENTRAL_REMOTE_ADMINISTRATION_ACTIONS,
|
||||||
CENTRAL_REMOTE_ADMINISTRATION_FAILURE_REASONS,
|
CENTRAL_REMOTE_ADMINISTRATION_FAILURE_REASONS,
|
||||||
@@ -20,6 +21,7 @@ import {
|
|||||||
CENTRAL_COMPATIBILITY_REJECTION_REASONS,
|
CENTRAL_COMPATIBILITY_REJECTION_REASONS,
|
||||||
REPLICATION_COMPLETED,
|
REPLICATION_COMPLETED,
|
||||||
REPLICATION_PROGRESS_PRESENTATIONS,
|
REPLICATION_PROGRESS_PRESENTATIONS,
|
||||||
|
USER_INITIATED_REPLICATION_AUTHORITY,
|
||||||
replicationFailed,
|
replicationFailed,
|
||||||
} from "@vrtmrz/livesync-commonlib/replication";
|
} from "@vrtmrz/livesync-commonlib/replication";
|
||||||
|
|
||||||
@@ -137,6 +139,17 @@ function makeOptions(command: CLIOptions["command"], commandArgs: string[]): CLI
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function makeJournalSkipOptions(journalKey: string): CLIOptions {
|
||||||
|
return makeOptions("journal-skip", [journalKey]);
|
||||||
|
}
|
||||||
|
|
||||||
|
function createJournalReplicatorMock(stoppedJournal: unknown) {
|
||||||
|
const replicator = Object.create(LiveSyncJournalReplicator.prototype) as any;
|
||||||
|
replicator.getStoppedJournal = vi.fn(() => stoppedJournal);
|
||||||
|
replicator.skipStoppedJournalAndResume = vi.fn(async () => true);
|
||||||
|
return replicator;
|
||||||
|
}
|
||||||
|
|
||||||
async function createSetupURI(passphrase: string): Promise<string> {
|
async function createSetupURI(passphrase: string): Promise<string> {
|
||||||
const settings = {
|
const settings = {
|
||||||
...DEFAULT_SETTINGS,
|
...DEFAULT_SETTINGS,
|
||||||
@@ -354,6 +367,108 @@ describe("runCommand abnormal cases", () => {
|
|||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
describe("journal-skip command", () => {
|
||||||
|
const snapshot = { key: "7-docs.jsonl.gz", stage: "parse" } as const;
|
||||||
|
|
||||||
|
it("skips only the exact stopped Journal after a normal user-initiated replication", async () => {
|
||||||
|
const core = createCoreMock();
|
||||||
|
const replicator = createJournalReplicatorMock(snapshot);
|
||||||
|
core.services.replicator.getActiveReplicator.mockReturnValue(replicator);
|
||||||
|
core.services.replication.replicateUserInitiated.mockResolvedValue(
|
||||||
|
replicationFailed(new Error("stopped Journal"))
|
||||||
|
);
|
||||||
|
|
||||||
|
await expect(runCommand(makeJournalSkipOptions(snapshot.key), { ...context, core })).resolves.toBe(true);
|
||||||
|
|
||||||
|
expect(core.services.replication.replicateUserInitiated).toHaveBeenCalledWith(
|
||||||
|
expect.objectContaining({
|
||||||
|
trigger: "manual",
|
||||||
|
progressPresentation: REPLICATION_PROGRESS_PRESENTATIONS.NOTICE,
|
||||||
|
interaction: USER_INITIATED_REPLICATION_AUTHORITY,
|
||||||
|
})
|
||||||
|
);
|
||||||
|
expect(replicator.getStoppedJournal).toHaveBeenCalledOnce();
|
||||||
|
expect(replicator.skipStoppedJournalAndResume).toHaveBeenCalledWith(
|
||||||
|
core.services.setting.currentSettings(),
|
||||||
|
snapshot,
|
||||||
|
true
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("refuses a key which does not match the stopped Journal", async () => {
|
||||||
|
const core = createCoreMock();
|
||||||
|
const replicator = createJournalReplicatorMock(snapshot);
|
||||||
|
core.services.replicator.getActiveReplicator.mockReturnValue(replicator);
|
||||||
|
core.services.replication.replicateUserInitiated.mockResolvedValue(
|
||||||
|
replicationFailed(new Error("stopped Journal"))
|
||||||
|
);
|
||||||
|
|
||||||
|
await expect(runCommand(makeJournalSkipOptions("8-docs.jsonl.gz"), { ...context, core })).resolves.toBe(
|
||||||
|
false
|
||||||
|
);
|
||||||
|
|
||||||
|
expect(replicator.skipStoppedJournalAndResume).not.toHaveBeenCalled();
|
||||||
|
expect(core.services.context.standardIo.writeStderr).toHaveBeenCalledWith(
|
||||||
|
expect.stringContaining("7-docs.jsonl.gz")
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("refuses when normal replication does not establish a stopped Journal", async () => {
|
||||||
|
const core = createCoreMock();
|
||||||
|
const replicator = createJournalReplicatorMock(undefined);
|
||||||
|
core.services.replicator.getActiveReplicator.mockReturnValue(replicator);
|
||||||
|
|
||||||
|
await expect(runCommand(makeJournalSkipOptions(snapshot.key), { ...context, core })).resolves.toBe(false);
|
||||||
|
|
||||||
|
expect(replicator.skipStoppedJournalAndResume).not.toHaveBeenCalled();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("refuses when the active replicator is not a Journal replicator", async () => {
|
||||||
|
const core = createCoreMock();
|
||||||
|
core.services.replicator.getActiveReplicator.mockReturnValue({ getStoppedJournal: vi.fn(() => snapshot) });
|
||||||
|
core.services.replication.replicateUserInitiated.mockResolvedValue(
|
||||||
|
replicationFailed(new Error("stopped Journal"))
|
||||||
|
);
|
||||||
|
|
||||||
|
await expect(runCommand(makeJournalSkipOptions(snapshot.key), { ...context, core })).resolves.toBe(false);
|
||||||
|
|
||||||
|
expect(core.services.context.standardIo.writeStderr).toHaveBeenCalledWith(
|
||||||
|
expect.stringContaining("Journal replicator")
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("propagates a failure while resuming after the requested Journal was skipped", async () => {
|
||||||
|
const core = createCoreMock();
|
||||||
|
const replicator = createJournalReplicatorMock(snapshot);
|
||||||
|
replicator.skipStoppedJournalAndResume.mockResolvedValue(false);
|
||||||
|
core.services.replicator.getActiveReplicator.mockReturnValue(replicator);
|
||||||
|
core.services.replication.replicateUserInitiated.mockResolvedValue(
|
||||||
|
replicationFailed(new Error("stopped Journal"))
|
||||||
|
);
|
||||||
|
|
||||||
|
await expect(runCommand(makeJournalSkipOptions(snapshot.key), { ...context, core })).resolves.toBe(false);
|
||||||
|
|
||||||
|
expect(replicator.skipStoppedJournalAndResume).toHaveBeenCalledOnce();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("refuses if the active replicator changes during the awaited replication", async () => {
|
||||||
|
const core = createCoreMock();
|
||||||
|
const replicator = createJournalReplicatorMock(snapshot);
|
||||||
|
const replacement = createJournalReplicatorMock(snapshot);
|
||||||
|
core.services.replicator.getActiveReplicator
|
||||||
|
.mockReturnValueOnce(replicator)
|
||||||
|
.mockReturnValueOnce(replacement);
|
||||||
|
core.services.replication.replicateUserInitiated.mockResolvedValue(
|
||||||
|
replicationFailed(new Error("stopped Journal"))
|
||||||
|
);
|
||||||
|
|
||||||
|
await expect(runCommand(makeJournalSkipOptions(snapshot.key), { ...context, core })).resolves.toBe(false);
|
||||||
|
|
||||||
|
expect(replicator.skipStoppedJournalAndResume).not.toHaveBeenCalled();
|
||||||
|
expect(replacement.skipStoppedJournalAndResume).not.toHaveBeenCalled();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
it("reports a lock from the exact sync outcome without inspecting a replacement Replicator", async () => {
|
it("reports a lock from the exact sync outcome without inspecting a replacement Replicator", async () => {
|
||||||
const core = createCoreMock();
|
const core = createCoreMock();
|
||||||
core.services.replication.replicateUserInitiated.mockResolvedValue(
|
core.services.replication.replicateUserInitiated.mockResolvedValue(
|
||||||
@@ -546,9 +661,9 @@ describe("runCommand abnormal cases", () => {
|
|||||||
clock.mockReturnValue(end);
|
clock.mockReturnValue(end);
|
||||||
const outOfWindow = createCoreMock();
|
const outOfWindow = createCoreMock();
|
||||||
outOfWindow.services.context.standardIo.prompt.mockResolvedValue(passphrase);
|
outOfWindow.services.context.standardIo.prompt.mockResolvedValue(passphrase);
|
||||||
await expect(runCommand(makeOptions("setup", [uri.trim()]), { ...context, core: outOfWindow })).rejects.toThrow(
|
await expect(
|
||||||
"Cannot open Setup URI"
|
runCommand(makeOptions("setup", [uri.trim()]), { ...context, core: outOfWindow })
|
||||||
);
|
).rejects.toThrow("Cannot open Setup URI");
|
||||||
expect(outOfWindow.services.setting.applyExternalSettings).not.toHaveBeenCalled();
|
expect(outOfWindow.services.setting.applyExternalSettings).not.toHaveBeenCalled();
|
||||||
expect(outOfWindow.services.control.applySettings).not.toHaveBeenCalled();
|
expect(outOfWindow.services.control.applySettings).not.toHaveBeenCalled();
|
||||||
} finally {
|
} finally {
|
||||||
@@ -559,8 +674,12 @@ describe("runCommand abnormal cases", () => {
|
|||||||
it("setup imports managed TURN through the existing encrypted URI", async () => {
|
it("setup imports managed TURN through the existing encrypted URI", async () => {
|
||||||
const core = createCoreMock();
|
const core = createCoreMock();
|
||||||
const profiles = {
|
const profiles = {
|
||||||
turn: { id: "turn", name: "TURN", isEncrypted: false,
|
turn: {
|
||||||
uri: "sls+p2p://room?managedType=CF&managedId=turn-key&token=private-token" },
|
id: "turn",
|
||||||
|
name: "TURN",
|
||||||
|
isEncrypted: false,
|
||||||
|
uri: "sls+p2p://room?managedType=CF&managedId=turn-key&token=private-token",
|
||||||
|
},
|
||||||
};
|
};
|
||||||
const passphrase = "setup-passphrase";
|
const passphrase = "setup-passphrase";
|
||||||
const setupURI = await processSetting.encodeSettingsToSetupURI(
|
const setupURI = await processSetting.encodeSettingsToSetupURI(
|
||||||
|
|||||||
@@ -7,6 +7,7 @@ import type { ReplicationSchedulingControl } from "@/serviceFeatures/replication
|
|||||||
export type CLICommand =
|
export type CLICommand =
|
||||||
| "daemon"
|
| "daemon"
|
||||||
| "sync"
|
| "sync"
|
||||||
|
| "journal-skip"
|
||||||
| "p2p-peers"
|
| "p2p-peers"
|
||||||
| "p2p-sync"
|
| "p2p-sync"
|
||||||
| "p2p-host"
|
| "p2p-host"
|
||||||
@@ -87,6 +88,7 @@ export interface CLICommandContext {
|
|||||||
export const VALID_COMMANDS = new Set([
|
export const VALID_COMMANDS = new Set([
|
||||||
"daemon",
|
"daemon",
|
||||||
"sync",
|
"sync",
|
||||||
|
"journal-skip",
|
||||||
"p2p-peers",
|
"p2p-peers",
|
||||||
"p2p-sync",
|
"p2p-sync",
|
||||||
"p2p-host",
|
"p2p-host",
|
||||||
|
|||||||
@@ -2,6 +2,16 @@ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
|||||||
import { mkdtemp, rm } from "node:fs/promises";
|
import { mkdtemp, rm } from "node:fs/promises";
|
||||||
import { tmpdir } from "node:os";
|
import { tmpdir } from "node:os";
|
||||||
import { join } from "node:path";
|
import { join } from "node:path";
|
||||||
|
import type { LOG_LEVEL } from "@vrtmrz/livesync-commonlib/compat/common/types";
|
||||||
|
import {
|
||||||
|
Logger,
|
||||||
|
LOG_LEVEL_DEBUG,
|
||||||
|
LOG_LEVEL_INFO,
|
||||||
|
LOG_LEVEL_NOTICE,
|
||||||
|
LOG_LEVEL_URGENT,
|
||||||
|
LOG_LEVEL_VERBOSE,
|
||||||
|
setGlobalLogFunction,
|
||||||
|
} from "octagonal-wheels/common/logger";
|
||||||
|
|
||||||
const mocks = vi.hoisted(() => ({
|
const mocks = vi.hoisted(() => ({
|
||||||
setLogHandler: vi.fn(),
|
setLogHandler: vi.fn(),
|
||||||
@@ -29,6 +39,18 @@ function createStandardIoMock() {
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async function invokeGlobalLogger(databasePath: string, message: string, level: LOG_LEVEL, debug = false) {
|
||||||
|
process.argv = ["node", "livesync-cli", databasePath, ...(debug ? ["--debug"] : []), "remote-ls"];
|
||||||
|
const standardIo = createStandardIoMock();
|
||||||
|
mocks.setLogHandler.mockImplementation(() => {
|
||||||
|
Logger(message, level);
|
||||||
|
throw new Error("__LOG_HANDLER_CONFIGURED__");
|
||||||
|
});
|
||||||
|
|
||||||
|
await expect(main(standardIo)).rejects.toThrow("__LOG_HANDLER_CONFIGURED__");
|
||||||
|
return standardIo;
|
||||||
|
}
|
||||||
|
|
||||||
describe("CLI log handler", () => {
|
describe("CLI log handler", () => {
|
||||||
const originalArgv = process.argv.slice();
|
const originalArgv = process.argv.slice();
|
||||||
let databasePath: string;
|
let databasePath: string;
|
||||||
@@ -43,6 +65,7 @@ describe("CLI log handler", () => {
|
|||||||
|
|
||||||
afterEach(async () => {
|
afterEach(async () => {
|
||||||
process.argv = originalArgv.slice();
|
process.argv = originalArgv.slice();
|
||||||
|
setGlobalLogFunction(() => {});
|
||||||
await rm(databasePath, { recursive: true, force: true });
|
await rm(databasePath, { recursive: true, force: true });
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -52,4 +75,50 @@ describe("CLI log handler", () => {
|
|||||||
await expect(main(createStandardIoMock())).rejects.toThrow("__LOG_HANDLER_CONFIGURED__");
|
await expect(main(createStandardIoMock())).rejects.toThrow("__LOG_HANDLER_CONFIGURED__");
|
||||||
expect(mocks.setLogHandler).toHaveBeenCalledWith(expect.any(Function), true);
|
expect(mocks.setLogHandler).toHaveBeenCalledWith(expect.any(Function), true);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it.each([
|
||||||
|
["NOTICE", LOG_LEVEL_NOTICE],
|
||||||
|
["URGENT", LOG_LEVEL_URGENT],
|
||||||
|
] as const)("writes Commonlib %s diagnostics to stderr by default", async (name, level) => {
|
||||||
|
const message = `Commonlib ${name} diagnostic`;
|
||||||
|
const standardIo = await invokeGlobalLogger(databasePath, message, level);
|
||||||
|
|
||||||
|
expect(standardIo.writeStderr).toHaveBeenCalledWith(expect.stringContaining(`[${level}] ${message}\n`));
|
||||||
|
expect(standardIo.writeStdout).not.toHaveBeenCalledWith(expect.stringContaining(message));
|
||||||
|
});
|
||||||
|
|
||||||
|
it.each([
|
||||||
|
["DEBUG", LOG_LEVEL_DEBUG],
|
||||||
|
["INFO", LOG_LEVEL_INFO],
|
||||||
|
["VERBOSE", LOG_LEVEL_VERBOSE],
|
||||||
|
] as const)("keeps Commonlib %s diagnostics hidden by default", async (name, level) => {
|
||||||
|
const message = `Commonlib ${name} diagnostic`;
|
||||||
|
const standardIo = await invokeGlobalLogger(databasePath, message, level);
|
||||||
|
|
||||||
|
expect(standardIo.writeStderr).not.toHaveBeenCalledWith(expect.stringContaining(message));
|
||||||
|
expect(standardIo.writeStdout).not.toHaveBeenCalledWith(expect.stringContaining(message));
|
||||||
|
});
|
||||||
|
|
||||||
|
it("writes detailed Commonlib diagnostics to stderr in debug mode", async () => {
|
||||||
|
const messages = [
|
||||||
|
["DEBUG", LOG_LEVEL_DEBUG],
|
||||||
|
["INFO", LOG_LEVEL_INFO],
|
||||||
|
["VERBOSE", LOG_LEVEL_VERBOSE],
|
||||||
|
] as const;
|
||||||
|
process.argv = ["node", "livesync-cli", databasePath, "--debug", "remote-ls"];
|
||||||
|
const standardIo = createStandardIoMock();
|
||||||
|
mocks.setLogHandler.mockImplementation(() => {
|
||||||
|
for (const [name, level] of messages) Logger(`Commonlib ${name} diagnostic`, level);
|
||||||
|
throw new Error("__LOG_HANDLER_CONFIGURED__");
|
||||||
|
});
|
||||||
|
|
||||||
|
await expect(main(standardIo)).rejects.toThrow("__LOG_HANDLER_CONFIGURED__");
|
||||||
|
|
||||||
|
for (const [name, level] of messages) {
|
||||||
|
expect(standardIo.writeStderr).toHaveBeenCalledWith(
|
||||||
|
expect.stringContaining(`[${level}] Commonlib ${name} diagnostic\n`)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
expect(standardIo.writeStdout).not.toHaveBeenCalledWith(expect.stringContaining("Commonlib"));
|
||||||
|
});
|
||||||
});
|
});
|
||||||
@@ -108,6 +108,8 @@ Arguments:
|
|||||||
Commands:
|
Commands:
|
||||||
daemon (default) Run mirror scan then continuously sync CouchDB <-> local filesystem
|
daemon (default) Run mirror scan then continuously sync CouchDB <-> local filesystem
|
||||||
sync Run one replication cycle and exit
|
sync Run one replication cycle and exit
|
||||||
|
journal-skip <journal-key>
|
||||||
|
Retry replication, skip only the matching stopped Journal, and resume
|
||||||
p2p-peers <timeout> Show discovered peers as [peer]\t<peer-id>\t<peer-name>
|
p2p-peers <timeout> Show discovered peers as [peer]\t<peer-id>\t<peer-name>
|
||||||
p2p-sync <peer> <timeout>
|
p2p-sync <peer> <timeout>
|
||||||
Sync with the specified peer-id or peer-name
|
Sync with the specified peer-id or peer-name
|
||||||
@@ -376,15 +378,10 @@ export async function main(
|
|||||||
const writeLine = avoidStdoutNoise ? writeStderrLine : writeStdoutLine;
|
const writeLine = avoidStdoutNoise ? writeStderrLine : writeStdoutLine;
|
||||||
writeLine(standardIo, ...values);
|
writeLine(standardIo, ...values);
|
||||||
};
|
};
|
||||||
if (options.debug) {
|
|
||||||
setGlobalLogFunction((msg, level) => {
|
setGlobalLogFunction((msg, level) => {
|
||||||
|
if (!options.debug && level !== LOG_LEVEL_NOTICE && level !== LOG_LEVEL_URGENT) return;
|
||||||
writeStderrLine(standardIo, `[${level}]`, msg);
|
writeStderrLine(standardIo, `[${level}]`, msg);
|
||||||
});
|
});
|
||||||
} else {
|
|
||||||
setGlobalLogFunction((msg, level) => {
|
|
||||||
// NO OP, leave it to logFunction
|
|
||||||
});
|
|
||||||
}
|
|
||||||
if (options.command === "init-settings") {
|
if (options.command === "init-settings") {
|
||||||
await createDefaultSettingsFile(options, standardIo);
|
await createDefaultSettingsFile(options, standardIo);
|
||||||
return;
|
return;
|
||||||
|
|||||||
@@ -72,6 +72,7 @@ describe("CLI parseArgs", () => {
|
|||||||
expect(combined).toContain("Usage:");
|
expect(combined).toContain("Usage:");
|
||||||
expect(combined).toContain("livesync-cli <database-path> [options] <command> [command-args]");
|
expect(combined).toContain("livesync-cli <database-path> [options] <command> [command-args]");
|
||||||
expect(combined).toContain("--compat-remote-admin-exit-zero");
|
expect(combined).toContain("--compat-remote-admin-exit-zero");
|
||||||
|
expect(combined).toContain("journal-skip <journal-key>");
|
||||||
});
|
});
|
||||||
|
|
||||||
it("parses p2p-peers command and timeout", () => {
|
it("parses p2p-peers command and timeout", () => {
|
||||||
@@ -101,6 +102,15 @@ describe("CLI parseArgs", () => {
|
|||||||
expect(parsed.commandArgs).toEqual([]);
|
expect(parsed.commandArgs).toEqual([]);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("parses journal-skip and its exact Journal key", () => {
|
||||||
|
process.argv = ["node", "livesync-cli", "./databasePath", "journal-skip", "7-docs.jsonl.gz"];
|
||||||
|
const parsed = parseArgs();
|
||||||
|
|
||||||
|
expect(parsed.databasePath).toBe("./databasePath");
|
||||||
|
expect(parsed.command).toBe("journal-skip");
|
||||||
|
expect(parsed.commandArgs).toEqual(["7-docs.jsonl.gz"]);
|
||||||
|
});
|
||||||
|
|
||||||
it.each(["rebuild", "fetch"] as const)("parses %s and its force flag", (command) => {
|
it.each(["rebuild", "fetch"] as const)("parses %s and its force flag", (command) => {
|
||||||
process.argv = ["node", "livesync-cli", "./databasePath", "--force", command];
|
process.argv = ["node", "livesync-cli", "./databasePath", "--force", command];
|
||||||
const parsed = parseArgs();
|
const parsed = parseArgs();
|
||||||
|
|||||||
@@ -7,6 +7,18 @@
|
|||||||
* remove it from this map in the same change.
|
* remove it from this map in the same change.
|
||||||
*/
|
*/
|
||||||
export const liveSyncProvisionalEnglishMessages = {
|
export const liveSyncProvisionalEnglishMessages = {
|
||||||
|
"Skip stopped Journal and resume": "Skip stopped Journal and resume",
|
||||||
|
"Object Storage only. Confirm to skip the currently stopped Journal and resume replication. The confirmation names the exact Journal key and failure stage. Skipping may leave content or history missing; the next Journal failure still stops replication.":
|
||||||
|
"Object Storage only. Confirm to skip the currently stopped Journal and resume replication. The confirmation names the exact Journal key and failure stage. Skipping may leave content or history missing; the next Journal failure still stops replication.",
|
||||||
|
"Skip and resume": "Skip and resume",
|
||||||
|
"No stopped Journal is available. Retry replication to identify the stopped Journal.":
|
||||||
|
"No stopped Journal is available. Retry replication to identify the stopped Journal.",
|
||||||
|
"Skip stopped Journal '${KEY}' at failure stage '${STAGE}' and resume replication? This may leave content or history missing.":
|
||||||
|
"Skip stopped Journal '${KEY}' at failure stage '${STAGE}' and resume replication? This may leave content or history missing.",
|
||||||
|
"The active replicator changed. Retry replication to identify the stopped Journal.":
|
||||||
|
"The active replicator changed. Retry replication to identify the stopped Journal.",
|
||||||
|
"Journal receipt did not complete after this action. Check the current failure before retrying.":
|
||||||
|
"Journal receipt did not complete after this action. Check the current failure before retrying.",
|
||||||
"File identity": "File identity",
|
"File identity": "File identity",
|
||||||
"Version 0 preserves legacy path identity; version 1 uses Unicode NFC. Physical file names stay unchanged. Update every device and stop synchronisation, then Rebuild from a complete authoritative Vault and Fetch on the other devices. With P2P, Rebuild prepares only the source device's local database. Resolve colliding file names first.":
|
"Version 0 preserves legacy path identity; version 1 uses Unicode NFC. Physical file names stay unchanged. Update every device and stop synchronisation, then Rebuild from a complete authoritative Vault and Fetch on the other devices. With P2P, Rebuild prepares only the source device's local database. Resolve colliding file names first.":
|
||||||
"Version 0 preserves legacy path identity; version 1 uses Unicode NFC. Physical file names stay unchanged. Update every device and stop synchronisation, then Rebuild from a complete authoritative Vault and Fetch on the other devices. With P2P, Rebuild prepares only the source device's local database. Resolve colliding file names first.",
|
"Version 0 preserves legacy path identity; version 1 uses Unicode NFC. Physical file names stay unchanged. Update every device and stop synchronisation, then Rebuild from a complete authoritative Vault and Fetch on the other devices. With P2P, Rebuild prepares only the source device's local database. Resolve colliding file names first.",
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
import { EVENT_REQUEST_PERFORM_GC_V3, eventHub } from "@/common/events.ts";
|
import { EVENT_REQUEST_PERFORM_GC_V3, eventHub } from "@/common/events.ts";
|
||||||
import { $msg } from "@/common/translation";
|
import { $msg } from "@/common/translation";
|
||||||
|
import { Notice } from "@/deps.ts";
|
||||||
import {
|
import {
|
||||||
createCoreSettingsAfterFullReset,
|
createCoreSettingsAfterFullReset,
|
||||||
createEditingSettingsAfterFullReset,
|
createEditingSettingsAfterFullReset,
|
||||||
@@ -8,6 +9,8 @@ import { LOG_LEVEL_NOTICE, Logger } from "@vrtmrz/livesync-commonlib/compat/comm
|
|||||||
import { FlagFilesHumanReadable, FlagFilesOriginal } from "@vrtmrz/livesync-commonlib/compat/common/types";
|
import { FlagFilesHumanReadable, FlagFilesOriginal } from "@vrtmrz/livesync-commonlib/compat/common/types";
|
||||||
import { fireAndForget } from "@vrtmrz/livesync-commonlib/compat/common/utils";
|
import { fireAndForget } from "@vrtmrz/livesync-commonlib/compat/common/utils";
|
||||||
import { LiveSyncCouchDBReplicator } from "@vrtmrz/livesync-commonlib/compat/replication/couchdb/LiveSyncReplicator";
|
import { LiveSyncCouchDBReplicator } from "@vrtmrz/livesync-commonlib/compat/replication/couchdb/LiveSyncReplicator";
|
||||||
|
import { LiveSyncJournalReplicator } from "@vrtmrz/livesync-commonlib/compat/replication/journal/LiveSyncJournalReplicator";
|
||||||
|
import type { StoppedJournal } from "@vrtmrz/livesync-commonlib/compat/replication/journal/JournalSyncTypes";
|
||||||
import { LiveSyncSetting as Setting } from "./LiveSyncSetting.ts";
|
import { LiveSyncSetting as Setting } from "./LiveSyncSetting.ts";
|
||||||
import type { ObsidianLiveSyncSettingTab } from "./ObsidianLiveSyncSettingTab";
|
import type { ObsidianLiveSyncSettingTab } from "./ObsidianLiveSyncSettingTab";
|
||||||
import { visibleOnly, type PageFunctions } from "./SettingPane";
|
import { visibleOnly, type PageFunctions } from "./SettingPane";
|
||||||
@@ -151,6 +154,73 @@ export function paneMaintenance(
|
|||||||
)
|
)
|
||||||
.addOnUpdate(this.onlyOnCouchDB);
|
.addOnUpdate(this.onlyOnCouchDB);
|
||||||
|
|
||||||
|
new Setting(paneEl)
|
||||||
|
.setName($msg("Skip stopped Journal and resume"))
|
||||||
|
.setDesc(
|
||||||
|
$msg(
|
||||||
|
"Object Storage only. Confirm to skip the currently stopped Journal and resume replication. The confirmation names the exact Journal key and failure stage. Skipping may leave content or history missing; the next Journal failure still stops replication."
|
||||||
|
)
|
||||||
|
)
|
||||||
|
.addButton((button) =>
|
||||||
|
setButtonDestructiveState(button)
|
||||||
|
.setButtonText($msg("Skip and resume"))
|
||||||
|
.setDisabled(false)
|
||||||
|
.onClick(async () => {
|
||||||
|
const replicator = this.core.replicator;
|
||||||
|
if (!(replicator instanceof LiveSyncJournalReplicator)) {
|
||||||
|
new Notice(
|
||||||
|
$msg(
|
||||||
|
"No stopped Journal is available. Retry replication to identify the stopped Journal."
|
||||||
|
)
|
||||||
|
);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const stoppedJournal: Readonly<StoppedJournal> | undefined = replicator.getStoppedJournal();
|
||||||
|
if (!stoppedJournal) {
|
||||||
|
new Notice(
|
||||||
|
$msg(
|
||||||
|
"No stopped Journal is available. Retry replication to identify the stopped Journal."
|
||||||
|
)
|
||||||
|
);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const confirmed =
|
||||||
|
(await this.core.confirm.askYesNoDialog(
|
||||||
|
$msg(
|
||||||
|
"Skip stopped Journal '${KEY}' at failure stage '${STAGE}' and resume replication? This may leave content or history missing.",
|
||||||
|
{ KEY: stoppedJournal.key, STAGE: stoppedJournal.stage }
|
||||||
|
),
|
||||||
|
{
|
||||||
|
title: $msg("Skip stopped Journal and resume"),
|
||||||
|
defaultOption: "No",
|
||||||
|
}
|
||||||
|
)) === "yes";
|
||||||
|
if (!confirmed) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (this.core.replicator !== replicator) {
|
||||||
|
new Notice(
|
||||||
|
$msg(
|
||||||
|
"The active replicator changed. Retry replication to identify the stopped Journal."
|
||||||
|
)
|
||||||
|
);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const resumed = await replicator.skipStoppedJournalAndResume(
|
||||||
|
this.core.settings,
|
||||||
|
stoppedJournal
|
||||||
|
);
|
||||||
|
if (!resumed) {
|
||||||
|
new Notice(
|
||||||
|
$msg(
|
||||||
|
"Journal receipt did not complete after this action. Check the current failure before retrying."
|
||||||
|
)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
})
|
||||||
|
)
|
||||||
|
.addOnUpdate(this.onlyOnMinIO);
|
||||||
|
|
||||||
new Setting(paneEl)
|
new Setting(paneEl)
|
||||||
.setName($msg("Reset journal received history"))
|
.setName($msg("Reset journal received history"))
|
||||||
.setDesc(
|
.setDesc(
|
||||||
|
|||||||
@@ -1,8 +1,14 @@
|
|||||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||||
|
|
||||||
const maintenanceHarness = vi.hoisted(() => ({
|
const maintenanceHarness = vi.hoisted(() => ({
|
||||||
createdSettings: [] as Array<{ name: string; click?: () => Promise<void> }>,
|
createdSettings: [] as Array<{
|
||||||
|
name: string;
|
||||||
|
click?: () => Promise<void>;
|
||||||
|
toggleKey?: string;
|
||||||
|
onUpdate?: () => { visibility?: boolean };
|
||||||
|
}>,
|
||||||
logger: vi.fn(),
|
logger: vi.fn(),
|
||||||
|
notice: vi.fn(),
|
||||||
}));
|
}));
|
||||||
|
|
||||||
vi.mock("@/common/events.ts", () => ({
|
vi.mock("@/common/events.ts", () => ({
|
||||||
@@ -10,7 +16,19 @@ vi.mock("@/common/events.ts", () => ({
|
|||||||
eventHub: { emitEvent: vi.fn() },
|
eventHub: { emitEvent: vi.fn() },
|
||||||
}));
|
}));
|
||||||
vi.mock("@/common/translation", () => ({
|
vi.mock("@/common/translation", () => ({
|
||||||
$msg: (message: string) => message,
|
$msg: (message: string, replacements?: Record<string, unknown>) => {
|
||||||
|
for (const [key, value] of Object.entries(replacements ?? {})) {
|
||||||
|
message = message.split(String.fromCharCode(36) + "{" + key + "}").join(String(value));
|
||||||
|
}
|
||||||
|
return message;
|
||||||
|
},
|
||||||
|
}));
|
||||||
|
vi.mock("@/deps.ts", () => ({
|
||||||
|
Notice: class {
|
||||||
|
constructor(message: string) {
|
||||||
|
maintenanceHarness.notice(message);
|
||||||
|
}
|
||||||
|
},
|
||||||
}));
|
}));
|
||||||
vi.mock("@/serviceFeatures/setupObsidian/settingsReset.ts", () => ({
|
vi.mock("@/serviceFeatures/setupObsidian/settingsReset.ts", () => ({
|
||||||
createCoreSettingsAfterFullReset: vi.fn(),
|
createCoreSettingsAfterFullReset: vi.fn(),
|
||||||
@@ -33,10 +51,15 @@ vi.mock("@vrtmrz/livesync-commonlib/compat/common/utils", () => ({
|
|||||||
vi.mock("@vrtmrz/livesync-commonlib/compat/replication/couchdb/LiveSyncReplicator", () => ({
|
vi.mock("@vrtmrz/livesync-commonlib/compat/replication/couchdb/LiveSyncReplicator", () => ({
|
||||||
LiveSyncCouchDBReplicator: class {},
|
LiveSyncCouchDBReplicator: class {},
|
||||||
}));
|
}));
|
||||||
|
vi.mock("@vrtmrz/livesync-commonlib/compat/replication/journal/LiveSyncJournalReplicator", () => ({
|
||||||
|
LiveSyncJournalReplicator: class {},
|
||||||
|
}));
|
||||||
vi.mock("./LiveSyncSetting.ts", () => ({
|
vi.mock("./LiveSyncSetting.ts", () => ({
|
||||||
LiveSyncSetting: class {
|
LiveSyncSetting: class {
|
||||||
name = "";
|
name = "";
|
||||||
click?: () => Promise<void>;
|
click?: () => Promise<void>;
|
||||||
|
toggleKey?: string;
|
||||||
|
onUpdate?: () => { visibility?: boolean };
|
||||||
|
|
||||||
constructor() {
|
constructor() {
|
||||||
maintenanceHarness.createdSettings.push(this);
|
maintenanceHarness.createdSettings.push(this);
|
||||||
@@ -51,6 +74,11 @@ vi.mock("./LiveSyncSetting.ts", () => ({
|
|||||||
return this;
|
return this;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
autoWireToggle(key: string) {
|
||||||
|
this.toggleKey = key;
|
||||||
|
return this;
|
||||||
|
}
|
||||||
|
|
||||||
addButton(callback: (button: this) => void) {
|
addButton(callback: (button: this) => void) {
|
||||||
callback(this);
|
callback(this);
|
||||||
return this;
|
return this;
|
||||||
@@ -73,7 +101,8 @@ vi.mock("./LiveSyncSetting.ts", () => ({
|
|||||||
return this;
|
return this;
|
||||||
}
|
}
|
||||||
|
|
||||||
addOnUpdate() {
|
addOnUpdate(onUpdate: () => { visibility?: boolean }) {
|
||||||
|
this.onUpdate = onUpdate;
|
||||||
return this;
|
return this;
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
@@ -85,14 +114,61 @@ vi.mock("./settingComponentStyles.ts", () => ({
|
|||||||
setButtonDestructiveState: <T>(button: T) => button,
|
setButtonDestructiveState: <T>(button: T) => button,
|
||||||
}));
|
}));
|
||||||
|
|
||||||
|
import { LiveSyncJournalReplicator } from "@vrtmrz/livesync-commonlib/compat/replication/journal/LiveSyncJournalReplicator";
|
||||||
import { paneMaintenance } from "./PaneMaintenance.ts";
|
import { paneMaintenance } from "./PaneMaintenance.ts";
|
||||||
|
|
||||||
afterEach(() => {
|
afterEach(() => {
|
||||||
maintenanceHarness.createdSettings.length = 0;
|
maintenanceHarness.createdSettings.length = 0;
|
||||||
maintenanceHarness.logger.mockClear();
|
maintenanceHarness.logger.mockClear();
|
||||||
|
maintenanceHarness.notice.mockClear();
|
||||||
vi.clearAllMocks();
|
vi.clearAllMocks();
|
||||||
});
|
});
|
||||||
|
|
||||||
|
type TestStoppedJournal = Readonly<{ key: string; stage: string }>;
|
||||||
|
type ConfirmationOptions = { title: string; defaultOption: "No" };
|
||||||
|
type AskYesNoDialog = (message: string, options: ConfirmationOptions) => Promise<"yes" | "no">;
|
||||||
|
|
||||||
|
function createJournalReplicator(stoppedJournal: TestStoppedJournal | undefined, result = true) {
|
||||||
|
const getStoppedJournal = vi.fn(() => stoppedJournal);
|
||||||
|
const skipStoppedJournalAndResume = vi.fn(async (_settings: unknown, _snapshot: unknown) => result);
|
||||||
|
const replicator = Object.assign(new LiveSyncJournalReplicator({} as never), {
|
||||||
|
getStoppedJournal,
|
||||||
|
skipStoppedJournalAndResume,
|
||||||
|
});
|
||||||
|
return { replicator, getStoppedJournal, skipStoppedJournalAndResume };
|
||||||
|
}
|
||||||
|
|
||||||
|
function createSyncingPane(replicator: unknown, askYesNoDialog: AskYesNoDialog = async () => "no") {
|
||||||
|
const settings = { databaseName: "test-database" };
|
||||||
|
const onlyOnMinIO = vi.fn(() => ({ visibility: true }));
|
||||||
|
const confirmation = vi.fn(askYesNoDialog);
|
||||||
|
const host = {
|
||||||
|
core: { replicator, settings, confirm: { askYesNoDialog: confirmation } },
|
||||||
|
createEl: vi.fn(),
|
||||||
|
getMinioJournalSyncClient: vi.fn(),
|
||||||
|
onlyOnCouchDB: vi.fn(),
|
||||||
|
onlyOnCouchDBOrMinIO: vi.fn(),
|
||||||
|
onlyOnMinIO,
|
||||||
|
services: { replication: { markLocked: vi.fn(), markUnlocked: vi.fn() } },
|
||||||
|
};
|
||||||
|
const addPanel = vi.fn((_parent: HTMLElement, heading: string) => ({
|
||||||
|
then(callback: (paneEl: HTMLElement) => void) {
|
||||||
|
if (heading === "Syncing") {
|
||||||
|
callback({} as HTMLElement);
|
||||||
|
}
|
||||||
|
return Promise.resolve();
|
||||||
|
},
|
||||||
|
}));
|
||||||
|
|
||||||
|
paneMaintenance.call(host as never, {} as HTMLElement, { addPanel } as never);
|
||||||
|
|
||||||
|
const setting = maintenanceHarness.createdSettings.find(({ name }) => name === "Skip stopped Journal and resume");
|
||||||
|
if (!setting?.click) {
|
||||||
|
throw new Error("Skip stopped Journal action was not registered");
|
||||||
|
}
|
||||||
|
return { setting, click: setting.click, host, settings, askYesNoDialog: confirmation, onlyOnMinIO };
|
||||||
|
}
|
||||||
|
|
||||||
describe("paneMaintenance", () => {
|
describe("paneMaintenance", () => {
|
||||||
it("does not announce success when the remote wipe reports failure", async () => {
|
it("does not announce success when the remote wipe reports failure", async () => {
|
||||||
const updateCheckPointInfo = vi.fn(async () => undefined);
|
const updateCheckPointInfo = vi.fn(async () => undefined);
|
||||||
@@ -180,9 +256,139 @@ describe("paneMaintenance", () => {
|
|||||||
|
|
||||||
expect(resetDatabase).toHaveBeenCalledOnce();
|
expect(resetDatabase).toHaveBeenCalledOnce();
|
||||||
expect(initialiseDatabase).toHaveBeenCalledOnce();
|
expect(initialiseDatabase).toHaveBeenCalledOnce();
|
||||||
expect(maintenanceHarness.logger).toHaveBeenCalledWith(
|
expect(maintenanceHarness.logger).toHaveBeenCalledWith("Ui.Common.LocalDatabaseInitialisationFailed", "notice");
|
||||||
"Ui.Common.LocalDatabaseInitialisationFailed",
|
});
|
||||||
"notice"
|
|
||||||
|
it("removes the broken journal toggle and preserves received history reset semantics", async () => {
|
||||||
|
const checkpoint = {
|
||||||
|
receivedFiles: new Set(["received-file"]),
|
||||||
|
knownIDs: new Set(["known-id"]),
|
||||||
|
sentIDs: new Set(["sent-id"]),
|
||||||
|
sentFiles: new Set(["sent-file"]),
|
||||||
|
lastLocalSeq: 23,
|
||||||
|
};
|
||||||
|
let updatedCheckpoint: typeof checkpoint | undefined;
|
||||||
|
const updateCheckPointInfo = vi.fn(async (update: (info: typeof checkpoint) => typeof checkpoint) => {
|
||||||
|
updatedCheckpoint = update(checkpoint);
|
||||||
|
});
|
||||||
|
const onlyOnMinIO = vi.fn(() => ({ visibility: true }));
|
||||||
|
const addPanel = vi.fn((_parent: HTMLElement, heading: string) => ({
|
||||||
|
then(callback: (paneEl: HTMLElement) => void) {
|
||||||
|
if (heading === "Syncing") {
|
||||||
|
callback({} as HTMLElement);
|
||||||
|
}
|
||||||
|
return Promise.resolve();
|
||||||
|
},
|
||||||
|
}));
|
||||||
|
const host = {
|
||||||
|
core: { replicator: {}, storageAccess: {} },
|
||||||
|
createEl: vi.fn(),
|
||||||
|
getMinioJournalSyncClient: vi.fn(() => ({ updateCheckPointInfo })),
|
||||||
|
onlyOnCouchDB: vi.fn(),
|
||||||
|
onlyOnCouchDBOrMinIO: vi.fn(),
|
||||||
|
onlyOnMinIO,
|
||||||
|
services: { replication: { markLocked: vi.fn(), markUnlocked: vi.fn() } },
|
||||||
|
};
|
||||||
|
|
||||||
|
paneMaintenance.call(host as never, {} as HTMLElement, { addPanel } as never);
|
||||||
|
|
||||||
|
expect(maintenanceHarness.createdSettings.some(({ toggleKey }) => toggleKey === "ignoreBrokenJournals")).toBe(
|
||||||
|
false
|
||||||
|
);
|
||||||
|
const resetReceivedHistory = maintenanceHarness.createdSettings.find(
|
||||||
|
({ name }) => name === "Reset journal received history"
|
||||||
|
);
|
||||||
|
if (!resetReceivedHistory?.click) {
|
||||||
|
throw new Error("Reset journal received history action was not registered");
|
||||||
|
}
|
||||||
|
await resetReceivedHistory.click();
|
||||||
|
|
||||||
|
expect(updateCheckPointInfo).toHaveBeenCalledOnce();
|
||||||
|
expect(updatedCheckpoint).toEqual({
|
||||||
|
...checkpoint,
|
||||||
|
receivedFiles: new Set(),
|
||||||
|
knownIDs: new Set(),
|
||||||
|
});
|
||||||
|
expect(maintenanceHarness.logger).toHaveBeenCalledWith("Journal received history has been cleared.", "notice");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("asks the user to retry when there is no stopped Journal", async () => {
|
||||||
|
const { replicator, getStoppedJournal, skipStoppedJournalAndResume } = createJournalReplicator(undefined);
|
||||||
|
const rendered = createSyncingPane(replicator);
|
||||||
|
|
||||||
|
expect(rendered.setting.onUpdate).toBe(rendered.onlyOnMinIO);
|
||||||
|
await rendered.click();
|
||||||
|
|
||||||
|
expect(getStoppedJournal).toHaveBeenCalledOnce();
|
||||||
|
expect(rendered.askYesNoDialog).not.toHaveBeenCalled();
|
||||||
|
expect(skipStoppedJournalAndResume).not.toHaveBeenCalled();
|
||||||
|
expect(maintenanceHarness.notice).toHaveBeenCalledWith(
|
||||||
|
"No stopped Journal is available. Retry replication to identify the stopped Journal."
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not skip the stopped Journal when confirmation is cancelled", async () => {
|
||||||
|
const stoppedJournal = { key: "journal/one", stage: "save-records" } as const;
|
||||||
|
const { replicator, skipStoppedJournalAndResume } = createJournalReplicator(stoppedJournal);
|
||||||
|
const rendered = createSyncingPane(replicator);
|
||||||
|
|
||||||
|
await rendered.click();
|
||||||
|
|
||||||
|
expect(rendered.askYesNoDialog).toHaveBeenCalledOnce();
|
||||||
|
const call = rendered.askYesNoDialog.mock.calls[0];
|
||||||
|
if (!call) {
|
||||||
|
throw new Error("Stopped Journal confirmation was not requested");
|
||||||
|
}
|
||||||
|
const [message, options] = call;
|
||||||
|
expect(message).toContain(stoppedJournal.key);
|
||||||
|
expect(message).toContain(stoppedJournal.stage);
|
||||||
|
expect(options).toEqual({ title: "Skip stopped Journal and resume", defaultOption: "No" });
|
||||||
|
expect(skipStoppedJournalAndResume).not.toHaveBeenCalled();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("skips and resumes with the exact confirmed stopped Journal snapshot", async () => {
|
||||||
|
const stoppedJournal = { key: "journal/one", stage: "save-records" } as const;
|
||||||
|
const { replicator, skipStoppedJournalAndResume } = createJournalReplicator(stoppedJournal);
|
||||||
|
const confirm = vi.fn(async () => "yes" as const);
|
||||||
|
const rendered = createSyncingPane(replicator, confirm);
|
||||||
|
|
||||||
|
await rendered.click();
|
||||||
|
|
||||||
|
expect(skipStoppedJournalAndResume).toHaveBeenCalledOnce();
|
||||||
|
expect(skipStoppedJournalAndResume.mock.calls[0]?.[0]).toBe(rendered.settings);
|
||||||
|
expect(skipStoppedJournalAndResume.mock.calls[0]?.[1]).toBe(stoppedJournal);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not skip after the active replicator changes during confirmation", async () => {
|
||||||
|
const stoppedJournal = { key: "journal/one", stage: "save-records" } as const;
|
||||||
|
const active = createJournalReplicator(stoppedJournal);
|
||||||
|
const replacement = createJournalReplicator(stoppedJournal);
|
||||||
|
const confirm = vi.fn(async () => "yes" as const);
|
||||||
|
const rendered = createSyncingPane(active.replicator, confirm);
|
||||||
|
confirm.mockImplementation(async () => {
|
||||||
|
rendered.host.core.replicator = replacement.replicator;
|
||||||
|
return "yes";
|
||||||
|
});
|
||||||
|
|
||||||
|
await rendered.click();
|
||||||
|
|
||||||
|
expect(active.skipStoppedJournalAndResume).not.toHaveBeenCalled();
|
||||||
|
expect(replacement.skipStoppedJournalAndResume).not.toHaveBeenCalled();
|
||||||
|
expect(maintenanceHarness.notice).toHaveBeenCalledWith(
|
||||||
|
"The active replicator changed. Retry replication to identify the stopped Journal."
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("reports when receipt does not complete after skip and resume", async () => {
|
||||||
|
const stoppedJournal = { key: "journal/one", stage: "save-records" } as const;
|
||||||
|
const { replicator, skipStoppedJournalAndResume } = createJournalReplicator(stoppedJournal, false);
|
||||||
|
const rendered = createSyncingPane(replicator, async () => "yes");
|
||||||
|
|
||||||
|
await rendered.click();
|
||||||
|
|
||||||
|
expect(skipStoppedJournalAndResume).toHaveBeenCalledOnce();
|
||||||
|
expect(maintenanceHarness.notice).toHaveBeenCalledWith(
|
||||||
|
"Journal receipt did not complete after this action. Check the current failure before retrying."
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
@@ -16,5 +16,6 @@
|
|||||||
"test:e2e-matrix:minio-enc0",
|
"test:e2e-matrix:minio-enc0",
|
||||||
"test:e2e-matrix:minio-enc1",
|
"test:e2e-matrix:minio-enc1",
|
||||||
"test:fetch-database-continuity",
|
"test:fetch-database-continuity",
|
||||||
"test:journal-chunk-recovery"
|
"test:journal-chunk-recovery",
|
||||||
|
"test:journal-receive-failure"
|
||||||
]
|
]
|
||||||
+35
-99
@@ -1,14 +1,9 @@
|
|||||||
{
|
{
|
||||||
"test:helpers": {
|
"test:helpers": {
|
||||||
"test": [
|
"test": ["helpers/process.test.ts", "test-ci-task-contract.ts"]
|
||||||
"helpers/process.test.ts",
|
|
||||||
"test-ci-task-contract.ts"
|
|
||||||
]
|
|
||||||
},
|
},
|
||||||
"test": {
|
"test": {
|
||||||
"tasks": [
|
"tasks": ["test:ci"]
|
||||||
"test:ci"
|
|
||||||
]
|
|
||||||
},
|
},
|
||||||
"test:ci": {
|
"test:ci": {
|
||||||
"script": "run-ci-suite.ts"
|
"script": "run-ci-suite.ts"
|
||||||
@@ -17,101 +12,61 @@
|
|||||||
"script": "run-compose-p2p.ts"
|
"script": "run-compose-p2p.ts"
|
||||||
},
|
},
|
||||||
"test:local": {
|
"test:local": {
|
||||||
"tasks": [
|
"tasks": ["test:setup-put-cat", "test:mirror", "test:daemon"]
|
||||||
"test:setup-put-cat",
|
|
||||||
"test:mirror",
|
|
||||||
"test:daemon"
|
|
||||||
]
|
|
||||||
},
|
},
|
||||||
"test:daemon": {
|
"test:daemon": {
|
||||||
"test": [
|
"test": ["test-daemon.ts"]
|
||||||
"test-daemon.ts"
|
|
||||||
]
|
|
||||||
},
|
},
|
||||||
"test:daemon-startup": {
|
"test:daemon-startup": {
|
||||||
"test": [
|
"test": ["test-daemon-startup.ts"]
|
||||||
"test-daemon-startup.ts"
|
|
||||||
]
|
|
||||||
},
|
},
|
||||||
"test:decoupled-vault": {
|
"test:decoupled-vault": {
|
||||||
"test": [
|
"test": ["test-decoupled-vault.ts"]
|
||||||
"test-decoupled-vault.ts"
|
|
||||||
]
|
|
||||||
},
|
},
|
||||||
"test:remote-commands": {
|
"test:remote-commands": {
|
||||||
"test": [
|
"test": ["test-remote-commands.ts"]
|
||||||
"test-remote-commands.ts"
|
|
||||||
]
|
|
||||||
},
|
},
|
||||||
"test:settings-writeback": {
|
"test:settings-writeback": {
|
||||||
"test": [
|
"test": ["test-settings-writeback.ts"]
|
||||||
"test-settings-writeback.ts"
|
|
||||||
]
|
|
||||||
},
|
},
|
||||||
"test:fetch-database-continuity": {
|
"test:fetch-database-continuity": {
|
||||||
"test": [
|
"test": ["test-fetch-database-continuity.ts"]
|
||||||
"test-fetch-database-continuity.ts"
|
|
||||||
]
|
|
||||||
},
|
},
|
||||||
"test:remote-administration-exit-codes": {
|
"test:remote-administration-exit-codes": {
|
||||||
"test": [
|
"test": ["test-remote-administration-exit-codes.ts"]
|
||||||
"test-remote-administration-exit-codes.ts"
|
|
||||||
]
|
|
||||||
},
|
},
|
||||||
"test:push-pull": {
|
"test:push-pull": {
|
||||||
"test": [
|
"test": ["test-push-pull.ts"]
|
||||||
"test-push-pull.ts"
|
|
||||||
]
|
|
||||||
},
|
},
|
||||||
"test:setup-put-cat": {
|
"test:setup-put-cat": {
|
||||||
"test": [
|
"test": ["test-setup-put-cat.ts"]
|
||||||
"test-setup-put-cat.ts"
|
|
||||||
]
|
|
||||||
},
|
},
|
||||||
"test:mirror": {
|
"test:mirror": {
|
||||||
"test": [
|
"test": ["test-mirror.ts"]
|
||||||
"test-mirror.ts"
|
|
||||||
]
|
|
||||||
},
|
},
|
||||||
"test:sync-two-local": {
|
"test:sync-two-local": {
|
||||||
"test": [
|
"test": ["test-sync-two-local-databases.ts"]
|
||||||
"test-sync-two-local-databases.ts"
|
|
||||||
]
|
|
||||||
},
|
},
|
||||||
"test:sync-locked-remote": {
|
"test:sync-locked-remote": {
|
||||||
"test": [
|
"test": ["test-sync-locked-remote.ts"]
|
||||||
"test-sync-locked-remote.ts"
|
|
||||||
]
|
|
||||||
},
|
},
|
||||||
"test:p2p-host": {
|
"test:p2p-host": {
|
||||||
"test": [
|
"test": ["test-p2p-host.ts"]
|
||||||
"test-p2p-host.ts"
|
|
||||||
]
|
|
||||||
},
|
},
|
||||||
"test:p2p-peers": {
|
"test:p2p-peers": {
|
||||||
"test": [
|
"test": ["test-p2p-peers-local-relay.ts"]
|
||||||
"test-p2p-peers-local-relay.ts"
|
|
||||||
]
|
|
||||||
},
|
},
|
||||||
"test:p2p-sync": {
|
"test:p2p-sync": {
|
||||||
"test": [
|
"test": ["test-p2p-sync.ts"]
|
||||||
"test-p2p-sync.ts"
|
|
||||||
]
|
|
||||||
},
|
},
|
||||||
"test:p2p-automatic-host": {
|
"test:p2p-automatic-host": {
|
||||||
"test": [
|
"test": ["test-p2p-automatic-host.ts"]
|
||||||
"test-p2p-automatic-host.ts"
|
|
||||||
]
|
|
||||||
},
|
},
|
||||||
"test:p2p-replacement": {
|
"test:p2p-replacement": {
|
||||||
"test": [
|
"test": ["test-p2p-replicator-replacement.ts"]
|
||||||
"test-p2p-replicator-replacement.ts"
|
|
||||||
]
|
|
||||||
},
|
},
|
||||||
"test:p2p-relay-disconnect": {
|
"test:p2p-relay-disconnect": {
|
||||||
"test": [
|
"test": ["test-p2p-relay-disconnect.ts"]
|
||||||
"test-p2p-relay-disconnect.ts"
|
|
||||||
]
|
|
||||||
},
|
},
|
||||||
"test:p2p:ci": {
|
"test:p2p:ci": {
|
||||||
"test": [
|
"test": [
|
||||||
@@ -122,19 +77,13 @@
|
|||||||
]
|
]
|
||||||
},
|
},
|
||||||
"test:p2p-three-nodes": {
|
"test:p2p-three-nodes": {
|
||||||
"test": [
|
"test": ["test-p2p-three-nodes-conflict.ts"]
|
||||||
"test-p2p-three-nodes-conflict.ts"
|
|
||||||
]
|
|
||||||
},
|
},
|
||||||
"test:p2p-upload-download": {
|
"test:p2p-upload-download": {
|
||||||
"test": [
|
"test": ["test-p2p-upload-download-repro.ts"]
|
||||||
"test-p2p-upload-download-repro.ts"
|
|
||||||
]
|
|
||||||
},
|
},
|
||||||
"test:benchmark-contract": {
|
"test:benchmark-contract": {
|
||||||
"test": [
|
"test": ["test-benchmark-contract.ts"]
|
||||||
"test-benchmark-contract.ts"
|
|
||||||
]
|
|
||||||
},
|
},
|
||||||
"bench:p2p": {
|
"bench:p2p": {
|
||||||
"script": "bench-p2p.ts"
|
"script": "bench-p2p.ts"
|
||||||
@@ -156,9 +105,7 @@
|
|||||||
},
|
},
|
||||||
"bench:item1": {
|
"bench:item1": {
|
||||||
"command": "bash",
|
"command": "bash",
|
||||||
"args": [
|
"args": ["./bench-run-item1.sh"]
|
||||||
"./bench-run-item1.sh"
|
|
||||||
]
|
|
||||||
},
|
},
|
||||||
"bench:item1:full": {
|
"bench:item1:full": {
|
||||||
"command": "bash",
|
"command": "bash",
|
||||||
@@ -168,42 +115,31 @@
|
|||||||
]
|
]
|
||||||
},
|
},
|
||||||
"test:e2e-couchdb": {
|
"test:e2e-couchdb": {
|
||||||
"test": [
|
"test": ["test-e2e-two-vaults-couchdb.ts"]
|
||||||
"test-e2e-two-vaults-couchdb.ts"
|
|
||||||
]
|
|
||||||
},
|
},
|
||||||
"test:journal-chunk-recovery": {
|
"test:journal-chunk-recovery": {
|
||||||
"test": [
|
"test": ["test-journal-chunk-recovery.ts"]
|
||||||
"test-journal-chunk-recovery.ts"
|
},
|
||||||
]
|
"test:journal-receive-failure": {
|
||||||
|
"test": ["test-journal-receive-failure.ts"]
|
||||||
},
|
},
|
||||||
"test:e2e-matrix": {
|
"test:e2e-matrix": {
|
||||||
"test": [
|
"test": ["test-e2e-two-vaults-matrix.ts"]
|
||||||
"test-e2e-two-vaults-matrix.ts"
|
|
||||||
]
|
|
||||||
},
|
},
|
||||||
"test:e2e-matrix:couchdb-enc0": {
|
"test:e2e-matrix:couchdb-enc0": {
|
||||||
"test": [
|
"test": ["test-e2e-two-vaults-matrix.ts"],
|
||||||
"test-e2e-two-vaults-matrix.ts"
|
|
||||||
],
|
|
||||||
"filter": "e2e matrix: COUCHDB-enc0"
|
"filter": "e2e matrix: COUCHDB-enc0"
|
||||||
},
|
},
|
||||||
"test:e2e-matrix:couchdb-enc1": {
|
"test:e2e-matrix:couchdb-enc1": {
|
||||||
"test": [
|
"test": ["test-e2e-two-vaults-matrix.ts"],
|
||||||
"test-e2e-two-vaults-matrix.ts"
|
|
||||||
],
|
|
||||||
"filter": "e2e matrix: COUCHDB-enc1"
|
"filter": "e2e matrix: COUCHDB-enc1"
|
||||||
},
|
},
|
||||||
"test:e2e-matrix:minio-enc0": {
|
"test:e2e-matrix:minio-enc0": {
|
||||||
"test": [
|
"test": ["test-e2e-two-vaults-matrix.ts"],
|
||||||
"test-e2e-two-vaults-matrix.ts"
|
|
||||||
],
|
|
||||||
"filter": "e2e matrix: MINIO-enc0"
|
"filter": "e2e matrix: MINIO-enc0"
|
||||||
},
|
},
|
||||||
"test:e2e-matrix:minio-enc1": {
|
"test:e2e-matrix:minio-enc1": {
|
||||||
"test": [
|
"test": ["test-e2e-two-vaults-matrix.ts"],
|
||||||
"test-e2e-two-vaults-matrix.ts"
|
|
||||||
],
|
|
||||||
"filter": "e2e matrix: MINIO-enc1"
|
"filter": "e2e matrix: MINIO-enc1"
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -0,0 +1,369 @@
|
|||||||
|
import { randomUUID } from "node:crypto";
|
||||||
|
import { mkdir, readFile, readdir, writeFile } from "node:fs/promises";
|
||||||
|
import { dirname, join } from "node:path";
|
||||||
|
import { GetObjectCommand, ListObjectsV2Command, PutObjectCommand, S3Client } from "@aws-sdk/client-s3";
|
||||||
|
import assert from "node:assert/strict";
|
||||||
|
import { test } from "node:test";
|
||||||
|
import { runCli, runCliOrFail, sanitiseCatStdout } from "./helpers/cli.ts";
|
||||||
|
import { startMinio, stopMinio } from "./helpers/docker.ts";
|
||||||
|
import { TempDir } from "./helpers/temp.ts";
|
||||||
|
import { applyRemoteSyncSettings, initSettingsFile } from "./helpers/settings.ts";
|
||||||
|
|
||||||
|
type JsonObject = Record<string, unknown>;
|
||||||
|
|
||||||
|
function requireEnv(...keys: string[]): string {
|
||||||
|
for (const key of keys) {
|
||||||
|
const value = process.env[key]?.trim();
|
||||||
|
if (value) return value;
|
||||||
|
}
|
||||||
|
throw new Error(`Required env var is missing: ${keys.join(" or ")}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
function createJournalReceiveBucketName(baseBucket: string, suffix: string): string {
|
||||||
|
const bucketSuffix = `journal-receive-${suffix}`;
|
||||||
|
const maxBaseLength = 63 - bucketSuffix.length - 1;
|
||||||
|
const truncatedBase = baseBucket.slice(0, maxBaseLength).replace(/-+$/, "");
|
||||||
|
return truncatedBase ? `${truncatedBase}-${bucketSuffix}` : bucketSuffix;
|
||||||
|
}
|
||||||
|
|
||||||
|
function isJsonObject(value: unknown): value is JsonObject {
|
||||||
|
return typeof value === "object" && value !== null && !Array.isArray(value);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function listJournalKeys(client: S3Client, bucket: string): Promise<string[]> {
|
||||||
|
const result = await client.send(new ListObjectsV2Command({ Bucket: bucket }));
|
||||||
|
assert.equal(result.IsTruncated, false, "The disposable test bucket should fit in one S3 listing page");
|
||||||
|
return (result.Contents ?? [])
|
||||||
|
.flatMap(({ Key }) => (Key && /^\d+-docs\.jsonl\.gz$/u.test(Key) ? [Key] : []))
|
||||||
|
.sort((left, right) => left.localeCompare(right, undefined, { numeric: true }));
|
||||||
|
}
|
||||||
|
|
||||||
|
async function addedJournalKey(client: S3Client, bucket: string, before: Set<string>): Promise<string> {
|
||||||
|
const added = (await listJournalKeys(client, bucket)).filter((key) => !before.has(key));
|
||||||
|
assert.equal(added.length, 1, `Expected one new Journal object, got: ${added.join(", ")}`);
|
||||||
|
const [key] = added;
|
||||||
|
assert.ok(key);
|
||||||
|
return key;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function withCorruptedJournals(
|
||||||
|
client: S3Client,
|
||||||
|
bucket: string,
|
||||||
|
keys: string[],
|
||||||
|
action: () => Promise<void>
|
||||||
|
): Promise<void> {
|
||||||
|
const originals = await Promise.all(
|
||||||
|
keys.map(async (key) => {
|
||||||
|
const response = await client.send(new GetObjectCommand({ Bucket: bucket, Key: key }));
|
||||||
|
const body = response.Body;
|
||||||
|
assert.ok(body, `Could not read the original Journal bytes for ${key}`);
|
||||||
|
const bytes = await body.transformToByteArray();
|
||||||
|
assert.ok(bytes.length > 0, `The owned Journal object ${key} was empty`);
|
||||||
|
return { key, bytes };
|
||||||
|
})
|
||||||
|
);
|
||||||
|
|
||||||
|
try {
|
||||||
|
await Promise.all(
|
||||||
|
originals.map(({ key }) =>
|
||||||
|
client.send(
|
||||||
|
new PutObjectCommand({
|
||||||
|
Bucket: bucket,
|
||||||
|
Key: key,
|
||||||
|
Body: Uint8Array.of(0xff, 0xff, 0xff),
|
||||||
|
})
|
||||||
|
)
|
||||||
|
)
|
||||||
|
);
|
||||||
|
await action();
|
||||||
|
} finally {
|
||||||
|
await Promise.all(
|
||||||
|
originals.map(({ key, bytes }) =>
|
||||||
|
client.send(new PutObjectCommand({ Bucket: bucket, Key: key, Body: bytes }))
|
||||||
|
)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function readReceivedJournalKeys(databasePath: string): Promise<Set<string>> {
|
||||||
|
const scopeRoot = join(databasePath, ".livesync", "runtime", "scopes");
|
||||||
|
const files = await readdir(scopeRoot, { recursive: true });
|
||||||
|
for (const relativePath of files.filter((file) => file.endsWith("keyvalue-db.json"))) {
|
||||||
|
const stored: unknown = JSON.parse(await readFile(join(scopeRoot, relativePath), "utf8"));
|
||||||
|
if (!isJsonObject(stored)) continue;
|
||||||
|
const entry = Object.entries(stored).find(([key]) => key.includes("bucketsync-checkpoint-"));
|
||||||
|
if (!entry || !isJsonObject(entry[1])) continue;
|
||||||
|
const receivedFiles = entry[1]["receivedFiles"];
|
||||||
|
if (!isJsonObject(receivedFiles)) continue;
|
||||||
|
const values = receivedFiles["values"];
|
||||||
|
if (receivedFiles["__nodeKvType"] !== "Set" || !Array.isArray(values)) continue;
|
||||||
|
return new Set(values.filter((value): value is string => typeof value === "string"));
|
||||||
|
}
|
||||||
|
throw new Error(`Journal receive checkpoint was not found under ${scopeRoot}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function runScenario(encrypt: boolean): Promise<void> {
|
||||||
|
const suffix = `${encrypt ? "enc1" : "enc0"}-${Date.now()}-${randomUUID().replaceAll("-", "").slice(0, 8)}`;
|
||||||
|
const minioEndpoint = requireEnv("MINIO_ENDPOINT", "minioEndpoint").replace(/\/$/, "");
|
||||||
|
const minioAccessKey = requireEnv("MINIO_ACCESS_KEY", "accessKey");
|
||||||
|
const minioSecretKey = requireEnv("MINIO_SECRET_KEY", "secretKey");
|
||||||
|
const minioBucket = createJournalReceiveBucketName(requireEnv("MINIO_BUCKET_NAME", "bucketName"), suffix);
|
||||||
|
const passphrase = "journal-receive-e2e-passphrase";
|
||||||
|
const keepDocker = process.env["LIVESYNC_DEBUG_KEEP_DOCKER"] === "1";
|
||||||
|
const stablePath = "e2e/stable.md";
|
||||||
|
const middlePath = "e2e/middle.md";
|
||||||
|
const secondBrokenPath = "e2e/second-broken.md";
|
||||||
|
const laterPath = "e2e/later.md";
|
||||||
|
const stableV1 = "Stable file revision one.\n";
|
||||||
|
const middleContent = "Content in the intermediate Journal.\n";
|
||||||
|
const secondBrokenContent = "Content in the next intermediate Journal.\n";
|
||||||
|
const stableV2 = "Stable file revision two.\n";
|
||||||
|
const laterContent = "Later Journal content.\n";
|
||||||
|
|
||||||
|
await using workDir = await TempDir.create(`livesync-cli-journal-receive-${suffix}`);
|
||||||
|
const sourceDatabase = workDir.join("source-db");
|
||||||
|
const sourceVault = workDir.join("source-vault");
|
||||||
|
const sourceSettings = workDir.join("source-settings.json");
|
||||||
|
const defaultDatabase = workDir.join("default-receiver-db");
|
||||||
|
const defaultVault = workDir.join("default-receiver-vault");
|
||||||
|
const defaultSettings = workDir.join("default-receiver-settings.json");
|
||||||
|
const skipDatabase = workDir.join("skip-receiver-db");
|
||||||
|
const skipVault = workDir.join("skip-receiver-vault");
|
||||||
|
const skipSettings = workDir.join("skip-receiver-settings.json");
|
||||||
|
const sourceStableFile = join(sourceVault, stablePath);
|
||||||
|
const sourceMiddleFile = join(sourceVault, middlePath);
|
||||||
|
const sourceSecondBrokenFile = join(sourceVault, secondBrokenPath);
|
||||||
|
const sourceLaterFile = join(sourceVault, laterPath);
|
||||||
|
const commonSettings = {
|
||||||
|
remoteType: "MINIO" as const,
|
||||||
|
minioBucket,
|
||||||
|
minioEndpoint,
|
||||||
|
minioAccessKey,
|
||||||
|
minioSecretKey,
|
||||||
|
encrypt,
|
||||||
|
passphrase,
|
||||||
|
enableCompression: false,
|
||||||
|
};
|
||||||
|
const client = new S3Client({
|
||||||
|
endpoint: minioEndpoint,
|
||||||
|
region: "us-east-1",
|
||||||
|
forcePathStyle: true,
|
||||||
|
credentials: { accessKeyId: minioAccessKey, secretAccessKey: minioSecretKey },
|
||||||
|
});
|
||||||
|
|
||||||
|
await Promise.all([
|
||||||
|
mkdir(sourceDatabase, { recursive: true }),
|
||||||
|
mkdir(sourceVault, { recursive: true }),
|
||||||
|
mkdir(defaultDatabase, { recursive: true }),
|
||||||
|
mkdir(defaultVault, { recursive: true }),
|
||||||
|
mkdir(skipDatabase, { recursive: true }),
|
||||||
|
mkdir(skipVault, { recursive: true }),
|
||||||
|
]);
|
||||||
|
await mkdir(dirname(sourceStableFile), { recursive: true });
|
||||||
|
await writeFile(sourceStableFile, stableV1);
|
||||||
|
|
||||||
|
try {
|
||||||
|
await startMinio(minioEndpoint, minioAccessKey, minioSecretKey, minioBucket);
|
||||||
|
await Promise.all([
|
||||||
|
initSettingsFile(sourceSettings),
|
||||||
|
initSettingsFile(defaultSettings),
|
||||||
|
initSettingsFile(skipSettings),
|
||||||
|
]);
|
||||||
|
await Promise.all([
|
||||||
|
applyRemoteSyncSettings(sourceSettings, commonSettings),
|
||||||
|
applyRemoteSyncSettings(defaultSettings, commonSettings),
|
||||||
|
applyRemoteSyncSettings(skipSettings, commonSettings),
|
||||||
|
]);
|
||||||
|
const runSource = (...args: string[]) =>
|
||||||
|
runCliOrFail(sourceDatabase, "--settings", sourceSettings, "--vault", sourceVault, ...args);
|
||||||
|
const runDefault = (...args: string[]) =>
|
||||||
|
runCliOrFail(defaultDatabase, "--settings", defaultSettings, "--vault", defaultVault, ...args);
|
||||||
|
const tryDefault = (...args: string[]) =>
|
||||||
|
runCli(defaultDatabase, "--settings", defaultSettings, "--vault", defaultVault, ...args);
|
||||||
|
const runSkip = (...args: string[]) =>
|
||||||
|
runCliOrFail(skipDatabase, "--settings", skipSettings, "--vault", skipVault, ...args);
|
||||||
|
const trySkip = (...args: string[]) =>
|
||||||
|
runCli(skipDatabase, "--settings", skipSettings, "--vault", skipVault, ...args);
|
||||||
|
|
||||||
|
const beforeInitial = new Set(await listJournalKeys(client, minioBucket));
|
||||||
|
await runSource("rebuild", "--force", "--write-settings");
|
||||||
|
const firstJournalKey = await addedJournalKey(client, minioBucket, beforeInitial);
|
||||||
|
await runDefault("fetch", "--force", "--write-settings");
|
||||||
|
await runSkip("fetch", "--force", "--write-settings");
|
||||||
|
const firstInfo = JSON.parse(await runSource("info", stablePath)) as JsonObject;
|
||||||
|
assert.equal(typeof firstInfo["revision"], "string");
|
||||||
|
const firstRevision = firstInfo["revision"] as string;
|
||||||
|
for (const receiver of [
|
||||||
|
{ name: "default receiver", database: defaultDatabase, run: runDefault },
|
||||||
|
{ name: "skip receiver", database: skipDatabase, run: runSkip },
|
||||||
|
]) {
|
||||||
|
assert.equal(
|
||||||
|
sanitiseCatStdout(await receiver.run("cat", stablePath)),
|
||||||
|
stableV1,
|
||||||
|
`${receiver.name} should fetch the initial healthy Journal content`
|
||||||
|
);
|
||||||
|
const checkpoint = await readReceivedJournalKeys(receiver.database);
|
||||||
|
assert.ok(
|
||||||
|
checkpoint.has(firstJournalKey),
|
||||||
|
`${receiver.name} should checkpoint the initial healthy Journal`
|
||||||
|
);
|
||||||
|
const info = JSON.parse(await receiver.run("info", stablePath)) as JsonObject;
|
||||||
|
assert.equal(info["revision"], firstRevision, `${receiver.name} should retain the first source revision`);
|
||||||
|
assert.equal(
|
||||||
|
sanitiseCatStdout(await receiver.run("cat-rev", stablePath, firstRevision)),
|
||||||
|
stableV1,
|
||||||
|
`${receiver.name} should retain the first revision contents`
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
await mkdir(dirname(sourceMiddleFile), { recursive: true });
|
||||||
|
await writeFile(sourceMiddleFile, middleContent);
|
||||||
|
await runSource("push", sourceMiddleFile, middlePath);
|
||||||
|
const beforeMiddle = new Set(await listJournalKeys(client, minioBucket));
|
||||||
|
await runSource("sync");
|
||||||
|
const brokenJournalKey = await addedJournalKey(client, minioBucket, beforeMiddle);
|
||||||
|
|
||||||
|
await mkdir(dirname(sourceSecondBrokenFile), { recursive: true });
|
||||||
|
await writeFile(sourceSecondBrokenFile, secondBrokenContent);
|
||||||
|
await runSource("push", sourceSecondBrokenFile, secondBrokenPath);
|
||||||
|
const beforeSecondBroken = new Set(await listJournalKeys(client, minioBucket));
|
||||||
|
await runSource("sync");
|
||||||
|
const secondBrokenJournalKey = await addedJournalKey(client, minioBucket, beforeSecondBroken);
|
||||||
|
|
||||||
|
await writeFile(sourceStableFile, stableV2);
|
||||||
|
await writeFile(sourceLaterFile, laterContent);
|
||||||
|
await runSource("push", sourceStableFile, stablePath);
|
||||||
|
await runSource("push", sourceLaterFile, laterPath);
|
||||||
|
const beforeLater = new Set(await listJournalKeys(client, minioBucket));
|
||||||
|
await runSource("sync");
|
||||||
|
const laterJournalKey = await addedJournalKey(client, minioBucket, beforeLater);
|
||||||
|
assert.ok(firstJournalKey.localeCompare(brokenJournalKey, undefined, { numeric: true }) < 0);
|
||||||
|
assert.ok(brokenJournalKey.localeCompare(secondBrokenJournalKey, undefined, { numeric: true }) < 0);
|
||||||
|
assert.ok(secondBrokenJournalKey.localeCompare(laterJournalKey, undefined, { numeric: true }) < 0);
|
||||||
|
|
||||||
|
const finalInfo = JSON.parse(await runSource("info", stablePath)) as JsonObject;
|
||||||
|
assert.equal(typeof finalInfo["revision"], "string");
|
||||||
|
const finalRevision = finalInfo["revision"] as string;
|
||||||
|
assert.notEqual(finalRevision, firstRevision);
|
||||||
|
|
||||||
|
await withCorruptedJournals(client, minioBucket, [brokenJournalKey, secondBrokenJournalKey], async () => {
|
||||||
|
const failedSync = await tryDefault("--debug", "sync");
|
||||||
|
assert.notEqual(failedSync.code, 0, "Default receive must report failure for a corrupt Journal");
|
||||||
|
assert.match(failedSync.combined, /receive journal error|journal.*failed|replication.*failed/i);
|
||||||
|
|
||||||
|
const checkpoint = await readReceivedJournalKeys(defaultDatabase);
|
||||||
|
assert.ok(checkpoint.has(firstJournalKey), "Completed earlier Journal must remain checkpointed");
|
||||||
|
assert.equal(checkpoint.has(brokenJournalKey), false, "Failed Journal must not be checkpointed");
|
||||||
|
assert.equal(
|
||||||
|
checkpoint.has(secondBrokenJournalKey),
|
||||||
|
false,
|
||||||
|
"A later failed Journal must not be checkpointed"
|
||||||
|
);
|
||||||
|
assert.equal(checkpoint.has(laterJournalKey), false, "Failure must prevent checkpointing a later Journal");
|
||||||
|
assert.equal(
|
||||||
|
sanitiseCatStdout(await runDefault("cat", stablePath)),
|
||||||
|
stableV1,
|
||||||
|
"The earlier Journal content must remain available after the later failure"
|
||||||
|
);
|
||||||
|
assert.notEqual(
|
||||||
|
(await tryDefault("cat", middlePath)).code,
|
||||||
|
0,
|
||||||
|
"The broken Journal content must not be applied"
|
||||||
|
);
|
||||||
|
assert.notEqual(
|
||||||
|
(await tryDefault("cat", secondBrokenPath)).code,
|
||||||
|
0,
|
||||||
|
"The next broken Journal content must not be applied"
|
||||||
|
);
|
||||||
|
assert.notEqual(
|
||||||
|
(await tryDefault("cat", laterPath)).code,
|
||||||
|
0,
|
||||||
|
"A later Journal must not pass the failed Journal"
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
await runDefault("sync");
|
||||||
|
assert.equal(sanitiseCatStdout(await runDefault("cat", middlePath)), middleContent);
|
||||||
|
assert.equal(sanitiseCatStdout(await runDefault("cat", secondBrokenPath)), secondBrokenContent);
|
||||||
|
assert.equal(sanitiseCatStdout(await runDefault("cat", stablePath)), stableV2);
|
||||||
|
assert.equal(sanitiseCatStdout(await runDefault("cat", laterPath)), laterContent);
|
||||||
|
const retriedCheckpoint = await readReceivedJournalKeys(defaultDatabase);
|
||||||
|
for (const key of [firstJournalKey, brokenJournalKey, secondBrokenJournalKey, laterJournalKey]) {
|
||||||
|
assert.ok(retriedCheckpoint.has(key), `Retry should checkpoint Journal ${key}`);
|
||||||
|
}
|
||||||
|
const retriedInfo = JSON.parse(await runDefault("info", stablePath)) as JsonObject;
|
||||||
|
assert.equal(retriedInfo["revision"], finalRevision, "Retry should preserve the source stable-file revision");
|
||||||
|
assert.equal(
|
||||||
|
sanitiseCatStdout(await runDefault("cat-rev", stablePath, firstRevision)),
|
||||||
|
stableV1,
|
||||||
|
"Retry should retain the earlier stable-file revision"
|
||||||
|
);
|
||||||
|
|
||||||
|
await withCorruptedJournals(client, minioBucket, [brokenJournalKey, secondBrokenJournalKey], async () => {
|
||||||
|
const firstSkip = await trySkip("journal-skip", brokenJournalKey);
|
||||||
|
assert.notEqual(
|
||||||
|
firstSkip.code,
|
||||||
|
0,
|
||||||
|
`Skipping the first Journal should stop at the next broken Journal: ${firstSkip.combined}`
|
||||||
|
);
|
||||||
|
assert.ok(firstSkip.combined.includes(brokenJournalKey), "The first skip must identify its Journal key");
|
||||||
|
assert.ok(
|
||||||
|
firstSkip.combined.includes(secondBrokenJournalKey),
|
||||||
|
"Resumed replication must stop and identify the next broken Journal"
|
||||||
|
);
|
||||||
|
assert.match(
|
||||||
|
firstSkip.combined,
|
||||||
|
/decrypt|inflate|decompress|decode|parse|persist|apply/i,
|
||||||
|
"The first skip attempt must report the failed Journal processing stage"
|
||||||
|
);
|
||||||
|
|
||||||
|
const firstSkipCheckpoint = await readReceivedJournalKeys(skipDatabase);
|
||||||
|
assert.ok(firstSkipCheckpoint.has(firstJournalKey));
|
||||||
|
assert.ok(firstSkipCheckpoint.has(brokenJournalKey), "The requested Journal should be skipped");
|
||||||
|
assert.equal(
|
||||||
|
firstSkipCheckpoint.has(secondBrokenJournalKey),
|
||||||
|
false,
|
||||||
|
"The first skip must not acknowledge the next failed Journal"
|
||||||
|
);
|
||||||
|
assert.equal(firstSkipCheckpoint.has(laterJournalKey), false);
|
||||||
|
assert.equal(sanitiseCatStdout(await runSkip("cat", stablePath)), stableV1);
|
||||||
|
assert.notEqual((await trySkip("cat", middlePath)).code, 0);
|
||||||
|
assert.notEqual((await trySkip("cat", secondBrokenPath)).code, 0);
|
||||||
|
assert.notEqual((await trySkip("cat", laterPath)).code, 0);
|
||||||
|
|
||||||
|
const secondSkip = await trySkip("journal-skip", secondBrokenJournalKey);
|
||||||
|
assert.equal(secondSkip.code, 0, `Explicit second skip should resume replication: ${secondSkip.combined}`);
|
||||||
|
assert.ok(secondSkip.combined.includes(secondBrokenJournalKey), "The second skip must identify its key");
|
||||||
|
assert.match(
|
||||||
|
secondSkip.combined,
|
||||||
|
/decrypt|inflate|decompress|decode|parse|persist|apply/i,
|
||||||
|
"The second skip attempt must report the failed Journal processing stage"
|
||||||
|
);
|
||||||
|
const checkpoint = await readReceivedJournalKeys(skipDatabase);
|
||||||
|
for (const key of [firstJournalKey, brokenJournalKey, secondBrokenJournalKey, laterJournalKey]) {
|
||||||
|
assert.ok(checkpoint.has(key), `Explicit skips should advance past Journal ${key}`);
|
||||||
|
}
|
||||||
|
assert.equal(sanitiseCatStdout(await runSkip("cat", stablePath)), stableV2);
|
||||||
|
assert.equal(sanitiseCatStdout(await runSkip("cat", laterPath)), laterContent);
|
||||||
|
assert.notEqual((await trySkip("cat", middlePath)).code, 0);
|
||||||
|
assert.notEqual((await trySkip("cat", secondBrokenPath)).code, 0);
|
||||||
|
const skippedInfo = JSON.parse(await runSkip("info", stablePath)) as JsonObject;
|
||||||
|
assert.equal(skippedInfo["revision"], finalRevision, "Later valid data should retain its source revision");
|
||||||
|
assert.equal(
|
||||||
|
sanitiseCatStdout(await runSkip("cat-rev", stablePath, firstRevision)),
|
||||||
|
stableV1,
|
||||||
|
"Later valid data should retain earlier revision history"
|
||||||
|
);
|
||||||
|
});
|
||||||
|
} finally {
|
||||||
|
client.destroy();
|
||||||
|
if (!keepDocker) {
|
||||||
|
await stopMinio().catch(() => {});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
test("journal receive failure, retry, and targeted skips through MINIO", async (t) => {
|
||||||
|
await t.test("without encryption", async () => runScenario(false));
|
||||||
|
await t.test("with encryption", async () => runScenario(true));
|
||||||
|
});
|
||||||
@@ -2,6 +2,7 @@ import { randomBytes } from "node:crypto";
|
|||||||
import { createServer } from "node:net";
|
import { createServer } from "node:net";
|
||||||
import { readFile, readdir } from "node:fs/promises";
|
import { readFile, readdir } from "node:fs/promises";
|
||||||
import { join } from "node:path";
|
import { join } from "node:path";
|
||||||
|
import { PutObjectCommand } from "@aws-sdk/client-s3";
|
||||||
import { deriveIdKey } from "@vrtmrz/livesync-commonlib/settings";
|
import { deriveIdKey } from "@vrtmrz/livesync-commonlib/settings";
|
||||||
import { evalObsidianJson } from "../runner/cli.ts";
|
import { evalObsidianJson } from "../runner/cli.ts";
|
||||||
import { discoverObsidianCli, requireObsidianBinary } from "../runner/environment.ts";
|
import { discoverObsidianCli, requireObsidianBinary } from "../runner/environment.ts";
|
||||||
@@ -17,6 +18,8 @@ import {
|
|||||||
type LocalDatabaseEntry,
|
type LocalDatabaseEntry,
|
||||||
} from "../runner/liveSyncWorkflow.ts";
|
} from "../runner/liveSyncWorkflow.ts";
|
||||||
import {
|
import {
|
||||||
|
createObjectStorageClient,
|
||||||
|
readObjectStorageObject,
|
||||||
deleteObjectStoragePrefix,
|
deleteObjectStoragePrefix,
|
||||||
ensureObjectStorageBucket,
|
ensureObjectStorageBucket,
|
||||||
listObjectStorageObjects,
|
listObjectStorageObjects,
|
||||||
@@ -27,7 +30,7 @@ import {
|
|||||||
} from "../runner/objectStorage.ts";
|
} from "../runner/objectStorage.ts";
|
||||||
import { startObsidianLiveSyncSession, type ObsidianLiveSyncSession } from "../runner/session.ts";
|
import { startObsidianLiveSyncSession, type ObsidianLiveSyncSession } from "../runner/session.ts";
|
||||||
import { createTemporaryVault, type TemporaryVault } from "../runner/vault.ts";
|
import { createTemporaryVault, type TemporaryVault } from "../runner/vault.ts";
|
||||||
import { withObsidianPage } from "../runner/ui.ts";
|
import { openLiveSyncSettings, waitForVisibleObsidianDialogue, withObsidianPage } from "../runner/ui.ts";
|
||||||
|
|
||||||
process.env.E2E_OBSIDIAN_CLI_TIMEOUT_MS ??= "60000";
|
process.env.E2E_OBSIDIAN_CLI_TIMEOUT_MS ??= "60000";
|
||||||
|
|
||||||
@@ -557,6 +560,141 @@ async function waitForJournalObjects(
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async function verifySingleJournalRecovery(
|
||||||
|
cliBinary: string,
|
||||||
|
session: ObsidianLiveSyncSession,
|
||||||
|
vaultPath: string,
|
||||||
|
storage: ObjectStorageConfig,
|
||||||
|
prefix: string,
|
||||||
|
validJournalKey: string
|
||||||
|
): Promise<void> {
|
||||||
|
const objects = await listObjectStorageObjects(storage, prefix);
|
||||||
|
const timestamp =
|
||||||
|
Math.max(Date.now(), ...objects.map(({ Key }) => Number(Key?.slice(prefix.length).split("-")[0]) || 0)) + 1;
|
||||||
|
const firstKey = `${timestamp}-docs.jsonl.gz`;
|
||||||
|
const secondKey = `${timestamp + 1}-docs.jsonl.gz`;
|
||||||
|
const laterKey = `${timestamp + 2}-docs.jsonl.gz`;
|
||||||
|
const validBytes = await readObjectStorageObject(storage, validJournalKey);
|
||||||
|
const client = createObjectStorageClient(storage);
|
||||||
|
try {
|
||||||
|
for (const [key, bytes] of [
|
||||||
|
[firstKey, new Uint8Array([1])],
|
||||||
|
[secondKey, new Uint8Array([1])],
|
||||||
|
[laterKey, validBytes],
|
||||||
|
] as const) {
|
||||||
|
await client.send(new PutObjectCommand({ Bucket: storage.bucket, Key: prefix + key, Body: bytes }));
|
||||||
|
}
|
||||||
|
} finally {
|
||||||
|
client.destroy();
|
||||||
|
}
|
||||||
|
|
||||||
|
type RecoveryState = { key: string | null; stage: string | null; received: string[] };
|
||||||
|
const readState = async (): Promise<RecoveryState> =>
|
||||||
|
await evalObsidianJson<RecoveryState>(
|
||||||
|
cliBinary,
|
||||||
|
[
|
||||||
|
"(async()=>{",
|
||||||
|
"const core=app.plugins.plugins['obsidian-livesync'].core;",
|
||||||
|
"const replicator=core.services.replicator.getActiveReplicator();",
|
||||||
|
"const stopped=replicator.getStoppedJournal();",
|
||||||
|
"const checkpoint=await replicator.client.getCheckpointInfo();",
|
||||||
|
"return JSON.stringify({key:stopped?.key??null,stage:stopped?.stage??null,received:[...checkpoint.receivedFiles]});",
|
||||||
|
"})()",
|
||||||
|
].join(""),
|
||||||
|
session.cliEnv,
|
||||||
|
timeoutMs
|
||||||
|
);
|
||||||
|
const waitForState = async (expectedKey: string | null, expectedReceived: string[]): Promise<void> => {
|
||||||
|
const deadline = Date.now() + timeoutMs;
|
||||||
|
let state: RecoveryState | undefined;
|
||||||
|
while (Date.now() < deadline) {
|
||||||
|
state = await readState();
|
||||||
|
if (state.key === expectedKey && expectedReceived.every((key) => state?.received.includes(key))) return;
|
||||||
|
await new Promise((resolve) => setTimeout(resolve, 100));
|
||||||
|
}
|
||||||
|
throw new Error(`Journal recovery did not reach the expected state: ${JSON.stringify(state)}`);
|
||||||
|
};
|
||||||
|
const result = await evalObsidianJson<boolean>(
|
||||||
|
cliBinary,
|
||||||
|
[
|
||||||
|
"(async()=>{",
|
||||||
|
"const core=app.plugins.plugins['obsidian-livesync'].core;",
|
||||||
|
"const replicator=core.services.replicator.getActiveReplicator();",
|
||||||
|
"return JSON.stringify(await replicator.replicateAllFromServer(core.settings,true));",
|
||||||
|
"})()",
|
||||||
|
].join(""),
|
||||||
|
session.cliEnv,
|
||||||
|
timeoutMs
|
||||||
|
);
|
||||||
|
assertEqual(result, false, "Receipt must stop at the first malformed Journal.");
|
||||||
|
const initial = await readState();
|
||||||
|
assertEqual(initial.key, firstKey, "The first failed Journal must be offered for recovery.");
|
||||||
|
for (const key of [firstKey, secondKey, laterKey]) {
|
||||||
|
assertEqual(initial.received.includes(key), false, "Failed and later Journals must remain unacknowledged.");
|
||||||
|
}
|
||||||
|
|
||||||
|
await withObsidianPage(session.remoteDebuggingPort, async (page) => {
|
||||||
|
const navigator = await openLiveSyncSettings(page, uiTimeoutMs);
|
||||||
|
try {
|
||||||
|
const maintenance = await navigator.openPage("Maintenance");
|
||||||
|
assertEqual(
|
||||||
|
await maintenance.getByText("Ignore broken journals (risk of data loss)", { exact: true }).count(),
|
||||||
|
0,
|
||||||
|
"The persistent skip toggle must not be offered."
|
||||||
|
);
|
||||||
|
const setting = maintenance.locator(".setting-item").filter({
|
||||||
|
has: navigator.page.getByText("Skip stopped Journal and resume", { exact: true }),
|
||||||
|
});
|
||||||
|
await setting.waitFor({ state: "visible", timeout: uiTimeoutMs });
|
||||||
|
const choose = async (key: string, accept: boolean): Promise<void> => {
|
||||||
|
await setting
|
||||||
|
.getByRole("button", { name: "Skip and resume", exact: true })
|
||||||
|
.click({ timeout: uiTimeoutMs });
|
||||||
|
const dialogue = await waitForVisibleObsidianDialogue(page, key, uiTimeoutMs);
|
||||||
|
const text = await dialogue.innerText();
|
||||||
|
if (!text.includes(key) || !/missing/i.test(text)) {
|
||||||
|
throw new Error(
|
||||||
|
"The skip confirmation must identify the exact Journal and its missing-content risk."
|
||||||
|
);
|
||||||
|
}
|
||||||
|
await dialogue
|
||||||
|
.getByRole("button", { name: accept ? "Yes" : "No", exact: true })
|
||||||
|
.click({ timeout: uiTimeoutMs });
|
||||||
|
};
|
||||||
|
await choose(firstKey, false);
|
||||||
|
assertEqual(
|
||||||
|
JSON.stringify(await readState()),
|
||||||
|
JSON.stringify(initial),
|
||||||
|
"Cancelling must preserve the failure and checkpoint."
|
||||||
|
);
|
||||||
|
await choose(firstKey, true);
|
||||||
|
await waitForState(secondKey, [firstKey]);
|
||||||
|
const afterFirst = await readState();
|
||||||
|
assertEqual(
|
||||||
|
afterFirst.received.includes(secondKey),
|
||||||
|
false,
|
||||||
|
"One action must not skip the next failed Journal."
|
||||||
|
);
|
||||||
|
assertEqual(
|
||||||
|
afterFirst.received.includes(laterKey),
|
||||||
|
false,
|
||||||
|
"Receipt must stop before the later valid Journal."
|
||||||
|
);
|
||||||
|
await choose(secondKey, true);
|
||||||
|
await waitForState(null, [firstKey, secondKey, laterKey]);
|
||||||
|
} finally {
|
||||||
|
await navigator.close();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
const saved = JSON.parse(
|
||||||
|
await readFile(join(vaultPath, ".obsidian/plugins/obsidian-livesync/data.json"), "utf8")
|
||||||
|
) as Record<string, unknown>;
|
||||||
|
assertEqual("ignoreBrokenJournals" in saved, false, "Recovery must not persist a skip toggle.");
|
||||||
|
console.log(
|
||||||
|
"Journal recovery UI: cancellation preserved the checkpoint, each action skipped one named Journal, and the next failure stopped receipt."
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
async function postponeConflictIfOpen(session: ObsidianLiveSyncSession): Promise<void> {
|
async function postponeConflictIfOpen(session: ObsidianLiveSyncSession): Promise<void> {
|
||||||
await withObsidianPage(session.remoteDebuggingPort, async (page) => {
|
await withObsidianPage(session.remoteDebuggingPort, async (page) => {
|
||||||
const dialogue = page.locator(".modal-container:visible").filter({ hasText: "Conflicting changes" });
|
const dialogue = page.locator(".modal-container:visible").filter({ hasText: "Conflicting changes" });
|
||||||
@@ -794,6 +932,24 @@ async function runCase(
|
|||||||
);
|
);
|
||||||
await waitForSinglePhysicalUnicodeFile(receiverVault.path, controlPathNfd, controlContent);
|
await waitForSinglePhysicalUnicodeFile(receiverVault.path, controlPathNfd, controlContent);
|
||||||
|
|
||||||
|
const validJournalKey = rebuiltKeys.find((key) => key.endsWith("-docs.jsonl.gz"));
|
||||||
|
if (!validJournalKey) throw new Error("A valid Journal fixture is required for recovery testing.");
|
||||||
|
await verifySingleJournalRecovery(
|
||||||
|
cliBinary,
|
||||||
|
receiverSession,
|
||||||
|
receiverVault.path,
|
||||||
|
objectStorage,
|
||||||
|
bucketPrefix,
|
||||||
|
validJournalKey
|
||||||
|
);
|
||||||
|
await postponeConflictIfOpen(receiverSession);
|
||||||
|
assertSameConflictTree(
|
||||||
|
await readConflictTree(cliBinary, receiverSession.cliEnv, conflictPathNfc, initialEntry.rev, false),
|
||||||
|
sourceTree,
|
||||||
|
"Receiver after explicit Journal skips"
|
||||||
|
);
|
||||||
|
await waitForSinglePhysicalUnicodeFile(receiverVault.path, controlPathNfd, controlContent);
|
||||||
|
|
||||||
const initialJournalObjects = initialKeys.filter((key) => key.endsWith("-docs.jsonl.gz")).length;
|
const initialJournalObjects = initialKeys.filter((key) => key.endsWith("-docs.jsonl.gz")).length;
|
||||||
const rebuiltJournalObjects = rebuiltKeys.filter((key) => key.endsWith("-docs.jsonl.gz")).length;
|
const rebuiltJournalObjects = rebuiltKeys.filter((key) => key.endsWith("-docs.jsonl.gz")).length;
|
||||||
console.log(
|
console.log(
|
||||||
|
|||||||
@@ -18,6 +18,12 @@ Earlier releases remain available in the 1.0 release history, the 1.0 preview hi
|
|||||||
|
|
||||||
### Fixed
|
### Fixed
|
||||||
|
|
||||||
|
#### Object Storage
|
||||||
|
|
||||||
|
- Failed Journal writes now stop replication instead of silently passing over missing records. After correcting the cause, the next replication with the same local database retries the failed Journal.
|
||||||
|
- If a Journal cannot be recovered and missing content or history is acceptable, **Maintenance → Syncing → Skip stopped Journal and resume** asks for confirmation of its exact key and failure stage, then skips only that Journal and resumes receipt. The next Journal failure still stops replication. The action does not add a persistent ignore setting.
|
||||||
|
- **Reset journal received history** replays Journals which remain available remotely, except those sent by this device. It cannot recreate missing remote Journals; earlier failures already silently acknowledged also require this explicit replay.
|
||||||
|
|
||||||
#### Conflict resolution
|
#### Conflict resolution
|
||||||
|
|
||||||
- Disabling or reloading the plug-in no longer causes a secondary error when pending work writes to the log. Logs arriving after the log display closes go to Obsidian's console.
|
- Disabling or reloading the plug-in no longer causes a secondary error when pending work writes to the log. Logs arriving after the log display closes go to Obsidian's console.
|
||||||
@@ -36,6 +42,8 @@ Earlier releases remain available in the 1.0 release history, the 1.0 preview hi
|
|||||||
|
|
||||||
#### Fixed
|
#### Fixed
|
||||||
|
|
||||||
|
- Commonlib warnings now appear on stderr without `--debug`, so deliberately skipped Journals and other important notices remain visible during ordinary CLI use.
|
||||||
|
|
||||||
- NFC file lookups and directory scans report permission and I/O failures instead of treating unreadable files as absent or returning a successful partial scan. A scan retains the comparison settings with which it began.
|
- NFC file lookups and directory scans report permission and I/O failures instead of treating unreadable files as absent or returning a successful partial scan. A scan retains the comparison settings with which it began.
|
||||||
|
|
||||||
## 1.0.44
|
## 1.0.44
|
||||||
|
|||||||
Reference in new issue
Block a user