Prepare 1.0.41 notes, dependencies, and recovery coverage

Align release metadata and audited development dependencies, and consolidate
release notes with contributor acknowledgements.

Cover upgrades, start-up integrity, text reflection, and interrupted saves
in real Obsidian. Document the accepted provenance persistence interval:
identical replayed content converges, safe merge retains the later remote
content and ancestry, and overlapping edits remain available for review.
This commit is contained in:
vorotamoroz committed 2026-10-26 13:35:14 +00:00
1 parent 935511c69b
commit b449dfd2bb
16 files changed
+1583 -292

No files matched your search

+13 -1
View File
@@ -98,6 +98,10 @@ The same scenario checks the five actions in Reporting Tools, the separate Check
`folder-batch` needs no remote service. It creates 24 notes and imports three notes with colons in their names into nested folders, reflects the database content into the existing files, then renames and deletes the parent through the Obsidian Vault API. It checks exact paths, descendant events, content, Chunks, deletion markers, provenance, and the absence of unexpected files. A note outside the parent must remain writable. The colon fixtures use the adapter to represent externally created files because Obsidian's Vault creation API rejects those names. The scenario also seeds a database-only colon-named note and verifies that Obsidian's refusal to create it preserves its Metadata without writing a differently named Vault file. It does not establish successful restoration of that absent note or behaviour on other operating systems.
`startup-integrity` uses the registered Hatch integrity-check event with real Obsidian storage and a local database. It verifies that healthy files are inspected without reading their contents, that a controlled stale stat followed by matching real bytes does not produce a repair dialogue, and that a changed final stat leaves the check incomplete until a settled retry. The stat variations are injected; this scenario exercises the checker used during start-up rather than restarting the process. Natural configured start-up remains covered by `startup-scan`.
`third-party-chunks` reflects synthetic externally generated text through the installed Commonlib and the real Obsidian file handler. It supplies a split surrogate pair at the file-access read boundary, with size checking enabled, then verifies exact Vault and on-disk text, unchanged Metadata, and matching persisted provenance. Complete-pair text and a binary attachment provide controls. The underlying Metadata and Chunks are real local database entries; this does not establish that CouchDB or a particular third-party application generates this chunk layout. No remote service is required.
`stale-file-restart` needs no remote service. It advances the local database while old Vault bytes remain, persists pending storage events, and restarts the same isolated Vault and profile. It checks that an unchanged file whose recorded revision has been pruned receives the newer database content using persisted content evidence without creating a revision, that unknown-origin content is preserved on a fresh independent branch, and that losing provenance and processing the file again does not duplicate or automatically merge that branch. A third file begins with no provenance while its bytes still match the current database revision; start-up must record that revision without creating a new one, and a later incoming revision must reflect without a conflict. A fourth file has legacy revision-only provenance; start-up must verify its exact content and add a hash without creating a database revision or changing storage. The database advances and pending snapshot are controlled fixtures; start-up processing, persistence, file reflection, and conflict checking run in real Obsidian. The scenario does not simulate a mobile operating system suspending the application.
The principal entry points are:
@@ -265,7 +269,13 @@ The workflow also opens the Customisation Sync dialogue. `--case=visibility` che
`--case=automatic` separately checks that choosing **Automatic** while Hidden File Sync is disabled displays a Notice without changing modes, files, Metadata revisions, or processed markers. The fixture starts with a saved Automatic selection and requires it to survive start-up and both rejected actions. It also checks disabling the feature after opening **Initial Action**. After enablement, selecting a whole plug-in must initialise both **MAIN** and **DATA**, and transfer their files to the other Vault. Separate temporary Vaults exercise **Overwrite Remote**, **Overwrite Local**, and **Use newer**. The latter two start with different local and database contents; deterministic per-file modification times require **Use newer** to keep the newer database MAIN content and the newer local DATA content.
`test:e2e:obsidian:livesync-batch` uses two real Obsidian Vaults and a disposable local CouchDB to compare batching disabled, the older `batchSave` choice alone, and the explicit LiveSync batching choice. It covers repeated edits in both directions, template-like writes, deletion and recreation, incoming changes, maximum delay, concurrent paths, manual flush, and restart recovery before and after a database write. Restarts use the saved profile with natural plug-in start-up. `ISSUE1207_CASE_FILTER` selects case names; `ISSUE1207_RESTART_BUCKET=same|exact|different` exercises interrupted processing at the selected timestamp condition. `ISSUE1207_RESTART_SAVE_AFTER_EDIT=true` separately checks a settings save after the pending edit. Use `npm run test:e2e:obsidian:focused -- livesync-batch` to build and run the default matrix. This scenario requires a localhost CouchDB and is outside the complete local suite.
`test:e2e:obsidian:livesync-batch` uses two real Obsidian Vaults and a disposable local CouchDB to compare batching disabled, the older `batchSave` choice alone, and the explicit LiveSync batching choice. It covers repeated edits in both directions, template-like writes, deletion and recreation, incoming changes, maximum delay, concurrent paths, manual flush, and restart recovery before and after a local database write. Restarts use the saved profile with natural plug-in start-up.
`ISSUE1207_CASE_FILTER` selects case names; `ISSUE1207_RESTART_BUCKET=same|exact|different` exercises interrupted processing at the selected timestamp condition. `ISSUE1207_RESTART_SAVE_AFTER_EDIT=true` separately checks a settings save after the pending edit.
`ISSUE1207_RESTART_BEFORE_PROVENANCE=true` deliberately stops after the local database write and before device-local provenance is recorded. The case without a remote edit demonstrates replay creating an identical sibling and tombstoning one of the two branches; it must converge to one live leaf with exact saved content, and a second restart must add no revision.
`ISSUE1207_RESTART_REMOTE_EDIT=mergeable|conflicting` requires `ISSUE1207_RESTART_BEFORE_PROVENANCE=true`. These cases persist the enabled test settings before editing so continuous replication remains active, receive a remote descendant before stopping, and restart the same profile. Both remote cases enable sensible merge. The pause is deliberately injected to observe the ordering; it is not a measure of field frequency. The `mergeable` case requires the final bytes to equal the incoming descendant exactly, with that revision retained in ancestry and further restart stable. The `conflicting` case uses a whole-line replacement and requires both bodies to remain for the manual conflict dialogue. The cases are named `enabled-before-provenance-mergeable-restart` and `enabled-before-provenance-conflicting-restart`. Use `npm run test:e2e:obsidian:focused -- livesync-batch` to build and run the default matrix. This scenario requires a localhost CouchDB and is outside the complete local suite.
`test:e2e:obsidian:optional-file-save-completion` exercises Hidden File Sync and both Customisation Sync formats in one isolated real Obsidian Vault. It holds a local database write to verify that its event stays in the snapshot and that flush waits for the write, then checks unchanged notifications, subsequent edits, and repeated deletions. Injected save failures remain pending and must replay successfully before the next start reaches readiness. This local scenario disables replication and requires no remote service. It is outside `test:e2e:obsidian:local-suite`.
@@ -281,6 +291,8 @@ The workflow also opens the Customisation Sync dialogue. `--case=visibility` che
`test:e2e:obsidian:setting-markdown-roundtrip` generates Markdown with and without credentials through the real export command, copies each original file into a separate fresh Vault, and imports it through **Parse setting file** and the visible **Apply settings** dialogue. Each receiver starts with default plug-in settings and no connection profiles, with its own configuration encryption passphrase provided independently of the source through localStorage. The scenario verifies ordinary settings, conditional credentials and complete profiles, encrypted local persistence, and restoration after restarting the same Vault and profile without reseeding settings or localStorage. Imports without credentials may create new profiles through the existing legacy migration; those profiles must contain none of the omitted source credentials. This scenario requires no remote service and is included in the focused wrapper and local suite.
`test:e2e:obsidian:upgrade-livesync-batching` verifies the introduction of the separate LiveSync batching choice using the pinned, published 1.0.40 artefacts. It first saves and restarts the old release with legacy batching enabled and LiveSync mode selected. Replacing only the plug-in artefacts must retain the Vault, local database, device identity, remote profile, and existing Metadata revision, while leaving the new LiveSync batching choice disabled. The upgraded Vault and an independent current-version Vault exchange notes through CouchDB. Natural restarts retain both the disabled LiveSync choice and the original batching choice when switching to periodic mode. The source release is downloaded into the verified release cache; `E2E_LIVESYNC_TARGET_ARTIFACT_ROOT` selects an exact target build. This tests the update behaviour, while installation through BRAT remains a separate acceptance step.
`test:e2e:obsidian:upgrade-from-stable` is the release-acceptance upgrade workflow. It installs the exact published 0.25.83 artefacts into an isolated Vault, verifies their pinned SHA-256 values, and then replaces only the plug-in artefacts with the current target while retaining the same Vault and isolated Obsidian profile. The first run downloads the old release into the ignored `_testdata/releases` cache; every later run verifies the cached bytes before use.
The workflow first exercises a non-empty legacy settings document which has no `isConfigured` or file-name case value. It verifies that 0.25.83 treats a default-equivalent document as unconfigured. That release can persist the inferred boolean during a later, unrelated settings-save event, so the runner accepts either an absent value or the inferred `false` on disk, then restores the same minimal pre-flag document deliberately before installing 1.0. The target independently proves its direct migration: the Vault remains unconfigured instead of receiving new-Vault recommendations, case-insensitive handling becomes explicit, no compatibility pause or acknowledgement marker is created while onboarding remains pending, and a second 1.0 start is idempotent. The absent marker is deliberately deferred while onboarding is pending; a later configured start records the current version if no other review is required. This fixture rewrite is limited to the missing-flag boundary; the configured transport upgrades use only state created and saved by 0.25.83 itself.
@@ -180,12 +180,17 @@ async function waitFor(
throw new Error(`State did not settle: ${JSON.stringify(state)}`);
}
async function settled(pair: Pair, path: string, body: string): Promise<[Snapshot, Snapshot]> {
async function settled(
pair: Pair,
path: string,
body: string,
allowDeletedBranches = false
): Promise<[Snapshot, Snapshot]> {
const ready = (state: Snapshot) =>
state.content === body &&
liveLeaves(state).length === 1 &&
liveLeaves(state)[0].body === body &&
state.leaves.length === 1 &&
(allowDeletedBranches || state.leaves.length === 1) &&
!state.waiting.some((item) => item.path === path);
const deadline = Date.now() + 20000;
let states: [Snapshot, Snapshot];
@@ -234,6 +239,15 @@ async function main(): Promise<void> {
await assertCouchDbReachable(config);
const dbName = makeUniqueDatabaseName(config.dbPrefix, "live-batch");
const couch = { ...config, dbName };
const remoteRestartEdit = process.env.ISSUE1207_RESTART_REMOTE_EDIT;
check(
!remoteRestartEdit || ["mergeable", "conflicting"].includes(remoteRestartEdit),
"Unknown remote edit for the provenance restart case"
);
check(
!remoteRestartEdit || process.env.ISSUE1207_RESTART_BEFORE_PROVENANCE === "true",
"A remote restart edit requires the provenance gate"
);
const overrides = {
doctorProcessedVersion: "1.0.0",
notifyThresholdOfRemoteStorageSize: -1,
@@ -248,7 +262,7 @@ async function main(): Promise<void> {
syncAfterMerge: false,
checkConflictOnlyOnOpen: true,
showMergeDialogOnlyOnActive: true,
disableMarkdownAutoMerge: true,
disableMarkdownAutoMerge: !remoteRestartEdit,
resolveConflictsByNewerFile: false,
writeDocumentsIfConflicted: false,
writeLogToTheFile: false,
@@ -259,8 +273,6 @@ async function main(): Promise<void> {
evidence.commonlibVersion = JSON.parse(
await readFile("node_modules/@vrtmrz/livesync-commonlib/package.json", "utf8")
).version;
evidence.sourceBaseRevision = "73a14d4e4b0b414a1acfe170d01500abb3dca2e9";
evidence.unreleasedImplementation = true;
if (process.env.ISSUE1207_COMMONLIB_ARTEFACT) {
evidence.commonlibArtefactSha256 = createHash("sha256")
.update(new Uint8Array(readFileSync(process.env.ISSUE1207_COMMONLIB_ARTEFACT)))
@@ -370,6 +382,31 @@ async function main(): Promise<void> {
type:item.type,path:item.args?.file?.path,oldPath:item.args?.oldPath}))};`
),
});
if (name === "enabled-settings-save-restart" || name.startsWith("enabled-before-provenance")) {
observations.push({
name,
device,
revisionEvidence: await evaluate(
page,
`
const path=${JSON.stringify(`${prefix}restart.md`)};
const id=await core.services.path.path2id(path);
const leaves=await core.localDatabase.getRaw(id,{open_revs:'all',revs:true});
const revisions=new Set();
for(const leaf of leaves){const h=leaf.ok?._revisions;
for(let i=0;i<(h?.ids.length??0);i++)revisions.add((h.start-i)+'-'+h.ids[i]);}
const result=[];
for(const rev of revisions){
const raw=await core.localDatabase.getRaw(id,{rev,revs:true}).catch(error=>({error:String(error)}));
const doc=await core.localDatabase.getDBEntry(path,{rev,revs:true},false,true,true).catch(()=>false);
result.push({rev,deleted:raw._deleted??false,ctime:raw.ctime,mtime:raw.mtime,
size:raw.size,error:raw.error,body:doc?(Array.isArray(doc.data)?doc.data.join(''):doc.data):null});
}
return {revisions:result,provenance:await core.services.keyValueDB.openSimpleStore('file-reflection-provenance-v1').get(path)};
`
),
});
}
await page.screenshot({ path: join(diagnostics, `${name}-${device}-failure.png`) });
try {
await evaluate(page, "await manager.waitForIdle();");
@@ -663,27 +700,61 @@ async function main(): Promise<void> {
const saveAfterEdit = process.env.ISSUE1207_RESTART_SAVE_AFTER_EDIT === "true";
const restartBucket = process.env.ISSUE1207_RESTART_BUCKET;
const stopBeforeProvenance = process.env.ISSUE1207_RESTART_BEFORE_PROVENANCE === "true";
check(!stopBeforeProvenance || !restartBucket, "The provenance and timestamp gates are separate cases");
check(
!restartBucket || ["same", "exact", "different"].includes(restartBucket),
"Unknown restart timestamp condition"
);
const restartName = restartBucket
? `enabled-inflight-restart-${restartBucket}`
: saveAfterEdit
? "enabled-settings-save-restart"
: "enabled-restart";
const restartName = stopBeforeProvenance
? `enabled-before-provenance${remoteRestartEdit ? `-${remoteRestartEdit}` : ""}-restart`
: restartBucket
? `enabled-inflight-restart-${restartBucket}`
: saveAfterEdit
? "enabled-settings-save-restart"
: "enabled-restart";
await run(restartName, async () => {
const path = `${prefix}restart.md`;
const beforeContent =
remoteRestartEdit === "mergeable"
? "Heading\n\nLocal: original\n\nRemote: original\n\nFooter\n"
: "Before restart\n";
const localContent =
remoteRestartEdit === "mergeable"
? "Heading\n\nLocal: saved\n\nRemote: original\n\nFooter\n"
: "Retained after restart\n";
const remoteContent =
remoteRestartEdit === "mergeable"
? "Heading\n\nLocal: saved\n\nRemote: changed remotely\n\nFooter\n"
: "Remote edit after the durable save\n";
const beforeRestart = await withPair(async (pair) => {
await options(pair.a, "enabled-batch", 5, 60);
await options(pair.b, "normal");
if (!saveAfterEdit)
// Remote cases keep continuous replication running while the local save is held.
if (!saveAfterEdit || remoteRestartEdit)
await evaluate(
pair.a,
"await core.services.setting.saveSettingData(); await core.services.control.applySettings();"
);
await create(pair.b, path, "Before restart\n");
const initial = await settled(pair, path, "Before restart\n");
await create(pair.b, path, beforeContent);
const initial = await settled(pair, path, beforeContent);
if (stopBeforeProvenance) {
await evaluate(
pair.a,
`
const access=core.serviceModules.databaseFileAccess;
const original=access.storeWithBaseRevision.bind(access);
access.storeWithBaseRevision=async(file,...args)=>{
const revision=await original(file,...args);
if(file.path===${JSON.stringify(path)}&&revision){
globalThis.__issue1207Batch.beforeProvenanceRevision=revision;
await new Promise(resolve=>setTimeout(resolve,30000));
}
return revision;
};
`
);
}
if (restartBucket) {
check(initial[0].databaseMtime !== null, "Missing initial database mtime");
const mtime =
@@ -703,11 +774,27 @@ async function main(): Promise<void> {
};
await app.vault.modify(app.vault.getFileByPath(${JSON.stringify(path)}),"Retained after restart\\n",{mtime:${mtime}});`
);
} else await modify(pair.a, path, "Retained after restart\n");
} else await modify(pair.a, path, localContent);
const pending = await waitFor(pair.a, path, (state) =>
state.waiting.some((item) => item.path === path)
);
if (saveAfterEdit) await evaluate(pair.a, "await core.services.setting.saveSettingData();");
if (saveAfterEdit && !remoteRestartEdit) {
await evaluate(pair.a, "await core.services.setting.saveSettingData();");
observations.push({
phase: "settings-save-before-stop",
initial,
pending,
current: await Promise.all([snapshot(pair.a, path), snapshot(pair.b, path)]),
sourceProbe: await evaluate(
pair.a,
`return {
events:globalThis.__issue1207Batch.events,
changes:globalThis.__issue1207Batch.changes,
provenance:await core.services.keyValueDB.openSimpleStore('file-reflection-provenance-v1').get(${JSON.stringify(path)})
};`
),
});
}
if (restartBucket)
await evaluate(
pair.a,
@@ -715,6 +802,82 @@ async function main(): Promise<void> {
while(!globalThis.__issue1207Batch.inflightPaused&&Date.now()<deadline) await new Promise(resolve=>setTimeout(resolve,10));
if(!globalThis.__issue1207Batch.inflightPaused) throw new Error('The in-flight test gate did not start');`
);
let durableBeforeProvenance: Snapshot | undefined;
let remoteAdvanced: Snapshot | undefined;
if (stopBeforeProvenance) {
await evaluate(
pair.a,
`
const deadline=Date.now()+core.services.setting.currentSettings().batchSaveMinimumDelay*1000+5000;
while(!globalThis.__issue1207Batch.beforeProvenanceRevision&&Date.now()<deadline)
await new Promise(resolve=>setTimeout(resolve,10));
if(!globalThis.__issue1207Batch.beforeProvenanceRevision)throw new Error('The durable-write provenance gate did not start');
`
);
durableBeforeProvenance = await snapshot(pair.a, path);
const provenance = await evaluate<{ revision: string }>(
pair.a,
`return await core.services.keyValueDB.openSimpleStore('file-reflection-provenance-v1').get(${JSON.stringify(path)});`
);
check(
durableBeforeProvenance.rev !== initial[0].rev &&
liveLeaves(durableBeforeProvenance)[0]?.body === localContent,
"The gated database write is not durable"
);
check(
provenance?.revision === initial[0].rev,
"The fixture did not stop before provenance was updated"
);
observations.push({
phase: "durable-before-provenance",
state: durableBeforeProvenance,
provenance,
});
if (remoteRestartEdit) {
const receivedOriginal = await waitFor(
pair.b,
path,
(state) => state.rev === durableBeforeProvenance!.rev && state.content === localContent
);
await modify(pair.b, path, remoteContent);
remoteAdvanced = await waitFor(
pair.b,
path,
(state) =>
liveLeaves(state).length === 1 &&
liveLeaves(state)[0].body === remoteContent &&
state.rev === liveLeaves(state)[0].rev &&
state.content === remoteContent
);
check(
liveLeaves(remoteAdvanced)[0].ancestry.includes(durableBeforeProvenance.rev!),
"The remote edit did not descend from the already durable local save"
);
const localDbAdvanced = await waitFor(
pair.a,
path,
(state) =>
state.rev === remoteAdvanced!.rev &&
liveLeaves(state).length === 1 &&
liveLeaves(state)[0].body === remoteContent
);
const stillOldProvenance = await evaluate<{ revision: string }>(
pair.a,
`return await core.services.keyValueDB.openSimpleStore('file-reflection-provenance-v1').get(${JSON.stringify(path)});`
);
check(
localDbAdvanced.content === localContent && stillOldProvenance.revision === initial[0].rev,
"The incoming DB change did not precede Vault reflection and provenance persistence"
);
observations.push({
phase: "remote-edit-local-db-ahead-of-storage",
receivedOriginal,
remoteAdvanced,
localDbAdvanced,
stillOldProvenance,
});
}
}
await evaluate(pair.a, "await manager._takeSnapshot();");
check(pending.rev === initial[0].rev, "Restart test did not retain a pending edit");
if (restartBucket) {
@@ -746,16 +909,18 @@ async function main(): Promise<void> {
sameBucket,
});
}
if (!saveAfterEdit && !restartBucket) {
if (stopBeforeProvenance || (!saveAfterEdit && !restartBucket)) {
const strictState = await snapshot(pair.a, path);
const saved = await evaluate<Array<{ path: string }>>(
pair.a,
`return (await manager.adapter.persistence.loadSnapshot()).map(item=>({path:item.args?.file?.path}));`
);
check(
strictState.rev === initial[0].rev && strictState.waiting.some((item) => item.path === path),
"Restart fixture flushed before shutdown"
);
if (!stopBeforeProvenance)
check(
strictState.rev === initial[0].rev &&
strictState.waiting.some((item) => item.path === path),
"Restart fixture flushed before shutdown"
);
check(
saved.some((item) => item.path === path),
"Restart fixture did not persist the pending event"
@@ -778,12 +943,12 @@ async function main(): Promise<void> {
const settings=core.services.setting.currentSettings();
return {initialized:await core.kvDB.get('initialized'),restorationRequested:manager.snapShotRestored!==null,
settings:{isConfigured:settings.isConfigured,liveSync:settings.liveSync,batchSave:settings.batchSave,batchSaveInLiveSync:settings.batchSaveInLiveSync,
suspendFileWatching:settings.suspendFileWatching},
suspendFileWatching:settings.suspendFileWatching,disableMarkdownAutoMerge:settings.disableMarkdownAutoMerge},
events:events?.map(item=>({type:item.type,path:item.args?.file?.path,
stat:item.args?.file?.stat,skipBatchWait:item.skipBatchWait}))};`
),
});
return { initial, pending };
return { initial, pending, durableBeforeProvenance, remoteAdvanced };
});
await stop(slots[0]);
const persistedSettings = JSON.parse(
@@ -817,11 +982,82 @@ async function main(): Promise<void> {
events:events?.map(item=>({type:item.type,path:item.args?.file?.path,stat:item.args?.file?.stat}))};`
),
});
const result = await settled(pair, path, "Retained after restart\n");
if (remoteRestartEdit === "conflicting") {
const preservesBothBodies = (state: Snapshot) => {
const leaves = liveLeaves(state);
return (
leaves.length === 2 &&
leaves.some((leaf) => leaf.body === localContent) &&
leaves.some(
(leaf) => leaf.body === remoteContent && leaf.rev === beforeRestart.remoteAdvanced!.rev
)
);
};
const result = await Promise.all([
waitFor(pair.a, path, preservesBothBodies),
waitFor(pair.b, path, preservesBothBodies),
]);
check(
result[0].content === localContent && result[1].content === remoteContent,
"An unresolved conflict overwrote one of the preserved Vault contents"
);
const manualDialogue = pair.a.locator(".modal").filter({ hasText: "Conflicting changes" });
await manualDialogue.waitFor({ state: "visible", timeout: 10000 });
await pair.a.screenshot({ path: join(diagnostics, `${restartName}-manual-conflict.png`) });
return { result, manualConflict: true, restoredGuard: result[0].shouldBatchSave };
}
const expectedContent = remoteRestartEdit ? remoteContent : localContent;
const result = await settled(pair, path, expectedContent, saveAfterEdit || stopBeforeProvenance);
check(result[0].shouldBatchSave, "The saved LiveSync batching choice was not restored");
if (beforeRestart.remoteAdvanced) {
for (const state of result)
check(
liveLeaves(state)[0].ancestry.includes(beforeRestart.remoteAdvanced.rev!),
"Sensible merge lost the later remote revision's ancestry"
);
check(
(await pair.a.locator(".modal").filter({ hasText: "Conflicting changes" }).count()) === 0,
"The mergeable recovery left a manual conflict dialogue"
);
}
for (const [index, page] of [pair.a, pair.b].entries()) {
const provenance = await evaluate<{ revision: string }>(
page,
`const store=core.services.keyValueDB.openSimpleStore('file-reflection-provenance-v1');
const deadline=Date.now()+5000;
let record;
do {
record=await store.get(${JSON.stringify(path)});
if(record?.revision===${JSON.stringify(result[index].rev)})return record;
await new Promise(resolve=>setTimeout(resolve,10));
}while(Date.now()<deadline);
return record;`
);
check(
provenance?.revision === result[index].rev,
"Recovery did not record the converged file revision"
);
}
return { result, restoredGuard: result[0].shouldBatchSave };
});
return { beforeRestart, afterRestart };
let repeatedRestart: Snapshot[] | undefined;
if (stopBeforeProvenance && remoteRestartEdit !== "conflicting") {
await stop(slots[0]);
await start(slots[0], true);
repeatedRestart = await withPair(async (pair) => {
const result = await settled(pair, path, remoteRestartEdit ? remoteContent : localContent, true);
check(
result[0].rev === afterRestart.result[0].rev,
"A repeated restart created another revision after recovery had converged"
);
check(
result[0].leaves.length === afterRestart.result[0].leaves.length,
"A repeated restart added another branch after recovery had converged"
);
return result;
});
}
return { beforeRestart, afterRestart, repeatedRestart };
});
await run("enabled-after-write-restart", async () => {
+3
View File
@@ -23,6 +23,7 @@ const focusedScenarios = new Set([
"p2p-pane",
"replication-status",
"vault-reflection",
"third-party-chunks",
"couchdb-upload",
"couchdb-http-cache",
"chunk-fetch-retry",
@@ -36,6 +37,7 @@ const focusedScenarios = new Set([
"p2p-setup-uri-workflow",
"partial-startup-file-failure",
"startup-scan",
"startup-integrity",
"stale-file-restart",
"folder-batch",
"setup-uri-workflow",
@@ -52,6 +54,7 @@ const focusedScenarios = new Set([
"setting-markdown-export",
"setting-markdown-roundtrip",
"upgrade-from-stable",
"upgrade-livesync-batching",
]);
function usage(): string {
@@ -0,0 +1,396 @@
/** Checks incomplete-document integrity against real Vault storage in Obsidian. */
import { createHash } from "node:crypto";
import { readFile } from "node:fs/promises";
import { evalObsidianJson } from "../runner/cli.ts";
import { requireObsidianBinary, requireObsidianCli } from "../runner/environment.ts";
import {
createE2eCouchDbPluginData,
createE2eObsidianDeviceLocalState,
waitForLiveSyncCoreReady,
waitForLocalDatabaseEntry,
} from "../runner/liveSyncWorkflow.ts";
import { startObsidianLiveSyncSession, type ObsidianLiveSyncSession } from "../runner/session.ts";
import { createTemporaryVault } from "../runner/vault.ts";
process.env.E2E_OBSIDIAN_CLI_TIMEOUT_MS ??= "60000";
const filePath = "startup-integrity.md";
const fileContent = "startup integrity remains healthy\n";
async function main(): Promise<void> {
const binary = requireObsidianBinary();
const cliBinary = requireObsidianCli();
const vault = await createTemporaryVault("obsidian-startup-integrity-");
let session: ObsidianLiveSyncSession | undefined;
const bundleHash = createHash("sha256")
.update(new Uint8Array(await readFile("main.js")))
.digest("hex");
const settings = createE2eCouchDbPluginData(
// No replication is enabled; this scenario uses only the local database.
{ uri: "http://127.0.0.1:1", username: "unused", password: "unused", dbName: "startup-integrity-local" },
{
deviceAndVaultName: "startup-integrity-device",
liveSync: false,
periodicReplication: false,
syncOnStart: false,
syncOnSave: false,
syncOnFileOpen: false,
syncOnEditorSave: false,
syncAfterMerge: false,
}
);
async function evaluate<T>(body: string): Promise<T> {
if (!session) throw new Error("The Obsidian session is not running");
return await evalObsidianJson<T>(
cliBinary,
`(async()=>{
const core=app.plugins.plugins['obsidian-livesync']?.core;
const sleep=ms=>new Promise(resolve=>setTimeout(resolve,ms));
const require=(condition,message)=>{if(!condition)throw new Error(message);};
const waitFor=async(predicate,message,timeoutMs=15000)=>{
const deadline=Date.now()+timeoutMs;
while(Date.now()<deadline){
if(await predicate())return;
await sleep(20);
}
throw new Error(message);
};
${body}
})()`,
session.cliEnv
);
}
try {
session = await startObsidianLiveSyncSession({
binary,
cliBinary,
vault,
pluginData: settings,
localStorageEntries: createE2eObsidianDeviceLocalState(vault.name),
});
await waitForLiveSyncCoreReady(cliBinary, session.cliEnv);
await evaluate(`
require(core, 'Self-hosted LiveSync did not load');
require(app.vault.getAbstractFileByPath(${JSON.stringify(filePath)})===null,
'The integrity fixture already exists in the Vault');
await app.vault.create(${JSON.stringify(filePath)},${JSON.stringify(fileContent)});
return JSON.stringify({created:${JSON.stringify(filePath)}});
`);
await waitForLocalDatabaseEntry(cliBinary, session.cliEnv, filePath);
const observations = await evaluate<unknown[]>(`
require(await core.services.fileProcessing.commitPendingFileEvents(), 'Initial local flush failed');
const path=${JSON.stringify(filePath)};
const expectedContent=${JSON.stringify(fileContent)};
const storage=core.serviceModules.storageAccess;
const fileHandler=core.serviceModules.fileHandler;
const keyValueDB=core.services.keyValueDB.kvDB;
const confirm=core.services.UI.confirm;
const noticeGroups=core.services.context.noticeGroups;
const events=core.services.context.events;
require(storage&&typeof storage.statHidden==='function'&&
typeof storage.readHiddenFileBinary==='function',
'The real StorageAccess methods were not available');
require(fileHandler&&typeof fileHandler.storeFileToDB==='function',
'The real file handler was not available');
const snapshot=async()=>{
const meta=await core.localDatabase.getDBEntryMeta(path,undefined,true);
require(meta&&meta._rev,'The integrity fixture Metadata was missing');
const doc=await core.localDatabase.getDBEntryFromMeta(meta);
require(doc&&doc.data!==undefined,'The integrity fixture Chunks could not be loaded');
const content=Array.isArray(doc.data)?doc.data.join(''):doc.data;
require(typeof content==='string','The Markdown fixture did not load as text');
const bytes=Array.from(new TextEncoder().encode(content));
require(meta.size===bytes.length,'Metadata size did not match the stored bytes');
require(doc.size===bytes.length,'Loaded document size did not match the stored bytes');
return {revision:meta._rev,metadataSize:meta.size,documentSize:doc.size,content,bytes};
};
const assertUnchanged=(before,after,label)=>{
require(before.revision===after.revision,label+': the Metadata revision changed');
require(before.metadataSize===after.metadataSize,label+': the Metadata size changed');
require(before.content===after.content,label+': the stored content changed');
require(JSON.stringify(before.bytes)===JSON.stringify(after.bytes),
label+': the stored bytes changed');
};
const activeBytes=async()=>{
const content=await app.vault.adapter.read(path);
return {content,bytes:Array.from(new TextEncoder().encode(content))};
};
const marker=async()=>await keyValueDB.get('checkIncompleteDocs')===true;
const activeState={current:null};
const originalStat=storage.statHidden;
const originalRead=storage.readHiddenFileBinary;
const originalStore=fileHandler.storeFileToDB;
const originalDialogue=confirm.askSelectStringDialogue;
const ownStat=Object.prototype.hasOwnProperty.call(storage,'statHidden');
const ownRead=Object.prototype.hasOwnProperty.call(storage,'readHiddenFileBinary');
const ownStore=Object.prototype.hasOwnProperty.call(fileHandler,'storeFileToDB');
const ownDialogue=Object.prototype.hasOwnProperty.call(confirm,'askSelectStringDialogue');
const unhandledErrors=[];
const onUnhandled=event=>{
const reason=event.reason;
unhandledErrors.push(String(reason&&reason.message?reason.message:reason));
event.preventDefault();
};
let installedStat=false;
let installedRead=false;
let installedStore=false;
let installedDialogue=false;
const restore=(owner,key,hadOwn,original)=>{
if(hadOwn)owner[key]=original;
else delete owner[key];
};
const onStat=async function(candidatePath){
const actual=await originalStat.call(this,candidatePath);
const current=activeState.current;
if(!current||candidatePath!==path)return actual;
current.statCalls++;
require(actual&&actual.type==='file','StorageAccess did not report the fixture as a file');
current.actualStats.push({size:actual.size,mtime:actual.mtime,ctime:actual.ctime});
if(current.mode==='stat-read-race'&&current.statCalls===1){
const oldStat={...actual,size:actual.size+1};
current.returnedStats.push({size:oldStat.size,mtime:oldStat.mtime,ctime:oldStat.ctime});
return oldStat;
}
if(current.mode==='unstable-final-stat'&&current.statCalls===2){
const changedStat={...actual,mtime:actual.mtime+1};
current.returnedStats.push({size:changedStat.size,mtime:changedStat.mtime,ctime:changedStat.ctime});
return changedStat;
}
current.returnedStats.push({size:actual.size,mtime:actual.mtime,ctime:actual.ctime});
return actual;
};
const onRead=async function(candidatePath){
const bytes=await originalRead.call(this,candidatePath);
const current=activeState.current;
if(current&&candidatePath===path){
current.readCalls++;
current.readBytes=Array.from(new Uint8Array(bytes));
}
return bytes;
};
const onStore=async function(file,...args){
const current=activeState.current;
if(current&&file?.path===path)current.repairCalls++;
return await originalStore.call(this,file,...args);
};
const onDialogue=async function(message,buttons,...args){
const current=activeState.current;
if(current){
current.dialogueCalls++;
current.dialogueMessages.push(String(message));
return Array.isArray(buttons)&&buttons.length>0?buttons[0]:false;
}
return await originalDialogue.call(this,message,buttons,...args);
};
const runRegisteredCheck=async current=>{
const originalFinish=noticeGroups.finish;
const ownFinish=Object.prototype.hasOwnProperty.call(noticeGroups,'finish');
let finished=false;
noticeGroups.finish=function(...args){
try{return originalFinish.apply(this,args);}
finally{
if(args[0]==='startup-integrity-check')finished=true;
}
};
activeState.current=current;
try{
events.emitEvent('request-run-fix-incomplete');
await waitFor(()=>finished,'The registered incomplete-document check did not finish');
}finally{
activeState.current=null;
restore(noticeGroups,'finish',ownFinish,originalFinish);
}
};
const makeObservation=mode=>({
mode,
statCalls:0,
actualStats:[],
returnedStats:[],
readCalls:0,
readBytes:[],
dialogueCalls:0,
dialogueMessages:[],
repairCalls:0,
});
window.addEventListener('unhandledrejection',onUnhandled);
try{
storage.statHidden=onStat;
installedStat=true;
storage.readHiddenFileBinary=onRead;
installedRead=true;
fileHandler.storeFileToDB=onStore;
installedStore=true;
confirm.askSelectStringDialogue=onDialogue;
installedDialogue=true;
const firstStorageStat=await storage.statHidden(path);
require(firstStorageStat&&firstStorageStat.type==='file',
'The real Vault file was not visible through StorageAccess');
const firstVault=await activeBytes();
const first=snapshot();
const healthyBefore=await first;
require(firstVault.content===expectedContent&&healthyBefore.content===expectedContent,
'The fixture content did not match between the Vault and Metadata');
require(firstStorageStat.size===healthyBefore.bytes.length,
'The real Vault size did not match the Metadata bytes');
await keyValueDB.set('checkIncompleteDocs',false);
const healthy=makeObservation('healthy');
const healthyUnhandledBefore=unhandledErrors.length;
await runRegisteredCheck(healthy);
const healthyAfter=await snapshot();
const healthyMarker=await marker();
const healthyVaultAfter=await activeBytes();
require(healthy.statCalls===2,'The healthy fixture was not checked with both StorageAccess stats');
require(healthy.readCalls===0,'The healthy fixture was read only to determine its size');
require(healthy.dialogueCalls===0,'The healthy fixture opened a repair dialogue');
require(healthy.repairCalls===0,'The healthy fixture invoked repair');
require(healthyMarker,'The healthy integrity check did not set its completion marker');
require(unhandledErrors.length===healthyUnhandledBefore,
'The healthy integrity check raised an unhandled error');
require(healthyVaultAfter.content===expectedContent,'The healthy Vault file changed');
assertUnchanged(healthyBefore,healthyAfter,'Healthy scan');
const observations=[
{
case:'healthy',
revision:healthyAfter.revision,
storedBytes:healthyAfter.bytes.length,
readCalls:healthy.readCalls,
statCalls:healthy.statCalls,
dialogueCalls:healthy.dialogueCalls,
repairCalls:healthy.repairCalls,
completionMarker:healthyMarker,
storedBytesUnchanged:true,
},
];
const raceBefore=await snapshot();
await keyValueDB.set('checkIncompleteDocs',false);
const race=makeObservation('stat-read-race');
const raceUnhandledBefore=unhandledErrors.length;
await runRegisteredCheck(race);
const raceAfter=await snapshot();
const raceMarker=await marker();
const raceVaultAfter=await activeBytes();
require(race.statCalls===1,'The controlled stat/read race did not use one initial stat');
require(race.actualStats[0]?.size===raceBefore.bytes.length,
'The real StorageAccess stat did not see the Metadata size');
require(race.returnedStats[0]?.size===raceBefore.bytes.length+1,
'The checker did not see the controlled old mismatched size');
require(race.readCalls===1,'The controlled mismatch did not read real Vault bytes');
require(JSON.stringify(race.readBytes)===JSON.stringify(raceBefore.bytes),
'The actual StorageAccess read did not match the database bytes');
require(race.dialogueCalls===0,'The settled stat/read race opened a repair dialogue');
require(race.repairCalls===0,'The settled stat/read race invoked repair');
require(raceMarker,'The settled stat/read race did not set its completion marker');
require(unhandledErrors.length===raceUnhandledBefore,
'The settled stat/read race raised an unhandled error');
require(raceVaultAfter.content===expectedContent,'The Vault file changed during the stat/read race');
assertUnchanged(raceBefore,raceAfter,'Settled stat/read race');
observations.push({
case:'stat-read-race',
revision:raceAfter.revision,
storedBytes:raceAfter.bytes.length,
observedStatSize:race.returnedStats[0]?.size,
readCalls:race.readCalls,
readMatchedDatabase:true,
dialogueCalls:race.dialogueCalls,
repairCalls:race.repairCalls,
completionMarker:raceMarker,
storedBytesUnchanged:true,
});
const unstableBefore=await snapshot();
await keyValueDB.set('checkIncompleteDocs',false);
const unstable=makeObservation('unstable-final-stat');
const unstableUnhandledBefore=unhandledErrors.length;
await runRegisteredCheck(unstable);
await waitFor(()=>unhandledErrors.length>unstableUnhandledBefore,
'The changed final stat did not reject the incomplete-document check');
const unstableError=unhandledErrors.slice(unstableUnhandledBefore).join(' | ');
const unstableAfter=await snapshot();
const unstableMarker=await marker();
require(unstable.statCalls===2,'The changed final stat was not checked');
require(unstable.returnedStats[1]?.mtime!==unstable.actualStats[1]?.mtime,
'The final StorageAccess stat was not changed by the race fixture');
require(unstable.readCalls===0,'The healthy path was read during the final-stat race');
require(unstableError.includes('Storage changed during the incomplete document check'),
'The final-stat race did not report the expected incomplete-check error');
require(unstableMarker===false,'The incomplete check was marked complete after storage changed');
require(unstable.dialogueCalls===0,'The final-stat race opened a repair dialogue');
require(unstable.repairCalls===0,'The final-stat race invoked repair');
assertUnchanged(unstableBefore,unstableAfter,'Final-stat race');
observations.push({
case:'unstable-final-stat',
revision:unstableAfter.revision,
statCalls:unstable.statCalls,
readCalls:unstable.readCalls,
reportedError:unstableError,
completionMarker:unstableMarker,
storedBytesUnchanged:true,
});
await keyValueDB.set('checkIncompleteDocs',false);
const retry=makeObservation('settled-retry');
const retryUnhandledBefore=unhandledErrors.length;
await runRegisteredCheck(retry);
const retryAfter=await snapshot();
const retryMarker=await marker();
require(retry.statCalls===2,'The settled retry did not complete both StorageAccess stats');
require(retry.readCalls===0,'The settled retry read healthy storage');
require(retry.dialogueCalls===0,'The settled retry opened a repair dialogue');
require(retry.repairCalls===0,'The settled retry invoked repair');
require(retryMarker,'The settled retry did not set its completion marker');
require(unhandledErrors.length===retryUnhandledBefore,
'The settled retry raised an unhandled error');
assertUnchanged(unstableBefore,retryAfter,'Settled retry');
observations.push({
case:'settled-retry',
revision:retryAfter.revision,
statCalls:retry.statCalls,
readCalls:retry.readCalls,
dialogueCalls:retry.dialogueCalls,
repairCalls:retry.repairCalls,
completionMarker:retryMarker,
storedBytesUnchanged:true,
});
return JSON.stringify(observations);
}finally{
window.removeEventListener('unhandledrejection',onUnhandled);
if(installedDialogue)restore(confirm,'askSelectStringDialogue',ownDialogue,originalDialogue);
if(installedStore)restore(fileHandler,'storeFileToDB',ownStore,originalStore);
if(installedRead)restore(storage,'readHiddenFileBinary',ownRead,originalRead);
if(installedStat)restore(storage,'statHidden',ownStat,originalStat);
}
`);
console.log(
JSON.stringify(
{
ok: true,
bundleHash,
trigger: "registered-request-run-fix-incomplete-event",
naturalRestart: false,
observations,
},
null,
2
)
);
} finally {
try {
await session?.app.stop();
} finally {
await vault.dispose();
}
}
}
await main();
@@ -0,0 +1,269 @@
/** Exercises synthetic split-surrogate input through real Obsidian file reflection. */
import { createHash } from "node:crypto";
import { readFile, writeFile } from "node:fs/promises";
import { join } from "node:path";
import { evalObsidianJson } from "../runner/cli.ts";
import { requireObsidianBinary, requireObsidianCli } from "../runner/environment.ts";
import {
assertEqual,
createE2eCouchDbPluginData,
createE2eObsidianDeviceLocalState,
waitForLiveSyncCoreReady,
} from "../runner/liveSyncWorkflow.ts";
import { startObsidianLiveSyncSession, type ObsidianLiveSyncSession } from "../runner/session.ts";
import { createTemporaryVault } from "../runner/vault.ts";
process.env.E2E_OBSIDIAN_CLI_TIMEOUT_MS ??= "60000";
async function main(): Promise<void> {
const binary = requireObsidianBinary();
const cliBinary = requireObsidianCli();
const textPath = "synthetic-split-pair.md";
const controlPath = "complete-pair.md";
const binaryPath = "attachment.bin";
const textContent = "prefix 🙂 suffix\n" + "normal chunk payload\n".repeat(50_000);
const controlContent = "prefix 🙂 suffix";
const binaryBytes = Buffer.from([0, 255, 31, 128]);
const textSize = Buffer.byteLength(textContent, "utf8");
const textHash = `sha256:${createHash("sha256").update(textContent, "utf8").digest("hex")}`;
const controlSize = Buffer.byteLength(controlContent, "utf8");
const controlHash = `sha256:${createHash("sha256").update(controlContent, "utf8").digest("hex")}`;
const settings = createE2eCouchDbPluginData(
{ uri: "http://127.0.0.1:1", username: "unused", password: "unused", dbName: "third-party-chunks-local" },
{
customChunkSize: 0,
processSizeMismatchedFiles: false,
liveSync: false,
periodicReplication: false,
syncOnStart: false,
syncOnSave: false,
syncOnFileOpen: false,
syncOnEditorSave: false,
syncAfterMerge: false,
}
);
const bundleHash = createHash("sha256")
.update(new Uint8Array(await readFile("main.js")))
.digest("hex");
const vault = await createTemporaryVault("obsidian-third-party-chunks-");
let session: ObsidianLiveSyncSession | undefined;
try {
session = await startObsidianLiveSyncSession({
binary,
cliBinary,
vault,
pluginData: settings,
localStorageEntries: createE2eObsidianDeviceLocalState(vault.name),
lifecycle: {
beforeLaunch: async () => {
const registryPath = join(vault.userDataPath, "obsidian.json");
const registry = JSON.parse(await readFile(registryPath, "utf8"));
registry.updateDisabled = true;
await writeFile(registryPath, JSON.stringify(registry));
},
},
});
await waitForLiveSyncCoreReady(cliBinary, session.cliEnv);
const reflection = await evalObsidianJson<{
sizeCheckingEnabled: boolean;
split: {
reflected: boolean;
metadataRevisionUnchanged: boolean;
chunkCount: number;
provenanceRevision: string;
contentSize: number;
contentHash: string;
};
control: {
reflected: boolean;
content: string;
metadataRevisionUnchanged: boolean;
provenanceRevision: string;
contentSize: number;
contentHash: string;
};
binary: { reflected: boolean; bytes: number[]; metadataRevisionUnchanged: boolean };
}>(
cliBinary,
[
"(async()=>{",
"const core=app.plugins.plugins['obsidian-livesync'].core;",
"const require=(condition,message)=>{if(!condition)throw new Error(message);};",
"const textPath='synthetic-split-pair.md';",
"const controlPath='complete-pair.md';",
"const binaryPath='attachment.bin';",
"const textContent='prefix 🙂 suffix\\n'+'normal chunk payload\\n'.repeat(50000);",
"const controlContent='prefix 🙂 suffix';",
"const binaryBytes=[0,255,31,128];",
"const store=core.services.keyValueDB.openSimpleStore('file-reflection-provenance-v1');",
"require(core.settings.processSizeMismatchedFiles===false,'Size mismatch checking must remain enabled');",
"const seed=async(path,blob,type,datatype)=>{",
" require(!app.vault.getAbstractFileByPath(path),'Fixture already exists in the Vault: '+path);",
" const id=await core.services.path.path2id(path);",
" const now=Date.now();",
" const saved=await core.localDatabase.putDBEntry({_id:id,path,data:blob,ctime:now,mtime:now,",
" size:blob.size,children:[],datatype,type,eden:{} });",
" require(saved?.ok,'Could not create Metadata and Chunks: '+path);",
" const meta=await core.localDatabase.getDBEntryMeta(path,undefined,true);",
" require(meta&&meta._rev===saved.rev,'Could not reload the created Metadata: '+path);",
" return meta;",
"};",
"const textMeta=await seed(textPath,new Blob([textContent],{type:'text/plain'}),'plain','plain');",
"require((textMeta.children??[]).length>0,'The primary fixture did not create normal Chunks');",
"const savedText=await core.localDatabase.getDBEntry(textPath,{rev:textMeta._rev,revs:true},false,true,true);",
"const storedText=Array.isArray(savedText?.data)?savedText.data.join(''):savedText?.data;",
"require(storedText===textContent,'The seeded local database content did not match the joined text');",
"const controlMeta=await seed(controlPath,new Blob([controlContent],{type:'text/plain'}),'plain','plain');",
"const binaryMeta=await seed(binaryPath,new Blob([new Uint8Array(binaryBytes)],{type:'application/octet-stream'}),'newnote','newnote');",
"const initialTextRevision=textMeta._rev;",
"const initialControlRevision=controlMeta._rev;",
"const database=core.serviceModules.databaseFileAccess;",
"require(typeof database?.fetchEntryFromMeta==='function','The database file-access provider is missing');",
"const descriptor=Object.getOwnPropertyDescriptor(database,'fetchEntryFromMeta');",
"const original=database.fetchEntryFromMeta;",
"let supplied=0;",
"const pairIndex=textContent.indexOf('🙂');",
"const syntheticChunks=[textContent.slice(0,pairIndex+1),textContent.slice(pairIndex+1)];",
"require(syntheticChunks[0]==='prefix \\uD83D','The synthetic first chunk did not end with the high surrogate');",
"require(syntheticChunks[1].startsWith('\\uDE42 suffix'),'The synthetic second chunk did not start with the low surrogate');",
"database.fetchEntryFromMeta=async function(meta,...args){",
" const loaded=await original.call(database,meta,...args);",
" if(meta?._id!==textMeta._id||meta?.path!==textPath||!loaded)return loaded;",
" supplied++;",
" return {...loaded,data:syntheticChunks};",
"};",
"let reflectedText=false;",
"try{reflectedText=await core.serviceModules.fileHandler.dbToStorage(textMeta,null,true);}",
"finally{if(descriptor)Object.defineProperty(database,'fetchEntryFromMeta',descriptor);else delete database.fetchEntryFromMeta;}",
"require(supplied===1,'The synthetic provider was not used exactly once');",
"require(reflectedText,'Real Obsidian reflection rejected the joined text');",
"const textFile=app.vault.getAbstractFileByPath(textPath);",
"require(textFile,'The reflected text file is missing from the Vault');",
"const vaultText=await app.vault.read(textFile);",
"require(vaultText===textContent,'The Vault text does not match the joined input');",
"const textMetaAfter=await core.localDatabase.getDBEntryMeta(textPath,undefined,true);",
"require(textMetaAfter?._rev===initialTextRevision,'Text reflection changed the Metadata revision');",
"const textProvenance=await store.get(textPath);",
"require(textProvenance?.revision===initialTextRevision,'Text provenance has the wrong revision');",
`require(textProvenance?.contentSize===${textSize},'Text provenance has the wrong UTF-8 byte size');`,
`require(textProvenance?.contentHash===${JSON.stringify(textHash)},'Text provenance has the wrong content hash');`,
"const reflectedControl=await core.serviceModules.fileHandler.dbToStorage(controlMeta,null,true);",
"require(reflectedControl,'Real Obsidian reflection rejected the complete-pair control');",
"const controlFile=app.vault.getAbstractFileByPath(controlPath);",
"require(controlFile&&await app.vault.read(controlFile)===controlContent,'The complete-pair control content changed');",
"const controlVaultText=await app.vault.read(controlFile);",
"const controlMetaAfter=await core.localDatabase.getDBEntryMeta(controlPath,undefined,true);",
"require(controlMetaAfter?._rev===initialControlRevision,'Control reflection changed the Metadata revision');",
"const controlProvenance=await store.get(controlPath);",
"require(controlProvenance?.revision===initialControlRevision,'Control provenance has the wrong revision');",
`require(controlProvenance?.contentSize===${controlSize},'Control provenance has the wrong UTF-8 byte size');`,
`require(controlProvenance?.contentHash===${JSON.stringify(controlHash)},'Control provenance has the wrong content hash');`,
"const reflectedBinary=await core.serviceModules.fileHandler.dbToStorage(binaryMeta,null,true);",
"require(reflectedBinary,'Real Obsidian reflection rejected the binary attachment');",
"const binaryFile=app.vault.getAbstractFileByPath(binaryPath);",
"require(binaryFile,'The reflected binary attachment is missing from the Vault');",
"const binaryVaultBytes=Array.from(new Uint8Array(await app.vault.readBinary(binaryFile)));",
"require(JSON.stringify(binaryVaultBytes)===JSON.stringify(binaryBytes),'The Vault binary bytes changed');",
"const binaryMetaAfter=await core.localDatabase.getDBEntryMeta(binaryPath,undefined,true);",
"require(binaryMetaAfter?._rev===binaryMeta._rev,'Binary reflection changed the Metadata revision');",
"return JSON.stringify({sizeCheckingEnabled:!core.settings.processSizeMismatchedFiles,",
" split:{reflected:reflectedText,metadataRevisionUnchanged:textMetaAfter._rev===initialTextRevision,chunkCount:textMeta.children.length,",
" provenanceRevision:textProvenance.revision,contentSize:textProvenance.contentSize,contentHash:textProvenance.contentHash},",
" control:{reflected:reflectedControl,content:controlVaultText,",
" metadataRevisionUnchanged:controlMetaAfter._rev===initialControlRevision,provenanceRevision:controlProvenance.revision,",
" contentSize:controlProvenance.contentSize,contentHash:controlProvenance.contentHash},",
" binary:{reflected:reflectedBinary,bytes:binaryVaultBytes,metadataRevisionUnchanged:binaryMetaAfter._rev===binaryMeta._rev}});",
"})()",
].join(""),
session.cliEnv
);
assertEqual(reflection.sizeCheckingEnabled, true, "Size mismatch checking was disabled.");
assertEqual(reflection.split.reflected, true, "Split-pair text was not reflected.");
assertEqual(
reflection.split.metadataRevisionUnchanged,
true,
"Split-pair reflection changed its Metadata revision."
);
assertEqual(reflection.split.provenanceRevision.length > 0, true, "Split-pair provenance is missing.");
assertEqual(reflection.split.contentSize, textSize, "Split-pair provenance has the wrong UTF-8 byte size.");
assertEqual(reflection.split.contentHash, textHash, "Split-pair provenance has the wrong content hash.");
assertEqual(reflection.control.reflected, true, "Complete-pair control was not reflected.");
assertEqual(reflection.control.content, controlContent, "Complete-pair control content changed.");
assertEqual(
reflection.control.metadataRevisionUnchanged,
true,
"Control reflection changed its Metadata revision."
);
assertEqual(reflection.control.contentSize, controlSize, "Control provenance has the wrong UTF-8 byte size.");
assertEqual(reflection.control.contentHash, controlHash, "Control provenance has the wrong content hash.");
assertEqual(reflection.binary.reflected, true, "Binary attachment was not reflected.");
if (!Buffer.from(reflection.binary.bytes).equals(binaryBytes)) {
throw new Error("The Obsidian Vault API returned different binary attachment bytes.");
}
assertEqual(
reflection.binary.metadataRevisionUnchanged,
true,
"Binary reflection changed its Metadata revision."
);
const diskText = await readFile(join(vault.path, textPath));
assertEqual(diskText.toString("utf8"), textContent, "On-disk UTF-8 text differs from the joined input.");
assertEqual(diskText.byteLength, textSize, "On-disk UTF-8 text has the wrong byte length.");
const diskControl = await readFile(join(vault.path, controlPath));
assertEqual(
diskControl.toString("utf8"),
controlContent,
"On-disk complete-pair control differs from the input."
);
const diskBinary = await readFile(join(vault.path, binaryPath));
if (!diskBinary.equals(binaryBytes)) throw new Error("On-disk binary attachment bytes changed.");
console.log(
JSON.stringify(
{
ok: true,
bundleHash,
syntheticInput:
"A local fetchEntryFromMeta result split the joined text between the emoji surrogates; this does not establish that CouchDB or any named third-party producer emits lone-surrogate chunks.",
sizeCheckingEnabled: reflection.sizeCheckingEnabled,
splitText: {
reflected: reflection.split.reflected,
metadataRevisionUnchanged: reflection.split.metadataRevisionUnchanged,
realChunkCount: reflection.split.chunkCount,
vaultTextMatches: true,
onDiskUtf8Bytes: diskText.byteLength,
provenanceRevision: reflection.split.provenanceRevision,
contentSize: reflection.split.contentSize,
contentHash: reflection.split.contentHash,
},
completePairControl: {
reflected: reflection.control.reflected,
metadataRevisionUnchanged: reflection.control.metadataRevisionUnchanged,
vaultAndDiskContentMatch: true,
provenanceContentSize: reflection.control.contentSize,
provenanceContentHash: reflection.control.contentHash,
},
binaryAttachment: {
reflected: reflection.binary.reflected,
exactVaultBytes: true,
exactDiskBytes: true,
metadataRevisionUnchanged: reflection.binary.metadataRevisionUnchanged,
},
},
null,
2
)
);
} finally {
try {
await session?.app.stop();
} finally {
await vault.dispose();
}
}
}
await main();
@@ -0,0 +1,312 @@
/** Keeps the 1.0.40 batching choice and Vault state when installing the current release. */
import { createHash } from "node:crypto";
import { readFile } from "node:fs/promises";
import { join, resolve } from "node:path";
import { evalObsidianJson } from "../runner/cli.ts";
import {
assertCouchDbReachable,
createCouchDbDatabase,
deleteCouchDbDatabase,
loadCouchDbConfig,
makeUniqueDatabaseName,
waitForCouchDbDocs,
} from "../runner/couchdb.ts";
import { requireObsidianBinary, requireObsidianCli } from "../runner/environment.ts";
import {
assertEqual,
createE2eCouchDbPluginData,
createE2eObsidianDeviceLocalState,
prepareRemote,
pushLocalChanges,
waitForLiveSyncCoreReady,
waitForLocalDatabaseEntry,
} from "../runner/liveSyncWorkflow.ts";
import { ensurePinnedReleaseArtifact, type PinnedPluginRelease } from "../runner/releaseArtifact.ts";
import { startObsidianLiveSyncSession, type ObsidianLiveSyncSession } from "../runner/session.ts";
import { readRuntimeUpgradeState, waitForPersistentNodeIdentity } from "../runner/upgradeWorkflow.ts";
import { createTemporaryVault, type TemporaryVault } from "../runner/vault.ts";
const sourceRelease: PinnedPluginRelease = {
pluginId: "obsidian-livesync",
version: "1.0.40",
files: [
{
name: "main.js",
url: "https://github.com/vrtmrz/obsidian-livesync/releases/download/1.0.40/main.js",
sha256: "bdd6b18ffe36a6a72ce774e857889b70d379559e4a5a3ccafee357a42daca6e9",
},
{
name: "manifest.json",
url: "https://github.com/vrtmrz/obsidian-livesync/releases/download/1.0.40/manifest.json",
sha256: "8f8f57fb105aba0ce7f9ea5104dbed92fea837d49fdf27b4f02b6099ddf1ba3b",
},
{
name: "styles.css",
url: "https://github.com/vrtmrz/obsidian-livesync/releases/download/1.0.40/styles.css",
sha256: "4ba00bef417ee98d9a62dd9febb792dc6555949cfc9e8e069450ad699ab2b9f0",
},
],
};
type BatchState = {
batchSave: boolean;
batchSaveInLiveSync: boolean | null;
liveSync: boolean;
periodicReplication: boolean;
shouldBatchSave: boolean;
minimum: number;
maximum: number;
};
const beforePath = "upgrade-batching-before.md";
const afterPath = "upgrade-batching-after.md";
const returnPath = "upgrade-batching-return.md";
const beforeContent = "Saved by the published 1.0.40 plug-in.\n";
const afterContent = "Saved after upgrading with LiveSync batching still disabled.\n";
const returnContent = "Returned by an independent current-release Vault.\n";
async function artefactHashes(root: string): Promise<Record<string, string>> {
return Object.fromEntries(
await Promise.all(
sourceRelease.files.map(async ({ name }) => [
name,
createHash("sha256")
.update(new Uint8Array(await readFile(join(root, name))))
.digest("hex"),
])
)
);
}
async function main(): Promise<void> {
const binary = requireObsidianBinary();
const cliBinary = requireObsidianCli();
const sourceRoot = await ensurePinnedReleaseArtifact(sourceRelease);
const targetRoot = resolve(process.env.E2E_LIVESYNC_TARGET_ARTIFACT_ROOT?.trim() || process.cwd());
const targetManifest = JSON.parse(await readFile(join(targetRoot, "manifest.json"), "utf8"));
if (targetManifest.id !== sourceRelease.pluginId || targetManifest.version === sourceRelease.version) {
throw new Error("The target must be a newer Self-hosted LiveSync build.");
}
const hashes = await artefactHashes(targetRoot);
const config = await loadCouchDbConfig();
if (!["127.0.0.1", "localhost"].includes(new URL(config.uri).hostname)) {
throw new Error("Use a disposable local CouchDB for the upgrade scenario.");
}
const dbName = makeUniqueDatabaseName(config.dbPrefix, "upgrade-batching");
const vault = await createTemporaryVault("obsidian-upgrade-batching-");
const verifier = await createTemporaryVault("obsidian-upgrade-verifier-");
let session: ObsidianLiveSyncSession | undefined;
let ownsDatabase = false;
const pluginData = createE2eCouchDbPluginData(
{ ...config, dbName },
{
batchSave: true,
batchSaveMinimumDelay: 3,
batchSaveMaximumDelay: 7,
liveSync: false,
periodicReplication: false,
syncOnFileOpen: false,
syncOnEditorSave: false,
syncAfterMerge: false,
doctorProcessedVersion: "1.0.0",
}
);
async function stop(): Promise<void> {
if (session) await session.app.stop();
session = undefined;
}
async function start(currentVault: TemporaryVault, artifactRoot: string, seed = false): Promise<void> {
session = await startObsidianLiveSyncSession({
binary,
cliBinary,
vault: currentVault,
artifactRoot,
pluginStartup: seed ? undefined : "natural",
pluginData: seed ? pluginData : undefined,
localStorageEntries: seed ? createE2eObsidianDeviceLocalState(currentVault.name) : undefined,
});
await waitForLiveSyncCoreReady(cliBinary, session.cliEnv);
}
async function evaluate<T>(body: string): Promise<T> {
if (!session) throw new Error("No active Obsidian session");
return await evalObsidianJson<T>(
cliBinary,
`(async()=>{
const core=app.plugins.plugins['obsidian-livesync'].core;
return JSON.stringify(await(async()=>{${body}})());
})()`,
session.cliEnv
);
}
async function batching(): Promise<BatchState> {
return await evaluate<BatchState>(`
const s=core.services.setting.currentSettings();
return {batchSave:s.batchSave,batchSaveInLiveSync:s.batchSaveInLiveSync??null,
liveSync:s.liveSync,periodicReplication:s.periodicReplication,
shouldBatchSave:core.serviceModules.storageAccess.vaultManager.shouldBatchSave,
minimum:s.batchSaveMinimumDelay,maximum:s.batchSaveMaximumDelay};
`);
}
async function createNote(path: string, content: string): Promise<void> {
await evaluate(`await app.vault.create(${JSON.stringify(path)},${JSON.stringify(content)}); return true;`);
const entry = await waitForLocalDatabaseEntry(cliBinary, session!.cliEnv, path);
if ((await batching()).liveSync) {
await waitForCouchDbDocs(config, dbName, (docs) =>
docs.some((doc) => doc._id === entry.id && doc._rev === entry.rev)
);
} else {
await pushLocalChanges(cliBinary, session!.cliEnv);
}
}
async function assertNote(currentVault: TemporaryVault, path: string, content: string): Promise<void> {
const actual = await evaluate(`
const until=Date.now()+15000;
while(Date.now()<until){
const file=app.vault.getAbstractFileByPath(${JSON.stringify(path)});
if(file?.stat){const value=await app.vault.read(file);if(value===${JSON.stringify(content)})return value;}
await new Promise(resolve=>setTimeout(resolve,100));
}
throw new Error('Expected Vault contents did not arrive: '+${JSON.stringify(path)});
`);
assertEqual(actual, content, "The Obsidian Vault did not contain the expected text.");
assertEqual(await readFile(join(currentVault.path, path), "utf8"), content, "On-disk text differed.");
}
function assertUpgradedBatching(state: BatchState, live: boolean): void {
assertEqual(state.batchSave, true, "The legacy batching choice changed during upgrade.");
assertEqual(state.batchSaveInLiveSync, false, "Upgrade unexpectedly enabled LiveSync batching.");
assertEqual(state.liveSync, live, "The selected synchronisation mode changed.");
assertEqual(state.shouldBatchSave, !live, "The retained choice applied to the wrong mode.");
assertEqual(state.minimum, 3, "The minimum batch delay changed.");
assertEqual(state.maximum, 7, "The maximum batch delay changed.");
}
try {
await assertCouchDbReachable(config);
await createCouchDbDatabase(config, dbName);
ownsDatabase = true;
await start(vault, sourceRoot, true);
await prepareRemote(cliBinary, session!.cliEnv);
await createNote(beforePath, beforeContent);
await evaluate(`
const s=core.services.setting.currentSettings();
s.liveSync=true;
await core.services.setting.saveSettingData();
await core.services.control.applySettings();
return true;
`);
await stop();
// Confirm that the old release itself persisted this state before replacing any artefacts.
await start(vault, sourceRoot);
const oldBatch = await batching();
assertEqual(oldBatch.batchSave, true, "The old release did not retain its batching choice.");
assertEqual(oldBatch.batchSaveInLiveSync, null, "The old release unexpectedly owns the new setting.");
assertEqual(oldBatch.liveSync, true, "The old release did not retain LiveSync mode.");
assertEqual(oldBatch.shouldBatchSave, false, "The old release unexpectedly batches LiveSync mode.");
await waitForPersistentNodeIdentity(cliBinary, session!.cliEnv);
const before = await readRuntimeUpgradeState(cliBinary, session!.cliEnv);
const originalEntry = await waitForLocalDatabaseEntry(cliBinary, session!.cliEnv, beforePath);
await assertNote(vault, beforePath, beforeContent);
await stop();
const savedOldData = JSON.parse(
await readFile(join(vault.path, ".obsidian/plugins/obsidian-livesync/data.json"), "utf8")
);
assertEqual(
Object.prototype.hasOwnProperty.call(savedOldData, "batchSaveInLiveSync"),
false,
"The old settings fixture contains the new option."
);
await start(vault, targetRoot);
const upgraded = await readRuntimeUpgradeState(cliBinary, session!.cliEnv);
assertEqual(upgraded.pluginVersion, targetManifest.version, "The requested target build was not loaded.");
assertEqual(upgraded.localDatabaseName, before.localDatabaseName, "Upgrade replaced the local database.");
assertEqual(upgraded.nodeId, before.nodeId, "Upgrade replaced the device identity.");
assertEqual(
upgraded.settings.activeConfigurationId,
before.settings.activeConfigurationId,
"Upgrade changed the remote profile."
);
assertEqual(upgraded.settings.couchDB_DBNAME, dbName, "Upgrade selected another remote database.");
assertEqual(upgraded.settings.versionUpFlash, "", "Upgrade unexpectedly paused for compatibility review.");
const firstBatch = await batching();
assertUpgradedBatching(firstBatch, true);
assertEqual(
(await waitForLocalDatabaseEntry(cliBinary, session!.cliEnv, beforePath)).rev,
originalEntry.rev,
"Upgrade rewrote existing Metadata."
);
await assertNote(vault, beforePath, beforeContent);
await createNote(afterPath, afterContent);
await stop();
await start(verifier, targetRoot, true);
await prepareRemote(cliBinary, session!.cliEnv);
await pushLocalChanges(cliBinary, session!.cliEnv);
await assertNote(verifier, beforePath, beforeContent);
await assertNote(verifier, afterPath, afterContent);
await createNote(returnPath, returnContent);
await stop();
await start(vault, targetRoot);
assertUpgradedBatching(await batching(), true);
await assertNote(vault, returnPath, returnContent);
await evaluate(`
const s=core.services.setting.currentSettings();
s.liveSync=false;s.periodicReplication=true;s.periodicReplicationInterval=60;
await core.services.setting.saveSettingData();
await core.services.control.applySettings();
return true;
`);
assertUpgradedBatching(await batching(), false);
await stop();
await start(vault, targetRoot);
const finalBatch = await batching();
assertUpgradedBatching(finalBatch, false);
assertEqual(finalBatch.periodicReplication, true, "The periodic mode choice did not survive restart.");
await assertNote(vault, returnPath, returnContent);
assertEqual(
JSON.stringify(await artefactHashes(targetRoot)),
JSON.stringify(hashes),
"Target artefacts changed during the test."
);
console.log(
JSON.stringify(
{
passed: true,
sourceVersion: sourceRelease.version,
targetVersion: targetManifest.version,
sourceHashes: Object.fromEntries(sourceRelease.files.map((file) => [file.name, file.sha256])),
targetHashes: hashes,
oldBatch,
firstBatch,
finalBatch,
retainedDatabase: true,
retainedNodeIdentity: true,
retainedMetadataRevision: true,
twoWayTransfer: true,
naturalRestarts: true,
},
null,
2
)
);
} finally {
await stop();
await vault.dispose();
await verifier.dispose();
if (ownsDatabase) await deleteCouchDbDatabase(config, dbName);
}
}
main().catch((error: unknown) => {
console.error(error instanceof Error ? error.stack : error);
process.exitCode = 1;
});