Integrate content evidence for unchanged files

test/README.md

@@ -98,7 +98,7 @@ npm run test:docker-couchdb:start
 NODE_OPTIONS=--max-old-space-size=3072 npm run test:integration -- src/serviceModules/FileHandler.multidevice.integration.spec.ts --maxWorkers=1
 ```
 
-The expected result is five passing tests. Each case creates a uniquely named remote database and removes it during teardown. Stop the fixture after the run, including when the test command fails:
+The expected result is seven passing tests. Each case creates a uniquely named remote database and removes it during teardown. Stop the fixture after the run, including when the test command fails:
 
 ```bash
 npm run test:docker-couchdb:stop
@@ -116,7 +116,9 @@ Each simulated device owns a separate real PouchDB database, `LiveSyncLocalDB` m
 | Four editing devices | Repeat the same sequence with four independently edited branches. | All four contents are initially preserved, and the same unchanged-save and convergence guarantees hold. |
 | A genuine edit on a losing device | After three-way resolution reaches its DB, a device adds content to its still-unreflected losing file, then saves and replicates it. | The new revision extends that device's recorded branch. The selected result and the new edit remain readable on every replica. |
 | Missing provenance | After four-way resolution, remove a losing device's provenance record and set its file to the original historical ancestor's content. Save, remove provenance again, and repeat the save. | The file becomes one fresh independent root, distinct from the historical root. Both contents remain readable after replication, and repeated saves create no duplicates. CLI `resolve` can select the independent root and propagate its resolution. |
-| Compacted base | After four-way resolution, compact a losing device's local DB and confirm that its recorded revision body is unavailable. Save the remaining file, then repeat after removing provenance. | The file is preserved as one independent conflict rather than discarded. Both contents remain readable on every replica, repeated saves create no duplicates, and CLI `resolve` can select the independent root and propagate its resolution. |
+| Compacted base with verified unchanged content | After four-way resolution, retain a losing device's content hash, compact its local DB, and confirm that its recorded revision body is unavailable. Save the unchanged file. | The selected content is reflected without adding a database revision or conflict, and all replicas retain only the selected branch. |
+| Compacted base with legacy provenance | After four-way resolution, retain only the losing device's revision and observed modification time, compact its local DB, and confirm that the recorded revision body is unavailable. Save the remaining file, then repeat after removing provenance. | The file is preserved as one independent conflict. Both contents remain readable on every replica, repeated saves create no duplicates, and CLI `resolve` can select the independent root and propagate its resolution. |
+| Compacted base with a genuine edit | After four-way resolution and local compaction, retain the losing device's content hash but edit its file without changing its byte length or modification time. | The changed content is preserved as one independent conflict, remains readable on every replica, and can be selected through CLI `resolve`. Repeated saves after provenance loss create no duplicates. |
 
 ### Coverage boundaries
 

test/e2e-obsidian/README.md

@@ -87,7 +87,7 @@ The wrapper accepts only maintained real-Obsidian scenario names; run it with `-
 
 `folder-batch` needs no remote service. It creates 24 notes and imports three notes with colons in their names into nested folders, reflects the database content into the existing files, then renames and deletes the parent through the Obsidian Vault API. It checks exact paths, descendant events, content, Chunks, deletion markers, provenance, and the absence of unexpected files. A note outside the parent must remain writable. The colon fixtures use the adapter to represent externally created files because Obsidian's Vault creation API rejects those names. The scenario also seeds a database-only colon-named note and verifies that Obsidian's refusal to create it preserves its Metadata without writing a differently named Vault file. It does not establish successful restoration of that absent note or behaviour on other operating systems.
 
-`stale-file-restart` needs no remote service. It advances the local database while old Vault bytes remain, persists pending storage events, and restarts the same isolated Vault and profile. It checks that an unchanged file with exact provenance receives the newer database content without creating a revision, that unknown-origin content is preserved on a fresh independent branch, and that losing provenance and processing the file again does not duplicate or automatically merge that branch. A third file begins with no provenance while its bytes still match the current database revision; start-up must record that revision without creating a new one, and a later incoming revision must reflect without a conflict. The database advances and pending snapshot are controlled fixtures; start-up processing, persistence, file reflection, and conflict checking run in real Obsidian. The scenario does not simulate a mobile operating system suspending the application.
+`stale-file-restart` needs no remote service. It advances the local database while old Vault bytes remain, persists pending storage events, and restarts the same isolated Vault and profile. It checks that an unchanged file whose recorded revision has been pruned receives the newer database content using persisted content evidence without creating a revision, that unknown-origin content is preserved on a fresh independent branch, and that losing provenance and processing the file again does not duplicate or automatically merge that branch. A third file begins with no provenance while its bytes still match the current database revision; start-up must record that revision without creating a new one, and a later incoming revision must reflect without a conflict. A fourth file has legacy revision-only provenance; start-up must verify its exact content and add a hash without creating a database revision or changing storage. The database advances and pending snapshot are controlled fixtures; start-up processing, persistence, file reflection, and conflict checking run in real Obsidian. The scenario does not simulate a mobile operating system suspending the application.
 
 The principal entry points are:
 

test/e2e-obsidian/scripts/review-harness.ts

@@ -204,6 +204,10 @@ async function runAutomaticScenarios(): Promise<void> {
                 .getByText("Passed:", { exact: false })
                 .waitFor({ state: "visible", timeout: uiTimeoutMs });
         }
+        await harness
+            .locator('[data-testid="review-harness-result-file-content-hash-performance"]')
+            .getByText("Not run:", { exact: false })
+            .waitFor({ state: "visible", timeout: uiTimeoutMs });
     });
 }
 
@@ -252,6 +256,52 @@ async function runIdBenchmark(): Promise<void> {
     });
 }
 
+async function runFileContentHashBenchmark(): Promise<void> {
+    await withObsidianPage(obsidianRemoteDebuggingPort(), async (page) => {
+        const snapshotSettings = () =>
+            page.evaluate(() => {
+                const plugin = (globalThis as ReviewHarnessTestGlobal).app?.plugins?.plugins["obsidian-livesync"] as {
+                    core: { services: { setting: { currentSettings(): unknown } } };
+                };
+                return JSON.stringify(plugin.core.services.setting.currentSettings());
+            });
+        const before = await snapshotSettings();
+        const harness = page.locator('[data-testid="review-harness"]');
+        await harness
+            .locator('[data-testid="review-harness-run-file-content-hash-performance"]')
+            .click({ timeout: uiTimeoutMs });
+        const result = harness.locator('[data-testid="review-harness-result-file-content-hash-performance"]');
+        await result.getByText("Passed:", { exact: false }).waitFor({ state: "visible", timeout: uiTimeoutMs * 10 });
+        const observations = await result.locator("li").allTextContents();
+        for (const label of ["4 KiB", "64 KiB", "1 MiB", "50 MiB"]) {
+            for (const algorithm of ["SHA-256", "xxHash64"]) {
+                if (
+                    !observations.some(
+                        (line) => line.startsWith(`${algorithm}, ${label}: median=`) && line.includes("; range=")
+                    )
+                ) {
+                    throw new Error(`Missing file content hash timing and range: ${algorithm}, ${label}`);
+                }
+            }
+        }
+        for (const label of [
+            "Input construction is excluded from timings.",
+            "xxHash64 engine initialisation:",
+            "Warm-up on 4 KiB",
+            "Calls run serially",
+            "JavaScript heap:",
+        ]) {
+            if (!observations.some((line) => line.includes(label))) {
+                throw new Error(`Missing file content hash measurement detail: ${label}`);
+            }
+        }
+        if ((await snapshotSettings()) !== before) throw new Error("The benchmark changed the live settings.");
+        await assertNoHorizontalOverflow(page, harness, { label: "file content hash benchmark results" });
+        const timings = observations.filter((line) => line.startsWith("SHA-256,") || line.startsWith("xxHash64,"));
+        console.log(`File content hash benchmark observations:\n${timings.join("\n")}`);
+    });
+}
+
 async function runVaultFixture(): Promise<string> {
     await withObsidianPage(obsidianRemoteDebuggingPort(), async (page) => {
         await page.locator('[data-testid="review-harness-run-vault-round-trip"]').click({ timeout: uiTimeoutMs });
@@ -409,6 +459,7 @@ async function verifyMobileHarness(): Promise<string> {
             "review-harness-run-full",
             "review-harness-copy-report",
             "review-harness-run-id-generation-performance",
+            "review-harness-run-file-content-hash-performance",
         ]) {
             await assertLocatorHasMinimumTouchTarget(page, harness.locator(`[data-testid="${testId}"]`), {
                 label: testId,
@@ -482,6 +533,7 @@ async function main(): Promise<void> {
         const resumedScreenshot = await restartAndResumeHarness();
         await completeResumedCompatibilityStep();
         await runIdBenchmark();
+        await runFileContentHashBenchmark();
         const report = await copyAndReadReport();
         if (!report.includes("## Self-hosted LiveSync Review Harness report")) {
             throw new Error("The copied Review Harness report was not Markdown evidence.");
@@ -491,6 +543,10 @@ async function main(): Promise<void> {
             "; per ID=",
             "ID key derivation at save time:",
             "JavaScript heap:",
+            "SHA-256, 4 KiB: median=",
+            "xxHash64, 4 KiB: median=",
+            "SHA-256, 50 MiB: median=",
+            "xxHash64, 50 MiB: median=",
         ]) {
             if (!report.includes(expected)) throw new Error(`Missing copied benchmark observation: ${expected}`);
         }

test/e2e-obsidian/scripts/stale-file-restart.ts

@@ -1,3 +1,4 @@
+import { createHash } from "node:crypto";
 import { evalObsidianJson } from "../runner/cli.ts";
 import { discoverObsidianCli, requireObsidianBinary } from "../runner/environment.ts";
 import {
@@ -11,6 +12,7 @@ import { createTemporaryVault } from "../runner/vault.ts";
 
 const paths = ["stale-known.md", "stale-unknown.md"];
 const recoveredPath = "stale-recovered.md";
+const legacyPath = "stale-legacy.md";
 const oldContent = "# Note\nKeep\n\nTail\n\nFooter\n";
 const newContent = oldContent.replace(
     "Footer\n",
@@ -18,7 +20,15 @@ const newContent = oldContent.replace(
 );
 
 type Branch = { rev: string; content: string; history: string[] };
-type FileState = { path: string; content: string; rev: string; branches: Branch[]; provenance: string | null };
+type FileState = {
+    path: string;
+    content: string;
+    rev: string;
+    branches: Branch[];
+    provenance: string | null;
+    contentHash: string | null;
+    contentSize: number | null;
+};
 
 async function readState(cliBinary: string, env: NodeJS.ProcessEnv): Promise<FileState[]> {
     return await evalObsidianJson<FileState[]>(
@@ -27,7 +37,7 @@ async function readState(cliBinary: string, env: NodeJS.ProcessEnv): Promise<Fil
             const core=app.plugins.plugins['obsidian-livesync'].core;
             const store=core.services.keyValueDB.openSimpleStore('file-reflection-provenance-v1');
             const states=[];
-            for(const path of ${JSON.stringify([...paths, recoveredPath])}){
+            for(const path of ${JSON.stringify([...paths, recoveredPath, legacyPath])}){
                 const meta=await core.localDatabase.getDBEntryMeta(path,{conflicts:true},true);
                 const branches=[];
                 for(const rev of [meta._rev,...(meta._conflicts??[])]){
@@ -38,7 +48,8 @@ async function readState(cliBinary: string, env: NodeJS.ProcessEnv): Promise<Fil
                 }
                 const file=app.vault.getAbstractFileByPath(path);
                 states.push({path,content:await app.vault.read(file),rev:meta._rev,branches,
-                    provenance:(await store.get(path))?.revision??null});
+                    provenance:(await store.get(path))?.revision??null,
+                    contentHash:(await store.get(path))?.contentHash??null,contentSize:(await store.get(path))?.contentSize??null});
             }
             return JSON.stringify(states);
         })()`,
@@ -83,16 +94,22 @@ async function main(): Promise<void> {
         await evalObsidianJson(
             cliBinary,
             `(async()=>{
-            for(const path of ${JSON.stringify([...paths, recoveredPath])}) await app.vault.create(path,${JSON.stringify(oldContent)});
+            for(const path of ${JSON.stringify([...paths, recoveredPath, legacyPath])}) await app.vault.create(path,${JSON.stringify(oldContent)});
             return JSON.stringify(true);
         })()`,
             session.cliEnv
         );
-        for (const path of [...paths, recoveredPath]) await waitForLocalDatabaseEntry(cliBinary, session.cliEnv, path);
+        for (const path of [...paths, recoveredPath, legacyPath])
+            await waitForLocalDatabaseEntry(cliBinary, session.cliEnv, path);
 
         // Drain real Vault events before creating a persisted pending-event fixture.
         // The DB advances without reflecting it in the Vault, as on an offline device.
-        const fixture = await evalObsidianJson<{ current: string[]; original: string[]; recovered: string }>(
+        const fixture = await evalObsidianJson<{
+            current: string[];
+            original: string[];
+            recovered: string;
+            legacy: string;
+        }>(
             cliBinary,
             `(async()=>{
             const core=app.plugins.plugins['obsidian-livesync'].core;
@@ -106,16 +123,32 @@ async function main(): Promise<void> {
                 const result=await core.localDatabase.putDBEntry({...meta,data,mtime:file.stat.mtime+60000,
                     size:data.size,children:[]},false,meta._rev);
                 if(!result?.ok) throw new Error('Could not advance '+path);
-                current.push(result.rev); original.push(meta._rev);
-                if(index===0) await store.set(path,{revision:meta._rev,observedStorageMtime:file.stat.mtime});
-                else await store.delete(path);
+                let currentRev=result.rev;
+                if(index===0){
+                    const record=await store.get(path);
+                    if(record?.revision!==meta._rev || !record.contentHash) throw new Error('Initial save did not record content evidence');
+                    // Advance beyond the local retention limit without reflecting the incoming bytes.
+                    let raw=await core.localDatabase.getRaw(meta._id,{rev:currentRev});
+                    for(let update=0;update<100;update++){
+                        const next=await core.localDatabase.putRaw({...raw,mtime:raw.mtime+1});
+                        currentRev=next.rev; raw={...raw,_rev:currentRev,mtime:raw.mtime+1};
+                    }
+                    let pruned=false;
+                    try { await core.localDatabase.getRaw(meta._id,{rev:meta._rev}); }
+                    catch(error) { if(error.status!==404) throw error; pruned=true; }
+                    if(!pruned) throw new Error('The recorded revision was not pruned');
+                } else await store.delete(path);
+                current.push(currentRev); original.push(meta._rev);
                 snapshot.push({type:'CHANGED',key:'CHANGED-'+path,args:{file}});
             }
             const recovered=${JSON.stringify(recoveredPath)};
             const recoveredMeta=await core.localDatabase.getDBEntryMeta(recovered,{},true);
             await store.delete(recovered);
+            const legacy=${JSON.stringify(legacyPath)};
+            const legacyMeta=await core.localDatabase.getDBEntryMeta(legacy,{},true);
+            await store.set(legacy,{revision:legacyMeta._rev});
             await core.kvDB.set('storage-event-manager-snapshot',snapshot);
-            return JSON.stringify({current,original,recovered:recoveredMeta._rev});
+            return JSON.stringify({current,original,recovered:recoveredMeta._rev,legacy:legacyMeta._rev});
         })()`,
             session.cliEnv
         );
@@ -124,7 +157,7 @@ async function main(): Promise<void> {
 
         session = await startObsidianLiveSyncSession({ binary, cliBinary, vault });
         await waitForLiveSyncCoreReady(cliBinary, session.cliEnv);
-        const [known, unknown, recovered] = await readState(cliBinary, session.cliEnv);
+        const [known, unknown, recovered, legacy] = await readState(cliBinary, session.cliEnv);
         assertEqual(known.rev, fixture.current[0], "An unchanged stale file created a revision during restart.");
         assertEqual(known.branches.length, 1, "An unchanged stale file created a conflict.");
         assertEqual(known.content, newContent, "The newer DB content was not reflected after suppressing the save.");
@@ -143,6 +176,15 @@ async function main(): Promise<void> {
         assertEqual(recovered.content, oldContent, "Unchanged recovery modified the Vault file.");
         assertEqual(recovered.provenance, fixture.recovered, "Start-up did not recover missing file provenance.");
 
+        const oldHash = `sha256:${createHash("sha256").update(oldContent).digest("hex")}`;
+        const newHash = `sha256:${createHash("sha256").update(newContent).digest("hex")}`;
+        assertEqual(known.contentHash, newHash, "Reflection did not record the incoming bytes.");
+        assertEqual(recovered.contentHash, oldHash, "Recovered provenance lacks verified content evidence.");
+        assertEqual(legacy.rev, fixture.legacy, "Legacy upgrade created a database revision.");
+        assertEqual(legacy.content, oldContent, "Legacy upgrade changed storage.");
+        assertEqual(legacy.contentHash, oldHash, "Legacy provenance was not upgraded after exact verification.");
+        assertEqual(legacy.contentSize, Buffer.byteLength(oldContent), "Legacy provenance has the wrong byte length.");
+
         const receivedRevision = await evalObsidianJson<string>(
             cliBinary,
             `(async()=>{
@@ -204,7 +246,7 @@ async function main(): Promise<void> {
         assertEqual(resolved.branches.length, 1, "The explicit newer-file option did not resolve the conflict.");
         assertEqual(resolved.content, newContent, "The newer-file option did not reflect the newer DB version.");
         console.log(
-            "Stale-file restart: missing unchanged provenance recovered; later update reflected; unknown content preserved; explicit newer-file resolution retained."
+            "Stale-file restart: pruned revision recognised from persisted content evidence; legacy provenance upgraded; missing unchanged provenance recovered; later update reflected; unknown content preserved; explicit newer-file resolution retained."
         );
     } finally {
         if (session) await session.app.stop();