Integrate content evidence for unchanged files

test/e2e-obsidian/README.md

@@ -87,7 +87,7 @@ The wrapper accepts only maintained real-Obsidian scenario names; run it with `-
 
 `folder-batch` needs no remote service. It creates 24 notes and imports three notes with colons in their names into nested folders, reflects the database content into the existing files, then renames and deletes the parent through the Obsidian Vault API. It checks exact paths, descendant events, content, Chunks, deletion markers, provenance, and the absence of unexpected files. A note outside the parent must remain writable. The colon fixtures use the adapter to represent externally created files because Obsidian's Vault creation API rejects those names. The scenario also seeds a database-only colon-named note and verifies that Obsidian's refusal to create it preserves its Metadata without writing a differently named Vault file. It does not establish successful restoration of that absent note or behaviour on other operating systems.
 
-`stale-file-restart` needs no remote service. It advances the local database while old Vault bytes remain, persists pending storage events, and restarts the same isolated Vault and profile. It checks that an unchanged file with exact provenance receives the newer database content without creating a revision, that unknown-origin content is preserved on a fresh independent branch, and that losing provenance and processing the file again does not duplicate or automatically merge that branch. A third file begins with no provenance while its bytes still match the current database revision; start-up must record that revision without creating a new one, and a later incoming revision must reflect without a conflict. The database advances and pending snapshot are controlled fixtures; start-up processing, persistence, file reflection, and conflict checking run in real Obsidian. The scenario does not simulate a mobile operating system suspending the application.
+`stale-file-restart` needs no remote service. It advances the local database while old Vault bytes remain, persists pending storage events, and restarts the same isolated Vault and profile. It checks that an unchanged file whose recorded revision has been pruned receives the newer database content using persisted content evidence without creating a revision, that unknown-origin content is preserved on a fresh independent branch, and that losing provenance and processing the file again does not duplicate or automatically merge that branch. A third file begins with no provenance while its bytes still match the current database revision; start-up must record that revision without creating a new one, and a later incoming revision must reflect without a conflict. A fourth file has legacy revision-only provenance; start-up must verify its exact content and add a hash without creating a database revision or changing storage. The database advances and pending snapshot are controlled fixtures; start-up processing, persistence, file reflection, and conflict checking run in real Obsidian. The scenario does not simulate a mobile operating system suspending the application.
 
 The principal entry points are:
 

test/e2e-obsidian/scripts/review-harness.ts

@@ -204,6 +204,10 @@ async function runAutomaticScenarios(): Promise<void> {
                 .getByText("Passed:", { exact: false })
                 .waitFor({ state: "visible", timeout: uiTimeoutMs });
         }
+        await harness
+            .locator('[data-testid="review-harness-result-file-content-hash-performance"]')
+            .getByText("Not run:", { exact: false })
+            .waitFor({ state: "visible", timeout: uiTimeoutMs });
     });
 }
 
@@ -252,6 +256,52 @@ async function runIdBenchmark(): Promise<void> {
     });
 }
 
+async function runFileContentHashBenchmark(): Promise<void> {
+    await withObsidianPage(obsidianRemoteDebuggingPort(), async (page) => {
+        const snapshotSettings = () =>
+            page.evaluate(() => {
+                const plugin = (globalThis as ReviewHarnessTestGlobal).app?.plugins?.plugins["obsidian-livesync"] as {
+                    core: { services: { setting: { currentSettings(): unknown } } };
+                };
+                return JSON.stringify(plugin.core.services.setting.currentSettings());
+            });
+        const before = await snapshotSettings();
+        const harness = page.locator('[data-testid="review-harness"]');
+        await harness
+            .locator('[data-testid="review-harness-run-file-content-hash-performance"]')
+            .click({ timeout: uiTimeoutMs });
+        const result = harness.locator('[data-testid="review-harness-result-file-content-hash-performance"]');
+        await result.getByText("Passed:", { exact: false }).waitFor({ state: "visible", timeout: uiTimeoutMs * 10 });
+        const observations = await result.locator("li").allTextContents();
+        for (const label of ["4 KiB", "64 KiB", "1 MiB", "50 MiB"]) {
+            for (const algorithm of ["SHA-256", "xxHash64"]) {
+                if (
+                    !observations.some(
+                        (line) => line.startsWith(`${algorithm}, ${label}: median=`) && line.includes("; range=")
+                    )
+                ) {
+                    throw new Error(`Missing file content hash timing and range: ${algorithm}, ${label}`);
+                }
+            }
+        }
+        for (const label of [
+            "Input construction is excluded from timings.",
+            "xxHash64 engine initialisation:",
+            "Warm-up on 4 KiB",
+            "Calls run serially",
+            "JavaScript heap:",
+        ]) {
+            if (!observations.some((line) => line.includes(label))) {
+                throw new Error(`Missing file content hash measurement detail: ${label}`);
+            }
+        }
+        if ((await snapshotSettings()) !== before) throw new Error("The benchmark changed the live settings.");
+        await assertNoHorizontalOverflow(page, harness, { label: "file content hash benchmark results" });
+        const timings = observations.filter((line) => line.startsWith("SHA-256,") || line.startsWith("xxHash64,"));
+        console.log(`File content hash benchmark observations:\n${timings.join("\n")}`);
+    });
+}
+
 async function runVaultFixture(): Promise<string> {
     await withObsidianPage(obsidianRemoteDebuggingPort(), async (page) => {
         await page.locator('[data-testid="review-harness-run-vault-round-trip"]').click({ timeout: uiTimeoutMs });
@@ -409,6 +459,7 @@ async function verifyMobileHarness(): Promise<string> {
             "review-harness-run-full",
             "review-harness-copy-report",
             "review-harness-run-id-generation-performance",
+            "review-harness-run-file-content-hash-performance",
         ]) {
             await assertLocatorHasMinimumTouchTarget(page, harness.locator(`[data-testid="${testId}"]`), {
                 label: testId,
@@ -482,6 +533,7 @@ async function main(): Promise<void> {
         const resumedScreenshot = await restartAndResumeHarness();
         await completeResumedCompatibilityStep();
         await runIdBenchmark();
+        await runFileContentHashBenchmark();
         const report = await copyAndReadReport();
         if (!report.includes("## Self-hosted LiveSync Review Harness report")) {
             throw new Error("The copied Review Harness report was not Markdown evidence.");
@@ -491,6 +543,10 @@ async function main(): Promise<void> {
             "; per ID=",
             "ID key derivation at save time:",
             "JavaScript heap:",
+            "SHA-256, 4 KiB: median=",
+            "xxHash64, 4 KiB: median=",
+            "SHA-256, 50 MiB: median=",
+            "xxHash64, 50 MiB: median=",
         ]) {
             if (!report.includes(expected)) throw new Error(`Missing copied benchmark observation: ${expected}`);
         }

test/e2e-obsidian/scripts/stale-file-restart.ts

@@ -1,3 +1,4 @@
+import { createHash } from "node:crypto";
 import { evalObsidianJson } from "../runner/cli.ts";
 import { discoverObsidianCli, requireObsidianBinary } from "../runner/environment.ts";
 import {
@@ -11,6 +12,7 @@ import { createTemporaryVault } from "../runner/vault.ts";
 
 const paths = ["stale-known.md", "stale-unknown.md"];
 const recoveredPath = "stale-recovered.md";
+const legacyPath = "stale-legacy.md";
 const oldContent = "# Note\nKeep\n\nTail\n\nFooter\n";
 const newContent = oldContent.replace(
     "Footer\n",
@@ -18,7 +20,15 @@ const newContent = oldContent.replace(
 );
 
 type Branch = { rev: string; content: string; history: string[] };
-type FileState = { path: string; content: string; rev: string; branches: Branch[]; provenance: string | null };
+type FileState = {
+    path: string;
+    content: string;
+    rev: string;
+    branches: Branch[];
+    provenance: string | null;
+    contentHash: string | null;
+    contentSize: number | null;
+};
 
 async function readState(cliBinary: string, env: NodeJS.ProcessEnv): Promise<FileState[]> {
     return await evalObsidianJson<FileState[]>(
@@ -27,7 +37,7 @@ async function readState(cliBinary: string, env: NodeJS.ProcessEnv): Promise<Fil
             const core=app.plugins.plugins['obsidian-livesync'].core;
             const store=core.services.keyValueDB.openSimpleStore('file-reflection-provenance-v1');
             const states=[];
-            for(const path of ${JSON.stringify([...paths, recoveredPath])}){
+            for(const path of ${JSON.stringify([...paths, recoveredPath, legacyPath])}){
                 const meta=await core.localDatabase.getDBEntryMeta(path,{conflicts:true},true);
                 const branches=[];
                 for(const rev of [meta._rev,...(meta._conflicts??[])]){
@@ -38,7 +48,8 @@ async function readState(cliBinary: string, env: NodeJS.ProcessEnv): Promise<Fil
                 }
                 const file=app.vault.getAbstractFileByPath(path);
                 states.push({path,content:await app.vault.read(file),rev:meta._rev,branches,
-                    provenance:(await store.get(path))?.revision??null});
+                    provenance:(await store.get(path))?.revision??null,
+                    contentHash:(await store.get(path))?.contentHash??null,contentSize:(await store.get(path))?.contentSize??null});
             }
             return JSON.stringify(states);
         })()`,
@@ -83,16 +94,22 @@ async function main(): Promise<void> {
         await evalObsidianJson(
             cliBinary,
             `(async()=>{
-            for(const path of ${JSON.stringify([...paths, recoveredPath])}) await app.vault.create(path,${JSON.stringify(oldContent)});
+            for(const path of ${JSON.stringify([...paths, recoveredPath, legacyPath])}) await app.vault.create(path,${JSON.stringify(oldContent)});
             return JSON.stringify(true);
         })()`,
             session.cliEnv
         );
-        for (const path of [...paths, recoveredPath]) await waitForLocalDatabaseEntry(cliBinary, session.cliEnv, path);
+        for (const path of [...paths, recoveredPath, legacyPath])
+            await waitForLocalDatabaseEntry(cliBinary, session.cliEnv, path);
 
         // Drain real Vault events before creating a persisted pending-event fixture.
         // The DB advances without reflecting it in the Vault, as on an offline device.
-        const fixture = await evalObsidianJson<{ current: string[]; original: string[]; recovered: string }>(
+        const fixture = await evalObsidianJson<{
+            current: string[];
+            original: string[];
+            recovered: string;
+            legacy: string;
+        }>(
             cliBinary,
             `(async()=>{
             const core=app.plugins.plugins['obsidian-livesync'].core;
@@ -106,16 +123,32 @@ async function main(): Promise<void> {
                 const result=await core.localDatabase.putDBEntry({...meta,data,mtime:file.stat.mtime+60000,
                     size:data.size,children:[]},false,meta._rev);
                 if(!result?.ok) throw new Error('Could not advance '+path);
-                current.push(result.rev); original.push(meta._rev);
-                if(index===0) await store.set(path,{revision:meta._rev,observedStorageMtime:file.stat.mtime});
-                else await store.delete(path);
+                let currentRev=result.rev;
+                if(index===0){
+                    const record=await store.get(path);
+                    if(record?.revision!==meta._rev || !record.contentHash) throw new Error('Initial save did not record content evidence');
+                    // Advance beyond the local retention limit without reflecting the incoming bytes.
+                    let raw=await core.localDatabase.getRaw(meta._id,{rev:currentRev});
+                    for(let update=0;update<100;update++){
+                        const next=await core.localDatabase.putRaw({...raw,mtime:raw.mtime+1});
+                        currentRev=next.rev; raw={...raw,_rev:currentRev,mtime:raw.mtime+1};
+                    }
+                    let pruned=false;
+                    try { await core.localDatabase.getRaw(meta._id,{rev:meta._rev}); }
+                    catch(error) { if(error.status!==404) throw error; pruned=true; }
+                    if(!pruned) throw new Error('The recorded revision was not pruned');
+                } else await store.delete(path);
+                current.push(currentRev); original.push(meta._rev);
                 snapshot.push({type:'CHANGED',key:'CHANGED-'+path,args:{file}});
             }
             const recovered=${JSON.stringify(recoveredPath)};
             const recoveredMeta=await core.localDatabase.getDBEntryMeta(recovered,{},true);
             await store.delete(recovered);
+            const legacy=${JSON.stringify(legacyPath)};
+            const legacyMeta=await core.localDatabase.getDBEntryMeta(legacy,{},true);
+            await store.set(legacy,{revision:legacyMeta._rev});
             await core.kvDB.set('storage-event-manager-snapshot',snapshot);
-            return JSON.stringify({current,original,recovered:recoveredMeta._rev});
+            return JSON.stringify({current,original,recovered:recoveredMeta._rev,legacy:legacyMeta._rev});
         })()`,
             session.cliEnv
         );
@@ -124,7 +157,7 @@ async function main(): Promise<void> {
 
         session = await startObsidianLiveSyncSession({ binary, cliBinary, vault });
         await waitForLiveSyncCoreReady(cliBinary, session.cliEnv);
-        const [known, unknown, recovered] = await readState(cliBinary, session.cliEnv);
+        const [known, unknown, recovered, legacy] = await readState(cliBinary, session.cliEnv);
         assertEqual(known.rev, fixture.current[0], "An unchanged stale file created a revision during restart.");
         assertEqual(known.branches.length, 1, "An unchanged stale file created a conflict.");
         assertEqual(known.content, newContent, "The newer DB content was not reflected after suppressing the save.");
@@ -143,6 +176,15 @@ async function main(): Promise<void> {
         assertEqual(recovered.content, oldContent, "Unchanged recovery modified the Vault file.");
         assertEqual(recovered.provenance, fixture.recovered, "Start-up did not recover missing file provenance.");
 
+        const oldHash = `sha256:${createHash("sha256").update(oldContent).digest("hex")}`;
+        const newHash = `sha256:${createHash("sha256").update(newContent).digest("hex")}`;
+        assertEqual(known.contentHash, newHash, "Reflection did not record the incoming bytes.");
+        assertEqual(recovered.contentHash, oldHash, "Recovered provenance lacks verified content evidence.");
+        assertEqual(legacy.rev, fixture.legacy, "Legacy upgrade created a database revision.");
+        assertEqual(legacy.content, oldContent, "Legacy upgrade changed storage.");
+        assertEqual(legacy.contentHash, oldHash, "Legacy provenance was not upgraded after exact verification.");
+        assertEqual(legacy.contentSize, Buffer.byteLength(oldContent), "Legacy provenance has the wrong byte length.");
+
         const receivedRevision = await evalObsidianJson<string>(
             cliBinary,
             `(async()=>{
@@ -204,7 +246,7 @@ async function main(): Promise<void> {
         assertEqual(resolved.branches.length, 1, "The explicit newer-file option did not resolve the conflict.");
         assertEqual(resolved.content, newContent, "The newer-file option did not reflect the newer DB version.");
         console.log(
-            "Stale-file restart: missing unchanged provenance recovered; later update reflected; unknown content preserved; explicit newer-file resolution retained."
+            "Stale-file restart: pruned revision recognised from persisted content evidence; legacy provenance upgraded; missing unchanged provenance recovered; later update reflected; unknown content preserved; explicit newer-file resolution retained."
         );
     } finally {
         if (session) await session.app.stop();