Merge origin/main into time-bound Setup URI branch

test/README.md

@@ -122,4 +122,4 @@ Each simulated device owns a separate real PouchDB database, `LiveSyncLocalDB` m
 
 This is a service integration test, not a CLI subprocess or Obsidian runtime test. The file and provenance stores are in-memory fixtures; automatic conflict callbacks are observed without running interactive or automatic merge policies. PouchDB revision creation, chunk storage and retrieval, local compaction, CouchDB replication, the CLI command dispatcher, and its resolution operations are real.
 
-Use the CLI and real-Obsidian procedures linked above for argument parsing, persistent host stores, file watchers, and dialogues. In particular, the real-Obsidian `stale-file-restart` scenario exercises persisted pending events and restart, and `folder-batch` exercises bulk Vault rename and deletion. These scenarios do not simulate a mobile operating system suspending the application.
+Use the CLI and real-Obsidian procedures linked above for argument parsing, persistent host stores, file watchers, and dialogues. In particular, the real-Obsidian `stale-file-restart` scenario exercises persisted pending events and restart. It checks that start-up records an unchanged file whose provenance is missing, then accepts a newer database revision without a conflict, while an unknown historical file remains protected. The `folder-batch` scenario exercises bulk Vault rename and deletion. These scenarios do not simulate a mobile operating system suspending the application.

test/e2e-obsidian/README.md

@@ -82,9 +82,9 @@ npm run test:e2e:obsidian:focused -- security-seed-reconnect
 
 The wrapper accepts only maintained real-Obsidian scenario names; run it with `--help` for the current list. It deliberately does not manage CouchDB, Object Storage, or the P2P signalling relay. Start the required fixture first, or use the complete service-managed suite.
 
-`folder-batch` needs no remote service. It creates 24 notes in nested folders, renames and deletes the parent through the Obsidian Vault API, and checks descendant events, content, Chunks, deletion markers, and provenance. A note outside the parent must remain writable.
+`folder-batch` needs no remote service. It creates 24 notes and imports three notes with colons in their names into nested folders, reflects the database content into the existing files, then renames and deletes the parent through the Obsidian Vault API. It checks exact paths, descendant events, content, Chunks, deletion markers, provenance, and the absence of unexpected files. A note outside the parent must remain writable. The colon fixtures use the adapter to represent externally created files because Obsidian's Vault creation API rejects those names. The scenario also seeds a database-only colon-named note and verifies that Obsidian's refusal to create it preserves its Metadata without writing a differently named Vault file. It does not establish successful restoration of that absent note or behaviour on other operating systems.
 
-`stale-file-restart` needs no remote service. It advances the local database while old Vault bytes remain, persists pending storage events, and restarts the same isolated Vault and profile. It checks that an unchanged file with exact provenance receives the newer database content without creating a revision, that unknown-origin content is preserved on a fresh independent branch, and that losing provenance and processing the file again does not duplicate or automatically merge that branch. The database advance and pending snapshot are controlled fixtures; startup processing, persistence, file reflection, and conflict checking run in real Obsidian. The scenario does not simulate a mobile operating system suspending the application.
+`stale-file-restart` needs no remote service. It advances the local database while old Vault bytes remain, persists pending storage events, and restarts the same isolated Vault and profile. It checks that an unchanged file with exact provenance receives the newer database content without creating a revision, that unknown-origin content is preserved on a fresh independent branch, and that losing provenance and processing the file again does not duplicate or automatically merge that branch. A third file begins with no provenance while its bytes still match the current database revision; start-up must record that revision without creating a new one, and a later incoming revision must reflect without a conflict. The database advances and pending snapshot are controlled fixtures; start-up processing, persistence, file reflection, and conflict checking run in real Obsidian. The scenario does not simulate a mobile operating system suspending the application.
 
 The principal entry points are:
 

test/e2e-obsidian/scripts/folder-batch.ts

@@ -8,11 +8,17 @@ process.env.E2E_OBSIDIAN_CLI_TIMEOUT_MS ??= "60000";
 const originalRoot = "batch/original";
 const renamedRoot = "batch/renamed";
 const outsidePath = "batch/outside.md";
+const missingColonPath = "batch/incoming/Poem: Example.md";
 const folders = ["alpha", "alpha/deep", "beta"];
 const notes = Array.from({ length: 24 }, (_, index) => ({
     relativePath: `${folders[index % folders.length]}/note-${index}.md`,
     body: `# Descendant ${index}\n\nThis body must survive a parent folder rename.\n`,
 }));
+notes.push(
+    { relativePath: "alpha/Poem: Example.md", body: "First poem\n" },
+    { relativePath: "beta/Poem: Example.md", body: "Second poem\n" },
+    { relativePath: "alpha/deep/Poem: Part: Example.md", body: "Poem with multiple colons\n" }
+);
 
 async function main(): Promise<void> {
     const binary = requireObsidianBinary();
@@ -45,7 +51,7 @@ async function main(): Promise<void> {
             localStorageEntries: createE2eObsidianDeviceLocalState(vault.name),
         });
         await waitForLiveSyncCoreReady(cliBinary, session.cliEnv);
-        const result = await evalObsidianJson<{ descendants: number; renamed: number; deleted: number }>(
+        const result = await evalObsidianJson<{ descendants: number; renamed: number; deleted: number; missingRejected: boolean }>(
             cliBinary,
             `(async()=>{
                 const core=app.plugins.plugins['obsidian-livesync'].core;
@@ -54,6 +60,7 @@ async function main(): Promise<void> {
                 const originalRoot=${JSON.stringify(originalRoot)};
                 const renamedRoot=${JSON.stringify(renamedRoot)};
                 const outsidePath=${JSON.stringify(outsidePath)};
+                const missingColonPath=${JSON.stringify(missingColonPath)};
                 const renamed=new Set(), deleted=new Set();
                 const refs=[
                     app.vault.on('rename',(file,oldPath)=>{
@@ -112,11 +119,23 @@ async function main(): Promise<void> {
                     await app.vault.createFolder(originalRoot);
                     for(const folder of ${JSON.stringify(folders)})
                         await app.vault.createFolder(originalRoot+'/'+folder);
-                    await Promise.all(notes.map(note=>app.vault.create(originalRoot+'/'+note.relativePath,note.body)));
+                    // Obsidian indexes imported colon names but rejects them in Vault.create.
+                    await Promise.all(notes.map(note=>note.relativePath.includes(':')
+                        ? app.vault.adapter.write(originalRoot+'/'+note.relativePath,note.body)
+                        : app.vault.create(originalRoot+'/'+note.relativePath,note.body)));
                     await app.vault.create(outsidePath,'Outside note');
                     await waitFor('Initial batch',async()=>[
                         ...await liveBatch(originalRoot), ...await liveErrors(outsidePath,'Outside note'),
                     ]);
+                    for(const note of notes){
+                        const path=originalRoot+'/'+note.relativePath;
+                        if(!await core.serviceModules.fileHandler.dbToStorage(await meta(path),null,true))
+                            throw new Error('Database reflection failed: '+path);
+                    }
+                    await waitFor('Reflected batch',()=>liveBatch(originalRoot));
+                    const expectedPaths=new Set([outsidePath,...notes.map(note=>originalRoot+'/'+note.relativePath)]);
+                    const unexpected=app.vault.getFiles().map(file=>file.path).filter(path=>!expectedPaths.has(path));
+                    if(unexpected.length) throw new Error('Unexpected reflected files: '+unexpected.join(', '));
                     const originalIds=await Promise.all(notes.map(async note=>(await meta(originalRoot+'/'+note.relativePath))._id));
 
                     // Rename the parent once: Obsidian must emit every descendant event.
@@ -144,7 +163,40 @@ async function main(): Promise<void> {
                     if(app.vault.getAbstractFileByPath(renamedRoot)) throw new Error('Deleted folder remains');
                     await app.vault.modify(app.vault.getAbstractFileByPath(outsidePath),'Outside note updated');
                     await waitFor('Outside update',()=>liveErrors(outsidePath,'Outside note updated'));
-                    return JSON.stringify({descendants:notes.length,renamed:renamed.size,deleted:deleted.size});
+
+                    // A received database entry must not create a different Vault file when Obsidian rejects its name.
+                    const filesBefore=new Set(app.vault.getFiles().map(file=>file.path));
+                    const incomingBody='Received colon note\\n';
+                    const incomingData=new Blob([incomingBody],{type:'text/plain'});
+                    const incomingId=await core.services.path.path2id(missingColonPath);
+                    const incomingTime=Date.now();
+                    const saved=await core.localDatabase.putDBEntry({
+                        _id:incomingId,path:missingColonPath,data:incomingData,
+                        ctime:incomingTime,mtime:incomingTime,size:incomingData.size,
+                        children:[],datatype:'plain',type:'plain',eden:{},
+                    });
+                    if(!saved?.ok) throw new Error('Could not seed received Metadata: '+missingColonPath);
+                    const incomingMeta=await meta(missingColonPath);
+                    if(!incomingMeta || incomingMeta._id!==incomingId || incomingMeta.path!==missingColonPath)
+                        throw new Error('Received Metadata has the wrong path: '+missingColonPath);
+                    const incomingEntry=await core.localDatabase.getDBEntry(missingColonPath,{rev:incomingMeta._rev},false,true,true);
+                    if(!incomingEntry || getContent(incomingEntry)!==incomingBody)
+                        throw new Error('Received content could not be read: '+missingColonPath);
+                    let creationFailure='';
+                    try{
+                        const reflected=await core.serviceModules.fileHandler.dbToStorage(incomingMeta,null,true);
+                        if(reflected) throw new Error('Obsidian unexpectedly created: '+missingColonPath);
+                    }catch(error){
+                        creationFailure=String(error);
+                        if(!creationFailure.includes('File name cannot contain')) throw error;
+                    }
+                    if(!creationFailure) throw new Error('Missing name rejection: '+missingColonPath);
+                    const filesAfter=app.vault.getFiles().map(file=>file.path);
+                    const newFiles=filesAfter.filter(path=>!filesBefore.has(path));
+                    if(newFiles.length) throw new Error('Received note was written under another name: '+newFiles.join(', '));
+                    if((await meta(missingColonPath))?.path!==missingColonPath)
+                        throw new Error('Received Metadata changed after rejection: '+missingColonPath);
+                    return JSON.stringify({descendants:notes.length,renamed:renamed.size,deleted:deleted.size,missingRejected:true});
                 }finally{
                     for(const ref of refs) app.vault.offref(ref);
                 }
@@ -153,7 +205,8 @@ async function main(): Promise<void> {
         );
         console.log(
             `Folder batch: ${result.descendants} descendants persisted, renamed, and deleted; ` +
-                `${result.renamed} rename and ${result.deleted} delete events observed; outside note remained writable.`
+                `${result.renamed} rename and ${result.deleted} delete events observed; outside note remained writable; ` +
+                `missing colon note rejected without an alternate file: ${result.missingRejected}.`
         );
     } finally {
         if (session) await session.app.stop();

test/e2e-obsidian/scripts/stale-file-restart.ts

@@ -10,6 +10,7 @@ import { startObsidianLiveSyncSession, type ObsidianLiveSyncSession } from "../r
 import { createTemporaryVault } from "../runner/vault.ts";
 
 const paths = ["stale-known.md", "stale-unknown.md"];
+const recoveredPath = "stale-recovered.md";
 const oldContent = "# Note\nKeep\n\nTail\n\nFooter\n";
 const newContent = oldContent.replace(
     "Footer\n",
@@ -26,7 +27,7 @@ async function readState(cliBinary: string, env: NodeJS.ProcessEnv): Promise<Fil
             const core=app.plugins.plugins['obsidian-livesync'].core;
             const store=core.services.keyValueDB.openSimpleStore('file-reflection-provenance-v1');
             const states=[];
-            for(const path of ${JSON.stringify(paths)}){
+            for(const path of ${JSON.stringify([...paths, recoveredPath])}){
                 const meta=await core.localDatabase.getDBEntryMeta(path,{conflicts:true},true);
                 const branches=[];
                 for(const rev of [meta._rev,...(meta._conflicts??[])]){
@@ -82,16 +83,16 @@ async function main(): Promise<void> {
         await evalObsidianJson(
             cliBinary,
             `(async()=>{
-            for(const path of ${JSON.stringify(paths)}) await app.vault.create(path,${JSON.stringify(oldContent)});
+            for(const path of ${JSON.stringify([...paths, recoveredPath])}) await app.vault.create(path,${JSON.stringify(oldContent)});
             return JSON.stringify(true);
         })()`,
             session.cliEnv
         );
-        for (const path of paths) await waitForLocalDatabaseEntry(cliBinary, session.cliEnv, path);
+        for (const path of [...paths, recoveredPath]) await waitForLocalDatabaseEntry(cliBinary, session.cliEnv, path);
 
         // Drain real Vault events before creating a persisted pending-event fixture.
         // The DB advances without reflecting it in the Vault, as on an offline device.
-        const fixture = await evalObsidianJson<{ current: string[]; original: string[] }>(
+        const fixture = await evalObsidianJson<{ current: string[]; original: string[]; recovered: string }>(
             cliBinary,
             `(async()=>{
             const core=app.plugins.plugins['obsidian-livesync'].core;
@@ -110,8 +111,11 @@ async function main(): Promise<void> {
                 else await store.delete(path);
                 snapshot.push({type:'CHANGED',key:'CHANGED-'+path,args:{file}});
             }
+            const recovered=${JSON.stringify(recoveredPath)};
+            const recoveredMeta=await core.localDatabase.getDBEntryMeta(recovered,{},true);
+            await store.delete(recovered);
             await core.kvDB.set('storage-event-manager-snapshot',snapshot);
-            return JSON.stringify({current,original});
+            return JSON.stringify({current,original,recovered:recoveredMeta._rev});
         })()`,
             session.cliEnv
         );
@@ -120,7 +124,7 @@ async function main(): Promise<void> {
 
         session = await startObsidianLiveSyncSession({ binary, cliBinary, vault });
         await waitForLiveSyncCoreReady(cliBinary, session.cliEnv);
-        const [known, unknown] = await readState(cliBinary, session.cliEnv);
+        const [known, unknown, recovered] = await readState(cliBinary, session.cliEnv);
         assertEqual(known.rev, fixture.current[0], "An unchanged stale file created a revision during restart.");
         assertEqual(known.branches.length, 1, "An unchanged stale file created a conflict.");
         assertEqual(known.content, newContent, "The newer DB content was not reflected after suppressing the save.");
@@ -134,6 +138,32 @@ async function main(): Promise<void> {
         if (!unknown.branches.some((branch) => branch.content === newContent)) {
             throw new Error("The remote additions were lost.");
         }
+        assertEqual(recovered.rev, fixture.recovered, "Unchanged recovery added a database revision.");
+        assertEqual(recovered.branches.length, 1, "Unchanged recovery created a conflict.");
+        assertEqual(recovered.content, oldContent, "Unchanged recovery modified the Vault file.");
+        assertEqual(recovered.provenance, fixture.recovered, "Start-up did not recover missing file provenance.");
+
+        const receivedRevision = await evalObsidianJson<string>(
+            cliBinary,
+            `(async()=>{
+                const core=app.plugins.plugins['obsidian-livesync'].core;
+                const path=${JSON.stringify(recoveredPath)};
+                const meta=await core.localDatabase.getDBEntryMeta(path,{},true);
+                const file=await core.storageAccess.getFileStub(path);
+                const data=new Blob([${JSON.stringify(newContent)}],{type:'text/plain'});
+                const result=await core.localDatabase.putDBEntry({...meta,data,mtime:file.stat.mtime+60000,
+                    size:data.size,children:[]},false,meta._rev);
+                if(!result?.ok) throw new Error('Could not advance '+path);
+                if(!await core.fileHandler.dbToStorage(path,path,false)) throw new Error('Could not reflect '+path);
+                return JSON.stringify(result.rev);
+            })()`,
+            session.cliEnv
+        );
+        const [, , received] = await readState(cliBinary, session.cliEnv);
+        assertEqual(received.rev, receivedRevision, "Incoming revision was not retained.");
+        assertEqual(received.branches.length, 1, "Incoming revision created a conflict after recovery.");
+        assertEqual(received.content, newContent, "Incoming revision was not reflected after recovery.");
+        assertEqual(received.provenance, receivedRevision, "Incoming reflection did not record its revision.");
 
         await evalObsidianJson(
             cliBinary,
@@ -174,7 +204,7 @@ async function main(): Promise<void> {
         assertEqual(resolved.branches.length, 1, "The explicit newer-file option did not resolve the conflict.");
         assertEqual(resolved.content, newContent, "The newer-file option did not reflect the newer DB version.");
         console.log(
-            "Stale-file restart: known content reflected; unknown content preserved without duplicate branches; explicit newer-file resolution retained."
+            "Stale-file restart: missing unchanged provenance recovered; later update reflected; unknown content preserved; explicit newer-file resolution retained."
         );
     } finally {
         if (session) await session.app.stop();