Integrate opt-in NFC path identity and combined regression coverage

test/README.md

@@ -116,7 +116,7 @@ Each simulated device owns a separate real PouchDB database, `LiveSyncLocalDB` m
 | Four editing devices | Repeat the same sequence with four independently edited branches. | All four contents are initially preserved, and the same unchanged-save and convergence guarantees hold. |
 | A genuine edit on a losing device | After three-way resolution reaches its DB, a device adds content to its still-unreflected losing file, then saves and replicates it. | The new revision extends that device's recorded branch. The selected result and the new edit remain readable on every replica. |
 | Missing provenance | After four-way resolution, remove a losing device's provenance record and set its file to the original historical ancestor's content. Save, remove provenance again, and repeat the save. | The file becomes one fresh independent root, distinct from the historical root. Both contents remain readable after replication, and repeated saves create no duplicates. CLI `resolve` can select the independent root and propagate its resolution. |
-| Compacted base with verified unchanged content | After four-way resolution, retain a losing device's content hash, compact its local DB, and confirm that its recorded revision body is unavailable. Save the unchanged file. | The selected content is reflected without adding a database revision or conflict, and all replicas retain only the selected branch. |
+| Compacted base with verified unchanged content | After four-way resolution, retain a losing device's content hash, compact its local DB, and confirm that its recorded revision body is unavailable. Process the unchanged file twice. | The selected content is reflected without adding a database revision or conflict, provenance advances to that revision, and all replicas retain only the selected branch. |
 | Compacted base with legacy provenance | After four-way resolution, retain only the losing device's revision and observed modification time, compact its local DB, and confirm that the recorded revision body is unavailable. Save the remaining file, then repeat after removing provenance. | The file is preserved as one independent conflict. Both contents remain readable on every replica, repeated saves create no duplicates, and CLI `resolve` can select the independent root and propagate its resolution. |
 | Compacted base with a genuine edit | After four-way resolution and local compaction, retain the losing device's content hash but edit its file without changing its byte length or modification time. | The changed content is preserved as one independent conflict, remains readable on every replica, and can be selected through CLI `resolve`. Repeated saves after provenance loss create no duplicates. |
 

test/e2e-obsidian/README.md

@@ -170,6 +170,10 @@ Set `E2E_OBSIDIAN_ONLY_DIFFERENT_CHUNK_ID_KEYS=true` to run the focused case whe
 
 `test:e2e:obsidian:cli-to-obsidian-sync` is the cross-runtime compatibility check for the official LiveSync CLI and the real Obsidian plug-in. Build the plug-in first, and build the local CLI too when no external CLI command is selected. The script uses E2EE, Path Obfuscation, and the current preferred chunk settings to create and synchronise a note through the CLI, starts real Obsidian with an isolated Vault and profile, synchronises the same CouchDB database, and verifies that the plug-in materialises identical note content. This covers the boundary that CLI-only and plug-in-only round trips do not exercise.
 
+Set `E2E_OBSIDIAN_UNICODE_PATHS=true` to exercise path identity version 1 on Linux. This variant uses an NFD directory and a mixed NFC/NFD file name, checks reflection at Obsidian's NFC path, and returns an update and deletion to the original CLI physical path. Updates use `sync` and `mirror`; deletions use the polling daemon, because `mirror` retains tombstoned local files by default. It uses unencrypted Metadata by default to match issue #1259; add `E2E_OBSIDIAN_UNICODE_ENCRYPTION=true` for the encrypted variant. These checks do not establish native macOS or iOS filesystem behaviour.
+
+Set `E2E_OBSIDIAN_UNICODE_MIGRATION=true` to begin with two CLI databases using legacy path identity. The source runs `rebuild --force --write-settings` after selecting version 1, and the second client runs `fetch --force --write-settings`. The scenario checks that the old document ID is replaced, the NFC feature is declared, and both physical NFD filenames are preserved before continuing the real Obsidian update and deletion checks.
+
 The isolated Obsidian session starts with its CouchDB settings and device-local compatibility acknowledgement already in place. This keeps the scenario focused on cross-runtime data compatibility; unconfigured start-up and visible CouchDB onboarding are covered by their dedicated workflows.
 
 By default, the compatibility check runs `node src/apps/cli/dist/index.cjs`. Set `LIVESYNC_CLI_COMMAND` to test another CLI build or distribution. The value may be a quoted command line or a JSON array of executable and prefix arguments; the scenario arguments are appended without going through a shell.

test/e2e-obsidian/scripts/cli-to-obsidian-sync.ts

@@ -28,7 +28,11 @@ process.env.E2E_OBSIDIAN_COUCHDB_TIMEOUT_MS ??= "30000";
 process.env.E2E_OBSIDIAN_FILE_TIMEOUT_MS ??= "30000";
 
 const liveSyncCli = resolve("src/apps/cli/dist/index.cjs");
-const notePath = "E2E/cli-to-obsidian.md";
+const migrateUnicodePaths = process.env.E2E_OBSIDIAN_UNICODE_MIGRATION === "true";
+const unicodePaths = migrateUnicodePaths || process.env.E2E_OBSIDIAN_UNICODE_PATHS === "true";
+const useEncryption = !unicodePaths || process.env.E2E_OBSIDIAN_UNICODE_ENCRYPTION === "true";
+const notePath = unicodePaths ? "Cafe\u0301/r\u00e9sume\u0301.md" : "E2E/cli-to-obsidian.md";
+const receivedNotePath = unicodePaths ? notePath.normalize("NFC") : notePath;
 const noteContent = [
     "# CLI to real Obsidian",
     "",
@@ -183,7 +187,8 @@ async function configureLiveSyncCli(
     command: LiveSyncCliCommand,
     settingsPath: string,
     couchDb: Awaited<ReturnType<typeof loadCouchDbConfig>>,
-    dbName: string
+    dbName: string,
+    pathNormalizationVersion = unicodePaths ? 1 : 0
 ): Promise<void> {
     await runLiveSyncCli(command, ["init-settings", "--force", settingsPath]);
     const settings = JSON.parse(await readFile(settingsPath, "utf-8")) as Record<string, unknown>;
@@ -210,16 +215,61 @@ async function configureLiveSyncCli(
         handleFilenameCaseSensitive: false,
         doNotUseFixedRevisionForChunks: true,
         E2EEAlgorithm: "v2",
-        encrypt: true,
+        encrypt: useEncryption,
         passphrase: e2eePassphrase,
-        usePathObfuscation: true,
-        encryptInternalMetadata: true,
+        usePathObfuscation: useEncryption,
+        encryptInternalMetadata: useEncryption,
+        pathNormalizationVersion,
         doctorProcessedVersion: "0.25.27",
         isConfigured: true,
     });
     await writeFile(settingsPath, `${JSON.stringify(settings, null, 2)}\n`, "utf-8");
 }
 
+async function startLiveSyncCliDaemon(command: LiveSyncCliCommand, databasePath: string, settingsPath: string) {
+    const child = spawn(
+        command.executable,
+        [...command.prefixArgs, databasePath, "--settings", settingsPath, "--interval", "1", "daemon"],
+        { cwd: process.cwd(), env: process.env, stdio: ["ignore", "pipe", "pipe"] }
+    );
+    let output = "";
+    let spawnError: Error | undefined;
+    child.stdout.on("data", (chunk: Buffer) => {
+        output += chunk.toString("utf-8");
+    });
+    child.stderr.on("data", (chunk: Buffer) => {
+        output += chunk.toString("utf-8");
+    });
+    child.on("error", (error) => {
+        spawnError = error;
+    });
+    const stop = async () => {
+        if (child.exitCode !== null || child.signalCode !== null || spawnError) return;
+        await new Promise<void>((resolvePromise) => {
+            const timeout = setTimeout(() => child.kill("SIGKILL"), 10000);
+            child.once("exit", () => {
+                clearTimeout(timeout);
+                resolvePromise();
+            });
+            child.kill("SIGTERM");
+        });
+    };
+    const deadline = Date.now() + Number(process.env.E2E_LIVESYNC_CLI_TIMEOUT_MS ?? 60000);
+    try {
+        while (!output.includes("[Daemon] Polling mode:")) {
+            if (spawnError) throw spawnError;
+            if (child.exitCode !== null || child.signalCode !== null || Date.now() >= deadline) {
+                throw new Error(`The CLI daemon did not become ready.\n${output}`);
+            }
+            await new Promise((resolvePromise) => setTimeout(resolvePromise, 100));
+        }
+        return { stop, getOutput: () => output };
+    } catch (error) {
+        await stop();
+        throw error;
+    }
+}
+
 async function writeCliNote(
     command: LiveSyncCliCommand,
     databasePath: string,
@@ -280,6 +330,86 @@ async function waitForVaultContent(
     throw new Error(`Timed out waiting for CLI-created note at ${fullPath}. Last content:\n${lastContent}`);
 }
 
+async function migrateCliDatabases(
+    command: LiveSyncCliCommand,
+    cliState: string,
+    databasePath: string,
+    settingsPath: string,
+    couchDb: Awaited<ReturnType<typeof loadCouchDbConfig>>,
+    dbName: string,
+    legacyFileInfo: CliFileInfo
+): Promise<CliFileInfo> {
+    const secondaryDatabasePath = join(cliState, "secondary-database");
+    const secondarySettingsPath = join(cliState, "secondary-settings.json");
+    await mkdir(secondaryDatabasePath, { recursive: true });
+    await configureLiveSyncCli(command, secondarySettingsPath, couchDb, dbName, 0);
+    await runLiveSyncCli(command, [secondaryDatabasePath, "--settings", secondarySettingsPath, "sync"]);
+    await runLiveSyncCli(command, [secondaryDatabasePath, "--settings", secondarySettingsPath, "mirror"]);
+    assertEqual(
+        await readFile(join(secondaryDatabasePath, notePath), "utf-8"),
+        editedNoteContent,
+        "The secondary CLI did not materialise the legacy NFD note before migration."
+    );
+
+    for (const path of [settingsPath, secondarySettingsPath]) {
+        const settings = JSON.parse(await readFile(path, "utf-8"));
+        settings.pathNormalizationVersion = 1;
+        await writeFile(path, `${JSON.stringify(settings, null, 2)}\n`, "utf-8");
+    }
+    await runLiveSyncCli(command, [databasePath, "--settings", settingsPath, "rebuild", "--force", "--write-settings"]);
+    const info = await runLiveSyncCli(command, [databasePath, "--settings", settingsPath, "info", notePath]);
+    const fileInfo = JSON.parse(info.stdout) as CliFileInfo;
+    if (fileInfo.id === legacyFileInfo.id) {
+        throw new Error("Rebuild retained the legacy NFD document ID.");
+    }
+    await waitForCouchDbDocs(couchDb, dbName, (docs) => {
+        const ids = new Set(docs.map((doc) => doc._id));
+        return (
+            !ids.has(legacyFileInfo.id) &&
+            ids.has(fileInfo.id) &&
+            fileInfo.children.every((childId) => ids.has(childId)) &&
+            docs.some((doc) => Array.isArray(doc.used_features) && doc.used_features.includes("nfc-path-identity-v1"))
+        );
+    });
+    await runLiveSyncCli(command, [
+        secondaryDatabasePath,
+        "--settings",
+        secondarySettingsPath,
+        "fetch",
+        "--force",
+        "--write-settings",
+    ]);
+    const secondaryInfo = await runLiveSyncCli(command, [
+        secondaryDatabasePath,
+        "--settings",
+        secondarySettingsPath,
+        "info",
+        notePath,
+    ]);
+    assertEqual(
+        (JSON.parse(secondaryInfo.stdout) as CliFileInfo).id,
+        fileInfo.id,
+        "Fetch did not replace the secondary CLI's legacy document identity."
+    );
+    for (const path of [databasePath, secondaryDatabasePath]) {
+        assertEqual(
+            await readFile(join(path, notePath), "utf-8"),
+            editedNoteContent,
+            "Migration changed the content of the original NFD physical file."
+        );
+        await access(join(path, receivedNotePath)).then(
+            () => {
+                throw new Error("Migration created a second NFC physical file.");
+            },
+            (error: NodeJS.ErrnoException) => {
+                if (error.code !== "ENOENT") throw error;
+            }
+        );
+    }
+    console.log("CLI Rebuild and Fetch migrated legacy document IDs while preserving both NFD physical files.");
+    return { ...fileInfo, expectedCtime: legacyFileInfo.expectedCtime };
+}
+
 async function main(): Promise<void> {
     const liveSyncCliCommand = resolveLiveSyncCliCommand();
     if (process.env.LIVESYNC_CLI_COMMAND === undefined) {
@@ -304,12 +434,19 @@ async function main(): Promise<void> {
     const cliSourcePath = join(cliState, "source", "cli-to-obsidian.md");
     const vault = await createTemporaryVault();
     let session: ObsidianLiveSyncSession | undefined;
+    let daemon: Awaited<ReturnType<typeof startLiveSyncCliDaemon>> | undefined;
 
     try {
         await assertCouchDbReachable(couchDb);
         await createCouchDbDatabase(couchDb, dbName);
         await mkdir(cliDatabasePath, { recursive: true });
-        await configureLiveSyncCli(liveSyncCliCommand, cliSettingsPath, couchDb, dbName);
+        await configureLiveSyncCli(
+            liveSyncCliCommand,
+            cliSettingsPath,
+            couchDb,
+            dbName,
+            migrateUnicodePaths ? 0 : undefined
+        );
 
         if (process.env.LIVESYNC_CLI_COMMAND === undefined) {
             console.log(`Using locally built LiveSync CLI: ${liveSyncCli}`);
@@ -323,7 +460,21 @@ async function main(): Promise<void> {
         console.log(`Temporary Obsidian vault: ${vault.path}`);
         console.log(`Temporary CouchDB database: ${dbName}`);
 
-        const cliFileInfo = await writeCliNote(liveSyncCliCommand, cliDatabasePath, cliSettingsPath, cliSourcePath);
+        let cliFileInfo = await writeCliNote(liveSyncCliCommand, cliDatabasePath, cliSettingsPath, cliSourcePath);
+        if (migrateUnicodePaths) {
+            cliFileInfo = await migrateCliDatabases(
+                liveSyncCliCommand,
+                cliState,
+                cliDatabasePath,
+                cliSettingsPath,
+                couchDb,
+                dbName,
+                cliFileInfo
+            );
+        }
+        if (unicodePaths && !useEncryption) {
+            assertEqual(cliFileInfo.id, receivedNotePath.toLowerCase(), "The CLI did not derive an NFC document ID.");
+        }
         await waitForCouchDbDocs(couchDb, dbName, (docs) => {
             const ids = new Set(docs.map((doc) => doc._id));
             return ids.has(cliFileInfo.id) && cliFileInfo.children.every((childId) => ids.has(childId));
@@ -342,10 +493,11 @@ async function main(): Promise<void> {
                     dbName,
                 },
                 {
-                    encrypt: true,
+                    encrypt: useEncryption,
                     passphrase: e2eePassphrase,
-                    usePathObfuscation: true,
-                    encryptInternalMetadata: true,
+                    usePathObfuscation: useEncryption,
+                    encryptInternalMetadata: useEncryption,
+                    pathNormalizationVersion: unicodePaths ? 1 : 0,
                     E2EEAlgorithm: "v2",
                 }
             ),
@@ -355,7 +507,7 @@ async function main(): Promise<void> {
         await prepareRemote(obsidianCli.binary, session.cliEnv);
         await pushLocalChanges(obsidianCli.binary, session.cliEnv);
 
-        const received = await waitForVaultContent(vault.path, notePath);
+        const received = await waitForVaultContent(vault.path, receivedNotePath);
         assertEqual(received, editedNoteContent, "The real Obsidian plug-in did not materialise the CLI-edited note.");
         if (cliFileInfo.expectedCtime !== undefined) {
             const receivedCtime = await evalObsidianJson<number>(
@@ -363,7 +515,7 @@ async function main(): Promise<void> {
                 [
                     "(async()=>{",
                     "const core=app.plugins.plugins['obsidian-livesync'].core;",
-                    `const entry=await core.localDatabase.getDBEntry(${JSON.stringify(notePath)},undefined,false,true);`,
+                    `const entry=await core.localDatabase.getDBEntry(${JSON.stringify(receivedNotePath)},undefined,false,true);`,
                     "return JSON.stringify(entry.ctime);",
                     "})()",
                 ].join(""),
@@ -376,12 +528,78 @@ async function main(): Promise<void> {
             );
             console.log("The CLI edit retained the filesystem creation time in the real Obsidian client's Metadata.");
         } else {
-            console.log("Creation-time verification skipped: native stats did not expose distinct creation and change times.");
+            console.log(
+                "Creation-time verification skipped: native stats did not expose distinct creation and change times."
+            );
         }
-        console.log(
-            "CLI-edited encrypted note was retrieved by the real Obsidian plug-in with identical content."
-        );
+        if (unicodePaths) {
+            const returnedContent = `${editedNoteContent}\nUpdated in Obsidian.\n`;
+            await evalObsidianJson(
+                obsidianCli.binary,
+                [
+                    "(async()=>{",
+                    `const path=${JSON.stringify(receivedNotePath)};`,
+                    "const file=app.vault.getAbstractFileByPath(path);",
+                    "await new Promise(resolve=>setTimeout(resolve,Math.max(0,file.stat.mtime+2500-Date.now())));",
+                    `await app.vault.modify(file,${JSON.stringify(returnedContent)});`,
+                    "return JSON.stringify(true);})()",
+                ].join(""),
+                session.cliEnv
+            );
+            await pushLocalChanges(obsidianCli.binary, session.cliEnv);
+            await runLiveSyncCli(liveSyncCliCommand, [cliDatabasePath, "--settings", cliSettingsPath, "sync"]);
+            await runLiveSyncCli(liveSyncCliCommand, [cliDatabasePath, "--settings", cliSettingsPath, "mirror"]);
+            assertEqual(
+                await readFile(join(cliDatabasePath, notePath), "utf-8"),
+                returnedContent,
+                "The Obsidian update did not reach the original NFD physical file."
+            );
+            await access(join(cliDatabasePath, receivedNotePath)).then(
+                () => {
+                    throw new Error("The CLI created a second NFC physical file.");
+                },
+                (error: NodeJS.ErrnoException) => {
+                    if (error.code !== "ENOENT") throw error;
+                }
+            );
+            // The mirror command intentionally keeps tombstoned local files. The daemon applies remote deletions.
+            daemon = await startLiveSyncCliDaemon(liveSyncCliCommand, cliDatabasePath, cliSettingsPath);
+            await evalObsidianJson(
+                obsidianCli.binary,
+                [
+                    "(async()=>{",
+                    `const file=app.vault.getAbstractFileByPath(${JSON.stringify(receivedNotePath)});`,
+                    "await new Promise(resolve=>setTimeout(resolve,Math.max(0,file.stat.mtime+2500-Date.now())));",
+                    "await app.vault.delete(file);",
+                    "return JSON.stringify(true);})()",
+                ].join(""),
+                session.cliEnv
+            );
+            await pushLocalChanges(obsidianCli.binary, session.cliEnv);
+            const deletionDeadline = Date.now() + Number(process.env.E2E_OBSIDIAN_FILE_TIMEOUT_MS);
+            for (;;) {
+                try {
+                    await access(join(cliDatabasePath, notePath));
+                } catch (error) {
+                    if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error;
+                    break;
+                }
+                if (Date.now() >= deletionDeadline) {
+                    throw new Error(
+                        `The Obsidian deletion did not remove the original NFD physical file.\n${daemon.getOutput()}`
+                    );
+                }
+                await new Promise((resolvePromise) => setTimeout(resolvePromise, 100));
+            }
+            await daemon.stop();
+            daemon = undefined;
+            console.log(
+                "NFC path identity preserved the NFD physical name through creation, remote update, and deletion."
+            );
+        }
+        console.log("CLI-edited note was retrieved by the real Obsidian plug-in with identical content.");
     } finally {
+        await daemon?.stop();
         if (session) {
             await session.app.stop();
         }