Merge NFC reconstruction fix into database diagnostics

docs/design_docs/unicode_path_identity.md

@@ -1,7 +1,7 @@
 ---
-date: 2026-10-17
+date: 2026-10-19
-commonlib-version: "0.1.42"
+commonlib-version: "0.1.43"
-self-hosted-livesync-version: "1.0.37"
+self-hosted-livesync-version: "1.0.38"
 status: unreleased
 ---
 
@@ -16,8 +16,9 @@ the [Unicode path identity ADR](../adr/2026_10_unicode_path_identity.md).
 Commonlib's [consumer contract](https://github.com/vrtmrz/livesync-commonlib/blob/main/docs/unicode-path-identity.md)
 defines the shared package boundary.
 
-This records an unreleased change after Self-hosted LiveSync `1.0.37`, using the
+NFC identity shipped in Self-hosted LiveSync `1.0.38` with Commonlib `0.1.42`.
-published Commonlib `0.1.42` package as an exact dependency.
+The deletion handling in reconstruction below uses Commonlib `0.1.43` and
+remains unreleased in Self-hosted LiveSync.
 
 ## Scope
 
@@ -117,6 +118,16 @@ must resolve the ambiguous names or exclude one file from the synchronisation
 scope, then retry the reconstruction. This check is bounded by the host's
 listing contract and does not imply that every raw physical alias is visible.
 
+For full Rebuild, Standard Fetch, and Fast Fetch, which replace the local
+database, the Metadata enumeration requests conflict information. It excludes
+`deleted: true` documents only when they have no unresolved conflict leaves.
+A deleted winner with any unresolved conflict remains a collision candidate,
+including when the other leaves are also logically deleted. Remote-only
+Rebuild retains logically deleted Metadata and requires stored IDs to match
+the selected identity. Fully tombstoned documents are already absent from
+ordinary enumeration; a still non-deleted local copy remains subject to the
+check until a remote deletion has been replicated.
+
 ## Reconstruction and client operations
 
 Mode changes are never applied by an ordinary Metadata write. In Obsidian, the
@@ -230,6 +241,14 @@ NFC duplicate, and remote deletion through the polling CLI daemon. The `mirror`
 command retains tombstoned local files by default; its update behaviour was
 tested separately from daemon deletion reflection.
 
+The published Commonlib `0.1.43` package also passed plain and encrypted NFC
+migration with obsolete logical deletions, plus the legacy encrypted ASCII
+scenario, on Linux with Obsidian `1.12.7` and CouchDB `3.5.0`. Both migration
+variants checked CLI Rebuild, second-client Standard Fetch, and real Obsidian
+Streaming Fast Fetch. Fast Fetch removed the obsolete Metadata and retained
+the current NFC document, its active state, and its stored path. The existing
+content, creation-time, physical-name, update, and deletion checks also passed.
+
 ## Non-goals
 
 This design does not:

docs/recovery.md

@@ -30,6 +30,10 @@ This migration reconstructs the databases from one authoritative Vault. It does
 4. On each remaining client, select **Unicode NFC path identity** (`pathNormalizationVersion: 1`) before fetching. In Obsidian, choose **Reset Synchronisation on This Device**. In the CLI, set the same value in its settings and run `fetch` with `--force --write-settings`.
 5. Confirm that every client uses version `1` and that the expected files are present before resuming synchronisation. Do not reconnect a version `0` client to the migrated database.
 
+Rebuild and both Fetch paths ignore local logical deletions without unresolved conflicts when replacing the local database. Deletion or rename history alone therefore does not block this migration. Fully tombstoned documents are already absent from the Metadata check. Unresolved conflicts and ambiguous current files still require review, even when a conflict's winning revision is logically deleted. Remote-only Rebuild retains its existing ID checks and is not a substitute for the source Rebuild above.
+
+A remote deletion which has not reached this device does not establish that its local Metadata is deleted. Inspect the local state and preserve unsynchronised content before resolving a reported collision. Manual deletion of every database is not a prerequisite for this procedure. Clients already using NFC identity do not need to repeat the migration for this change.
+
 The Inspector's **Repair this Metadata document ID** action repairs one local entry; it is not a bulk migration tool. Use the Rebuild and Fetch procedure above for path identity changes.
 
 This procedure does not establish native macOS or iOS filesystem behaviour. Obsidian's collision checks use the paths exposed by its Vault APIs; see [Unicode path identity](settings.md#unicode-path-identity) for that visibility limit.

docs/settings.md

@@ -1062,7 +1062,7 @@ Version `1` requires a coordinated migration: set it on the source client before
 
 The selected value stays pending through ordinary settings saves and reloads. It becomes active only when the Rebuild or Fetch operation is successfully reserved; cancellation leaves it pending.
 
-Distinct files or Metadata documents which derive the same identity stop reconstruction before either database is reset. This check uses the host's supported file listing; Obsidian checks the paths exposed by its Vault APIs and cannot detect physical aliases hidden by its index. Resolve the duplicate names or synchronisation scope first; Self-hosted LiveSync does not merge them automatically. Case-insensitive handling also applies the existing letter-case rule.
+Distinct files or retained Metadata documents which derive the same identity stop reconstruction before either database is reset. When replacing the local database, Rebuild and Fetch exclude logical deletions without unresolved conflicts; a deleted winner with unresolved conflicts still participates in the check. This check uses the host's supported file listing; Obsidian checks the paths exposed by its Vault APIs and cannot detect physical aliases hidden by its index. Resolve the duplicate names or synchronisation scope first; Self-hosted LiveSync does not merge them automatically. Case-insensitive handling also applies the existing letter-case rule.
 
 ### 4. Compatibility (Internal API Usage)
 

test/e2e-obsidian/README.md

@@ -178,7 +178,7 @@ Set `E2E_OBSIDIAN_ONLY_DIFFERENT_CHUNK_ID_KEYS=true` to run the focused case whe
 
 Set `E2E_OBSIDIAN_UNICODE_PATHS=true` to exercise path identity version 1 on Linux. This variant uses an NFD directory and a mixed NFC/NFD file name, checks reflection at Obsidian's NFC path, and returns an update and deletion to the original CLI physical path. Updates use `sync` and `mirror`; deletions use the polling daemon, because `mirror` retains tombstoned local files by default. It uses unencrypted Metadata by default to match issue #1259; add `E2E_OBSIDIAN_UNICODE_ENCRYPTION=true` for the encrypted variant. These checks do not establish native macOS or iOS filesystem behaviour.
 
-Set `E2E_OBSIDIAN_UNICODE_MIGRATION=true` to begin with two CLI databases using legacy path identity. The source runs `rebuild --force --write-settings` after selecting version 1, and the second client runs `fetch --force --write-settings`. The scenario checks that the old document ID is replaced, the NFC feature is declared, and both physical NFD filenames are preserved before continuing the real Obsidian update and deletion checks.
+Set `E2E_OBSIDIAN_UNICODE_MIGRATION=true` to begin with two CLI databases using legacy path identity and an obsolete, logically deleted NFC-equivalent Metadata document. The source runs `rebuild --force --write-settings` after selecting version 1, and the second client runs `fetch --force --write-settings`. The scenario checks that the old document ID is replaced, the NFC feature is declared, and both physical NFD filenames are preserved. In real Obsidian it also seeds an unconflicted logical deletion under the obsolete ID, runs Streaming Fast Fetch, and verifies replacement of that local Metadata with unchanged file content before continuing the update and deletion checks. The CouchDB fixture must allow browser CORS requests; this case rejects the native Request API fallback.
 
 The isolated Obsidian session starts with its CouchDB settings and device-local compatibility acknowledgement already in place. This keeps the scenario focused on cross-runtime data compatibility; unconfigured start-up and visible CouchDB onboarding are covered by their dedicated workflows.
 

test/e2e-obsidian/scripts/cli-to-obsidian-sync.ts

@@ -58,6 +58,7 @@ type CliFileInfo = {
     id: string;
     children: string[];
     expectedCtime?: number;
+    legacyId?: string;
 };
 
 function parseCommandLine(value: string): string[] {
@@ -130,14 +131,15 @@ function resolveLiveSyncCliCommand(): LiveSyncCliCommand {
     return { executable: process.execPath, prefixArgs: [liveSyncCli] };
 }
 
-async function runLiveSyncCli(command: LiveSyncCliCommand, args: string[]): Promise<CliResult> {
+async function runLiveSyncCli(command: LiveSyncCliCommand, args: string[], input?: string): Promise<CliResult> {
     return await new Promise((resolvePromise, reject) => {
         const timeoutMs = Number(process.env.E2E_LIVESYNC_CLI_TIMEOUT_MS ?? 60000);
         const child = spawn(command.executable, [...command.prefixArgs, ...args], {
             cwd: process.cwd(),
             env: process.env,
-            stdio: ["ignore", "pipe", "pipe"],
+            stdio: ["pipe", "pipe", "pipe"],
         });
+        child.stdin.end(input);
         let stdout = "";
         let stderr = "";
         child.stdout.on("data", (chunk: Buffer) => {
@@ -219,6 +221,7 @@ async function configureLiveSyncCli(
         passphrase: e2eePassphrase,
         usePathObfuscation: useEncryption,
         encryptInternalMetadata: useEncryption,
+        deleteMetadataOfDeletedFiles: false,
         pathNormalizationVersion,
         doctorProcessedVersion: "0.25.27",
         isConfigured: true,
@@ -339,6 +342,27 @@ async function migrateCliDatabases(
     dbName: string,
     legacyFileInfo: CliFileInfo
 ): Promise<CliFileInfo> {
+    await runLiveSyncCli(
+        command,
+        [databasePath, "--settings", settingsPath, "put", receivedNotePath],
+        "Obsolete composed-path content.\n"
+    );
+    const aliasInfo = await runLiveSyncCli(command, [
+        databasePath,
+        "--settings",
+        settingsPath,
+        "info",
+        receivedNotePath,
+    ]);
+    const deletedAliasId = (JSON.parse(aliasInfo.stdout) as CliFileInfo).id;
+    if (deletedAliasId === legacyFileInfo.id)
+        throw new Error("The legacy Unicode fixture needs distinct Metadata IDs.");
+    await runLiveSyncCli(command, [databasePath, "--settings", settingsPath, "rm", receivedNotePath]);
+    await runLiveSyncCli(command, [databasePath, "--settings", settingsPath, "sync"]);
+    await waitForCouchDbDocs(couchDb, dbName, (docs) =>
+        docs.some((doc) => doc._id === deletedAliasId && doc.deleted === true)
+    );
+
     const secondaryDatabasePath = join(cliState, "secondary-database");
     const secondarySettingsPath = join(cliState, "secondary-settings.json");
     await mkdir(secondaryDatabasePath, { recursive: true });
@@ -406,8 +430,8 @@ async function migrateCliDatabases(
             }
         );
     }
-    console.log("CLI Rebuild and Fetch migrated legacy document IDs while preserving both NFD physical files.");
+    console.log("CLI Rebuild and Fetch passed with obsolete logical deletions and preserved both NFD physical files.");
-    return { ...fileInfo, expectedCtime: legacyFileInfo.expectedCtime };
+    return { ...fileInfo, expectedCtime: legacyFileInfo.expectedCtime, legacyId: legacyFileInfo.id };
 }
 
 async function main(): Promise<void> {
@@ -509,6 +533,45 @@ async function main(): Promise<void> {
 
         const received = await waitForVaultContent(vault.path, receivedNotePath);
         assertEqual(received, editedNoteContent, "The real Obsidian plug-in did not materialise the CLI-edited note.");
+        if (cliFileInfo.legacyId) {
+            const fetched = await evalObsidianJson<{
+                obsoleteRetained: boolean;
+                currentDeleted: boolean;
+                currentPath: string;
+                previousPath: string;
+                content: string;
+            }>(
+                obsidianCli.binary,
+                [
+                    "(async()=>{",
+                    "const core=app.plugins.plugins['obsidian-livesync'].core;",
+                    "if(core.settings.useRequestAPI) throw new Error('This case requires Streaming Fast Fetch.');",
+                    "await core.services.setting.applyPartial({suspendParseReplicationResult:true,suspendFileWatching:true});",
+                    `const legacyId=${JSON.stringify(cliFileInfo.legacyId)};`,
+                    `const {_rev,...current}=await core.localDatabase.localDatabase.get(${JSON.stringify(cliFileInfo.id)});`,
+                    `await core.localDatabase.localDatabase.put({...current,_id:legacyId,path:${JSON.stringify(notePath)},deleted:true});`,
+                    "await core.serviceModules.rebuilder.$fetchLocalDBFast(true);",
+                    `const restored=await core.localDatabase.localDatabase.get(${JSON.stringify(cliFileInfo.id)});`,
+                    "const rows=await core.localDatabase.localDatabase.allDocs();",
+                    "return JSON.stringify({",
+                    "obsoleteRetained:rows.rows.some(row=>row.id===legacyId),",
+                    "currentDeleted:restored.deleted===true,",
+                    "currentPath:restored.path,",
+                    "previousPath:current.path,",
+                    `content:await app.vault.adapter.read(${JSON.stringify(receivedNotePath)}),`,
+                    "});",
+                    "})()",
+                ].join(""),
+                session.cliEnv
+            );
+            assertEqual(fetched.obsoleteRetained, false, "Fast Fetch retained obsolete local Metadata.");
+            assertEqual(fetched.currentDeleted, false, "Fast Fetch returned logically deleted current Metadata.");
+            assertEqual(fetched.currentPath, fetched.previousPath, "Fast Fetch changed the current Metadata path.");
+            assertEqual(fetched.content, editedNoteContent, "Fast Fetch changed the synchronised file content.");
+            console.log(
+                "Real Obsidian Fast Fetch replaced obsolete logical deletions and retained current Metadata and file content."
+            );
+        }
         if (cliFileInfo.expectedCtime !== undefined) {
             const receivedCtime = await evalObsidianJson<number>(
                 obsidianCli.binary,

updates.md

@@ -12,6 +12,13 @@ Earlier releases remain available in the 1.0 release history, the 1.0 preview hi
 
 ## Unreleased
 
+### Synchronisation
+
+#### Fixed
+
+- We can migrate to NFC path identity when deletion or rename history would otherwise block Rebuild or Fetch. (#1259)
+    - Logical deletions without unresolved conflicts no longer block operations which replace the local database. Conflicting histories and ambiguous current files still require review.
+
 ### Diagnostics
 
 - We can inspect normal-file Metadata and current conflicts in the local database and CouchDB together, with byte comparisons for suspect groups. Partial results retain the available evidence when a source is unavailable. The inspection leaves databases and settings unchanged.