Merge main into NFC transport support

2026-11-05 07:48:18 +00:00 · 2026-11-01 14:11:17 +00:00
commit 4957f8d28f
@@ -38,6 +38,7 @@ Use these references for the area being changed, together with its relevant ADR.
 | Encryption and sharing | [Metadata encryption](design_docs/internal_metadata_encryption.md) and [time-bound Setup URIs](design_docs/time_bound_setup_uri.md) |
 | P2P connection preparation | [TURN connection settings](design_docs/renewable_turn_credentials.md) |
 | Journal storage internals | [Journal Replicator 2nd Edition](design_docs_of_journalsync_2nd.md) |
+| CLI database selection and interrupted reconstruction | [CLI database continuity](design_docs/cli_database_continuity.md) |

 The installed `@vrtmrz/livesync-commonlib` package is authoritative for shared runtime contracts. Use the exact dependency selected by [package.json](../package.json) and its lockfile when checking behaviour. The [Commonlib documentation](https://github.com/vrtmrz/livesync-commonlib/tree/main/docs) is maintained in that repository; this index does not replace its package contracts.

@@ -0,0 +1,101 @@
+---
+date: 2026-11-01
+commonlib-version: "0.2.2"
+self-hosted-livesync-version: "1.0.43"
+status: accepted
+---
+
+# CLI database continuity
+
+Publication note: the correction for #1304 is implemented and locally validated,
+but has not been released. The reviewed baseline is Self-hosted LiveSync `0e458e5e` with
+Commonlib `0.2.2`. The [CLI guide](../../src/apps/cli/README.md) describes the
+command interface; the [test procedure](../../test/README.md#cli-fetch-database-continuity)
+describes the Object Storage reproduction.
+
+## Required behaviour
+
+A successful `fetch` or `rebuild` must leave subsequent CLI commands using the
+reconstructed local database. The Commonlib Rebuilder selects the host app ID
+as the database suffix. The CLI saves that selection on success even without
+`--write-settings`; the option continues to govern other lasting settings
+changes. Temporary suspension values remain runtime state.
+
+Before reconstruction can mutate a database, the CLI records the operation,
+database directory, Vault path, and settings path. An interrupted operation
+blocks ordinary mutation and synchronisation. Retrying the same reconstruction
+with the same paths and `--force` is an explicit recovery action. It can repeat
+the destructive operation, so users must preserve unique edits and establish
+the authoritative copy before retrying.
+
+Completion records the settings update before applying it. A later invocation
+can finish a pending settings commit; if the settings file has independently
+changed, it must stop for review rather than overwrite those changes. The
+pending record is removed only after completion and settings persistence.
+
+## Ownership of persistent state
+
+The selected local database name identifies the dataset. Its runtime state is
+further scoped by the canonical physical Vault path. This includes Journal
+checkpoints, scanner state, replication-result queues, file-reflection
+provenance, and storage-event snapshots. Paths identify local ownership; they
+are not sent to the remote as a new document identity.
+
+A local database reset invalidates every Vault scope associated with that
+database, because each scope refers to the same discarded revision history.
+Other local database suffixes retain their state. PouchDB database names and
+Commonlib's reconstruction contract remain unchanged.
+
+KV writes replace the file atomically. Invalid persisted data must fail
+initialisation instead of becoming an empty checkpoint history. Closing a KV
+handle must not overwrite the file from an old in-memory snapshot.
+
+## Legacy state
+
+The former shared `keyvalue-db.json` and Vault-local `.livesync-snapshot.json`
+do not record their complete owner. A single surviving database directory does
+not prove which physical Vault produced those records. Automatically importing
+them can replay events against the wrong database; dropping them and immediately
+scanning can misinterpret file absence before remote history is recovered.
+
+Preserve unowned legacy state. When it contains work, stop ordinary mutating
+commands and require explicit reconstruction from a verified source. Inspection
+remains available. This deliberately requires more user action than automatic
+migration, and must not silently delete the old records.
+
+Check the selected scope's completion flag before reading legacy files. For an
+unreviewed scope, inspect both legacy files after settings and scoped KV loading,
+but before readiness starts scanning or event replay. Distinguish non-empty
+unowned state, invalid JSON or shape, and read failures. Report affected paths
+once per invocation on stderr without logging file contents. Read commands may
+continue inspection; ordinary writes stop, and explicit reconstruction retains
+the files. Once reconstruction has succeeded for that scope, legacy parse or
+read failures must not block it again. The active scoped KV still fails
+initialisation on corruption.
+
+## Evidence and limits
+
+The original Object Storage scenario uses encryption, path obfuscation, and a
+second-generation file revision. Without settings writeback, A reopens an empty
+database while sharing the fetched database's Journal history, creates an
+independent root, and receives B's same-content resolution tombstone without
+the original surviving revision. The original assertion fails because A's
+physical file disappears. The explicit `--write-settings` control preserves
+the file and the original revision. The remote and B retain the content in this
+reproduction; it does not demonstrate irreversible loss.
+
+Validation of the correction must check bytes and revision ancestry after
+restart, interrupted reconstruction admission, pending settings completion,
+legacy-state preservation, and reset isolation. A successful exit or an empty
+queue alone does not establish preservation.
+
+Local validation passed the encrypted Object Storage CLI regression and the
+explicit-settings control on Node.js 26. Unit tests cover failed-operation
+admission, saved running/completed markers, settings-edit preservation, scoped
+stores, reset isolation, corrupt KV files, and legacy-state refusal. The marker
+tests inject interruption states; they do not emulate a machine power failure.
+
+The shared device-value JSON file retains its existing concurrency limitations.
+This change does not establish concurrent CLI writers to one database directory
+as a supported workflow. Native Obsidian and other transport behaviour require
+their own runtime evidence.
@@ -1,7 +1,7 @@
 ---
-date: 2026-09-04
-commonlib-version: "0.1.21"
-self-hosted-livesync-version: "1.0.24"
+date: 2026-10-25
+commonlib-version: "0.2.0"
+self-hosted-livesync-version: "1.0.40"
 status: accepted
 ---

@@ -104,14 +104,22 @@ For new behaviour, keep Obsidian-specific presentation or registration in an Obs

 ### A small serviceFeature: language initialisation

-`src/serviceFeatures/onLayoutReady/enablei18n.ts` declares only `setting`, `API`, and `appLifecycle`:
+`src/serviceFeatures/onLayoutReady/enablei18n.ts` declares only `setting`, `API`, and `appLifecycle`, with an explicit dependency type:

 ```typescript
-export const enableI18nFeature = createServiceFeature(async ({ services: { setting, API, appLifecycle } }) => {
-    // Apply the language, persist a change, and register unload clean-up.
-});
+type EnableI18nHost = NecessaryServices<"setting" | "API" | "appLifecycle", never>;
+
+export const enableI18nFeature = createServiceFeature(
+    async (host: EnableI18nHost): Promise<boolean> => {
+        const { setting, API, appLifecycle } = host.services;
+        // Apply the language, persist a change, and register unload clean-up.
+        return true;
+    }
+);
 ```

+The host annotation keeps the dependency contract visible because `createServiceFeature()` does not infer dependency keys from the callback body. This feature retains its `Promise<boolean>` result.
+
 The local `ObsidianLanguageAppliedNotice` class is still appropriate. It owns one replaceable Obsidian `Notice` and has an explicit `clear()` lifetime operation. The class is not used as a service locator, and the serviceFeature owns its construction and host binding.

 ### Operation and composition: database preparation
@@ -130,6 +138,22 @@ The operation does not need an application Module identity.

 ### Ordered start-up composition and registration-only features

+Compatibility review uses distinct controller and composer dependencies:
+
+```typescript
+type CompatibilityReviewDependencies = NecessaryServicesInterfaces<
+    "setting" | "config" | "vault" | "control",
+    never
+>;
+
+type CompatibilityReviewHost = NecessaryServices<
+    "setting" | "config" | "vault" | "control" | "appLifecycle" | "API",
+    never
+>;
+```
+
+The controller receives the four interface-only Services it uses. The composer adds `appLifecycle` and `API` for ordered lifecycle handlers and command registration; the priority-30 compatibility review remains after flag-file recovery.
+
 Configured Vault admission and the checks which follow database preparation are composed by `src/serviceFeatures/startupLifecycle/`. The directory keeps onboarding admission, compromised-chunk inspection, incomplete-document repair, Config Doctor, and the obsolete bulk-send setting migration as separate operations. One feature composer owns their order and receives the compatibility-review wait operation explicitly; an individual operation does not call the composer.

 The layout-ready admission handler uses priority 1. This preserves ordinary priority-0 host integration before admission, while keeping an unconfigured Vault outside the flag-file recovery handlers at priorities 5, 10, and 20, and the compatibility review at priority 30. Admission belongs to one plug-in process: an initially unconfigured process remains inert until setup restarts it, and declining the requested restart does not trigger an in-process reconfiguration. Changing an admitted process back to unconfigured retires its Config Doctor and incomplete-document repair request handlers. The handlers also recheck the current configured state and database readiness when invoked, so a pending restart cannot expose partially initialised or retired state. The first-initialise handler rechecks admission before retaining the established order after the file watcher has been started: database readiness, compromised chunks, incomplete documents, compatibility review, Config Doctor, and the bulk-send setting migration.
@@ -144,7 +168,19 @@ The state remains private to the composed feature. It does not become a `LiveSyn

 ### Implemented composition: conflict resolution

-Conflict checking and resolution are composed for every host by `useConflictResolutionFeature`. The feature owns its `QueueProcessor` privately and registers the conflict Service handlers directly. Its operations receive explicit collaborators for settings, active-file state, database and storage access, replication, logging, and host events. No consumer locates a conflict Module or retains the queue.
+Conflict checking and resolution are composed for every host by `useConflictResolutionFeature`. The feature owns its `QueueProcessor` privately and registers the conflict Service handlers directly. Its host type selects the required service and module contracts and adds the borrowed logger explicitly:
+
+```typescript
+type ConflictResolutionHost = NecessaryServices<
+    "appLifecycle" | "conflict" | "database" | "replication" | "setting" | "vault",
+    "databaseFileAccess" | "fileHandler" | "storageAccess",
+    InjectableServiceHub
+> & {
+    readonly log: LogFunction;
+};
+```
+
+The operations receive explicit collaborators for settings, active-file state, database and storage access, replication, a borrowed `LogFunction`, and host events. `LiveSyncBaseCore` supplies `log` using `createInstanceLogFunction("SF:ConflictResolution", this.services.API)`. Because feature composition precedes legacy Module binding, the host supplies the existing lazy service logger and does not read `logResource.value` before binding. `ModuleLog` and the host retain resource ownership. No consumer locates a conflict Module or retains the queue.

 The scheduling queue remains one state owner. It publishes `conflictProcessQueueCount`, coalesces pending checks for the same path, and makes `ensureAllProcessed()` wait for conflict resolution to finish. Repeated resolver invocations for one path retain only the newest waiting request and close an active comparison for that path before waiting for the per-file resolver, while comparisons for other paths remain open. Resolution remains host-neutral and communicates dialogue cancellation through `services.context.events`, so CLI, WebApp, and Obsidian compositions use their own selected event channel.

@@ -160,10 +160,11 @@ preflight failure. If a required setup decision cannot be made in a headless
 CLI invocation, the command must fail with a clear remediation message rather
 than report success after cancellation.

-`--write-settings` retains the Rebuilder's selected local database name and
-lasting settings changes for subsequent invocations. Without this option, the
-existing CLI policy leaves the settings file unchanged, so a later command can
-reopen the previously selected database.
+The unreleased CLI correction for #1304 always retains the Rebuilder's selected
+local database suffix after successful `fetch` or `rebuild`. `--write-settings`
+still controls other lasting settings changes. Earlier versions required that
+option to retain the database selection; omitting it could reopen the previous
+database on the next invocation. The examples retain the option for compatibility.

 ## State which follows the selected identity

@@ -18,6 +18,37 @@ Use the least destructive operation which matches the evidence:

 Do not switch transport, enable P2P, or run Garbage Collection as a substitute for diagnosing a stopped CouchDB or Object Storage setup.

+## Recover files truncated during Object Storage reception
+
+This procedure addresses [issue #1303](https://github.com/vrtmrz/obsidian-livesync/issues/1303): an older Journal receiver can truncate a Chunk at a raw unit separator (`U+001F`) within its data. This can damage a received text file, including a plug-in's `main.js`, while the original batch remains intact in the bucket.
+
+The receiver correction is available in Commonlib 0.2.3 and is included in the unreleased Self-hosted LiveSync build. Use a build containing that correction on every participating client before recovery; versions 1.0.42 and 1.0.43 still contain the defect. Updating alone does not repair Chunks already stored locally. Resetting only the Journal received history is also insufficient because ordinary reception skips existing Chunk IDs.
+
+### Preserve the intact source
+
+1. Stop editing and synchronisation on every device. If necessary, use the [emergency suspension flag](#suspend-before-diagnosis).
+2. Back up every Vault, including hidden files such as `.obsidian`, and the complete remote bucket before resetting anything.
+3. Verify the actual contents of the proposed recovery source. A file's existence, recorded size, or successful synchronisation status does not prove that its content is intact. Preserve unique unsynchronised edits and any needed content which exists only in database history separately.
+4. Update all participating clients to a build containing the correction, keeping the other devices stopped while recovering one source and then one receiving device at a time.
+
+### Rebuild from a complete, intact Vault
+
+When one Vault contains all the intended current files, a deliberate Rebuild avoids relying on potentially damaged local databases. Use [Overwrite Server Data with This Device's Files](#overwrite-server-data-with-this-devices-files) on that authoritative device, then prepare an empty recovery Vault on each other device and [Reset Synchronisation on This Device](#reset-synchronisation-on-this-device) from the rebuilt remote. Keep the previous Vaults as backups until verification is complete.
+
+Rebuild reconstructs the local and remote databases from the current Vault files. It does not retain revisions available only in the previous local database, remote history, or another device. Keep the backups and preserve any required historical content before proceeding. An intact copy of one affected file does not establish that the whole Vault is a complete recovery source.
+
+### Reuse a verified intact remote
+
+If the selected remote file contents are verified intact, the remote need not be overwritten. Prepare a new empty recovery Vault with the same remote configuration and use [Reset Synchronisation on This Device](#reset-synchronisation-on-this-device). Keep the previous Vault, including its hidden files and unsynchronised work, as a separate backup. This reconstructs the local database and its receive checkpoints, allowing the corrected receiver to read the original batches again.
+
+Fetch does not unconditionally overwrite existing Vault files. A differing local file can remain and be preserved as a separate conflict branch alongside the intact remote content. An empty recovery Vault avoids retaining damaged local files during this process. In the CLI, use the empty Vault with `fetch --force --write-settings`; `--force` confirms database replacement, not forced file overwriting.
+
+### Restore optional files and verify the result
+
+Rebuild and Fetch disable Hidden File Sync and Customisation Sync. Complete ordinary note recovery first, then initialise the optional feature deliberately. For Hidden File Sync, follow its [initial source selection](tips/hidden-file-sync.md#choose-the-initial-source): after a full Rebuild, use `Overwrite` on the device with the intact hidden files, then `Fetch` on additional devices. When retaining an intact remote hidden-file state, use `Fetch` on the recovering device. Restore the intended Customisation Sync selections separately; do not enable both features for the same files.
+
+Compare the recovered files with the preserved originals, including the affected hidden files, then restart Obsidian and synchronise again. Confirm that the contents remain intact and that any affected plug-in loads before resuming ordinary work on the remaining devices. Keep the backups until this verification is complete.
+
 ## Migrate to Unicode NFC path identity

 Use this coordinated migration when CLI and Obsidian clients should identify composed and decomposed Unicode spellings as the same path. Version `0` remains the default and preserves legacy identity. CouchDB support is released; Object Storage and P2P support belong to the unreleased extension described in the [Unicode path identity design](design_docs/unicode_path_identity.md).
@@ -72,6 +72,12 @@ Hidden File Sync must be enabled independently after ordinary setup on each devi

 If you later run Rebuild or Fetch as a recovery operation, expect optional features to be disabled again. Complete ordinary recovery first, then repeat this guide deliberately.

+## Resolve a JSON conflict
+
+When automatic merging cannot combine conflicting JSON changes and no overwrite pattern applies, the **Conflicted Setting** dialogue compares two revisions. Review the proposed contents before choosing **Apply**. Closing the dialogue leaves the conflict unresolved.
+
+A later conflict check for the same file cancels the earlier comparison and checks the latest revisions again. If manual resolution is still needed, a replacement dialogue opens. A cancelled dialogue cannot apply a later selection. A resolution received through replication closes the comparison for that file; a remaining conflict is checked again. Dialogues for other files wait their turn. An **Apply** operation which has already started finishes before the replacement proceeds.
+
 ## Automatic in Customisation Sync

 **Automatic** delegates an item's files to Hidden File Sync. Enable Hidden File Sync on this device through the steps above before choosing Automatic. If Hidden File Sync is disabled, the choice displays a Notice without changing the item's mode or starting initialisation. Previously saved Automatic assignments remain saved, so this prerequisite also applies after importing settings from another device.