Merge internal Metadata integration for stacked Setup URI validation

docs/adr/2026_07_release_notes_and_database_compatibility.md

@@ -57,6 +57,8 @@ Keep configured-state inference separate from new-Vault initialisation. If an ex
 - On resume, clear `versionUpFlash` and persist that fail-closed change before recording the current `VER` as acknowledged. If saving fails, restore the gate. Reapply settings only after the marker has advanced so that the previously configured synchronisation behaviour can resume without reconstruction.
 - Preserve the original legacy review message as a structured reason when no more specific database or settings-schema reason is available. Escape it before including it in Markdown UI.
 - Continue to reject a remote version document which is newer than the running implementation. That receiver-side check is independent of the local upgrade review.
+- From remote generation 13, assess the `used_features` list in that document as a separate compatibility dimension. A client must recognise every listed feature before it interprets the database or runs maintenance which depends on Metadata. Declare a feature before writing its representation, and retain the declaration while older data may depend on it. An unknown identifier is reported as text without requiring a descriptive label in that client.
+- Do not advance the device-local `VER` acknowledgement merely because a remote feature is introduced. The remote generation and its feature list govern remote admission; `VER` remains the local compatibility review gate. Connecting to a generation-12 database does not promote it solely because the client understands generation 13.
 
 ### Onboarding activation and initialisation
 
@@ -90,7 +92,7 @@ Keep configured-state inference separate from new-Vault initialisation. If an ex
 - Accepted new-device and existing-device setup cannot enable ordinary processing before the selected Rebuild or Fetch has been reserved.
 - An older installation cannot dismiss evidence that a newer implementation or settings schema has already been used on the device.
 - The Obsidian-specific dialogue depends only on a host-neutral compatibility result and the injected confirmation capability. Commonlib remains responsible for settings migration, device-local storage, and the replication gate.
-- A future incompatible database change must increment `VER`, provide an actionable review message, verify the remote version negotiation, and test both the pending and acknowledged states. A major SemVer increase without those changes has no database-compatibility effect.
+- A future local database change which requires compatibility review must increment `VER`, provide an actionable review message, and test both the pending and acknowledged states. A new remote representation must declare its feature before use and verify remote admission independently. A major SemVer increase alone has no database-compatibility effect.
 
 ## Verification
 

docs/design_docs/internal_metadata_encryption.md

@@ -0,0 +1,163 @@
+---
+date: 2026-09-27
+commonlib-version: "0.1.30"
+self-hosted-livesync-version: "1.0.32"
+status: unreleased
+---
+
+# Internal Metadata encryption and remote feature changes
+
+This document defines the LiveSync integration of Commonlib's remote feature
+contract and encrypted Metadata for Hidden File Sync and Customisation Sync.
+It describes unreleased behaviour being implemented in this branch.
+
+Commonlib's companion `docs/remote-feature-compatibility.md` is the
+source of truth for the wire document, identifiers, validation, and shared
+assessment. This document owns the application behaviour, settings, Doctor
+recommendation, and verification of the Obsidian and CLI integrations.
+
+## Scope and settings
+
+Add `encryptInternalMetadata` to the shared encryption settings. A genuinely new
+Vault or CLI configuration defaults to true. Existing stored settings and old
+Setup URI or QR imports complete an absent value as false. Ordinary partial
+setting updates retain the current value.
+
+The preference applies to CouchDB with E2EE V2 and Property Encryption enabled.
+Show the preference as unavailable and explain its prerequisites when they are
+absent. Keep Journal and P2P's existing
+transport protection and avoid unrelated setting mismatches for those remotes.
+
+Use the existing HKDF Metadata representation to protect path, creation and
+modification times, size, and Chunk references for obfuscated internal entries.
+Keep the `i:`, `ix:`, and supported legacy `ps:` document IDs, path conversion,
+and content Chunk representation. Read encrypted Metadata independently of the
+write preference, including after that preference is disabled.
+
+The protection leaves document IDs, namespaces, revisions, deletion state,
+document counts, and ciphertext lengths visible. It does not encrypt device or
+Vault names stored in separate participant records.
+
+## Enabling the preference
+
+Changing the preference does not automatically reconstruct a database or gather
+all devices' data. It affects subsequent Metadata writes. Unchanged documents
+and old revisions can retain plaintext; mixed plaintext and encrypted Metadata
+are a supported transition state.
+
+Strongly recommend the existing manual remote Rebuild workflow when the person
+wants existing Metadata protected as well. The person prepares the authoritative
+data for that workflow. Describe this distinction in the setting, Doctor reason,
+and operational documentation. Do not advertise complete historical protection
+merely because the preference is enabled.
+
+Copy the preference with the other encryption settings when preparing a remote
+profile. Recreate a connection when its effective encryption settings change.
+Use the existing Tweak assessment and manual mismatch resolution; do not change
+the remote's shared policy silently when importing or loading settings.
+
+## Doctor
+
+Use Commonlib's existing conditional recommendation rules. Recommend true when
+the selected CouchDB settings have E2EE V2 and Property Encryption enabled and
+the new preference is false. Do not require Hidden File Sync or Customisation
+Sync to be active before offering the recommendation.
+
+Retain the existing E2EE V2 recommendation for a legacy algorithm. After that
+change, ensure the newly applicable Metadata recommendation is not hidden by a
+premature `doctorProcessedVersion` update. Advance the Doctor rule revision so
+an older completed consultation does not suppress this new recommendation.
+
+Apply the preference only when the person accepts the recommendation. Include
+the existing-data limitation, the manual Rebuild recommendation, and the need
+for compatible clients in the explanation. Do not set `requireRebuild` or
+`requireRebuildLocal` for this rule: the current host wrapper can schedule those
+operations and restart. `recommendRebuild` currently exists only as an unused
+rule field, so setting it alone does not display an explanation.
+
+## Admission and received version documents
+
+The remote version document is the source of feature requirements. Commonlib
+checks it before replication, Fast Fetch, and direct access. The milestone keeps
+the existing Tweak comparison and Rebuild lock. An accepted writer declares the
+feature before using it, including the writer admitted to a locked rebuilt
+remote; an unaccepted device remains blocked by that lock.
+
+Retain the existing received-version path through `parseSynchroniseResult`,
+`enqueueAll`, and `processIfNonDocumentChange`. Replace its numeric comparison
+with the shared assessment so unknown names at the same generation are also
+reported. Known features, reordered lists, and ordinary revision updates do not
+retire the connection. Unsupported or malformed control documents request
+retirement through the existing Replicator owner and display the reason.
+The callback must not await retirement of the operation which delivered it.
+
+This is an admission check and a best-effort stop for exceptional changes during
+an active connection. It does not fence every queued file application, roll back
+accepted writes, or guarantee an atomic change across live devices. Feature
+changes are an infrequent administrative operation: update all devices first,
+then enable the preference and use the recommended manual Rebuild. Rebuild
+locks the remote using the existing workflow; changing this preference alone
+does not lock it. The action to proceed without rebuilding explicitly reminds
+the person to update every other device, including currently connected devices.
+
+## Persistence and recovery boundaries
+
+Do not retain a second feature list, highest generation, or rejection flag in
+KV storage. Do not add compatibility checks to pending-work snapshot recovery
+or make that recovery a new prerequisite for application readiness. Preserve
+the existing queue and startup behaviour. A later attempt checks the current
+remote declaration, including after restart. Declared features remain on the
+remote when the write preference is disabled because older data can still use
+them; manually shortening that declaration is not a supported migration.
+
+After updating clients, use normal reconnection and the existing Hatch
+inspection or Fetch workflow if reconciliation is needed. This feature does not
+repair unrelated KV inconsistencies or the existing readiness queue behaviour.
+
+Garbage Collection V3 is a beta manual operation which begins with an ordinary
+bidirectional synchronisation. That admission checks the remote feature
+contract; no additional per-step GC checks are introduced. The separate
+cleaned-remote recovery path checks the local version document before its
+first Chunk-reference count because it does not start with that synchronisation.
+
+Use the same Commonlib assessment at the CLI, Fast Fetch, and direct-access
+boundaries. The Obsidian result processor is one consumer, not the only place
+which determines compatibility. Keep unrelated Vaults and databases operational.
+Fast Fetch checks the remote declaration before opening or resetting the local
+database, both for a fresh Fetch and for checkpoint resumption.
+
+## Verification and documentation
+
+Keep focused tests for settings defaults and imports, the Doctor condition
+matrix, acceptance and dismissal, connection replacement, and absence of an
+automatic Rebuild, Fetch, or restart for this rule.
+
+Keep unit tests for known and unknown feature notifications, generic identifier
+presentation, retirement without a circular wait, and the unchanged snapshot
+behaviour after KV failure or obsolete snapshot fields. The previous batch
+fences, physical-database tracking, and persistent rejection tests are outside
+this design; they must not imply an atomic live migration guarantee.
+
+Use real Obsidian Hidden File Sync and Customisation Sync scenarios to inspect
+raw CouchDB Metadata and restore content in another Vault. Check the admitted
+writer on a locked remote, unknown-feature rejection before and during
+replication, and remote-based rejection after restart. Retain the encrypted
+CLI-to-Obsidian interoperability scenario. A future client upgrade that adds
+support for an unknown feature is a separate validation boundary.
+
+Also exercise enabling the preference through the settings UI without Rebuild:
+retain unchanged plaintext Metadata, encrypt rewritten entries with stable IDs,
+reject a second device's mismatched preference, and restore both representations
+after alignment. With the preference subsequently OFF, verify that Fast Fetch
+still decodes encrypted Metadata and preserves the remote feature declaration.
+
+Keep the primary-language settings and troubleshooting guides, the
+database-compatibility ADR, and Unreleased notes aligned with this behaviour.
+Keep the detailed shared protocol in Commonlib and link to it after publication;
+do not maintain another copy of its wire schema here. Translations are a separate
+change. Update tested-version evidence when the implementation and its
+validation have been accepted.
+
+Related application contracts: [Replicator architecture](replicator_architecture.md),
+[Tweak compatibility](tweak_compatibility.md), and
+[database compatibility](../adr/2026_07_release_notes_and_database_compatibility.md).

docs/settings.md

@@ -245,6 +245,14 @@ Setting key: usePathObfuscation
 
 In default, the path of the file is not obfuscated to improve the performance. If you enable this, the path of the file will be obfuscated. This is useful when you want to hide the path of the file.
 
+#### Encrypt internal file Properties
+
+Setting key: encryptInternalMetadata
+
+For CouchDB, this encrypts paths, times, sizes, and Chunk references in the Metadata used by Hidden File Sync and Customisation Sync. It requires E2EE V2 and **Property Encryption**. New Vaults enable the preference by default, but it has no effect until those prerequisites are enabled. Existing Vaults and older Setup URIs and QR codes keep it disabled unless you enable it.
+
+Enabling the preference protects future Metadata writes. Existing Metadata and earlier revisions can remain readable in the remote database. If you want to protect existing Metadata too, prepare the authoritative data, update every device to a compatible version, and manually Rebuild the remote database. LiveSync does not gather data or start a Rebuild when you change this preference. The action to enable it without rebuilding explicitly reminds you to update every other synchronising device first, including devices currently running LiveSync. Plaintext and encrypted Metadata can coexist during the transition. Document IDs, revision information, document counts, and ciphertext lengths remain visible.
+
 #### Encryption Algorithm
 
 Setting key: E2EEAlgorithm

docs/troubleshooting.md

@@ -96,6 +96,8 @@ Current releases automatically align compatible settings which control how new c
 
 A missing legacy file-name case setting means case-insensitive handling. It matches an explicit disabled setting and does not require a rebuild for that difference. An explicitly enabled setting can use different document IDs and still requires a compatibility decision against either value. Other configuration differences shown in the dialogue must still be resolved.
 
+If the mismatch names **Encrypt internal file Properties**, update every device before accepting that preference. It affects subsequent Metadata writes for Hidden File Sync and Customisation Sync; it does not automatically protect existing Metadata. A manual remote Rebuild is strongly recommended if you need to protect existing paths, times, sizes, and Chunk references.
+
 The `Sync now` command keeps routine replication progress quiet so that it is convenient to assign to a keyboard shortcut; assign one in Obsidian if that suits your workflow. A quiet command may still open this dialogue when a mismatch or another decision requires your attention.
 
 The available actions depend on when the mismatch is found:
@@ -109,6 +111,10 @@ The available actions depend on when the mismatch is found:
 
 Historic defect notices and renamed controls are retained in the [0.25 release history](releases/0.25.md) and [legacy release history](releases/legacy.md), rather than in the current troubleshooting path.
 
+## The remote database uses an unknown feature
+
+When a notice identifies an unknown feature, update this device and every other client of the same CouchDB database, including the CLI. The notice includes the feature identifier even if this version has no descriptive name for it. New synchronisation is refused, and receiving an unsupported requirement stops active replication, because an older client may not interpret the Metadata and its Chunk references correctly. Already queued file changes are not rolled back. The cleaned-remote recovery path also checks compatibility before counting Chunk references. Do not remove the feature name from the remote version document to bypass the check. After updating, reconnect and review any pending file changes before running Garbage Collection.
+
 ## Setup and settings questions
 
 ### Share a configuration with another device