Merge main into CouchDB cache compatibility changes

This commit is contained in:
vorotamoroz committed 2026-11-05 09:48:00 +00:00
commit c306146dcd
78 files changed
+6177 -929

No files matched your search

+15 -1
View File
@@ -13,6 +13,10 @@ Accepted on 2026-10-17. Unicode NFC path identity was released in Self-hosted
LiveSync 1.0.38 with Commonlib 0.1.42. Local validation evidence and platform
limits are recorded in the design document.
The maintained design now also describes an unreleased Journal and P2P
extension and the move to General Settings. The decision below retains the
initial CouchDB support boundary and original acceptance date.
## Context
Self-hosted LiveSync derives normal-file Metadata document IDs from Vault paths.
@@ -102,9 +106,19 @@ preflight uses the supported host listing and local Metadata. It refuses
ambiguous file or parent-directory candidates before database reconstruction;
it does not claim to detect physical aliases which a host does not expose.
## Unreleased extension: defaults for new environments
The later Journal and P2P extension uses NFC for newly generated CLI settings
and Obsidian manual onboarding. It retains legacy defaults for stored settings
and old Setup URIs, and reads an existing central remote's declaration before
first participation. This refines the original default decision without
implicitly changing existing document IDs. Shared Commonlib defaults and other
hosts remain unchanged. See the maintained [design](../design_docs/unicode_path_identity.md#initial-settings-and-existing-environments)
for the initial-setup and failure rules.
## Alternatives considered
- **Use NFC by default:** rejected because existing clients and databases must
- **Use NFC by default for all settings:** rejected because existing clients and databases must
retain their current IDs until their owners deliberately migrate them.
- **Normalise only lookups:** rejected because new Metadata writes and deletes
could continue using legacy IDs and create duplicate documents.
+11 -5
View File
@@ -34,10 +34,15 @@ admission checks and supported transports.
The central replication gate, mismatch dialogues, and RedFlag Fetch preparation
consume the same effective differences. The P2P transport retains its separate
policy: ordinary representation differences warn without rejecting transfer,
while its existing passphrase and peer checks still apply. NFC path identity
is outside that warning-only policy: P2P refuses version `1`. Sharing assessment
does not make the central replication policy appropriate for every transport.
policy: ordinary representation and case differences warn without rejecting
transfer, while its existing passphrase and peer checks still apply. Path
identity has a separate required check: both peers must use the same version,
and a version mismatch rejects before replication. Commonlib `0.3.0` includes
NFC protocol support for Journal and P2P, but the matching CLI and Obsidian host
adoption remains an unreleased extension tracked by
[the Unicode path identity design](unicode_path_identity.md). Sharing
assessment does not make the central replication policy appropriate for every
transport.
## Host responsibilities
@@ -96,7 +101,8 @@ failed transfer into a completed transfer.
## Verification boundaries
Commonlib tests protect missing-value interpretation, representation differences,
directional consequences, immutable results, central admission, and P2P policy.
directional consequences, immutable results, central admission, and P2P policy,
including the distinction between ordinary warnings and a path-version mismatch.
LiveSync unit tests protect ordinary versus reconstruction choices, trial-setting
selection, partial-setting adoption, and invalidated decisions. RedFlag tests
protect the distinction between adopting settings for Fetch and retaining local
+177 -16
View File
@@ -22,14 +22,33 @@ and is included in current Commonlib `0.2.2`, used by Self-hosted LiveSync
`1.0.42`. This later handling is distinct from the initial NFC identity
release.
Commonlib `0.3.0` publishes the Journal and P2P path-identity protocol
extension. The matching Self-hosted LiveSync CLI and Obsidian host adoption,
including moving the setting to General Settings and defaulting new
synchronisation environments to NFC, remains an unreleased candidate tracked
by [pull request #1309](https://github.com/vrtmrz/obsidian-livesync/pull/1309).
The original dates and reviewed versions above remain the historical baseline.
## Unreleased shared storage boundary
Commonlib `0.3.1` publishes the shared storage-path contract, including
its Unicode resolver and typed physical-I/O ports. The LiveSync candidate
adopts the shared resolver and typed physical-I/O ports in the CLI. Obsidian
adopts shared namespace and comparison helpers while retaining its Vault APIs
and existing alias-guard policy. Each NFC scan retains one comparison-policy
snapshot; directory and permission errors propagate instead of producing a
successful partial scan. LiveSync's host adoption remains unreleased. The
[developer guide](../../devs.md#file-path-handling) links the Commonlib design
and ADR.
## Scope
| Area | Contract |
| --- | --- |
| Logical identity | Version `1` uses NFC for Metadata ID and path comparison keys. Existing case handling remains a separate rule. Missing and `0` retain legacy behaviour. |
| Stored paths and content | Keep the spelling in `Metadata.path` and existing physical Vault names. NFC does not change Chunk IDs or content. |
| Initial hosts and remote | Support CLI and Obsidian with CouchDB. |
| Other transports | Journal and P2P reject version `1`; Direct file access refuses a remote declaring the NFC feature. Other host adapters remain outside the support matrix. |
| Maintained hosts and transports | CLI and Obsidian with CouchDB are released; their Journal and P2P host adoption remains unreleased. See the support matrix below. |
| Other consumers | Direct-file access and other host adapters are covered separately below; shared path primitives alone do not enable NFC in an application. |
| Migration | Rebuild both databases from one complete authoritative Vault, then Fetch each remaining client. Do not rewrite IDs or merge histories in place. |
## Path identity contract
@@ -63,6 +82,31 @@ under legacy identity cannot be resumed under version `1`, or vice versa. This
reuses the existing checkpoint identity and does not add a separate recovery
store.
## Initial settings and existing environments
The unreleased extension uses version `1` for newly generated CLI settings and
Obsidian manual onboarding. Stored settings and old Setup URIs with no path
version retain `0`. Commonlib's shared settings defaults remain `0`, so this
host policy does not extend to WebApp or WebPeer. An absent settings file alone
does not establish that an existing local database is new.
Before saving initial manual CouchDB or Object Storage setup, the host reads
the remote declaration through its owned connection resource. A configured
remote supplies `0` or `1`; only a confirmed empty remote retains the new
settings' selected version. Read failures and unsupported or contradictory
declarations stop setup without saving a guessed value. The CLI makes the same
read before opening the local database for an unconfigured client's first
explicit Fetch. Ordinary configured startup and source Rebuild do not adopt a
remote's mode automatically.
CouchDB inspection reads the version document without preparing encryption or
creating a database. Without a version document, a non-empty database remains
legacy. Journal inspection reads its existing milestone ranges and preferred
Tweaks without registering a node, preparing a Security Seed, or resetting a
checkpoint. P2P Setup URIs carry the selected version. Manual P2P configuration
requires the owner to match the source; the signalling probe does not establish
the remote database's identity, and replication still rejects a mismatch.
## Admission and database compatibility
Clients validate local and remote database identity before accepting ordinary
@@ -133,8 +177,8 @@ check until a remote deletion has been replicated.
## Reconstruction and client operations
Mode changes are never applied by an ordinary Metadata write. In Obsidian, the
Advanced page offers **Legacy path identity** (`0`) and **Unicode NFC path
identity** (`1`) for CouchDB. A changed `pathNormalizationVersion` remains
General Settings → **File identity** offers **Legacy path identity** (`0`)
and **Unicode NFC path identity** (`1`) independently of the selected transport. A changed `pathNormalizationVersion` remains
pending through ordinary settings saves, the settings-only Apply fallback, and
reload. It becomes active only after the existing Rebuild or Fetch operation
has been successfully reserved; cancellation or failed reservation leaves it
@@ -144,7 +188,8 @@ the settings-only fallback is not offered.
The [user migration procedure](../recovery.md#migrate-to-unicode-nfc-path-identity)
owns the operational sequence: updated and stopped clients, a complete backed-up
authoritative Vault, source Rebuild, then Fetch on each remaining client. The
source operation reconstructs both databases; a receiving client's Fetch
source operation reconstructs both databases for CouchDB and Journal. With
P2P, it reconstructs only the source database. A receiving client's Fetch
replaces only its local database. The [CLI guide](../../src/apps/cli/README.md)
provides the equivalent `rebuild` and `fetch` commands.
@@ -187,15 +232,53 @@ updates and deletions.
## Compatibility matrix
| Client and remote | Version `0` | Version `1` |
| Host or API | Published baseline | Unreleased or host-specific status |
| --- | --- | --- |
| CLI with CouchDB | Supported with legacy identity. | Supported with NFC path resolution. |
| Obsidian with CouchDB | Supported with legacy identity. | Supported with NFC path resolution. |
| Journal or P2P | Existing behaviour. | Rejected. |
| Direct file access | Existing behaviour. | Refuses a remote requiring `nfc-path-identity-v1`. |
| Other host adapters, including WebApp | Existing behaviour, subject to their existing support. | Outside the initial support matrix until physical-path handling is verified. |
| CLI and Obsidian with CouchDB | NFC identity is supported from Self-hosted LiveSync `1.0.38` with Commonlib `0.1.42`. | Commonlib `0.3.1` publishes the shared storage-path contract. LiveSync's CLI adoption of the shared resolver and typed physical-I/O ports, and its Obsidian adoption of shared namespace and comparison helpers, remain unreleased. |
| CLI and Obsidian with Journal | Commonlib `0.3.0` publishes the NFC protocol support. | Matching LiveSync host adoption remains an unreleased candidate tracked by [pull request #1309](https://github.com/vrtmrz/obsidian-livesync/pull/1309). Update and migrate every client before using version `1`. |
| CLI and Obsidian with P2P | Commonlib `0.3.0` publishes the NFC protocol support. | Matching LiveSync host adoption remains an unreleased candidate. All peers must use the same path version; P2P warns and continues for ordinary representation or case differences. |
| Bridge | The published baseline uses Legacy version `0`. | The Bridge integration is an unreleased candidate with explicit per-peer opt-in to version `1`; every peer in a group must use the same version. Its ordinary case mismatch check remains enabled unless the Bridge-wide `disableCheckingConfigMismatch: true` override disables that case check. The override does not bypass a path-version mismatch. |
| `DirectFileManipulator` | Commonlib `0.3.1` accepts explicit `pathNormalizationVersion` values `0` or `1`, defaults to Legacy (`0`), and rejects a populated database in the opposite mode. | This admission setting does not resolve physical paths; callers own physical path resolution and migration. |
| WebApp | Legacy version `0` only. | WebApp remains Legacy-only and rejects an explicit version `1` at startup. The browser resolver primitive does not enable NFC in WebApp. |
| WebPeer | A vaultless P2P utility, not a filesystem host. | No NFC filesystem support is implied by its P2P role. |
| Older client without this feature | Can use a legacy database. | Cannot join a database declaring `nfc-path-identity-v1`. |
## Journal and P2P extension
The transport moves the Metadata IDs already generated in the local database;
it does not derive a second remote ID. Journal uses its existing compatibility
range with NFC value `4`, plus a required path-version comparison. A legacy
client with ordinary version checking cannot join the rebuilt Journal. The
explicit version override in older clients remains outside this guarantee.
Journal refuses to initialise NFC compatibility when journal objects exist
without their milestone.
Journal shares one read-only identity assessment across ordinary admission,
Fetch before local reset, and ordinary milestone updates. It checks stored
node ranges even when a Rebuild lock has retired the previous accepted nodes.
Rebuild locks the existing milestone for its source node while retaining an
existing compatibility range, and declares the replacement identity only after
the remote journals have
been deleted. If the milestone is missing, the pre-reset lock does not create
one over the existing journals. A failed or interrupted reset therefore keeps
the old identity or the missing declaration.
P2P compares the effective path version through its existing Tweak exchange,
checks local database identity, and reads the peer version document before
replication. Fresh control-only databases can declare the existing NFC feature.
Every participating peer must be updated and migrated; this is not a new
authentication protocol for already deployed clients.
The existing Rebuild and Fetch paths own migration and state reset. Journal
reconstructs the local database and remote objects; P2P Rebuild reconstructs
only the source database. Other devices Fetch the source. Local reset clears
Journal KV checkpoints, while P2P uses its existing source mark and target
checkpoint. No new migration store, checkpoint format, or persistent admission
state is added. An explicit Journal bucket reset also clears the existing
transfer checkpoint when the bucket is already empty, so a remote-only Rebuild
resends the complete local database. P2P retains its existing peer selection
during Fetch.
## Verification contract and limits
The implementation acceptance matrix covers:
@@ -212,8 +295,11 @@ The implementation acceptance matrix covers:
- refusal to write when an existing legacy database has Metadata, refusal of
version-`0` clients against an NFC database, and enforcement when ordinary
mismatch warnings are disabled;
- URI round trips, checkpoint invalidation across path versions, and transport
rejection for Journal, P2P, and Direct file access;
- URI round trips, checkpoint invalidation across path versions, and
explicit Legacy/NFC admission by `DirectFileManipulator` in Commonlib `0.3.1`,
including rejection of a populated database in the opposite mode. Journal and
P2P protocol support is published in Commonlib `0.3.0`, while matching CLI and
Obsidian host adoption and end-to-end checks remain unreleased;
- refusal for ambiguous file or parent-directory candidates before local or
remote reset, preserving both databases on rejection, within the paths
returned by each host's supported listing; and
@@ -252,6 +338,81 @@ Streaming Fast Fetch. Fast Fetch removed the obsolete Metadata and retained
the current NFC document, its active state, and its stored path. The existing
content, creation-time, physical-name, update, and deletion checks also passed.
### Unreleased Journal and P2P validation
The extension was initially checked on Linux with a locally packed Commonlib
`0.2.3` candidate and real Obsidian `1.12.7`. The Self-hosted LiveSync extension
remains unreleased. The initial checks covered:
- real S3-compatible Journal transfer with and without E2EE and Path
Obfuscation, including restart, logical rename and deletion, both conflict
leaves and their revision ancestry, and a full resend after bucket reset;
- real Obsidian and CLI Journal round trips with plain IDs and independent
encrypted IDs: Standard Fetch, identical Metadata IDs and Chunk content,
retention of a pre-existing NFD physical name, edits after CLI process
restart, and deletion reflected back into Obsidian;
- real CLI P2P peers in legacy mode and NFC mode, with and without encryption
in NFC mode: canonical path reads, incompatible-peer refusal, reverse edits
after restart, retained original revision content, logical rename, and
deletion. Existing automatic-host, adapter-replacement, and explicit
disconnect scenarios also passed; and
- two real Obsidian `1.12.7` processes in an isolated Compose project, with NFC
identity in plain and E2EE/Path Obfuscation modes using an independent ID key:
initial Fetch, identical Metadata IDs and Chunk-backed content, reverse edits,
retained original revision bodies, a full application restart and subsequent
transfer, and explicit path-version refusal after ordinary peer permission.
The refused peer retained its sentinel ID, revision, and content and imported
no source Metadata or file. These cases passed on 2026-11-02 with the updated
locally packed candidate after integrating the current main branches.
The Journal migration scenarios also start with existing legacy CLI databases.
The source Rebuild changes its Metadata ID and declares compatibility range
`4`. A receiving client's legacy Fetch is refused while preserving its local
Metadata ID and revision and the remote milestone. After selecting NFC, its
Fetch converges to the source ID and content while preserving the NFD physical
path. Both plain and independently encrypted ID scenarios passed.
On 2026-11-02, the exact published Commonlib `0.3.0` package passed the
consumer source checks, unit tests, and plug-in, CLI, WebApp, and WebPeer builds.
The real Obsidian P2P cases above were repeated successfully with that package.
Real Obsidian and CLI Journal round trips and the legacy-to-NFC migration were
also repeated with plain IDs and independently encrypted IDs, retaining the
physical NFD names and verifying edits after restart and deletion propagation.
The CLI Compose P2P suite also passed with the published package, including
legacy and NFC identity, plain and encrypted NFC transfer, logical rename and
deletion, automatic hosting, adapter replacement, and relay disconnection.
Additional real Obsidian cases with the published package passed legacy-to-NFC
P2P migration with plain IDs and independently encrypted IDs. They seed legacy
NFD Metadata IDs using real Chunks, since Obsidian's file cache normalises
names. The checks covered source Rebuild and restart, Standard Fetch on the
receiving device, removal of obsolete active IDs, exact content, and retention
of physical NFD names. Both cases also passed transfer of a Unicode note,
logical rename with matching new IDs, and deletion after restart. An
incompatible P2P Fetch reset the local database before a peer was selected.
Its physical note retained its exact bytes, and a Fetch with matching settings
completed recovery. This check does not establish preservation of the old
database's revision history.
Real Obsidian Journal cases also passed remote-only Rebuild after deleting
their isolated Object Storage prefix while leaving the source database and
populated transfer checkpoint intact. With both plain IDs and independently
encrypted IDs, a fresh receiving database and its subsequent restart retained
the exact bodies of both conflict leaves, Chunk references, revision IDs, and
each leaf's `_revisions`. The source retained the readable body of its common
ancestor. Fresh receivers retained the ancestor ID without its historical
body; ancestry IDs alone do not establish that an older body is readable. The
[conflict policy](../specs_conflict_resolution.md#file-saving-and-reflection-guarantees)
continues to require an available shared base for conservative automatic merge.
The settings, first-join identity inspection, legacy Setup URI defaults, and
initial CLI Fetch persistence are also covered by unit tests. The additional
migration cases call the running plug-in's production Rebuild and Fetch
methods with real peer selection; reservation through the settings dialogue
has separate coverage.
These checks do not extend NFC support to other hosts or establish native iOS
or macOS filesystem behaviour.
## Non-goals
This design does not:
@@ -262,9 +423,9 @@ This design does not:
- migrate remote document IDs or revision histories in place;
- choose among ambiguous NFC-equivalent files or parents, or merge their
content;
- change the default away from legacy path identity;
- enable version `1` for Journal, P2P, Direct file access, WebApp, or an
unverified host adapter; or
- automatically change existing settings or database identity;
- automatically enable version `1` for WebApp or an unverified host adapter;
or
- claim native iOS or macOS filesystem compatibility.
## References
+3 -3
View File
@@ -51,13 +51,13 @@ Compare the recovered files with the preserved originals, including the affected
## Migrate to Unicode NFC path identity
Use this coordinated migration when CLI and Obsidian clients share a CouchDB database and composed and decomposed Unicode spellings should identify the same path. Version `0` remains the default and preserves legacy identity. Version `1` is supported with CouchDB only; it is unsupported with Object Storage and P2P.
Use this coordinated migration when CLI and Obsidian clients should identify composed and decomposed Unicode spellings as the same path. Existing settings and old Setup URIs retain version `0`, which preserves legacy identity. The unreleased extension defaults new CLI settings and Obsidian manual onboarding to version `1`; this does not migrate an existing database. CouchDB support is released; Object Storage and P2P support belong to the unreleased extension described in the [Unicode path identity design](design_docs/unicode_path_identity.md).
This migration reconstructs the databases from one authoritative Vault. It does not rewrite document IDs in place or rename files.
1. Update every participating CLI and Obsidian client to a version which supports Unicode NFC path identity. Back up a complete authoritative Vault and the CouchDB database. Stop synchronisation and editing on every device until the migration is complete.
1. Update every participating CLI and Obsidian client to a version which supports Unicode NFC path identity. Back up a complete authoritative Vault and the selected remote database or Object Storage bucket. For P2P, preserve the participating devices because there is no central remote backup. Stop synchronisation and editing on every device until the migration is complete.
2. Choose the complete Vault which contains the intended files. Review any composed and decomposed names which would map to the same path identity, and resolve them before reconstruction. If Rebuild or Fetch reports an ambiguous path or Metadata ID, stop and resolve that collision before retrying; LiveSync does not choose or merge the files for you.
3. On the source client, select **Unicode NFC path identity** (`pathNormalizationVersion: 1`). In Obsidian, choose **Overwrite Server Data with This Device's Files**. In the CLI, set the same value in its settings and run `rebuild` with `--force --write-settings`.
3. On the source client, open **General Settings** → **File identity** and select **Unicode NFC path identity** (`pathNormalizationVersion: 1`). In Obsidian, choose **Overwrite Server Data with This Device's Files**. In the CLI, set the same value in its settings and run `rebuild` with `--force --write-settings`. For Object Storage, this replaces the remote journals as well as the local database. For P2P, it prepares only the source device's local database; keep that device online for the other devices to Fetch.
4. On each remaining client, select **Unicode NFC path identity** (`pathNormalizationVersion: 1`) before fetching. In Obsidian, choose **Reset Synchronisation on This Device**. In the CLI, set the same value in its settings and run `fetch` with `--force --write-settings`.
5. Confirm that every client uses version `1` and that the expected files are present before resuming synchronisation. Do not reconnect a version `0` client to the migrated database.
+16 -12
View File
@@ -236,6 +236,21 @@ Setting key: showVerboseLog
Show verbose log. Please enable when you report the logs
### 3. File identity
#### Unicode path identity
Setting key: pathNormalizationVersion
Version `0` preserves legacy file identity. Existing settings and Setup URIs which omit this field retain version `0`. New CLI settings and Obsidian manual onboarding use version `1` in the unreleased extension. On first manual setup or an unconfigured CLI's initial Fetch, an existing CouchDB or Object Storage declaration takes precedence; an unavailable or unsupported declaration stops setup. P2P Setup URIs carry the selected version; manual P2P setup requires matching settings on every peer. Version `1` uses Unicode NFC when deriving document IDs, so equivalent composed and decomposed paths identify the same file across the CLI and Obsidian. Existing physical file names are retained. CouchDB support is released. The unreleased extension also supports Object Storage and P2P for the maintained CLI and Obsidian hosts. This is a common local database setting, independently of the active transport.
Version `1` requires a coordinated migration: set it on the source client before Rebuild, then set it on each remaining client before Fetch. Changing the setting alone does not migrate existing Metadata. Follow [Migrate to Unicode NFC path identity](recovery.md#migrate-to-unicode-nfc-path-identity) for the complete procedure. Do not resume a version `0` client against the migrated remote.
The selected value stays pending through ordinary settings saves and reloads. It becomes active only when the Rebuild or Fetch operation is successfully reserved; cancellation leaves it pending.
Distinct files or retained Metadata documents which derive the same identity stop reconstruction before either database is reset. When replacing the local database, Rebuild and Fetch exclude logical deletions without unresolved conflicts; a deleted winner with unresolved conflicts still participates in the check. This check uses the host's supported file listing; Obsidian checks the paths exposed by its Vault APIs and cannot detect physical aliases hidden by its index. Resolve the duplicate names or synchronisation scope first; Self-hosted LiveSync does not merge them automatically. Case-insensitive handling also applies the existing letter-case rule.
## 3. Remote Configuration
### 1. Connection settings
@@ -1176,17 +1191,6 @@ When this setting is disabled, changing only the letter case of a file name with
New Vaults use case-insensitive handling for cross-platform compatibility. Existing settings with an explicit value preserve that choice. Earlier releases also followed the case-insensitive branch when this value was absent, so 1.0 saves a missing legacy value as `false` without requiring a compatibility review or database rebuild.
#### Unicode path identity
Setting key: pathNormalizationVersion
Version `0` preserves legacy file identity and is the default for existing and new settings. Version `1` uses Unicode NFC when deriving document IDs, so equivalent composed and decomposed paths identify the same file across the CLI and Obsidian. Existing physical file names are retained. This mode currently supports CouchDB only.
Version `1` requires a coordinated migration: set it on the source client before Rebuild, then set it on each remaining client before Fetch. Changing the setting alone does not migrate existing Metadata. Follow [Migrate to Unicode NFC path identity](recovery.md#migrate-to-unicode-nfc-path-identity) for the complete procedure. Do not resume a version `0` client against the migrated remote.
The selected value stays pending through ordinary settings saves and reloads. It becomes active only when the Rebuild or Fetch operation is successfully reserved; cancellation leaves it pending.
Distinct files or retained Metadata documents which derive the same identity stop reconstruction before either database is reset. When replacing the local database, Rebuild and Fetch exclude logical deletions without unresolved conflicts; a deleted winner with unresolved conflicts still participates in the check. This check uses the host's supported file listing; Obsidian checks the paths exposed by its Vault APIs and cannot detect physical aliases hidden by its index. Resolve the duplicate names or synchronisation scope first; Self-hosted LiveSync does not merge them automatically. Case-insensitive handling also applies the existing letter-case rule.
### 4. Compatibility (Internal API Usage)
@@ -1234,7 +1238,7 @@ Enable this setting to process files with size mismatches, which can sometimes b
#### Do not split chunks in the background
Setting key: disableWorkerForGeneratingChunks
If disabled(toggled), chunks will be split on the UI thread (Previous behaviour).
If enabled, chunks will be split on the UI thread (previous behaviour).
#### Process small files in the foreground