public/obsidian-livesync
1
0 Fork 0
mirror of https://github.com/vrtmrz/obsidian-livesync.git synced 2026-11-06 00:08:23 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #1309 from vrtmrz/feature/nfc-journal-p2p

Browse Source 
Expose NFC path identity for Object Storage and P2P
This commit is contained in:
vorotamoroz authored and GitHub committed 2026-11-05 18:39:57 +09:00
2 parents 0fe7d82f46 2716d68b3d
commit 5a21255fb7
47 files changed
+3711 -461
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/adr/2026_10_unicode_path_identity.md
+15 -1
View File
@@ -13,6 +13,10 @@ Accepted on 2026-10-17. Unicode NFC path identity was released in Self-hosted
LiveSync 1.0.38 with Commonlib 0.1.42. Local validation evidence and platform
limits are recorded in the design document.
The maintained design now also describes an unreleased Journal and P2P
extension and the move to General Settings. The decision below retains the
initial CouchDB support boundary and original acceptance date.
## Context
Self-hosted LiveSync derives normal-file Metadata document IDs from Vault paths.
@@ -102,9 +106,19 @@ preflight uses the supported host listing and local Metadata. It refuses
ambiguous file or parent-directory candidates before database reconstruction;
it does not claim to detect physical aliases which a host does not expose.
## Unreleased extension: defaults for new environments
The later Journal and P2P extension uses NFC for newly generated CLI settings
and Obsidian manual onboarding. It retains legacy defaults for stored settings
and old Setup URIs, and reads an existing central remote's declaration before
first participation. This refines the original default decision without
implicitly changing existing document IDs. Shared Commonlib defaults and other
hosts remain unchanged. See the maintained [design](../design_docs/unicode_path_identity.md#initial-settings-and-existing-environments)
for the initial-setup and failure rules.
## Alternatives considered
- **Use NFC by default:** rejected because existing clients and databases must
- **Use NFC by default for all settings:** rejected because existing clients and databases must
retain their current IDs until their owners deliberately migrate them.
- **Normalise only lookups:** rejected because new Metadata writes and deletes
could continue using legacy IDs and create duplicate documents.
docs/design_docs/unicode_path_identity.md
+154 -10
View File
@@ -22,14 +22,19 @@ and is included in current Commonlib `0.2.2`, used by Self-hosted LiveSync
`1.0.42`. This later handling is distinct from the initial NFC identity
release.
An unreleased extension adds Journal and P2P for the same maintained CLI and
Obsidian hosts, moves the setting to General Settings, and defaults new
synchronisation environments to NFC. The original dates
and reviewed versions above remain the historical baseline.
## Scope
| Area | Contract |
| --- | --- |
| Logical identity | Version `1` uses NFC for Metadata ID and path comparison keys. Existing case handling remains a separate rule. Missing and `0` retain legacy behaviour. |
| Stored paths and content | Keep the spelling in `Metadata.path` and existing physical Vault names. NFC does not change Chunk IDs or content. |
| Initial hosts and remote | Support CLI and Obsidian with CouchDB. |
| Other transports | Journal and P2P reject version `1`; Direct file access refuses a remote declaring the NFC feature. Other host adapters remain outside the support matrix. |
| Maintained hosts and transports | CLI and Obsidian with CouchDB; Journal and P2P are added by the unreleased extension. |
| Other consumers | Direct file access refuses a remote declaring the NFC feature. Other host adapters remain outside the support matrix. |
| Migration | Rebuild both databases from one complete authoritative Vault, then Fetch each remaining client. Do not rewrite IDs or merge histories in place. |
## Path identity contract
@@ -63,6 +68,31 @@ under legacy identity cannot be resumed under version `1`, or vice versa. This
reuses the existing checkpoint identity and does not add a separate recovery
store.
## Initial settings and existing environments
The unreleased extension uses version `1` for newly generated CLI settings and
Obsidian manual onboarding. Stored settings and old Setup URIs with no path
version retain `0`. Commonlib's shared settings defaults remain `0`, so this
host policy does not extend to WebApp or WebPeer. An absent settings file alone
does not establish that an existing local database is new.
Before saving initial manual CouchDB or Object Storage setup, the host reads
the remote declaration through its owned connection resource. A configured
remote supplies `0` or `1`; only a confirmed empty remote retains the new
settings' selected version. Read failures and unsupported or contradictory
declarations stop setup without saving a guessed value. The CLI makes the same
read before opening the local database for an unconfigured client's first
explicit Fetch. Ordinary configured startup and source Rebuild do not adopt a
remote's mode automatically.
CouchDB inspection reads the version document without preparing encryption or
creating a database. Without a version document, a non-empty database remains
legacy. Journal inspection reads its existing milestone ranges and preferred
Tweaks without registering a node, preparing a Security Seed, or resetting a
checkpoint. P2P Setup URIs carry the selected version. Manual P2P configuration
requires the owner to match the source; the signalling probe does not establish
the remote database's identity, and replication still rejects a mismatch.
## Admission and database compatibility
Clients validate local and remote database identity before accepting ordinary
@@ -133,8 +163,8 @@ check until a remote deletion has been replicated.
## Reconstruction and client operations
Mode changes are never applied by an ordinary Metadata write. In Obsidian, the
Advanced page offers **Legacy path identity** (`0`) and **Unicode NFC path
identity** (`1`) for CouchDB. A changed `pathNormalizationVersion` remains
General Settings → **File identity** offers **Legacy path identity** (`0`)
and **Unicode NFC path identity** (`1`) independently of the selected transport. A changed `pathNormalizationVersion` remains
pending through ordinary settings saves, the settings-only Apply fallback, and
reload. It becomes active only after the existing Rebuild or Fetch operation
has been successfully reserved; cancellation or failed reservation leaves it
@@ -144,7 +174,8 @@ the settings-only fallback is not offered.
The [user migration procedure](../recovery.md#migrate-to-unicode-nfc-path-identity)
owns the operational sequence: updated and stopped clients, a complete backed-up
authoritative Vault, source Rebuild, then Fetch on each remaining client. The
source operation reconstructs both databases; a receiving client's Fetch
source operation reconstructs both databases for CouchDB and Journal. With
P2P, it reconstructs only the source database. A receiving client's Fetch
replaces only its local database. The [CLI guide](../../src/apps/cli/README.md)
provides the equivalent `rebuild` and `fetch` commands.
@@ -191,11 +222,48 @@ updates and deletions.
| --- | --- | --- |
| CLI with CouchDB | Supported with legacy identity. | Supported with NFC path resolution. |
| Obsidian with CouchDB | Supported with legacy identity. | Supported with NFC path resolution. |
| Journal or P2P | Existing behaviour. | Rejected. |
| CLI or Obsidian with Journal | Existing behaviour. | Added by the unreleased extension; Rebuild remote journals and Fetch other devices. |
| CLI or Obsidian with P2P | Existing behaviour. | Added by the unreleased extension; update and migrate every peer. |
| Direct file access | Existing behaviour. | Refuses a remote requiring `nfc-path-identity-v1`. |
| Other host adapters, including WebApp | Existing behaviour, subject to their existing support. | Outside the initial support matrix until physical-path handling is verified. |
| Older client without this feature | Can use a legacy database. | Cannot join a database declaring `nfc-path-identity-v1`. |
## Journal and P2P extension
The transport moves the Metadata IDs already generated in the local database;
it does not derive a second remote ID. Journal uses its existing compatibility
range with NFC value `4`, plus a required path-version comparison. A legacy
client with ordinary version checking cannot join the rebuilt Journal. The
explicit version override in older clients remains outside this guarantee.
Journal refuses to initialise NFC compatibility when journal objects exist
without their milestone.
Journal shares one read-only identity assessment across ordinary admission,
Fetch before local reset, and ordinary milestone updates. It checks stored
node ranges even when a Rebuild lock has retired the previous accepted nodes.
Rebuild locks the existing milestone for its source node while retaining an
existing compatibility range, and declares the replacement identity only after
the remote journals have
been deleted. If the milestone is missing, the pre-reset lock does not create
one over the existing journals. A failed or interrupted reset therefore keeps
the old identity or the missing declaration.
P2P compares the effective path version through its existing Tweak exchange,
checks local database identity, and reads the peer version document before
replication. Fresh control-only databases can declare the existing NFC feature.
Every participating peer must be updated and migrated; this is not a new
authentication protocol for already deployed clients.
The existing Rebuild and Fetch paths own migration and state reset. Journal
reconstructs the local database and remote objects; P2P Rebuild reconstructs
only the source database. Other devices Fetch the source. Local reset clears
Journal KV checkpoints, while P2P uses its existing source mark and target
checkpoint. No new migration store, checkpoint format, or persistent admission
state is added. An explicit Journal bucket reset also clears the existing
transfer checkpoint when the bucket is already empty, so a remote-only Rebuild
resends the complete local database. P2P retains its existing peer selection
during Fetch.
## Verification contract and limits
The implementation acceptance matrix covers:
@@ -212,8 +280,9 @@ The implementation acceptance matrix covers:
- refusal to write when an existing legacy database has Metadata, refusal of
version-`0` clients against an NFC database, and enforcement when ordinary
mismatch warnings are disabled;
- URI round trips, checkpoint invalidation across path versions, and transport
rejection for Journal, P2P, and Direct file access;
- URI round trips, checkpoint invalidation across path versions, and
rejection for Direct file access; Journal and P2P acceptance and identity
mismatch checks belong to the unreleased extension;
- refusal for ambiguous file or parent-directory candidates before local or
remote reset, preserving both databases on rejection, within the paths
returned by each host's supported listing; and
@@ -252,6 +321,81 @@ Streaming Fast Fetch. Fast Fetch removed the obsolete Metadata and retained
the current NFC document, its active state, and its stored path. The existing
content, creation-time, physical-name, update, and deletion checks also passed.
### Unreleased Journal and P2P validation
The extension was initially checked on Linux with a locally packed Commonlib
`0.2.3` candidate and real Obsidian `1.12.7`. The Self-hosted LiveSync extension
remains unreleased. The initial checks covered:
- real S3-compatible Journal transfer with and without E2EE and Path
Obfuscation, including restart, logical rename and deletion, both conflict
leaves and their revision ancestry, and a full resend after bucket reset;
- real Obsidian and CLI Journal round trips with plain IDs and independent
encrypted IDs: Standard Fetch, identical Metadata IDs and Chunk content,
retention of a pre-existing NFD physical name, edits after CLI process
restart, and deletion reflected back into Obsidian;
- real CLI P2P peers in legacy mode and NFC mode, with and without encryption
in NFC mode: canonical path reads, incompatible-peer refusal, reverse edits
after restart, retained original revision content, logical rename, and
deletion. Existing automatic-host, adapter-replacement, and explicit
disconnect scenarios also passed; and
- two real Obsidian `1.12.7` processes in an isolated Compose project, with NFC
identity in plain and E2EE/Path Obfuscation modes using an independent ID key:
initial Fetch, identical Metadata IDs and Chunk-backed content, reverse edits,
retained original revision bodies, a full application restart and subsequent
transfer, and explicit path-version refusal after ordinary peer permission.
The refused peer retained its sentinel ID, revision, and content and imported
no source Metadata or file. These cases passed on 2026-11-02 with the updated
locally packed candidate after integrating the current main branches.
The Journal migration scenarios also start with existing legacy CLI databases.
The source Rebuild changes its Metadata ID and declares compatibility range
`4`. A receiving client's legacy Fetch is refused while preserving its local
Metadata ID and revision and the remote milestone. After selecting NFC, its
Fetch converges to the source ID and content while preserving the NFD physical
path. Both plain and independently encrypted ID scenarios passed.
On 2026-11-02, the exact published Commonlib `0.3.0` package passed the
consumer source checks, unit tests, and plug-in, CLI, WebApp, and WebPeer builds.
The real Obsidian P2P cases above were repeated successfully with that package.
Real Obsidian and CLI Journal round trips and the legacy-to-NFC migration were
also repeated with plain IDs and independently encrypted IDs, retaining the
physical NFD names and verifying edits after restart and deletion propagation.
The CLI Compose P2P suite also passed with the published package, including
legacy and NFC identity, plain and encrypted NFC transfer, logical rename and
deletion, automatic hosting, adapter replacement, and relay disconnection.
Additional real Obsidian cases with the published package passed legacy-to-NFC
P2P migration with plain IDs and independently encrypted IDs. They seed legacy
NFD Metadata IDs using real Chunks, since Obsidian's file cache normalises
names. The checks covered source Rebuild and restart, Standard Fetch on the
receiving device, removal of obsolete active IDs, exact content, and retention
of physical NFD names. Both cases also passed transfer of a Unicode note,
logical rename with matching new IDs, and deletion after restart. An
incompatible P2P Fetch reset the local database before a peer was selected.
Its physical note retained its exact bytes, and a Fetch with matching settings
completed recovery. This check does not establish preservation of the old
database's revision history.
Real Obsidian Journal cases also passed remote-only Rebuild after deleting
their isolated Object Storage prefix while leaving the source database and
populated transfer checkpoint intact. With both plain IDs and independently
encrypted IDs, a fresh receiving database and its subsequent restart retained
the exact bodies of both conflict leaves, Chunk references, revision IDs, and
each leaf's `_revisions`. The source retained the readable body of its common
ancestor. Fresh receivers retained the ancestor ID without its historical
body; ancestry IDs alone do not establish that an older body is readable. The
[conflict policy](../specs_conflict_resolution.md#file-saving-and-reflection-guarantees)
continues to require an available shared base for conservative automatic merge.
The settings, first-join identity inspection, legacy Setup URI defaults, and
initial CLI Fetch persistence are also covered by unit tests. The additional
migration cases call the running plug-in's production Rebuild and Fetch
methods with real peer selection; reservation through the settings dialogue
has separate coverage.
These checks do not extend NFC support to other hosts or establish native iOS
or macOS filesystem behaviour.
## Non-goals
This design does not:
@@ -262,8 +406,8 @@ This design does not:
- migrate remote document IDs or revision histories in place;
- choose among ambiguous NFC-equivalent files or parents, or merge their
content;
- change the default away from legacy path identity;
- enable version `1` for Journal, P2P, Direct file access, WebApp, or an
- automatically change existing settings or database identity;
- enable version `1` for Direct file access, WebApp, or an
unverified host adapter; or
- claim native iOS or macOS filesystem compatibility.
docs/recovery.md
+3 -3
View File
@@ -51,13 +51,13 @@ Compare the recovered files with the preserved originals, including the affected
## Migrate to Unicode NFC path identity
Use this coordinated migration when CLI and Obsidian clients share a CouchDB database and composed and decomposed Unicode spellings should identify the same path. Version `0` remains the default and preserves legacy identity. Version `1` is supported with CouchDB only; it is unsupported with Object Storage and P2P.
Use this coordinated migration when CLI and Obsidian clients should identify composed and decomposed Unicode spellings as the same path. Existing settings and old Setup URIs retain version `0`, which preserves legacy identity. The unreleased extension defaults new CLI settings and Obsidian manual onboarding to version `1`; this does not migrate an existing database. CouchDB support is released; Object Storage and P2P support belong to the unreleased extension described in the [Unicode path identity design](design_docs/unicode_path_identity.md).
This migration reconstructs the databases from one authoritative Vault. It does not rewrite document IDs in place or rename files.
1. Update every participating CLI and Obsidian client to a version which supports Unicode NFC path identity. Back up a complete authoritative Vault and the CouchDB database. Stop synchronisation and editing on every device until the migration is complete.
1. Update every participating CLI and Obsidian client to a version which supports Unicode NFC path identity. Back up a complete authoritative Vault and the selected remote database or Object Storage bucket. For P2P, preserve the participating devices because there is no central remote backup. Stop synchronisation and editing on every device until the migration is complete.
2. Choose the complete Vault which contains the intended files. Review any composed and decomposed names which would map to the same path identity, and resolve them before reconstruction. If Rebuild or Fetch reports an ambiguous path or Metadata ID, stop and resolve that collision before retrying; LiveSync does not choose or merge the files for you.
3. On the source client, select **Unicode NFC path identity** (`pathNormalizationVersion: 1`). In Obsidian, choose **Overwrite Server Data with This Device's Files**. In the CLI, set the same value in its settings and run `rebuild` with `--force --write-settings`.
3. On the source client, open **General Settings** → **File identity** and select **Unicode NFC path identity** (`pathNormalizationVersion: 1`). In Obsidian, choose **Overwrite Server Data with This Device's Files**. In the CLI, set the same value in its settings and run `rebuild` with `--force --write-settings`. For Object Storage, this replaces the remote journals as well as the local database. For P2P, it prepares only the source device's local database; keep that device online for the other devices to Fetch.
4. On each remaining client, select **Unicode NFC path identity** (`pathNormalizationVersion: 1`) before fetching. In Obsidian, choose **Reset Synchronisation on This Device**. In the CLI, set the same value in its settings and run `fetch` with `--force --write-settings`.
5. Confirm that every client uses version `1` and that the expected files are present before resuming synchronisation. Do not reconnect a version `0` client to the migrated database.
docs/settings.md
+15 -11
View File
@@ -236,6 +236,21 @@ Setting key: showVerboseLog
Show verbose log. Please enable when you report the logs
### 3. File identity
#### Unicode path identity
Setting key: pathNormalizationVersion
Version `0` preserves legacy file identity. Existing settings and Setup URIs which omit this field retain version `0`. New CLI settings and Obsidian manual onboarding use version `1` in the unreleased extension. On first manual setup or an unconfigured CLI's initial Fetch, an existing CouchDB or Object Storage declaration takes precedence; an unavailable or unsupported declaration stops setup. P2P Setup URIs carry the selected version; manual P2P setup requires matching settings on every peer. Version `1` uses Unicode NFC when deriving document IDs, so equivalent composed and decomposed paths identify the same file across the CLI and Obsidian. Existing physical file names are retained. CouchDB support is released. The unreleased extension also supports Object Storage and P2P for the maintained CLI and Obsidian hosts. This is a common local database setting, independently of the active transport.
Version `1` requires a coordinated migration: set it on the source client before Rebuild, then set it on each remaining client before Fetch. Changing the setting alone does not migrate existing Metadata. Follow [Migrate to Unicode NFC path identity](recovery.md#migrate-to-unicode-nfc-path-identity) for the complete procedure. Do not resume a version `0` client against the migrated remote.
The selected value stays pending through ordinary settings saves and reloads. It becomes active only when the Rebuild or Fetch operation is successfully reserved; cancellation leaves it pending.
Distinct files or retained Metadata documents which derive the same identity stop reconstruction before either database is reset. When replacing the local database, Rebuild and Fetch exclude logical deletions without unresolved conflicts; a deleted winner with unresolved conflicts still participates in the check. This check uses the host's supported file listing; Obsidian checks the paths exposed by its Vault APIs and cannot detect physical aliases hidden by its index. Resolve the duplicate names or synchronisation scope first; Self-hosted LiveSync does not merge them automatically. Case-insensitive handling also applies the existing letter-case rule.
## 3. Remote Configuration
### 1. Connection settings
@@ -1166,17 +1181,6 @@ When this setting is disabled, changing only the letter case of a file name with
New Vaults use case-insensitive handling for cross-platform compatibility. Existing settings with an explicit value preserve that choice. Earlier releases also followed the case-insensitive branch when this value was absent, so 1.0 saves a missing legacy value as `false` without requiring a compatibility review or database rebuild.
#### Unicode path identity
Setting key: pathNormalizationVersion
Version `0` preserves legacy file identity and is the default for existing and new settings. Version `1` uses Unicode NFC when deriving document IDs, so equivalent composed and decomposed paths identify the same file across the CLI and Obsidian. Existing physical file names are retained. This mode currently supports CouchDB only.
Version `1` requires a coordinated migration: set it on the source client before Rebuild, then set it on each remaining client before Fetch. Changing the setting alone does not migrate existing Metadata. Follow [Migrate to Unicode NFC path identity](recovery.md#migrate-to-unicode-nfc-path-identity) for the complete procedure. Do not resume a version `0` client against the migrated remote.
The selected value stays pending through ordinary settings saves and reloads. It becomes active only when the Rebuild or Fetch operation is successfully reserved; cancellation leaves it pending.
Distinct files or retained Metadata documents which derive the same identity stop reconstruction before either database is reset. When replacing the local database, Rebuild and Fetch exclude logical deletions without unresolved conflicts; a deleted winner with unresolved conflicts still participates in the check. This check uses the host's supported file listing; Obsidian checks the paths exposed by its Vault APIs and cannot detect physical aliases hidden by its index. Resolve the duplicate names or synchronisation scope first; Self-hosted LiveSync does not merge them automatically. Case-insensitive handling also applies the existing letter-case rule.
### 4. Compatibility (Internal API Usage)