public/obsidian-livesync
1
0
Fork 0
mirror of https://github.com/vrtmrz/obsidian-livesync.git synced 2026-10-01 23:12:32 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge current main to document the Spanish translation update

Browse Source
This commit is contained in:
vorotamoroz
2026-09-30 17:38:50 +00:00
parent 990a026e85 c83fe1b08c
commit bff70e6ab8
83 changed files with 5033 additions and 304 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+2 -1
View File
@@ -98,7 +98,8 @@ Synchronisation status is shown in the status bar with the following icons.
- 🛫 Pending read storage processes
- 📬 Batched read storage processes
- ⚙️ Working or pending storage processes for hidden files
- 🧩 Waiting chunks
- 🛄 Pending initial on-demand chunk requests
- 🔁 Pending chunk retries, including retry delays
- 🔌 Working customisation items (configuration, snippets, and plug-ins)
To prevent file and database corruption, please avoid closing Obsidian until all progress indicators have disappeared as much as possible (although the plug-in will attempt to resume if interrupted). This is especially important if you have deleted or renamed files.
devs.md
+12
View File
@@ -235,6 +235,18 @@ Commonlib owns the typed English fallback for messages requested by its services
### Logging & Debugging
#### ID generation measurements on a device
Enable **Enable Developers' Debug Tools.**, restart Obsidian, and run **Open review harness** from the command palette. Choose **Run** beside **ID generation performance**, keep Obsidian in the foreground, and use **Copy Markdown report** to retain the results. The **Automatic** action does not run this measurement; **Full review** includes it.
The measurement uses fixed in-memory inputs and keys, with no Vault, database, settings, or remote writes. It compares legacy `xxhash64` and independent Chunk IDs for 256-byte, 4-KiB, and 32-KiB inputs, and compares obfuscated document IDs. Each result reports the median and range of three 1,000-ID samples and the median time per ID. Key derivation at save time is measured separately. Warm-up and pauses between batches are excluded from the timings. These measurements do not represent a full Rebuild.
Where `performance.memory` is available, the report includes approximate JavaScript heap samples before, during, and after measurement. These may include other Obsidian activity and garbage collection; they are neither total process RAM nor an exact peak. Unsupported devices explicitly report that heap measurements are unavailable.
The developer-only adapter in `src/features/ReviewHarness/reviewHarnessIdBenchmarkRuntime.ts` imports `HashManager` from Commonlib's public `/hashing` entry. Compilation, packed-package checks, and runtime tests cover this boundary. The algorithms remain owned by Commonlib.
#### Logs
- Use `this._log(msg, LOG_LEVEL_INFO)` in modules (automatically prefixes with module name)
- Log levels: `LOG_LEVEL_DEBUG`, `LOG_LEVEL_VERBOSE`, `LOG_LEVEL_INFO`, `LOG_LEVEL_NOTICE`, `LOG_LEVEL_URGENT`
- LOG_LEVEL_NOTICE and above are reported to the user via Obsidian notices
docs/adr/2026_07_bounded_remote_activity.md
+2 -2
View File
@@ -43,7 +43,7 @@ Applying downloaded document changes enters one local-application boundary from
Manual P2P commands which bypass `ReplicationService` enter the broad boundary. Direct P2P pull and push entry points are therefore both protected as finite remote work, covering the Obsidian panes, CLI, and Webapp. A pull or bidirectional synchronisation also enters the narrower finite-replication boundary because it can place documents in the local database. A push-only request remains broad-only: it cannot satisfy a local missing-chunk read and must not present itself as a delivery source. Automatic synchronisation on peer discovery, a pull requested by a remote peer, and a watched pull following a peer progress notification enter both boundaries because each can deliver local documents. A normal P2P peer-selection dialogue represents one broad finite session: it remains inside the boundary while waiting for a peer and while the person may perform repeated synchronisations, then settles when the dialogue closes and any in-flight synchronisation has finished. Closing without synchronising returns a failed result and releases the boundary. The 'Start Sync & Close' action completes its synchronisation before closing. This deliberately protects peer discovery and selection, because display sleep can interrupt discovery or connection establishment and require the person to start detection again. It may therefore retain a Wake Lock longer than the network transfer alone. A transfer performed inside that session temporarily adds a nested activity; the count remains a logical-operation count rather than a connection total.
`ChunkFetcher` enters the broad boundary synchronously when it accepts newly missing chunk identifiers, but it does not increment the finite-replication count. A typed per-identifier claim keeps the broad boundary active through queue waiting, interval throttling, `fetchRemoteChunks`, validation, local persistence, and terminal event delivery. Duplicate requests share the existing claim. Explicit absence, failure, cancellation, or a conservative five-minute period without fetcher-observable progress settles the affected claim. The five-minute value is only a last-resort leak fuse: it prevents a never-settling integration from retaining the per-identifier claim and waiter indefinitely and, once the activity runner has entered the claim task, lets the associated Wake Lock, lifecycle deferral, and indicator finish. It is not an arrival estimate, proof of remote absence, or a transport deadline. This scope is defined in detail by the chunk-arrival-quiescence ADR.
`ChunkFetcher` enters the broad boundary synchronously when it accepts newly missing chunk identifiers, but it does not increment the finite-replication count. A typed per-identifier claim keeps the broad boundary active through queue waiting, interval throttling, `fetchRemoteChunks`, validation, local persistence, and terminal event delivery. Duplicate requests share the existing claim. A first successful omission schedules a two-second retry; further omissions increase the per-identifier delay up to ten seconds while finite replication remains active. Backoff releases physical concurrency without releasing the bounded task. Observed finite completion interrupts backoff for a local recheck and, if needed, a final remote probe. A current final successful omission settles the claim; failure, cancellation, or a conservative five-minute period without fetcher-observable progress also settles it. The five-minute value is only an inactivity leak fuse, not a total retry budget: it prevents a never-settling integration from retaining the per-identifier claim and waiter indefinitely and, once the activity runner has entered the claim task, lets the associated Wake Lock, lifecycle deferral, and indicator finish. It is not an arrival estimate, proof of remote absence, or a transport deadline. This scope is defined in detail by the chunk-arrival-quiescence ADR.
Rebuild operations use the same boundary at their destructive or remote phase:
@@ -105,7 +105,7 @@ Unit tests cover:
- direct P2P pull and push entry points entering the broad boundary, while only pull and bidirectional operations enter the finite-delivery boundary;
- automatic synchronisation on peer discovery, remote pull requests, and watched peer progress entering the boundary;
- P2P peer-selection sessions settling on close, including cancellation, repeated synchronisation, and a close during in-flight work;
- remote chunk fetching remaining inside the shared boundary from synchronous queue acceptance through local persistence and terminal notification;
- remote chunk fetching remaining inside the shared boundary from synchronous queue acceptance through the delayed retry of omitted identifiers, local persistence, and terminal notification;
- missing-chunk waiters rechecking local storage when observed per-identifier claims and finite replication have settled;
- the finite-replication count excluding other bounded work;
- replicated document application sharing one local boundary, settling after the final recovery snapshot, and releasing around processing suspension;
docs/adr/2026_07_chunk_arrival_quiescence.md
+31 -7
View File
@@ -12,6 +12,8 @@ Before this decision, a missing-chunk waiter used a fixed 5-second or 30-second
Finite replication provides a stronger boundary than elapsed wall-clock time. In the absence of an error, a finite replication does not complete until it has reached the latest sequence in its scope. Once it completes, that replication cannot deliver another chunk. An on-demand fetch is a separate finite delivery path and needs its own per-identifier boundary.
A successful CouchDB lookup which omits a requested chunk is a point-in-time observation. Metadata and Chunk documents can become visible separately, so the first such result immediately after Metadata arrives does not prove that a second lookup will return the same result. The on-demand fetch lifecycle therefore needs to distinguish an initial omission from its terminal result.
## Decision
Wait only for a delivery lifecycle which is observable when the local miss is handled. Do not guess how long an unobserved producer might take.
@@ -29,7 +31,7 @@ The waiting layer follows these rules:
2. Dispatch `missingChunks` synchronously when direct fetch is permitted. `ChunkFetcher` must claim accepted identifiers before dispatch returns, closing the scheduling gap without a timer.
3. If a matching claim or finite replication is active, wait for that observable producer.
4. A valid chunk arrival resolves the waiter immediately.
5. An explicit remote-missing result resolves it as missing immediately.
5. A terminal explicit remote-missing result resolves it as missing immediately.
6. Once every observed producer completes, read the requested identifier from the local database once more, bypassing the cache.
7. Return the rechecked chunk, or return unavailable. Do not add a fixed grace period after the producer has stopped.
8. If no producer is observable after synchronous dispatch, return unavailable immediately. There is no operation for a duration to represent.
@@ -46,17 +48,24 @@ A successful finite replication completion is the authoritative ‘latest’ bou
- configured interval throttling;
- entry into the injected bounded-activity runner;
- `fetchRemoteChunks`;
- scheduled retries for identifiers omitted from successful responses;
- response validation;
- local database persistence; and
- fetched or missing event delivery.
The claim settles on every terminal path, including explicit absence, no active replicator, rejection, invalid results, destruction, and cancellation. Its completion Promise is the task passed to the bounded `chunk-fetch` activity, keeping Wake Lock, application lifecycle deferral, the remote-work indicator, and missing-chunk delivery aligned.
When the first successful response omits an identifier, `ChunkFetcher` keeps that identifier claimed and schedules a retry after two seconds, even if no finite replication is active. Identifiers present in a partial response are persisted and settled immediately. While finite replication remains active, subsequent omissions schedule per-identifier delays of four, six, eight, and then ten seconds, capped at ten seconds. Backoff returns the physical request slot to the queue without releasing the logical claim. Eligible retries can share a batch with new identifiers, and the queue wakes without requiring a new missing-chunk event.
When the observed finite count falls to zero, the fetcher interrupts any remaining backoff. It rechecks local persistence and, if the identifier is still absent, makes one final remote probe, subject to the configured request interval and concurrency. An already-running lookup cannot overlap another lookup for that identifier. If it began before the latest finite completion, an omitted identifier still needs a post-completion probe. A new finite operation which ends during that probe similarly makes its negative result stale. A successful omission from a current final probe emits `missingChunkRemote` and settles the current read; it does not leave a retry pending for a future synchronisation.
The retry policy observes only finite replication, not the claim itself or the broader bounded-activity count. It does not treat the continuous live channel as a finite producer. There is no fixed total duration or attempt limit while finite replication continues; the delay cap controls request frequency, not overall lifetime. No retry state is persisted, and transport errors do not enter this missing-result retry policy.
The claim settles on every terminal path, including explicit absence after the retry, no active Replicator, rejection, invalid results, destruction, and cancellation. Its completion Promise is the task passed to the bounded `chunk-fetch` activity, keeping Wake Lock, application lifecycle deferral, the remote-work indicator, and missing-chunk delivery aligned.
### Five-minute leak fuse
An accepted on-demand claim has a separate five-minute inactivity fuse. This is a last-resort leak safety valve, not a chunk-arrival budget or a remote-request timeout.
Its purpose is to prevent a faulty integration, a never-settling Promise, or a stalled transport from retaining logical ownership indefinitely. When it fires, the coordinator releases the per-identifier claim and its waiter. If the bounded activity callback has been entered, resolving the claim also allows the associated Wake Lock, application-lifecycle deferral, and remote-work indicator to be released. `ChunkFetcher` refreshes the fuse only at observable progress points, such as entering the activity boundary, beginning and completing throttling or transfer, and completing persistence.
Its purpose is to prevent a faulty integration, a never-settling Promise, or a stalled transport from retaining logical ownership indefinitely. When it fires, the coordinator releases the per-identifier claim and its waiter. If the bounded activity callback has been entered, resolving the claim also allows the associated Wake Lock, application-lifecycle deferral, and remote-work indicator to be released. `ChunkFetcher` refreshes the fuse at observable progress points, including entry into the activity boundary, request scheduling, transfer, and persistence. Repeated successful missing responses are progress for this fuse, so it is not a five-minute total limit on retries during active finite replication.
Five minutes is deliberately a conservative operational limit, not a value derived from a network protocol, a benchmark, or evidence that a missing chunk will arrive within that period. Firing the fuse neither proves remote absence nor makes the underlying request safe to abort. The current `fetchRemoteChunks` contract has no `AbortSignal`, so a physical request may still complete after its logical claim has been released. A future cancellable transport contract should add transport-specific deadlines and explicit cancellation without changing the lifecycle-based wait rule.
@@ -64,7 +73,7 @@ Five minutes is deliberately a conservative operational limit, not a value deriv
The unbounded live channel is not a quiescence gate because it has no natural end. Its initial pull-only catch-up is finite, however, and must enter `runFiniteReplicationActivity`. This includes the one-shot parameter fallback chain: every retry remains within the catch-up boundary until it succeeds or stops. If continuous replication later restarts with adjusted parameters, the new initial catch-up enters a new finite boundary.
Once the live channel has begun, a chunk delivered through it still resolves an existing waiter immediately, but the channel itself does not keep a new waiter open. CouchDB on-demand fetching supplies its own per-identifier claim. If a future defect demonstrates a delivery race inside a live batch, that batch lifecycle should be exposed explicitly rather than approximated with another elapsed delay.
Once the live channel has begun, a chunk delivered through it still resolves an existing waiter immediately, but the channel itself does not keep a new waiter open. CouchDB on-demand fetching supplies its own per-identifier claim and at least one delayed follow-up lookup. Further retries require active finite replication; the live channel does not extend them. If another future defect requires waiting for a live batch itself, that batch lifecycle should be exposed explicitly rather than approximated with another elapsed delay.
## Ownership
@@ -80,7 +89,7 @@ Once the live channel has begun, a chunk delivered through it still resolves an
- Treat a positive deprecated `timeout` only as source-compatible opt-in to lifecycle waiting. Its numeric value no longer represents an arrival duration.
- Preserve `preventRemoteRequest`: no on-demand request is dispatched, although an already-active finite replication may satisfy the waiter.
- Preserve Promise sharing for concurrent reads of the same chunk identifier.
- Preserve immediate explicit remote-missing results.
- Preserve immediate waiter resolution when `ChunkFetcher` emits a terminal explicit remote-missing result.
- Do not change which remote types support direct on-demand fetching.
## Historical Evidence and Scope
@@ -90,6 +99,7 @@ This decision addresses the lifecycle-race class rather than treating every ‘L
- [Issue #166](https://github.com/vrtmrz/obsidian-livesync/issues/166) contained logs where chunk collection failed shortly before related chunk writes appeared. It is evidence for the timing class, although that issue's hidden-file start-up path was repaired separately and is not claimed as a direct regression test here.
- The 2021 timing fixes in [commit `39e2eab0`](https://github.com/vrtmrz/obsidian-livesync/commit/39e2eab0238d9c37e3653cdec884cbeed543fc23) and the extended leaf timeout in [commit `9facb577`](https://github.com/vrtmrz/obsidian-livesync/commit/9facb577601d8aceff7df547cd2a6f9357fdaa29) show that elapsed timeout values have historically been used to absorb the same ordering uncertainty. They do not provide a protocol basis for retaining 5-second or 30-second delays.
- Replication pacing introduced by [commit `8d66c372`](https://github.com/vrtmrz/obsidian-livesync/commit/8d66c372e15c43a2de84a223c6385077b7724eec) and commonlib [commit `051b50c`](https://github.com/vrtmrz/livesync-commonlib/commit/051b50ca38ec4c05a11e8216ac259b4488b825f0) is a direct precedent for preventing replication progress from outrunning chunk collection. The present design expresses that dependency as an explicit lifecycle and completion recheck.
- [Issue #1224](https://github.com/vrtmrz/obsidian-livesync/issues/1224) reports repeatable burst edits where the receiving device observes Metadata, finds its Chunk absent in the first direct lookup, and then receives the Chunk after the file read has already failed. Retrying addresses that transient ordering case. It does not claim to reconstruct a Chunk which was never uploaded.
- [Issue #505](https://github.com/vrtmrz/obsidian-livesync/issues/505) was traced to chunks which were genuinely absent after the former bulk-send option broke the chunks-before-metadata guarantee. Waiting cannot recreate missing data, so this decision does not claim to fix it.
- [Issue #771](https://github.com/vrtmrz/obsidian-livesync/issues/771) and [Issue #986](https://github.com/vrtmrz/obsidian-livesync/issues/986) contain ambiguous or version-dependent `Load failed` reports. They remain unclaimed until the original writer and database state can be reproduced.
@@ -113,6 +123,8 @@ The broad count includes operations which cannot provide the requested chunk. Us
An unobserved producer has no defined start, progress, or completion semantics. A timer would therefore be a guess rather than a safety property. Relevant delivery paths must claim their work synchronously or expose a finite replication boundary; otherwise the read returns unavailable.
The on-demand backoff is not such a fallback timer. `ChunkFetcher` retains the identifier claim throughout each delay and owns the lookup it schedules. Once finite activity ends, the fetcher cuts the delay short and uses a current final probe rather than waiting for an unobserved future producer.
### Remove every timer
The arrival wait has no elapsed timer, but an implementation fault can leave a delivery claim unresolved forever. The five-minute inactivity fuse bounds that leaked logical state without being used as a successful delivery condition.
@@ -125,6 +137,15 @@ Unit tests use deterministic clocks and deferred Promises to cover:
- successful finite completion causing a cache-bypassing local database recheck;
- immediate unavailability when no producer is observable;
- per-identifier claims covering queueing, throttling, remote fetch, validation, persistence, and event delivery;
- an autonomous two-second retry after a first successful omission, retaining the claim and bounded remote activity;
- per-identifier backoff capped at ten seconds, with physical concurrency released during each delay;
- mixed retry stages sharing batches without resetting their individual delays or starving eligible retries;
- finite completion interrupting backoff, local persistence avoiding the final request, and pre-completion lookups requiring a current final probe;
- partial results settling available identifiers and retrying only absent identifiers;
- partial-batch completion and expired-request responses preserving replacement claims;
- concurrent request starts respecting the configured interval after retry and throttle waits;
- a final successful omission producing the terminal remote-missing result;
- disjoint initial and retry counts, aggregate ownership across fetchers, and timer cleanup;
- explicit missing, no-replicator, rejection, invalid response, cancellation, runner rejection, and teardown paths;
- overlapping claims and finite replications;
- a runner which never enters the task and a request which never settles;
@@ -132,13 +153,16 @@ Unit tests use deterministic clocks and deferred Promises to cover:
- continuous replication's finite initial catch-up, including its parameter fallback path; and
- the setting and replicator decision matrix.
Integration-style unit tests exercise `LayeredChunkManager`, `ChunkFetcher`, a memory-backed PouchDB database, and a deferred fake replicator together. A real Obsidian test is not required because the change remains behind the existing database, service, and event boundaries and does not alter platform UI or an adapter contract.
Integration-style unit tests exercise `LayeredChunkManager`, `ChunkFetcher`, a memory-backed PouchDB database, and a deferred fake replicator together. The downstream `chunk-fetch-retry` scenario exercises real CouchDB lookup results, local persistence, Vault reflection, and the actual status bar in Obsidian. Deterministic clock tests remain responsible for the exact backoff schedule and overlapping completion races; the real-runtime scenario does not substitute HTTP responses or synthesise finite-activity counts.
## Consequences
- A healthy finite replication or on-demand request no longer loses a race against an unrelated wall-clock estimate.
- A Chunk omitted from the first successful CouchDB lookup receives a follow-up lookup while the same delivery claim remains active, and further retries while finite replication continues.
- Successful finite replication completion provides a precise latest boundary for missing-chunk reads.
- The local recheck closes event-delivery and cache timing gaps without extending the wait after completion.
- Reads no longer pause for 5 or 30 seconds when no observable operation can deliver the chunk.
- Reads no longer pause for the former 5-second or 30-second arrival budgets. Backoff schedules observable requests rather than setting an elapsed arrival deadline.
- With no finite replication active, a genuinely absent remote Chunk takes one additional lookup after two seconds. Active finite replication can extend that lifetime; its completion expedites the final probe.
- The status bar separates initial pending identifiers (`🛄`) from retrying identifiers (`🔁`). Their sum remains the internal pending count used for restart deferral.
- Relevant producers must expose a lifecycle and must continue to prove cleanup on every exceptional path.
- The five-minute fuse bounds leaked logical activity, but it neither establishes remote absence nor cancels a physical request.
docs/adr/2026_07_release_notes_and_database_compatibility.md
+3 -1
View File
@@ -57,6 +57,8 @@ Keep configured-state inference separate from new-Vault initialisation. If an ex
- On resume, clear `versionUpFlash` and persist that fail-closed change before recording the current `VER` as acknowledged. If saving fails, restore the gate. Reapply settings only after the marker has advanced so that the previously configured synchronisation behaviour can resume without reconstruction.
- Preserve the original legacy review message as a structured reason when no more specific database or settings-schema reason is available. Escape it before including it in Markdown UI.
- Continue to reject a remote version document which is newer than the running implementation. That receiver-side check is independent of the local upgrade review.
- From remote generation 13, assess the `used_features` list in that document as a separate compatibility dimension. A client must recognise every listed feature before it interprets the database or runs maintenance which depends on Metadata. Declare a feature before writing its representation, and retain the declaration while older data may depend on it. An unknown identifier is reported as text without requiring a descriptive label in that client.
- Do not advance the device-local `VER` acknowledgement merely because a remote feature is introduced. The remote generation and its feature list govern remote admission; `VER` remains the local compatibility review gate. Connecting to a generation-12 database does not promote it solely because the client understands generation 13.
### Onboarding activation and initialisation
@@ -90,7 +92,7 @@ Keep configured-state inference separate from new-Vault initialisation. If an ex
- Accepted new-device and existing-device setup cannot enable ordinary processing before the selected Rebuild or Fetch has been reserved.
- An older installation cannot dismiss evidence that a newer implementation or settings schema has already been used on the device.
- The Obsidian-specific dialogue depends only on a host-neutral compatibility result and the injected confirmation capability. Commonlib remains responsible for settings migration, device-local storage, and the replication gate.
- A future incompatible database change must increment `VER`, provide an actionable review message, verify the remote version negotiation, and test both the pending and acknowledged states. A major SemVer increase without those changes has no database-compatibility effect.
- A future local database change which requires compatibility review must increment `VER`, provide an actionable review message, and test both the pending and acknowledged states. A new remote representation must declare its feature before use and verify remote admission independently. A major SemVer increase alone has no database-compatibility effect.
## Verification
docs/design_docs/chunk_retrieval_and_waiting.md
+35 -9
View File
@@ -53,7 +53,9 @@ For CouchDB, `readChunksOnline` changes what normal replication includes, not th
| `true` | No | Primary chunk delivery after metadata arrives. |
| `false` | Yes | Recovery fallback for a chunk which is unexpectedly unavailable locally. |
`concurrencyOfReadChunksOnline` and `minimumIntervalOfReadChunksOnline` affect only the scheduling of CouchDB on-demand requests. They do not change whether a request may be dispatched or which lifecycle a reader observes. Accepted identifiers remain claimed while they wait for a concurrency slot and while the configured interval is applied. A minimum interval of five minutes or more is an exceptional value: the inactivity fuse may release the logical claim before that deliberate pause completes. This safety precedence does not abort the delayed physical request.
`concurrencyOfReadChunksOnline` and `minimumIntervalOfReadChunksOnline` affect only the scheduling of CouchDB on-demand requests. They do not change whether a request may be dispatched or which lifecycle a reader observes. Accepted identifiers remain claimed while they wait for a concurrency slot and while the configured interval is applied. Backoff releases the physical concurrency slot but retains logical ownership. Eligible retries and new identifiers can share a batch of up to 100 identifiers. An owned timer wakes the queue at the earliest eligible retry without requiring a new event. New arrivals do not reset existing retry times, and eligible older work precedes newer queued work.
After every wait or asynchronous local recheck, the fetcher checks the configured interval against the latest shared request time and reserves its start synchronously. A minimum interval of five minutes or more is an exceptional value: the inactivity fuse may release the logical claim before that deliberate pause completes. An expired claim is not dispatched after the pause; an already-running physical request is not aborted by the fuse.
## Wait State Machine
@@ -62,11 +64,12 @@ For CouchDB, `readChunksOnline` changes what normal replication includes, not th
3. Register one shared waiter per missing identifier.
4. If policy permits direct fetch, emit `missingChunks`. `ChunkFetcher` synchronously creates the per-identifier claim before the event dispatch returns.
5. Observe both the matching claim and `finiteReplicationActivityCount`.
6. Resolve immediately if a valid chunk or explicit remote-missing event arrives.
7. If an observed producer remains active, do not charge elapsed time against an arrival budget.
8. When all observed producers end, bypass the cache and read the identifiers from the local database once.
9. Return the rechecked chunk, or return unavailable. Do not add another fixed grace after the authoritative boundary.
10. If no producer is observable after synchronous dispatch, return unavailable immediately.
6. Resolve immediately if a valid chunk or terminal explicit remote-missing event arrives.
7. If a successful direct-fetch response omits an identifier, apply the per-identifier retry policy below. Settle identifiers present in a partial result and retry only those which remain absent.
8. If an observed producer remains active, do not charge elapsed time against a general arrival budget.
9. When all observed producers end, bypass the cache and read the identifiers from the local database once.
10. Return the rechecked chunk, or return unavailable. Do not add another fixed grace after the authoritative boundary.
11. If no producer is observable after synchronous dispatch, return unavailable immediately.
If new relevant activity starts while the final database recheck is pending, that result becomes stale. The waiter remains active until the newer producer completes and a current recheck finishes.
@@ -84,18 +87,30 @@ The continuous live channel is intentionally excluded because it has no completi
## Meaning of an On-demand Claim
An accepted identifier remains claimed from synchronous queue acceptance through throttling, physical fetch, validation, local persistence, and terminal event delivery. The claim is identifier-scoped because a global remote-work count cannot say whether unrelated work can provide this chunk.
An accepted identifier remains claimed from synchronous queue acceptance through throttling, physical fetch, validation, local persistence, retry delays, and terminal event delivery. The claim is identifier-scoped because a global remote-work count cannot say whether unrelated work can provide this chunk.
The claim finishes when the fetcher has recorded an outcome for the identifier. A transport error, missing active replicator, or invalid result releases the claim without emitting an explicit remote-missing result unless the remote actually supplied that information.
The claim finishes when the fetcher has recorded an outcome for the identifier. A successful first omission schedules a two-second retry even if finite replication is already inactive. While finite replication remains active, each subsequent omission increases that identifier's delay by two seconds, up to ten seconds: `2, 4, 6, 8, 10, 10, ...`. Joining a different batch does not reset its retry stage. Each retry first bypasses the local cache and disables further remote dispatch and delivery waiting for its local recheck.
When the observed finite count falls to zero, remaining backoff is interrupted. The fetcher rechecks local persistence and makes a final remote probe only for still-absent identifiers, respecting concurrency and minimum request spacing. A pre-completion in-flight lookup cannot count as that final probe: if it omits the identifier, a subsequent post-completion lookup is needed, without overlapping requests for the same identifier. If another finite operation ends during the final probe, its negative result is stale too. The current final successful omission emits the terminal explicit remote-missing result and settles the current read. No retry remains for a future synchronisation.
A transport error, missing active Replicator, or invalid result releases the claim according to its existing terminal path rather than entering this missing-result retry. The retry gate is finite replication alone: neither the fetcher's own claim nor broader bounded remote work keeps it alive. There is no attempt limit or absolute elapsed limit while finite replication continues. All retry state is in memory; destruction and inactivity expiry remove queued work and its timer.
Each request retains the identity of the claims it accepted. If another read claims the same identifier after an earlier claim settles or expires, the earlier request cannot release the replacement claim, refresh its fuse, or report unavailability for it. This also applies when a partial batch has already settled one identifier but is still retrying another.
The status indicators count two disjoint sets of pending on-demand Chunk identifiers. `🛄` shows identifiers with no successful missing response yet; `🔁` shows identifiers omitted at least once, including backoff, retry requests, and the final probe. Thus `🛄3 🔁2` means five pending identifiers. The atomic `chunkFetchCounts` snapshot provides this classification, while `collectingChunks` retains their total for the existing restart-deferral check. Moving between categories does not change that total.
Each fetcher contributes its unique accepted identifiers from queueing through terminal delivery. Repeated requests and retry attempts do not increase the count. Settled, expired, and destroyed claims leave the count; one fetcher's teardown preserves another fetcher's contribution. These are not counts of replication connections or all missing Chunks. Zero means that no on-demand claims remain, not that every Chunk was retrieved successfully.
## Meaning of the Five-minute Value
The five-minute value is an inactivity leak fuse for an accepted on-demand claim. It is the only elapsed duration in this state machine, and it is not a normal terminal condition.
The five-minute value is an inactivity leak fuse for an accepted on-demand claim. It is not a normal terminal condition and is distinct from the backoff which schedules identified follow-up lookups.
The fuse bounds retention if a faulty activity runner never enters its task, a Promise never settles, or a transport stops making observable progress. It prevents the per-identifier claim and waiter from remaining live forever. Once the bounded activity callback has entered, releasing the claim also allows Wake Lock, application-lifecycle deferral, and the remote-work indicator associated with that callback to finish. Observable progress rearms the fuse.
Five minutes is a conservative operational ceiling rather than a measured chunk-arrival expectation. It must not be used to infer that the remote lacks a chunk, and it does not abort the physical request. `fetchRemoteChunks` does not yet accept an `AbortSignal`, so the request may complete after the logical state has been released. Transport cancellation and transport-specific deadlines are separate future work.
Backoff does not resolve the waiter by elapsed time or prove that another producer will deliver the Chunk. Successful missing responses refresh the inactivity fuse, so retries can continue beyond five minutes while finite replication remains active. This is an intentional distinction between inactivity protection and a total lifetime limit.
The old 5-second and 30-second constants remain exported for source compatibility only. A positive deprecated `ChunkReadOptions.timeout` opts into lifecycle waiting, but its numeric value is ignored. Zero or a negative value still requests an immediate result. New code uses `waitForDelivery` explicitly.
## Test Obligations
@@ -110,6 +125,17 @@ Changes to this behaviour must keep automated coverage for:
- overlapping finite operations and overlapping per-identifier claims;
- activity restarting while a local recheck is pending;
- direct fetch queueing, throttling, persistence, and terminal notification;
- an autonomous two-second retry after a first successful omission, with bounded remote activity retained throughout;
- per-identifier `2, 4, 6, 8, 10, 10, ...` backoff while finite replication is active;
- backoff releasing physical concurrency, mixed-stage batching, and eligible retries not being starved by new work;
- expiry of the earliest queued claim preserving the scheduled retry for later identifiers;
- finite completion interrupting backoff and pre-completion in-flight requests requiring a current final probe;
- local persistence avoiding an unnecessary retry, including completion during request-interval throttling;
- partial fetch results settling available identifiers and retrying only absent identifiers;
- partial-batch completion and expired-request responses preserving replacement claims;
- disjoint initial and retry counts retaining queued identifiers without duplicates and releasing only settled ownership;
- concurrent request starts respecting the configured interval after retry and throttle waits;
- terminal unavailability after a current final successful omission;
- explicit remote absence versus transport or replicator failure;
- runner rejection, cancellation, teardown, and an operation which never enters its task;
- leak-fuse refresh at observable progress points; and
docs/design_docs/configurable_id_derivation.md
+515
View File
@@ -0,0 +1,515 @@
---
date: 2026-09-29
commonlib-version: "0.1.33"
self-hosted-livesync-version: "1.0.32"
status: unreleased
---
# Configurable ID derivation
## Purpose and baseline
Introduce an optional, saved secret for deterministic Chunk IDs and obfuscated
Metadata document IDs. This allows an E2EE passphrase to change without also
changing those IDs, and allows their derivation to use an independent secret.
Identical inputs must produce identical IDs on participating devices so that
Chunks can be reused and edits to the same path share one document identity.
The baseline is [PR #1222](https://github.com/vrtmrz/obsidian-livesync/pull/1222),
including its passphrase-persistence correction at commit
`126d6eadb858a79a08ad7f600061e54fc8d31196`. Commonlib `0.1.33`, published with
the `next` tag, provides the construction described here. It replaces the
independent Chunk algorithm from the `0.1.32` prerelease without a
compatibility branch or a new settings version. Legacy ID generation remains
unchanged. LiveSync pins the published `0.1.33` package and its registry
integrity in the lockfile.
Its [Internal Metadata encryption design](https://github.com/vrtmrz/obsidian-livesync/blob/126d6eadb858a79a08ad7f600061e54fc8d31196/docs/design_docs/internal_metadata_encryption.md)
remains the basis for Properties encryption and CouchDB feature admission.
This document records an unreleased LiveSync feature. It does not select a
plug-in release version.
## Feasibility
The change is feasible within the existing architecture. Commonlib already
centralises Chunk hashing, path-to-ID conversion, settings persistence, and
Setup URI encoding. Chunk reads follow stored IDs, so changing the generator
does not require a new Chunk reader or content representation.
The work spans Commonlib and its consumers. The principal constraints are
agreement on document IDs, complete propagation of the saved secret, and cache
behaviour after a setting change. The construction and transport-specific
agreement checks are described below. No database migration framework is
required.
## Scope
| Value or operation | Proposed behaviour |
| -------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Encrypted Chunk IDs | Use the saved ID secret in the new mode. Keep legacy generation when the option is absent. |
| Obfuscated Metadata document IDs | Use the same saved secret with a separate derivation purpose. Preserve existing path normalisation and namespace handling, including ordinary files, `i:`, `ix:`, and supported legacy `ps:` entries. |
| Unobfuscated document IDs | Retain the current path-based identity. |
| Content and Properties encryption | Continue using the E2EE passphrase and existing encryption format. |
| CouchDB/PouchDB `_rev` | Retain existing revision generation and replication behaviour. |
| Internal content digests and transport bookkeeping | Retain existing behaviour unless they directly construct one of the IDs above. |
Document IDs are already assigned in the local database. Properties encryption
protects the path and other fields during transfer, while preserving `_id`.
Consequently, the saved ID secret must reach local path conversion as well as
remote-facing code. Path Obfuscation continues to control whether this
conversion is used; this proposal does not enable it automatically.
Journal keys which incorporate document IDs inherit the resulting IDs. They do
not need another secret. Content digests inside encrypted Customisation Sync
content also do not need a separate setting.
Automatic migration or enablement of existing or already migrated users,
Setup URI expiry or revocation, QR format security changes, revision redesign,
remote-only E2EE passphrase rotation, and a new remote configuration management
protocol are outside this change.
## Input and saved state
New Vault setup selects independent ID derivation and a random source by
default when E2EE is enabled. An existing Vault with no saved key retains
legacy mode until the user selects a new key explicitly. The setup dialogue
shows three radio choices:
1. Keep current configuration, selected by default for an existing Vault. It
retains the saved key when present and otherwise retains legacy ID
generation. A small description below this choice shows which configuration
is currently saved. In legacy mode, changing the E2EE passphrase still
changes IDs.
2. Generate a random ID key, selected by default for a new Vault.
3. Set an ID key. Three nested radio choices derive it once from the current
E2EE passphrase, accept a source string, or import a tagged recovery code.
Only the latter two show the text input.
The action is not persisted. An ordinary source is converted to a key when the
settings are applied, and then discarded. A tagged `sls-id-v1:` recovery code
imports its exact 256-bit key without deriving it again, including when pasted
into the source-string input. The recovery-code choice accepts only tagged
codes. A malformed tagged code is rejected. If either input is empty, a saved
key is kept; without a saved key, the dialogue requests input. Changing the
E2EE passphrase later preserves the saved ID key. Cancelling or failing to
save preserves the previous settings.
The source itself cannot be recovered from its key. A user can explicitly
display and copy the saved key as a tagged recovery code on the local device;
it is hidden when the dialogue opens. The dialogue warns that anyone who needs
recovery after losing every device should save that code or choose a source
they can reproduce.
Turning E2EE off retains the saved value but suspends its use for ID generation.
The setup dialogue disables new ID-key configuration while E2EE is off. Turning
E2EE on again reactivates the same value. Existing E2EE re-encryption and
Rebuild requirements still apply when the passphrase changes.
The source-input warning concerns only that input. The E2EE passphrase retains
its separate, existing storage behaviour. The recovery code contains the actual
saved ID key and must be handled as a secret.
Settings need to represent legacy mode or a supported version plus a derived
secret. Final field names belong in Commonlib. A declared new version with a
missing, malformed, or unavailable secret is an error; it must not silently
fall back to legacy generation. Loading or exporting an already derived value
must not derive it again.
## Deterministic derivation
The required contract is:
```text
source string --versioned derivation at save--> saved ID secret
saved ID secret + Chunk content ------------> Chunk ID
saved ID secret + canonical path -----------> obfuscated document ID
E2EE passphrase ----------------------------> content and Properties encryption
```
Derivation is offline and deterministic across supported runtimes. Its version
fixes the text encoding, treatment of Unicode and whitespace, salt, parameters,
and saved representation. It must not depend on server state, the E2EE Security
Seed, a device identifier, time, or device-specific iteration calibration.
Repeated saving of the same source under the same version produces the same
value. Reusing that source in another Vault consequently also reuses the value.
Version 1 uses PBKDF2-HMAC-SHA-256 with 310,000 iterations, the UTF-8 bytes of
the source after NFC normalisation, the fixed salt
`self-hosted-livesync:id-source:v1`, and a 256-bit output encoded as 64 lowercase
hexadecimal characters. Whitespace is preserved. The existing
`idDerivationVersion: 1` setting, saved-settings fields, and recovery-code format
remain unchanged; this implementation change does not add an ID format version
or migration path.
Obfuscated document IDs continue to use the saved 256-bit value directly as the
key for full HMAC-SHA-256. Their message remains UTF-8 encoding of
`self-hosted-livesync:id-v1:document`, a NUL byte, and the canonical path.
Agreement proofs likewise retain their existing full-HMAC messages. Neither
path uses the new Chunk-specific cache.
For encrypted Chunk IDs, first compute xxHash64 over the UTF-8 bytes of the
exact Chunk text with seed 0. Encode its result as a fixed 16-character
lowercase hexadecimal prehash. Derive a Chunk-specific subkey from the saved
32-byte value, then HMAC the domain-separated prehash:
```text
Kchunk = HMAC-SHA-256(
saved 32-byte key,
UTF8('self-hosted-livesync:id-v1:chunk-key:xxhash64')
)
prehash = fixed16lowerhex(xxHash64(UTF8(exact Chunk text), seed 0))
Chunk ID = full64lowerhex(HMAC-SHA-256(
Kchunk,
UTF8('self-hosted-livesync:id-v1:chunk:xxhash64' + NUL + prehash)
))
```
The resulting Chunk ID is the full 64-character lowercase hexadecimal HMAC
output. Existing namespace prefixes remain outside the digest. Keyed Chunk IDs
use this fixed prehash regardless of `hashAlg`; legacy mode and its existing
hash selection remain unchanged.
The Chunk-specific HMAC subkey and imported key, along with the WASM xxHash64
generator, are cached per `HashManager` and active saved key. Concurrent
preparation is shared. Replacing the manager or key, turning E2EE off, or
returning to legacy mode clears the cache; failed preparation can be retried.
This cache adds no persistent state. The current E2EE setting also selects the
legacy encrypted or plain Chunk route when a manager remains alive while E2EE is
turned off.
### Security properties and limits
A derived value remains a secret capable of generating IDs. Hashing does not
increase the entropy of its source. A password KDF adds guessing cost; it does
not make a weak source strong. HKDF alone does not provide that password
stretching. See [RFC 8018](https://www.rfc-editor.org/rfc/rfc8018.html#section-8)
and [RFC 5869](https://www.rfc-editor.org/rfc/rfc5869.html#section-4).
The random default separates ID generation from the E2EE passphrase. Deriving
both secrets from the same source retains a relationship with the original
passphrase, even after that passphrase changes. An independent source with
sufficient entropy provides the intended separation. HMAC with purpose
separation is the construction for using that secret; see
[RFC 2104](https://www.rfc-editor.org/rfc/rfc2104.html).
The fixed derivation salt means that reusing a source across Vaults reuses the
ID key; use separate sources when independent Vault identities are required.
CouchDB authentication and database access control remain the first access
boundary. This design also considers exposure through database credentials,
server administration, or backups. It does not promise to conceal equality,
document counts, revision history, or ciphertext lengths from database readers.
For Chunk IDs, xxHash64 is a public, non-cryptographic prehash. Distinct Chunk
texts which produce the same 64-bit prehash produce the same ID under the same
saved key. The final 256-bit HMAC does not restore distinctions lost at that
stage, so collision resistance for Chunk IDs is bounded by xxHash64 rather than
by the HMAC output width. This limit is an accepted trade-off for bounding the
content processed by HMAC.
The independent ID key preserves existing file contents, Chunk representation,
and `_rev` behaviour. It does not change the privacy properties of those
formats. Payload and virtual file padding remain outside this change.
## Sharing, import, and storage
Include the saved derived value and its version in Setup URIs, protected by the
existing, separate Setup URI passphrase. Import the saved value directly.
Additional devices therefore need neither the original source nor another
derivation step. Manual setup can reproduce it by entering the same source and
version, or by importing the tagged recovery code. After an E2EE passphrase
change, the current passphrase cannot be assumed to reproduce the old ID secret.
The standalone Setup URI generator uses a fresh random 256-bit ID key by
default, prints its tagged recovery code, and accepts that code for repeatable
generation for the same Vault. `id_mode=legacy` selects the old ID behaviour.
Running it again without the code produces a different key, so the generated
URI must not be treated as an update for an existing remote.
QR sharing includes the same fields through the existing QR representation and
warnings. Its current payload is not encrypted like a Setup URI. The agreed
scope accepts that existing sharing model and user responsibility for keeping
QR material private; it adds no QR storage or expiry mechanism.
| Boundary | Required handling |
| ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Existing settings and old complete Setup URI/QR/P2P imports | Missing fields select legacy behaviour. Do not inherit an unrelated value already present on the receiving device. |
| Ordinary partial setting updates | Preserve the current secret and version when neither is supplied. |
| New-format imports | Validate version and value together before applying or starting database work. |
| Local persistence | Integrate the secret explicitly with sensitive-configuration encryption and loading. Adding an arbitrary field does not currently provide this protection. |
| Reports, logs, and Markdown settings | Redact the secret in reports and logs; treat it as a credential in the existing Markdown export/import policy. An export which omits credentials must omit this secret. |
| Remote profiles, CLI, WebApp, WebPeer, and direct writers | Carry the effective value and version through every supported configuration path. A selected new mode must never degrade silently to legacy mode. |
Setup URI JSON encoding can carry ordinary new settings, but QR encoding uses
an explicit key-index table. Append stable QR entries without reordering old
ones. Complete imports and partial edits must have distinct missing-value
semantics even where the current implementation merges settings objects.
In particular, `SetupManager` currently merges decoded URI settings over the
receiving device's settings. Complete imports must normalise the new fields
before that merge to prevent accidental inheritance.
## Compatibility and changes to existing data
| Difference or change | Consequence |
| ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| Different Chunk derivation only | Existing content remains readable through Metadata `children`; new writes can duplicate Chunks and reduce reuse. |
| Different obfuscated document ID derivation | The same path can become separate documents. Treat this as an incompatible configuration requiring resolution. |
| New E2EE passphrase, unchanged saved ID secret | IDs remain stable for unchanged content, paths, and other ID settings. Re-encryption still requires the existing E2EE workflow. |
| Enabling, replacing, or disabling the option with Path Obfuscation active | Document identity changes. Use the established authoritative Rebuild and secondary-device Fetch workflow. |
| Existing installation with no new option | Preserve its exact legacy behaviour; do not derive or copy a value during upgrade. |
Copying the legacy passphrase into the new derivation does not preserve legacy
IDs, because the derivation itself changes. This proposal therefore makes no
automatic or seamless migration promise. Before an explicit transition, update
and stop the participating devices, select the authoritative data, and use the
existing [Rebuild and Fetch procedures](../recovery.md). Share the resulting
configuration before other devices rejoin. One-entry
[Metadata ID repair](metadata_document_id_validation_and_repair.md) does not
perform this transition.
Commonlib's Chunk cache includes content-to-ID lookup before hashing. Changing
the hash function alone can keep producing old IDs, and reading old Chunks can
populate that lookup again. The implementation must distinguish read reuse
from the ID selected for a new write, including after manager replacement and
restart. Readers continue accepting referenced legacy Chunks.
## Agreement checks and older clients
Keep checks focused on preventing incompatible document identities. Reuse the
existing configuration review and replication admission paths. A mismatch
must not be treated as an automatically alignable Chunk setting when document
IDs depend on it. Show a mismatch or unsupported version without exposing the
saved secret.
The advertised ID version is used for comparison only. Ordinary Tweak alignment
preserves each device's ID version and key together, including when only Chunk
IDs differ. A document ID mode mismatch requires explicit configuration through
a Setup URI or the matching key, rather than adopting a version without its key.
For CouchDB, extend the supported feature set in the remote feature contract
introduced by PR #1222, then declare the new requirement before writing data
under it. That mechanism rejects unsupported features at admission and provides
a best-effort stop when an unsupported requirement arrives later. It checks
format support, not equality of saved secrets, and does not make a live
migration atomic. Journal can extend its existing milestone compatibility path;
P2P needs its separate admission handling. The CouchDB feature contract alone
cannot protect those transports.
The implementation checks up to two remote documents in each ordinary and
internal obfuscated-ID namespace before CouchDB replication or direct writes.
This includes a legacy-mode caller connecting to a remote which uses keyed IDs.
For each available sample it recomputes the ID from the decrypted path; any
mismatch rejects the connection. An empty database, or one with no usable
sample, is reported as unverified and may proceed because there is no observed
document identity to conflict with. A sampled match is evidence, not a proof
that every document has the same identity; an unsampled mixture remains a
limitation of this bounded check.
When E2EE and Path Obfuscation are both active, Journal stores an
E2EE-encrypted, domain-separated proof in its existing milestone. It is
encrypted before the milestone is uploaded and compared on later connections.
An established milestone without this proof requires a Rebuild before the new
document IDs can be used. Journal advertises a new compatibility range for
keyed document IDs so older clients reject it. P2P compares a
purpose-separated HMAC over a fresh challenge during peer admission; the proof
is not stored. When Path Obfuscation is off, different keys affect only Chunk
IDs, so Journal keeps its legacy compatibility range and P2P does not require
key agreement. Neither transport publishes the key or a plaintext verifier in
Tweak values. CouchDB and direct writers use the document sample check above
rather than a stored verifier. The sample check uses the host's path service
with the attempted settings snapshot so stored non-canonical paths are treated
the same way as ID generation.
The existing `_rev` behaviour for ordinary content remains unchanged.
## Implementation responsibilities
The following are the confirmed integration points in the reviewed baseline.
Commonlib paths refer to its package implementation, not a source mirror in
this repository.
| Owner and entry points | Work |
| -------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Commonlib `HashManagerCore`, concrete hash managers, `PathService`, and `path2id_base` | Select legacy or new derivation consistently; preserve path and namespace semantics. |
| Commonlib `EntryManagerImpls`, `LayeredChunkManager`, and `LiveSyncManagers` | Keep referenced Chunks readable and invalidate or partition generation-dependent caches. |
| Commonlib settings definitions/lifecycle, `SettingService`, `pickEncryptionSettings`, and `API/processSetting` | Own version validation, persistence, copying, imports, Setup URI encoding, and QR slots. |
| Commonlib compatibility assessment and replication implementations | Classify identity differences, protect any comparison data, and enforce supported formats at each transport boundary. |
| Commonlib `API/DirectFileManipulatorV2` | Carry the option through its explicit settings and path-obfuscation configuration. |
| LiveSync `SetupRemoteE2EE.svelte`, `PaneRemoteConfig.ts`, and `SetupManager.ts` | Implement the three configuration actions and three nested ID-key inputs, configured state, local recovery-code reveal, and existing Apply/Rebuild/Fetch choices. |
| LiveSync `replicatorConfigurationIdentity.ts`, `reportTool.ts`, and `ModuleObsidianSettingAsMarkdown.ts` | Replace connections when effective settings change, redact the secret, and apply credential-sharing rules. |
| CLI, browser applications, and setup tools | Use the same Commonlib contract in manual setup and imports; generate new Setup URIs with a reusable random ID key by default. |
Implement Commonlib changes in its own repository, validate its packed artefact,
and validate LiveSync against that exact dependency before adopting a released
version. Translations remain outside this implementation scope.
## Validation
The fixed-vector and cache tests first failed against unchanged Commonlib
`0.1.32`, then passed after the implementation change. Commonlib's 2,036 Unit
tests, type check, package boundary, and isolated packed-package checks pass.
Three Integration tests against real CouchDB and Object Storage verify direct
access, Journal agreement, and rejection before control-document changes.
The same-manager E2EE-off regression was reproduced and fixed.
LiveSync's 1,049 Unit tests, type and lint checks, production build, and iOS 15
bundle syntax check pass after installing the exact published `0.1.33`
package. Its tarball matches the validated publication candidate, and all 559
installed package files match the registry artefact. The resulting bundle is
identical to the one checked before publication.
Real Obsidian two-Vault checks with the accepted Chunk construction cover
matching-key synchronisation, incompatible document key rejection, and
differing Chunk keys with visible paths. The Review Harness also passes with
the published package, verifying actual ID calculations and report copying
without changing live settings. Its adapter imports `HashManager` through
Commonlib's focused `/hashing` entry, whose package checks cover public types,
Node execution, and browser bundling.
### Current Chunk calculation performance
The actual Commonlib `HashManager` implementations were compared in Obsidian
1.12.7 on ARM64 Linux. Six samples rotate all three variants through each
execution position twice. The table reports median total ID calculation time;
1,000 means the total for 1,000 IDs, not the time per ID. Inputs are synthetic.
| Input | IDs per sample | Legacy xxHash64 | Previous independent HMAC | Updated independent ID |
| --------------------------------- | -------------: | --------------: | ------------------------: | ---------------------: |
| 256-byte text | 1,000 | 4.50 ms | 41.40 ms | 24.45 ms |
| 4 KiB text | 1,000 | 10.00 ms | 96.25 ms | 29.70 ms |
| 32 KiB text | 1,000 | 48.40 ms | 494.40 ms | 68.95 ms |
| 10 MiB binary, default splitting | 103 | 19.60 ms | 192.50 ms | 21.45 ms |
| 50 MiB binary, default splitting | 512 | 99.85 ms | 972.75 ms | 106.65 ms |
| 10 MiB binary, Self-hosted preset | 5 | 24.10 ms | 216.90 ms | 18.50 ms |
| 50 MiB binary, Self-hosted preset | 35 | 118.40 ms | 1,067.00 ms | 91.75 ms |
The first updated ID, including Chunk-key preparation, took 1.3 ms in this
run. Repeated measurements exclude preparation, warm-up, and pauses. Binary
cases use the actual splitter and Base64 representation; all decoded bytes
and repeated IDs were checked. Splitting, Base64 conversion, database work,
payload encryption, and transfer are outside the measured interval. These
results establish lower ID calculation cost on this host, not a complete
Rebuild speedup or native mobile performance.
### Native-device ID measurements
User-supplied Review Harness reports compare the previous and updated builds
on Android 13 and iOS 18.7. Each value is the median total time for 1,000
independent Chunk IDs, using three samples in each run.
| Input | Android, previous | Android, updated | iOS, previous | iOS, updated |
| ------------- | ----------------: | ---------------: | ------------: | -----------: |
| 256-byte text | 53.6 ms | 37.8 ms | 21 ms | 20 ms |
| 4 KiB text | 70.9 ms | 43.4 ms | 22 ms | 22 ms |
| 32 KiB text | 151.1 ms | 66.5 ms | 36 ms | 39 ms |
Android's 32-KiB result takes about 56% less time. The corresponding iOS
result increases by 3 ms per 1,000 IDs; separate runs with three samples do
not establish the cause of that difference. Across these sizes, the updated
independent calculation adds approximately 18–25 ms per 1,000 IDs over each
device's legacy xxHash64 calculation. Save-time key derivation has medians of
46.6 ms on Android and 51 ms on iOS.
These reports measure synthetic ID calculations, excluding database work,
payload encryption, and transfer. Android's heap samples remain constant,
and iOS does not expose them, so the reports do not establish memory usage or
improvement. The updated build has not been measured on Windows.
The checks below are historical reference evidence for the predecessor
independent-ID implementation, which used full-content HMAC-SHA-256 for Chunk
IDs. They do not validate the current xxHash64-prehash construction.
### Historical predecessor checks
Earlier consumer validation used a local Commonlib `0.1.32` candidate. Clean
installations with npm 10 and npm 11, type checking, lint, Svelte checks, the
production build, and the iOS 15 bundle compatibility check passed. LiveSync
had 1,039 passing Unit tests, six passing Integration tests against real
CouchDB, and seven passing Setup URI utility tests with the frozen Deno
lockfile.
Predecessor Commonlib candidate checks covered deterministic vectors, Unicode
normalisation, legacy behaviour, encrypted settings persistence, imports,
cache transitions, and transport admission. Its Integration tests against real
Object Storage accept a matching Journal key, reject a different document ID
key before changing the milestone, and allow different Chunk keys when paths
remain visible. A direct-access Integration test against real CouchDB reads
with the same key and rejects a different key before changing the version
document. These library tests complemented the consumer checks for that
predecessor; they were not additional LiveSync Unit tests.
Real Obsidian 1.12.7 on ARM64 Linux verifies the following consumer boundaries:
| Boundary | Verified behaviour |
| ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| CouchDB synchronisation | Two Vaults exchange notes in both directions with matching document and Chunk IDs. Different Chunk keys also work with Path Obfuscation off. |
| CouchDB rejection | Ordinary replication rejects different or legacy document ID keys before downloading files or changing remote documents and checkpoints. |
| Visible onboarding | Both a separate source and the random default persist an encrypted key, transfer it through a Setup URI, complete Fast Fetch, and synchronise in both directions. The `%`-prefixed E2EE passphrase survives restart. |
| Input and recovery | The radio controls and disabled styling are exercised. An empty first source keeps the dialogue open with an error; empty input keeps an existing key. A recovery code restores the same key. |
| Journal upload | The uploaded documents and Chunks have keyed IDs, and the first Object Storage milestone contains the encrypted agreement proof. |
| Credential-free Markdown | Neither the saved ID key nor its encrypted representation appears in exported settings Markdown. |
A CLI P2P E2E run with a local relay imports encrypted Setup URIs, transfers a
note with matching keys, and rejects a peer with a different document ID key.
A separate check loads the published `0.1.31` DirectFileManipulator in another
process: it reads a legacy remote, and rejects an independent-ID remote with
or without Path Obfuscation, leaving remote documents and checkpoints
unchanged. This checks the previous library API, rather than an older
Obsidian installation. Bounded sampling tests cover empty and mixed document
collections; they do not establish that every document in a remote is
compatible.
### Performance reference for the predecessor
The measurements below are historical results for a local predecessor
Commonlib `0.1.32` candidate which used full-content HMAC-SHA-256 for
independent Chunk IDs. They are reference evidence only, not performance
results for the accepted xxHash64-prehash construction. Both modes enable E2EE
and Path Obfuscation; the legacy baseline uses `xxhash64`. Three trials per
mode alternate their order. These are synthetic corpora and serial local
writes, excluding Vault enumeration, remote payload encryption, and transfer;
they are not timings of the complete Rebuild action.
Saving an ordinary ID source takes 52–57 ms in Node.js 24, with a median of
56 ms. This PBKDF2 operation happens once when saving the source. Per-Chunk
and per-document IDs use the saved key and do not repeat PBKDF2.
The actual Obsidian renderer gives these median times for 1,000 serial calls
to the Commonlib hash manager or Path Service, after warm-up:
| Input | Legacy | Independent ID |
| ------------------------ | ------: | -------------: |
| Distinct 256-byte Chunks | 5.4 ms | 43.3 ms |
| Distinct 4 KiB Chunks | 15.0 ms | 104.8 ms |
| Distinct 32 KiB Chunks | 52.8 ms | 581.7 ms |
| Distinct document paths | 36.5 ms | 50.0 ms |
These direct calls include no Chunk-content cache hits. The predecessor hash
had a measurable cost, especially when there were many small Chunks. The
local-write experiments exercise splitting, ID generation, Chunk reuse, and
PouchDB writes:
| Workload and adapter | Legacy median (range) | Independent median (range) |
| ------------------------------------------------------------- | --------------------: | -------------------------: |
| 5,000 text/binary files, 100 MiB, Node PouchDB memory adapter | 102.6 s (87.4–109.3) | 86.6 s (75.4–93.1) |
| 1,000 text files, 19.5 MiB, actual Obsidian local database | 78.7 s (52.0–82.8) | 63.2 s (62.8–63.8) |
The measurements do not show a large overall slowdown for these workloads,
but the variation does not support a general speedup claim. Both experiments
checked document counts, Chunk-reference counts, and sample content readback.
The Node experiment also checked that every referenced Chunk was present. The
100 MiB corpus includes 250 duplicate files and produces 213,788 Chunk
references to 199,468 distinct Chunks in both modes, preserving reuse.
For that corpus, stored document JSON grows from 133,157,581 to 154,342,743
UTF-8 bytes, an increase of 15.9%. The text-only Obsidian corpus produces many
small Chunks and grows from 27,893,656 to 33,594,994 bytes, or 20.4%, including
32 warm-up documents. Longer Chunk IDs occur in both Chunk documents and
Metadata references. Ordinary obfuscated document IDs remain 66 characters.
These totals measure serialised document JSON; physical database and index
growth depend on the adapter and have not been measured.
### Remaining validation
Larger binary workloads on mobile, a representative user's Vault, physical
storage growth, and the complete Rebuild wall time remain unmeasured. The
native-device reports above verify the synthetic ID calculation scenario;
desktop E2E and mobile viewport checks do not establish other mobile
operating-system behaviour. URI revocation, QR redesign, and automatic
migration remain outside this change.
docs/design_docs/internal_metadata_encryption.md
+163
View File
@@ -0,0 +1,163 @@
---
date: 2026-09-27
commonlib-version: "0.1.30"
self-hosted-livesync-version: "1.0.32"
status: unreleased
---
# Internal Metadata encryption and remote feature changes
This document defines the LiveSync integration of Commonlib's remote feature
contract and encrypted Metadata for Hidden File Sync and Customisation Sync.
It describes unreleased behaviour being implemented in this branch.
Commonlib's companion `docs/remote-feature-compatibility.md` is the
source of truth for the wire document, identifiers, validation, and shared
assessment. This document owns the application behaviour, settings, Doctor
recommendation, and verification of the Obsidian and CLI integrations.
## Scope and settings
Add `encryptInternalMetadata` to the shared encryption settings. A genuinely new
Vault or CLI configuration defaults to true. Existing stored settings and old
Setup URI or QR imports complete an absent value as false. Ordinary partial
setting updates retain the current value.
The preference applies to CouchDB with E2EE V2 and Property Encryption enabled.
Show the preference as unavailable and explain its prerequisites when they are
absent. Keep Journal and P2P's existing
transport protection and avoid unrelated setting mismatches for those remotes.
Use the existing HKDF Metadata representation to protect path, creation and
modification times, size, and Chunk references for obfuscated internal entries.
Keep the `i:`, `ix:`, and supported legacy `ps:` document IDs, path conversion,
and content Chunk representation. Read encrypted Metadata independently of the
write preference, including after that preference is disabled.
The protection leaves document IDs, namespaces, revisions, deletion state,
document counts, and ciphertext lengths visible. It does not encrypt device or
Vault names stored in separate participant records.
## Enabling the preference
Changing the preference does not automatically reconstruct a database or gather
all devices' data. It affects subsequent Metadata writes. Unchanged documents
and old revisions can retain plaintext; mixed plaintext and encrypted Metadata
are a supported transition state.
Strongly recommend the existing manual remote Rebuild workflow when the person
wants existing Metadata protected as well. The person prepares the authoritative
data for that workflow. Describe this distinction in the setting, Doctor reason,
and operational documentation. Do not advertise complete historical protection
merely because the preference is enabled.
Copy the preference with the other encryption settings when preparing a remote
profile. Recreate a connection when its effective encryption settings change.
Use the existing Tweak assessment and manual mismatch resolution; do not change
the remote's shared policy silently when importing or loading settings.
## Doctor
Use Commonlib's existing conditional recommendation rules. Recommend true when
the selected CouchDB settings have E2EE V2 and Property Encryption enabled and
the new preference is false. Do not require Hidden File Sync or Customisation
Sync to be active before offering the recommendation.
Retain the existing E2EE V2 recommendation for a legacy algorithm. After that
change, ensure the newly applicable Metadata recommendation is not hidden by a
premature `doctorProcessedVersion` update. Advance the Doctor rule revision so
an older completed consultation does not suppress this new recommendation.
Apply the preference only when the person accepts the recommendation. Include
the existing-data limitation, the manual Rebuild recommendation, and the need
for compatible clients in the explanation. Do not set `requireRebuild` or
`requireRebuildLocal` for this rule: the current host wrapper can schedule those
operations and restart. `recommendRebuild` currently exists only as an unused
rule field, so setting it alone does not display an explanation.
## Admission and received version documents
The remote version document is the source of feature requirements. Commonlib
checks it before replication, Fast Fetch, and direct access. The milestone keeps
the existing Tweak comparison and Rebuild lock. An accepted writer declares the
feature before using it, including the writer admitted to a locked rebuilt
remote; an unaccepted device remains blocked by that lock.
Retain the existing received-version path through `parseSynchroniseResult`,
`enqueueAll`, and `processIfNonDocumentChange`. Replace its numeric comparison
with the shared assessment so unknown names at the same generation are also
reported. Known features, reordered lists, and ordinary revision updates do not
retire the connection. Unsupported or malformed control documents request
retirement through the existing Replicator owner and display the reason.
The callback must not await retirement of the operation which delivered it.
This is an admission check and a best-effort stop for exceptional changes during
an active connection. It does not fence every queued file application, roll back
accepted writes, or guarantee an atomic change across live devices. Feature
changes are an infrequent administrative operation: update all devices first,
then enable the preference and use the recommended manual Rebuild. Rebuild
locks the remote using the existing workflow; changing this preference alone
does not lock it. The action to proceed without rebuilding explicitly reminds
the person to update every other device, including currently connected devices.
## Persistence and recovery boundaries
Do not retain a second feature list, highest generation, or rejection flag in
KV storage. Do not add compatibility checks to pending-work snapshot recovery
or make that recovery a new prerequisite for application readiness. Preserve
the existing queue and startup behaviour. A later attempt checks the current
remote declaration, including after restart. Declared features remain on the
remote when the write preference is disabled because older data can still use
them; manually shortening that declaration is not a supported migration.
After updating clients, use normal reconnection and the existing Hatch
inspection or Fetch workflow if reconciliation is needed. This feature does not
repair unrelated KV inconsistencies or the existing readiness queue behaviour.
Garbage Collection V3 is a beta manual operation which begins with an ordinary
bidirectional synchronisation. That admission checks the remote feature
contract; no additional per-step GC checks are introduced. The separate
cleaned-remote recovery path checks the local version document before its
first Chunk-reference count because it does not start with that synchronisation.
Use the same Commonlib assessment at the CLI, Fast Fetch, and direct-access
boundaries. The Obsidian result processor is one consumer, not the only place
which determines compatibility. Keep unrelated Vaults and databases operational.
Fast Fetch checks the remote declaration before opening or resetting the local
database, both for a fresh Fetch and for checkpoint resumption.
## Verification and documentation
Keep focused tests for settings defaults and imports, the Doctor condition
matrix, acceptance and dismissal, connection replacement, and absence of an
automatic Rebuild, Fetch, or restart for this rule.
Keep unit tests for known and unknown feature notifications, generic identifier
presentation, retirement without a circular wait, and the unchanged snapshot
behaviour after KV failure or obsolete snapshot fields. The previous batch
fences, physical-database tracking, and persistent rejection tests are outside
this design; they must not imply an atomic live migration guarantee.
Use real Obsidian Hidden File Sync and Customisation Sync scenarios to inspect
raw CouchDB Metadata and restore content in another Vault. Check the admitted
writer on a locked remote, unknown-feature rejection before and during
replication, and remote-based rejection after restart. Retain the encrypted
CLI-to-Obsidian interoperability scenario. A future client upgrade that adds
support for an unknown feature is a separate validation boundary.
Also exercise enabling the preference through the settings UI without Rebuild:
retain unchanged plaintext Metadata, encrypt rewritten entries with stable IDs,
reject a second device's mismatched preference, and restore both representations
after alignment. With the preference subsequently OFF, verify that Fast Fetch
still decodes encrypted Metadata and preserves the remote feature declaration.
Keep the primary-language settings and troubleshooting guides, the
database-compatibility ADR, and Unreleased notes aligned with this behaviour.
Keep the detailed shared protocol in Commonlib and link to it after publication;
do not maintain another copy of its wire schema here. Translations are a separate
change. Update tested-version evidence when the implementation and its
validation have been accepted.
Related application contracts: [Replicator architecture](replicator_architecture.md),
[Tweak compatibility](tweak_compatibility.md), and
[database compatibility](../adr/2026_07_release_notes_and_database_compatibility.md).
docs/glossary.md
+6
View File
@@ -92,6 +92,12 @@ recovery guidance, or diagnostics intended for users.
reports, and advanced edge-case settings.
- **Hidden File Sync:** The feature which synchronises files in hidden
directories, such as `.obsidian`.
- **ID key:** A saved secret used to generate encrypted Chunk IDs and
obfuscated Metadata document IDs when independent ID derivation is enabled.
It is separate from the current E2EE passphrase.
- **ID recovery code:** A versioned text form of the saved ID key which can be
shown on the current device and imported without deriving a different key.
Treat it as a secret.
- **JWT Authentication:** An experimental CouchDB authentication option which
uses a JSON Web Token instead of standard credentials. It requires a private
key or secret, algorithm, expiry duration, subject, and key ID.
docs/releases/1.0.md
+58
View File
@@ -2,6 +2,64 @@
This document contains earlier published releases from the 1.0 line of the [current Self-hosted LiveSync release history](../../updates.md). Beta and release-candidate builds published before 1.0.0 are recorded in the [1.0 preview history](1.0-previews.md). Earlier release lines continue in the [0.25 history](0.25.md) and the [legacy history](legacy.md).
## 1.0.26
~~1.0.25~~ was cancelled because pre-release validation found that LiveSync could appear to finish synchronising even though Android had not written a received file to the Vault; the warning appeared only after restart.
6th September, 2026
### Synchronisation and storage
#### Fixed
- Files inside a folder are no longer silently removed from synchronisation when an external tool changes only the letter case of that folder while Obsidian is running. This prevents the stale deletion from reaching other devices or later removing the local file. Moving files into ignored or otherwise excluded locations retains the existing behaviour, and the folder-name case itself may still differ between devices. (#1168)
- A problem processing one file during ordinary start-up no longer prevents every other file from synchronising. LiveSync warns about the affected files and can retry them later; Fetch and Rebuild still stop if they cannot finish safely. (#1164)
- When LiveSync cannot finish preparing this device for synchronisation, it now says that synchronisation is unavailable and directs you to generate a report, instead of remaining at 'Not ready'. (#1164)
#### Improved
- When LiveSync cannot write a received file to the Vault, it now warns immediately instead of appearing to have synchronised it successfully. The generated report identifies the affected path, and a later scan can try it again.
### Conflict handling and recovery
#### Improved
- Conflict resolution dialogues now close when the same file is resolved elsewhere or when the plug-in unloads. Requests for different files are shown one at a time, while a newer request for the same file replaces the older one.
### Setup and compatibility
#### Improved
- Unconfigured Vaults now stay focused on setup instead of running Config Doctor or incomplete-document checks before they can be used. Returning a configured Vault to an unconfigured state also stops those checks until the requested restart. (#1161)
- When the active file contains a file or folder name longer than 255 UTF-8 bytes, LiveSync now explains that the path may not work on some Android and Linux file systems. It does not rename or reject the file. (#1164)
## 1.0.24
3rd September, 2026
### Interface and translation
#### Fixed
- The Setup Wizard now correctly explains that the existing-device path adds this device to an existing synchronisation (PR #1118). Thank you to @nikhilmaddirala for the contribution!
- Spanish translations now resolve the **Display language** placeholder, cover previously untranslated Setup Wizard and CouchDB text, translate user-facing Config Doctor values and confirmation controls, and use Spanish sentence case (PR #1129). Thank you to @zeedif for the contribution!
#### Improved
- The Setup Wizard now shows the passphrase and **Obfuscate Properties** controls only after E2EE is enabled, provides a password-visibility button, allows longer translated labels to wrap, and keeps the invitation link compact on desktop while preserving its mobile touch target (PR #1130). Thank you to @zeedif for the contribution!
### Synchronisation and storage
#### Fixed
- **Overwrite Server Data with This Device's Files** now keeps this device's synchronisation settings instead of reapplying settings from the remote database which is about to be replaced. Enabling E2EE before a rebuild therefore remains enabled and uploads encrypted data. (#1146)
### Command-line tool
#### Fixed
- The systemd installer now finds the repository root correctly, installs every generated bundle chunk and required production dependency, checks the installed command before activation, and reports success only when the service remains active.
## 1.0.23
2nd September, 2026
docs/settings.md
+24
View File
@@ -239,12 +239,34 @@ Setting key: passphrase
Encrypting passphrase. If you change the passphrase, you need to rebuild databases (You will be informed).
#### Independent ID derivation
Setting keys: `idDerivationVersion`, `idDerivationKey`
This setting saves a separate key for encrypted Chunk IDs and obfuscated Metadata document IDs. New Vault setup selects **Generate a random ID key** by default when E2EE is enabled. Existing Vaults select **Keep current configuration** by default. The radio choices show the available configurations together. A small description under **Keep current configuration** identifies the saved configuration: an existing ID key, or legacy IDs linked to the E2EE passphrase. That choice retains either one; on a new Vault, choosing it explicitly uses legacy IDs. If the configuration is legacy, changing the E2EE passphrase also changes IDs.
To set a key yourself, choose **Set an ID key**. Three further radio choices then appear: **Derive from current E2EE passphrase**, **Enter an ID source**, and **Import an ID recovery code**. The last two choices show a text input. The source input also recognises a tagged recovery code. An empty input keeps an existing key; a first key requires input. An ordinary source is converted to a key when you apply the settings and cannot be shown again. A recovery code imports the saved key directly.
Use **Show current recovery code** to display and copy the saved key on this device. The code starts with `sls-id-v1:` and can be pasted into the manual input on another device without deriving a different key. A Setup URI carries the same saved key under its separate passphrase. If you need to restore the configuration after losing every device, save the recovery code or choose a source you can reproduce before relying on the random default. Keep the code private.
Using the E2EE passphrase as the source keeps IDs stable after later passphrase changes, but it does not separate the original passphrase from guesses based on known IDs. Use a long, unpredictable, separate source when that separation matters. Hashing a weak source does not make it strong.
Changing the E2EE passphrase later does not change the saved ID key, although the existing re-encryption and Rebuild procedure still applies to the encrypted data. While E2EE is off, the saved ID key is retained but is not used; existing legacy ID generation applies until E2EE is enabled again. Devices with different ID keys can synchronise when Path Obfuscation is off, although identical content may produce duplicate Chunks. Enabling, replacing, or disabling the ID key can change document IDs when Path Obfuscation is active. Update participating devices, Rebuild from the authoritative Vault, and Fetch on other devices before resuming ordinary synchronisation. A QR code includes the saved key under the existing QR sharing rules, so keep the QR code private.
#### Path Obfuscation
Setting key: usePathObfuscation
In default, the path of the file is not obfuscated to improve the performance. If you enable this, the path of the file will be obfuscated. This is useful when you want to hide the path of the file.
#### Encrypt internal file Properties
Setting key: encryptInternalMetadata
For CouchDB, this encrypts paths, times, sizes, and Chunk references in the Metadata used by Hidden File Sync and Customisation Sync. It requires E2EE V2 and **Property Encryption**. New Vaults enable the preference by default, but it has no effect until those prerequisites are enabled. Existing Vaults and older Setup URIs and QR codes keep it disabled unless you enable it.
Enabling the preference protects future Metadata writes. Existing Metadata and earlier revisions can remain readable in the remote database. If you want to protect existing Metadata too, prepare the authoritative data, update every device to a compatible version, and manually Rebuild the remote database. LiveSync does not gather data or start a Rebuild when you change this preference. The action to enable it without rebuilding explicitly reminds you to update every other synchronising device first, including devices currently running LiveSync. Plaintext and encrypted Metadata can coexist during the transition. Document IDs, revision information, document counts, and ciphertext lengths remain visible.
#### Encryption Algorithm
Setting key: E2EEAlgorithm
@@ -1025,6 +1047,8 @@ Setting key: hashAlg
`xxhash64` is the supported current value. Older algorithms remain selectable only as an edge-case compatibility path for existing databases. Changing the algorithm can reduce chunk reuse between devices and requires the normal tweak review.
When independent ID derivation is enabled, encrypted Chunk IDs use its versioned HMAC construction instead of `hashAlg`. The selected `hashAlg` continues to apply to legacy IDs.
### 6. Edge case addressing (Behaviour)
#### Fetch database with previous behaviour
docs/setup_object_storage.md
+1 -1
View File
@@ -31,7 +31,7 @@ deno run --minimum-dependency-age=0 --allow-env https://raw.githubusercontent.co
For providers which require them, set `force_path_style`, `use_custom_request_handler`, or `bucket_custom_headers` as described in the [setup utility reference](../utils/readme.md#object-storage).
Store the generated Setup URI and Setup URI passphrase separately. The URI is encrypted, but it contains the Object Storage credentials.
Store the generated Setup URI and Setup URI passphrase separately. The URI is encrypted, but it contains the Object Storage credentials. The generator also prints an ID recovery code; reuse it through `id_recovery_code` if you regenerate a URI for the same Vault. A new run without it creates a different ID key. The [setup utility reference](../utils/readme.md#setup-uri-generation) describes the `id_mode=legacy` option for existing Vaults.
## Set up the first device
docs/setup_own_server.md
+4
View File
@@ -190,6 +190,8 @@ deno run --minimum-dependency-age=0 --allow-env https://raw.githubusercontent.co
>
> If `uri_passphrase` is omitted, the generator creates a cryptographically random value and prints it once.
The generator also prints an ID recovery code for its random ID key. Save that code if you may need to regenerate a Setup URI for the same Vault. Pass it back as `id_recovery_code`; otherwise a later run creates a different key. Set `id_mode=legacy` only when connecting to a Vault which uses the previous ID behaviour. See the [setup utility reference](../utils/readme.md#setup-uri-generation).
The generator consumes the exact registry-pinned Commonlib release used by the provisioning utility. It creates a configured CouchDB remote profile, applies the current defaults for a new Vault, and encodes them with Commonlib's Setup URI contract.
You will then get the following output:
@@ -198,6 +200,8 @@ You will then get the following output:
Generated couchdb Setup URI.
Your passphrase for the Setup URI is: H7vX...a-random-32-character-value
This passphrase is never shown again, so store it safely.
ID recovery code: sls-id-v1:<64 lowercase hexadecimal characters>
Use id_recovery_code with this value and reuse the same remote settings when generating another Setup URI for the same Vault.
obsidian://setuplivesync?settings=%5B%22tm2DpsOE74nJAryprZO2M93wF%2Fvg.......4b26ed33230729%22%5D
```
docs/setup_p2p.md
+1 -1
View File
@@ -135,4 +135,4 @@ export uri_passphrase=<A SEPARATE SETUP URI PASSPHRASE>
deno run --minimum-dependency-age=0 --allow-env https://raw.githubusercontent.com/vrtmrz/obsidian-livesync/main/utils/setup/generate_setup_uri.ts
```
The generated Setup URI contains the encrypted room, relay, and Vault settings. It deliberately omits the device-specific name. Store the URI and its passphrase separately. After importing it on the first device, continue from the initialisation step above, then generate a fresh Setup URI for an additional device from that working device.
The generated Setup URI contains the encrypted room, relay, and Vault settings. It deliberately omits the device-specific name. Store the URI and its passphrase separately. The generator prints an ID recovery code; pass it as `id_recovery_code` if you regenerate a URI for the same Vault. A run without it creates a different ID key. Also reuse the original `p2p_room_id` and `p2p_passphrase`, since omitted values are generated afresh. The [setup utility reference](../utils/readme.md#setup-uri-generation) describes the `id_mode=legacy` option for existing Vaults. After importing the URI on the first device, continue from the initialisation step above, then generate a fresh Setup URI for an additional device from that working device.
docs/troubleshooting.md
+6
View File
@@ -96,6 +96,8 @@ Current releases automatically align compatible settings which control how new c
A missing legacy file-name case setting means case-insensitive handling. It matches an explicit disabled setting and does not require a rebuild for that difference. An explicitly enabled setting can use different document IDs and still requires a compatibility decision against either value. Other configuration differences shown in the dialogue must still be resolved.
If the mismatch names **Encrypt internal file Properties**, update every device before accepting that preference. It affects subsequent Metadata writes for Hidden File Sync and Customisation Sync; it does not automatically protect existing Metadata. A manual remote Rebuild is strongly recommended if you need to protect existing paths, times, sizes, and Chunk references.
The `Sync now` command keeps routine replication progress quiet so that it is convenient to assign to a keyboard shortcut; assign one in Obsidian if that suits your workflow. A quiet command may still open this dialogue when a mismatch or another decision requires your attention.
The available actions depend on when the mismatch is found:
@@ -109,6 +111,10 @@ The available actions depend on when the mismatch is found:
Historic defect notices and renamed controls are retained in the [0.25 release history](releases/0.25.md) and [legacy release history](releases/legacy.md), rather than in the current troubleshooting path.
## The remote database uses an unknown feature
When a notice identifies an unknown feature, update this device and every other client of the same CouchDB database, including the CLI. The notice includes the feature identifier even if this version has no descriptive name for it. New synchronisation is refused, and receiving an unsupported requirement stops active replication, because an older client may not interpret the Metadata and its Chunk references correctly. Already queued file changes are not rolled back. The cleaned-remote recovery path also checks compatibility before counting Chunk references. Do not remove the feature name from the remote version document to bypass the check. After updating, reconnect and review any pending file changes before running Garbage Collection.
## Setup and settings questions
### Share a configuration with another device
manifest.json
+1 -1
View File
@@ -1,7 +1,7 @@
{
"id": "obsidian-livesync",
"name": "Self-hosted LiveSync",
"version": "1.0.30",
"version": "1.0.32",
"minAppVersion": "1.7.2",
"description": "Community implementation of self-hosted livesync. Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.",
"author": "vorotamoroz",
package-lock.json
Generated
+9 -9
View File
@@ -1,12 +1,12 @@
{
"name": "obsidian-livesync",
"version": "1.0.30",
"version": "1.0.32",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "obsidian-livesync",
"version": "1.0.30",
"version": "1.0.32",
"license": "MIT",
"workspaces": [
"src/apps/cli",
@@ -23,7 +23,7 @@
"@smithy/types": "^4.14.3",
"@smithy/util-retry": "^4.4.5",
"@vrtmrz/browser-ui-kit": "0.1.0",
"@vrtmrz/livesync-commonlib": "0.1.28",
"@vrtmrz/livesync-commonlib": "0.1.34",
"@vrtmrz/obsidian-plugin-kit": "0.1.4",
"@vrtmrz/ui-interactions": "0.1.2",
"diff-match-patch": "^1.0.5",
@@ -4567,9 +4567,9 @@
}
},
"node_modules/@vrtmrz/livesync-commonlib": {
"version": "0.1.28",
"resolved": "https://registry.npmjs.org/@vrtmrz/livesync-commonlib/-/livesync-commonlib-0.1.28.tgz",
"integrity": "sha512-3NXswTtOU+fE4KRxBWHfkMx9r2Eu9IFH9N/ibj75C5eLh1tFPH7n87bGib6La7Twz9yCDIJRBsBUc2tx81rxOQ==",
"version": "0.1.34",
"resolved": "https://registry.npmjs.org/@vrtmrz/livesync-commonlib/-/livesync-commonlib-0.1.34.tgz",
"integrity": "sha512-EeVpeFg3W43cpN+x0BZvFj9fN+eQFil9vohvmLLV1z/8bCTT0Hsq/bfRV59oL0qFNggPetkwyCZUusdfjajNOw==",
"license": "MIT",
"dependencies": {
"@aws-sdk/client-s3": "^3.808.0",
@@ -12665,7 +12665,7 @@
},
"src/apps/cli": {
"name": "self-hosted-livesync-cli",
"version": "1.0.30-cli",
"version": "1.0.32-cli",
"dependencies": {
"chokidar": "^4.0.0",
"minimatch": "^10.2.5",
@@ -12690,7 +12690,7 @@
},
"src/apps/webapp": {
"name": "livesync-webapp",
"version": "1.0.30-webapp",
"version": "1.0.32-webapp",
"dependencies": {
"octagonal-wheels": "^0.1.54"
},
@@ -12702,7 +12702,7 @@
}
},
"src/apps/webpeer": {
"version": "1.0.30-webpeer",
"version": "1.0.32-webpeer",
"dependencies": {
"octagonal-wheels": "^0.1.54"
},
package.json
+6 -2
View File
@@ -1,6 +1,6 @@
{
"name": "obsidian-livesync",
"version": "1.0.30",
"version": "1.0.32",
"description": "Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.",
"main": "main.js",
"type": "module",
@@ -66,6 +66,7 @@
"test:e2e:obsidian:p2p-pane": "tsx test/e2e-obsidian/scripts/p2p-pane.ts",
"test:e2e:obsidian:vault-reflection": "tsx test/e2e-obsidian/scripts/vault-reflection.ts",
"test:e2e:obsidian:couchdb-upload": "tsx test/e2e-obsidian/scripts/couchdb-upload.ts",
"test:e2e:obsidian:chunk-fetch-retry": "tsx test/e2e-obsidian/scripts/chunk-fetch-retry.ts",
"test:e2e:obsidian:tweak-compatibility": "tsx test/e2e-obsidian/scripts/tweak-compatibility.ts",
"test:e2e:obsidian:couchdb-manual-setup-workflow": "tsx test/e2e-obsidian/scripts/couchdb-manual-setup-workflow.ts",
"test:e2e:obsidian:cli-to-obsidian-sync": "tsx test/e2e-obsidian/scripts/cli-to-obsidian-sync.ts",
@@ -85,6 +86,9 @@
"test:e2e:obsidian:security-seed-reconnect": "tsx test/e2e-obsidian/scripts/security-seed-reconnect.ts",
"test:e2e:obsidian:hidden-file-snippet-sync": "tsx test/e2e-obsidian/scripts/hidden-file-snippet-sync.ts",
"test:e2e:obsidian:customisation-sync": "tsx test/e2e-obsidian/scripts/customisation-sync.ts",
"test:e2e:obsidian:received-change-readiness": "tsx test/e2e-obsidian/scripts/received-change-readiness.ts",
"test:e2e:obsidian:remote-feature-change": "tsx test/e2e-obsidian/scripts/remote-feature-change.ts",
"test:e2e:obsidian:internal-metadata-migration": "tsx test/e2e-obsidian/scripts/internal-metadata-migration.ts",
"test:e2e:obsidian:setting-markdown-export": "tsx test/e2e-obsidian/scripts/setting-markdown-export.ts",
"test:e2e:obsidian:upgrade-from-stable": "tsx test/e2e-obsidian/scripts/upgrade-from-stable.ts",
"test:e2e:obsidian:local-suite": "tsx test/e2e-obsidian/scripts/local-suite.ts",
@@ -183,7 +187,7 @@
"@smithy/types": "^4.14.3",
"@smithy/util-retry": "^4.4.5",
"@vrtmrz/browser-ui-kit": "0.1.0",
"@vrtmrz/livesync-commonlib": "0.1.28",
"@vrtmrz/livesync-commonlib": "0.1.34",
"@vrtmrz/obsidian-plugin-kit": "0.1.4",
"@vrtmrz/ui-interactions": "0.1.2",
"diff-match-patch": "^1.0.5",
src/apps/cli/package.json
+1 -1
View File
@@ -1,7 +1,7 @@
{
"name": "self-hosted-livesync-cli",
"private": true,
"version": "1.0.30-cli",
"version": "1.0.32-cli",
"main": "dist/index.cjs",
"type": "module",
"scripts": {
src/apps/cli/testdeno/helpers/docker.ts
+1 -1
View File
@@ -628,7 +628,7 @@ export async function startP2pRelay(): Promise<void> {
//TODO: port mapping should be configurable.
"4000:7777",
"--tmpfs",
"/app/strfry-db:rw,size=256m",
"/app/strfry-db:rw,size=256m,mode=1777",
"--entrypoint",
"sh",
P2P_RELAY_IMAGE,
src/apps/cli/testdeno/helpers/settings.ts
+16 -8
View File
@@ -13,7 +13,11 @@ export async function initSettingsFile(settingsFile: string): Promise<void> {
* Generate a full setup URI from a settings file via the Commonlib package API.
* Mirrors the bash flow in test-setup-put-cat-linux.sh.
*/
export async function generateSetupUriFromSettings(settingsFile: string, setupPassphrase: string): Promise<string> {
export async function generateSetupUriFromSettings(
settingsFile: string,
setupPassphrase: string,
preserveRemoteSettings = false
): Promise<string> {
const script = [
"import { fs } from '@vrtmrz/livesync-commonlib/node';",
"import { encodeSettingsToSetupURI } from '@vrtmrz/livesync-commonlib/compat/API/processSetting';",
@@ -21,13 +25,17 @@ export async function generateSetupUriFromSettings(settingsFile: string, setupPa
" const settingsPath = process.env.SETTINGS_FILE;",
" const passphrase = process.env.SETUP_PASSPHRASE;",
" const settings = JSON.parse(fs.readFileSync(settingsPath, 'utf-8'));",
" settings.couchDB_DBNAME = 'setup-put-cat-db';",
" settings.couchDB_URI = 'http://127.0.0.1:5999';",
" settings.couchDB_USER = 'dummy';",
" settings.couchDB_PASSWORD = 'dummy';",
" settings.liveSync = false;",
" settings.syncOnStart = false;",
" settings.syncOnSave = false;",
...(preserveRemoteSettings
? []
: [
" settings.couchDB_DBNAME = 'setup-put-cat-db';",
" settings.couchDB_URI = 'http://127.0.0.1:5999';",
" settings.couchDB_USER = 'dummy';",
" settings.couchDB_PASSWORD = 'dummy';",
" settings.liveSync = false;",
" settings.syncOnStart = false;",
" settings.syncOnSave = false;",
]),
" const uri = await encodeSettingsToSetupURI(settings, passphrase);",
" process.stdout.write(uri.trim());",
"})();",
src/apps/cli/testdeno/test-p2p-sync.ts
+79 -3
View File
@@ -1,6 +1,11 @@
import { assert } from "@std/assert";
import { TempDir } from "./helpers/temp.ts";
import { initSettingsFile, applyP2pSettings, applyP2pTestTweaks } from "./helpers/settings.ts";
import {
initSettingsFile,
applyP2pSettings,
applyP2pTestTweaks,
generateSetupUriFromSettings,
} from "./helpers/settings.ts";
import { startCliInBackground } from "./helpers/backgroundCli.ts";
import {
discoverPeer,
@@ -9,10 +14,10 @@ import {
maybeStartCoturn,
stopCoturnIfStarted,
} from "./helpers/p2p.ts";
import { runCli } from "./helpers/cli.ts";
import { runCli, runCliOrFail, runCliWithInputOrFail, sanitiseCatStdout } from "./helpers/cli.ts";
import { getOptimalLoopbackIp } from "./helpers/net.ts";
Deno.test("p2p-sync: discovers peer and completes sync", async () => {
Deno.test("p2p-sync: transfers with the same ID key and rejects a different document ID key", async () => {
const loopbackIp = await getOptimalLoopbackIp();
const loopbackHost = loopbackIp === "::1" ? "[::1]" : loopbackIp;
@@ -32,14 +37,18 @@ Deno.test("p2p-sync: discovers peer and completes sync", async () => {
const hostSettings = workDir.join("settings-host.json");
const clientVault = workDir.join("vault-sync");
const clientSettings = workDir.join("settings-sync.json");
const rejectedVault = workDir.join("vault-rejected");
const rejectedSettings = workDir.join("settings-rejected.json");
await Deno.mkdir(hostVault, { recursive: true });
await Deno.mkdir(clientVault, { recursive: true });
await Deno.mkdir(rejectedVault, { recursive: true });
const relayStarted = await maybeStartLocalRelay(relay);
const coturnStarted = await maybeStartCoturn(turnServers);
try {
await initSettingsFile(hostSettings);
await initSettingsFile(clientSettings);
await initSettingsFile(rejectedSettings);
await applyP2pSettings(
hostSettings,
roomId,
@@ -58,8 +67,52 @@ Deno.test("p2p-sync: discovers peer and completes sync", async () => {
"~.*",
turnServers
);
await applyP2pSettings(
rejectedSettings,
roomId,
passphrase,
"self-hosted-livesync-cli-tests",
relay,
"~.*",
turnServers
);
await applyP2pTestTweaks(hostSettings, hostPeerName, passphrase);
await applyP2pTestTweaks(clientSettings, clientPeerName, passphrase);
await applyP2pTestTweaks(rejectedSettings, "p2p-rejected-" + nonce, passphrase);
for (const [vault, path, key, label] of [
[hostVault, hostSettings, "ab".repeat(32), "host"],
[clientVault, clientSettings, "ab".repeat(32), "client"],
[rejectedVault, rejectedSettings, "cd".repeat(32), "rejected"],
]) {
const settings = JSON.parse(await Deno.readTextFile(path));
settings.idDerivationVersion = 1;
settings.idDerivationKey = key;
const sourcePath = workDir.join("setup-source-" + label + ".json");
await Deno.writeTextFile(sourcePath, JSON.stringify(settings));
const setupPassphrase = "independent-id-setup-passphrase";
const setupUri = await generateSetupUriFromSettings(sourcePath, setupPassphrase, true);
await runCliWithInputOrFail(setupPassphrase + "\n", vault, "--settings", path, "setup", setupUri);
const persisted = JSON.parse(await Deno.readTextFile(path));
assert(persisted.idDerivationVersion === 1, "The Setup URI lost the ID derivation version.");
assert(persisted.idDerivationKey === "", "The CLI stored the ID key in plain text.");
assert(
typeof persisted.encryptedIdDerivationKey === "string" && persisted.encryptedIdDerivationKey.length > 0,
"The CLI did not encrypt the saved ID key."
);
assert(persisted.P2P_Enabled === true, "The Setup URI disabled P2P.");
assert(persisted.P2P_roomID === roomId, "The Setup URI changed the P2P room.");
assert(persisted.P2P_relays === relay, "The Setup URI changed the P2P relay.");
assert(persisted.remoteType === "ONLY_P2P", "The Setup URI changed the remote type.");
}
const notePath = "p2p/independent-id-note.md";
await runCliWithInputOrFail(
"A note transferred with the saved ID key.\n",
clientVault,
"--settings",
clientSettings,
"put",
notePath
);
const host = startCliInBackground(hostVault, "--settings", hostSettings, "p2p-host");
try {
@@ -82,9 +135,32 @@ Deno.test("p2p-sync: discovers peer and completes sync", async () => {
syncResult.code === 0,
`p2p-sync failed\nstdout: ${syncResult.stdout}\nstderr: ${syncResult.stderr}`
);
const rejectedPeer = await discoverPeer(rejectedVault, rejectedSettings, peersTimeout, hostPeerName);
const rejectedSync = await runCli(
rejectedVault,
"--settings",
rejectedSettings,
"p2p-sync",
rejectedPeer.id,
String(syncTimeout)
);
assert(
rejectedSync.code !== 0,
`P2P accepted a different key for obfuscated document IDs.\nstdout: ${rejectedSync.stdout}\nstderr: ${rejectedSync.stderr}`
);
assert(
rejectedSync.combined.includes("Tweak values are not matched"),
`P2P failed before checking peer settings.\nstdout: ${rejectedSync.stdout}\nstderr: ${rejectedSync.stderr}`
);
} finally {
await host.stop();
}
const received = sanitiseCatStdout(
await runCliOrFail(hostVault, "--settings", hostSettings, "cat", notePath)
).trimEnd();
assert(received === "A note transferred with the saved ID key.", "The host did not receive the keyed note.");
const rejectedRead = await runCli(rejectedVault, "--settings", rejectedSettings, "cat", notePath);
assert(rejectedRead.code !== 0, "The rejected device received the keyed note.");
} finally {
await stopLocalRelayIfStarted(relayStarted);
await stopCoturnIfStarted(coturnStarted);
src/apps/webapp/package.json
+1 -1
View File
@@ -1,7 +1,7 @@
{
"name": "livesync-webapp",
"private": true,
"version": "1.0.30-webapp",
"version": "1.0.32-webapp",
"type": "module",
"description": "Browser-based Self-hosted LiveSync using FileSystem API",
"scripts": {
src/apps/webpeer/package.json
+1 -1
View File
@@ -1,7 +1,7 @@
{
"name": "webpeer",
"private": true,
"version": "1.0.30-webpeer",
"version": "1.0.32-webpeer",
"type": "module",
"scripts": {
"dev": "vite",
src/common/messages/LiveSyncProvisionalMessages.ts
+44
View File
@@ -204,6 +204,50 @@ export const liveSyncProvisionalEnglishMessages = {
"Repair failed before the source was removed. Run inspection again before retrying.",
"Connection settings": "Connection settings",
"Saved connections": "Saved connections",
"ID generation": "ID generation",
"Keep current configuration": "Keep current configuration",
"Set an ID key": "Set an ID key",
"Current configuration: a saved ID key is used.": "Current configuration: a saved ID key is used.",
"Current configuration: the saved ID key is retained while E2EE is off.":
"Current configuration: the saved ID key is retained while E2EE is off.",
"Current configuration: no ID key is saved. With E2EE enabled, keeping it uses legacy IDs tied to the E2EE passphrase.":
"Current configuration: no ID key is saved. With E2EE enabled, keeping it uses legacy IDs tied to the E2EE passphrase.",
"Changing the E2EE passphrase changes IDs generated by the legacy configuration.":
"Changing the E2EE passphrase changes IDs generated by the legacy configuration.",
"This uses a saved key for new Chunk IDs and obfuscated Metadata document IDs, so changing the E2EE passphrase does not derive a new key automatically.":
"This uses a saved key for new Chunk IDs and obfuscated Metadata document IDs, so changing the E2EE passphrase does not derive a new key automatically.",
Configured: "Configured",
"The saved ID key is configured. Its source cannot be shown again.":
"The saved ID key is configured. Its source cannot be shown again.",
"Leave this input empty to keep the saved ID key.": "Leave this input empty to keep the saved ID key.",
"Generate a random ID key": "Generate a random ID key",
"How to set the ID key": "How to set the ID key",
"Derive from current E2EE passphrase": "Derive from current E2EE passphrase",
"Enter an ID source": "Enter an ID source",
"Import an ID recovery code": "Import an ID recovery code",
"ID source": "ID source",
"ID recovery code": "ID recovery code",
"Enter an ID recovery code": "Enter an ID recovery code",
"Choose a long, unpredictable source. It is used once and cannot be shown again after saving. A recovery code can be displayed on this device later. This input also accepts a tagged recovery code.":
"Choose a long, unpredictable source. It is used once and cannot be shown again after saving. A recovery code can be displayed on this device later. This input also accepts a tagged recovery code.",
"Paste a tagged recovery code from an existing device to restore the same ID key.":
"Paste a tagged recovery code from an existing device to restore the same ID key.",
"For recovery after losing every device, save the recovery code after setup or choose an ID source you can reproduce.":
"For recovery after losing every device, save the recovery code after setup or choose an ID source you can reproduce.",
"Show current recovery code": "Show current recovery code",
"Hide current recovery code": "Hide current recovery code",
"Current ID recovery code": "Current ID recovery code",
"Copy recovery code": "Copy recovery code",
"Recovery code copied.": "Recovery code copied.",
"The displayed recovery code belongs to the current key. Reopen this dialogue after saving to copy the replacement key.":
"The displayed recovery code belongs to the current key. Reopen this dialogue after saving to copy the replacement key.",
"The recovery code could not be copied. Select and copy the visible code instead.":
"The recovery code could not be copied. Select and copy the visible code instead.",
"The ID key is derived from the current E2EE passphrase and saved separately. Changing that passphrase later does not change the saved ID key. To reduce the risk of guessing that passphrase from known IDs, use a separate, unpredictable ID source instead.":
"The ID key is derived from the current E2EE passphrase and saved separately. Changing that passphrase later does not change the saved ID key. To reduce the risk of guessing that passphrase from known IDs, use a separate, unpredictable ID source instead.",
"An ID source is required to enable this option.": "An ID source is required to enable this option.",
"The ID source or recovery code is invalid. Check it and try again.":
"The ID source or recovery code is invalid. Check it and try again.",
} as const;
export type LiveSyncProvisionalMessageKey = keyof typeof liveSyncProvisionalEnglishMessages;
src/common/replicatorConfigurationIdentity.ts
+8 -2
View File
@@ -1,4 +1,5 @@
import type { RemoteDBSettings } from "@vrtmrz/livesync-commonlib/compat/common/types";
import { usesEncryptedInternalMetadata } from "@vrtmrz/livesync-commonlib/replication";
type EndpointProjection = readonly [kind: "url" | "invalid-url", value: string];
@@ -42,7 +43,10 @@ function projectHeaders(value: string): readonly (readonly [name: string, value:
}
function projectRemoteSecurity(settings: RemoteDBSettings) {
return settings.encrypt
return [
settings.idDerivationVersion,
settings.idDerivationKey,
settings.encrypt
? ([
"encrypted",
settings.passphrase,
@@ -50,7 +54,8 @@ function projectRemoteSecurity(settings: RemoteDBSettings) {
settings.E2EEAlgorithm,
settings.permitEmptyPassphrase,
] as const)
: (["plain"] as const);
: (["plain"] as const),
] as const;
}
/**
@@ -77,6 +82,7 @@ export function getCouchDBReplicatorConfigurationIdentity(settings: RemoteDBSett
settings.useRequestAPI,
settings.disableRequestURI,
projectRemoteSecurity(settings),
usesEncryptedInternalMetadata(settings),
settings.enableCompression,
]);
}
src/common/replicatorConfigurationIdentity.unit.spec.ts
+27
View File
@@ -31,6 +31,17 @@ describe("active Replicator configuration identity", () => {
});
}
it("replaces a connection when the independent ID key changes", () => {
const first = configuredSettings({ idDerivationVersion: 1, idDerivationKey: "a".repeat(64) });
const second = { ...first, idDerivationKey: "b".repeat(64) };
expect(getCouchDBReplicatorConfigurationIdentity(second)).not.toBe(
getCouchDBReplicatorConfigurationIdentity(first)
);
expect(getObjectStorageReplicatorConfigurationIdentity(second)).not.toBe(
getObjectStorageReplicatorConfigurationIdentity(first)
);
});
it.each([
["couchDB_URI", "https://other.example.test/base"],
["couchDB_DBNAME", "other-vault"],
@@ -67,6 +78,22 @@ describe("active Replicator configuration identity", () => {
);
});
it("recreates the CouchDB connection when internal Metadata encryption becomes effective", () => {
const active = configuredSettings({ usePathObfuscation: true, encryptInternalMetadata: false });
const enabled = { ...active, encryptInternalMetadata: true };
expect(getCouchDBReplicatorConfigurationIdentity(enabled)).not.toBe(
getCouchDBReplicatorConfigurationIdentity(active)
);
const inactive = { ...active, usePathObfuscation: false };
expect(getCouchDBReplicatorConfigurationIdentity({ ...inactive, encryptInternalMetadata: true })).toBe(
getCouchDBReplicatorConfigurationIdentity(inactive)
);
expect(getObjectStorageReplicatorConfigurationIdentity(enabled)).toBe(
getObjectStorageReplicatorConfigurationIdentity(active)
);
});
it("projects only the active CouchDB authentication mode", () => {
const basic = configuredSettings({ useJWT: false, jwtKey: "inactive-a" });
expect(getCouchDBReplicatorConfigurationIdentity({ ...basic, jwtKey: "inactive-b" })).toBe(
src/common/reportTool.ts
+2
View File
@@ -80,6 +80,8 @@ export async function generateReport(settings: ObsidianLiveSyncSettings, core: L
pluginConfig.couchDB_USER = REDACTED;
pluginConfig.passphrase = REDACTED;
pluginConfig.encryptedPassphrase = REDACTED;
pluginConfig.idDerivationKey = REDACTED;
pluginConfig.encryptedIdDerivationKey = REDACTED;
pluginConfig.encryptedCouchDBConnection = REDACTED;
pluginConfig.accessKey = REDACTED;
pluginConfig.secretKey = REDACTED;
src/common/reportTool.unit.spec.ts
+16
View File
@@ -10,6 +10,22 @@ vi.mock("@vrtmrz/livesync-commonlib/compat/common/coreEnvFunctions", () => ({
}));
describe("TURN credentials in diagnostic reports", () => {
it("redacts the derived ID key and its encrypted local wrapper", async () => {
const key = "f3205cc41d24116d8c2484993c9d9a2e667373af338ba02f2ee71199adb82f2e";
const wrapper = "encrypted-id-key-test-wrapper";
const settings = {
...DEFAULT_SETTINGS,
idDerivationVersion: 1 as const,
idDerivationKey: key,
encryptedIdDerivationKey: wrapper,
};
const core = { services: { vault: { isStorageInsensitive: () => false } } } as unknown as LiveSyncBaseCore;
const report = await generateReport(settings, core);
const text = JSON.stringify(report);
expect(text).not.toContain(key);
expect(text).not.toContain(wrapper);
});
it("redacts provider tokens in all profiles and runtime credentials", async () => {
const token = "private+token/with=symbols";
const provider = { P2P_managedType: "CF", P2P_managedId: "private-key", P2P_managedToken: token };
src/features/ReviewHarness/reviewHarnessContract.ts
+24 -14
View File
@@ -1,9 +1,6 @@
import type { ObsidianLiveSyncSettings, SettingsMigrationState } from "@vrtmrz/livesync-commonlib/settings";
import type { CompatibilityPause } from "@/common/databaseCompatibility.ts";
import type {
ReviewHarnessScenarioResult,
ReviewHarnessScenarioStatus,
} from "./reviewHarnessTypes";
import type { ReviewHarnessScenarioResult, ReviewHarnessScenarioStatus } from "./reviewHarnessTypes";
export type { ReviewHarnessScenarioResult, ReviewHarnessScenarioStatus } from "./reviewHarnessTypes";
@@ -32,6 +29,14 @@ export const REVIEW_HARNESS_SCENARIOS = [
mode: "automatic",
access: "dedicated-vault-fixtures",
},
{
id: "id-generation-performance",
title: "ID generation performance",
description:
"Measures legacy and independent IDs with fixed in-memory inputs. Reports time per 1,000 IDs and per ID, key derivation time, and JavaScript heap samples where available. Keep Obsidian in the foreground.",
mode: "automatic",
access: "read-only",
},
] as const;
export const REVIEW_HARNESS_SCENARIO_IDS = REVIEW_HARNESS_SCENARIOS.map(({ id }) => id);
@@ -114,7 +119,9 @@ const NEW_VAULT_RECOMMENDATION_KEYS = [
"E2EEAlgorithm",
] as const;
type LifecycleSettingKey = (typeof PRESERVED_SYNC_SETTING_KEYS)[number] | (typeof NEW_VAULT_RECOMMENDATION_KEYS)[number];
type LifecycleSettingKey =
| (typeof PRESERVED_SYNC_SETTING_KEYS)[number]
| (typeof NEW_VAULT_RECOMMENDATION_KEYS)[number];
type SettingsForLifecycleInspection = Partial<Pick<ObsidianLiveSyncSettings, LifecycleSettingKey>>;
export function inspectSettingsLifecycle(input: {
@@ -130,9 +137,7 @@ export function inspectSettingsLifecycle(input: {
};
}
const invalidSyncSettings = PRESERVED_SYNC_SETTING_KEYS.filter(
(key) => typeof input.settings[key] !== "boolean"
);
const invalidSyncSettings = PRESERVED_SYNC_SETTING_KEYS.filter((key) => typeof input.settings[key] !== "boolean");
if (invalidSyncSettings.length > 0) {
return {
status: "failed",
@@ -205,6 +210,7 @@ export interface ReviewHarnessReportScenario {
readonly mode: ReviewHarnessScenarioMode;
readonly status: ReviewHarnessScenarioStatus;
readonly detail: string;
readonly observations?: readonly string[];
}
export interface ReviewHarnessReportInput {
@@ -248,13 +254,15 @@ export function formatReviewHarnessReport(input: ReviewHarnessReportInput): stri
);
const scenarios = table(
["Scenario", "Mode", "Status", "Detail"],
input.scenarios.map(({ id, title, mode, status, detail }) => [
`${title} (${id})`,
mode,
status,
detail,
])
input.scenarios.map(({ id, title, mode, status, detail }) => [`${title} (${id})`, mode, status, detail])
);
const observations = input.scenarios
.filter((scenario) => scenario.observations?.length)
.map(
({ title, observations }) =>
`### ${title}\n\n${observations!.map((value) => `- ${tableCell(value)}`).join("\n")}`
)
.join("\n\n");
return `## Self-hosted LiveSync Review Harness report
Generated at \`${tableCell(input.generatedAt)}\`.
@@ -267,6 +275,8 @@ ${environment}
${scenarios}
${observations}
<details>
<summary>Event transcript</summary>
src/features/ReviewHarness/reviewHarnessContract.unit.spec.ts
+1
View File
@@ -75,6 +75,7 @@ describe("Review Harness contract", () => {
"settings-lifecycle",
"compatibility-review",
"vault-round-trip",
"id-generation-performance",
]);
});
src/features/ReviewHarness/reviewHarnessController.ts
+5 -4
View File
@@ -21,6 +21,7 @@ export interface ReviewHarnessRuntime {
getCompatibilityPause(): CompatibilityPause | undefined;
openCompatibilityReview(): Promise<void>;
runVaultRoundTrip(): Promise<ReviewHarnessScenarioResult>;
runIdBenchmark(): Promise<ReviewHarnessScenarioResult>;
readContinuation(): string | null;
writeContinuation(value: string): void;
deleteContinuation(): void;
@@ -159,6 +160,8 @@ export class ReviewHarnessController {
});
} else if (id === "vault-round-trip") {
result = await this.runtime.runVaultRoundTrip();
} else if (id === "id-generation-performance") {
result = await this.runtime.runIdBenchmark();
} else {
const inspection = this.inspectCompatibilityReview();
result =
@@ -206,10 +209,7 @@ export class ReviewHarnessController {
detail: "The device-local compatibility review remains pending.",
observations: inspection.observations,
};
this.record(
"compatibility-review-updated",
this.results["compatibility-review"].status
);
this.record("compatibility-review-updated", this.results["compatibility-review"].status);
} catch (error) {
this.setUnexpectedFailure("compatibility-review", error);
} finally {
@@ -259,6 +259,7 @@ export class ReviewHarnessController {
mode,
status: this.results[id].status,
detail: this.results[id].detail,
observations: this.results[id].observations,
})),
transcript: this.transcript,
});
src/features/ReviewHarness/reviewHarnessController.unit.spec.ts
+60 -3
View File
@@ -80,6 +80,11 @@ function createRuntime(): ReviewHarnessRuntime & {
detail: "The owned fixture tree was exercised and removed.",
observations: [],
})),
runIdBenchmark: vi.fn(async () => ({
status: "passed" as const,
detail: "ID generation measurements completed.",
observations: ["Chunk 256 B: 1000 IDs total=43.00 ms; per ID=0.0430 ms"],
})),
readContinuation() {
return this.continuation;
},
@@ -150,6 +155,60 @@ describe("ReviewHarnessController", () => {
expect(runtime.reportError).toHaveBeenCalledOnce();
});
it("runs ID measurements on request and includes their units in the copied report", async () => {
const runtime = createRuntime();
const controller = new ReviewHarnessController(runtime);
await controller.runAutomaticScenarios();
expect(runtime.runIdBenchmark).not.toHaveBeenCalled();
await controller.runScenario("id-generation-performance");
await controller.copyReport();
expect(runtime.runIdBenchmark).toHaveBeenCalledOnce();
expect(controller.snapshot().results["id-generation-performance"].status).toBe("passed");
expect(vi.mocked(runtime.copyText).mock.calls[0][0]).toContain("1000 IDs total=43.00 ms; per ID=0.0430 ms");
expect(runtime.runVaultRoundTrip).not.toHaveBeenCalled();
expect(runtime.events).toEqual([]);
expect(runtime.continuation).toBeNull();
});
it("excludes an unexpected measurement error from the copied report", async () => {
const runtime = createRuntime();
runtime.runIdBenchmark = vi.fn().mockRejectedValue(new Error("private measurement error"));
const controller = new ReviewHarnessController(runtime);
await controller.runScenario("id-generation-performance");
expect(controller.snapshot().results["id-generation-performance"].status).toBe("failed");
expect(controller.createReport()).not.toContain("private measurement error");
expect(runtime.reportError).toHaveBeenCalledOnce();
});
it("does not overlap an ID measurement with another scenario", async () => {
const runtime = createRuntime();
let finish!: () => void;
const pending = new Promise<void>((resolve) => {
finish = resolve;
});
runtime.runIdBenchmark = vi.fn(async () => {
await pending;
return { status: "passed" as const, detail: "Measured", observations: [] };
});
const controller = new ReviewHarnessController(runtime);
const running = controller.runScenario("id-generation-performance");
await controller.runScenario("id-generation-performance");
await controller.runScenario("vault-round-trip");
expect(runtime.runIdBenchmark).toHaveBeenCalledOnce();
expect(runtime.runVaultRoundTrip).not.toHaveBeenCalled();
expect(controller.snapshot().running).toBe(true);
finish();
await running;
expect(controller.snapshot().running).toBe(false);
});
it("deletes a one-shot continuation before exposing the resumed guided step", () => {
const runtime = createRuntime();
runtime.continuation = JSON.stringify({
@@ -167,9 +226,7 @@ describe("ReviewHarnessController", () => {
expect(controller.snapshot().results["compatibility-review"]).toMatchObject({
status: "waiting-for-user",
});
expect(controller.snapshot().resumedRequestId).toBe(
"compatibility-review-2026-07-18T11:59:00.000Z"
);
expect(controller.snapshot().resumedRequestId).toBe("compatibility-review-2026-07-18T11:59:00.000Z");
});
it("does not copy rejected continuation values into the report", () => {
src/features/ReviewHarness/reviewHarnessIdBenchmark.ts
+109
View File
@@ -0,0 +1,109 @@
import type { ReviewHarnessScenarioResult } from "./reviewHarnessTypes";
export interface IdBenchmarkOperations {
deriveKey(): Promise<unknown>;
chunkId(piece: string, independent: boolean): Promise<string>;
documentId(path: string, independent: boolean): Promise<string>;
}
type BenchmarkPerformance = Pick<Performance, "now"> & {
readonly memory?: { readonly usedJSHeapSize: number };
};
const ID_COUNT = 1000;
const SAMPLES = 3;
const BATCH_SIZE = 100;
const WARMUP_COUNT = 32;
function readHeap(clock: BenchmarkPerformance): number | undefined {
try {
const bytes = clock.memory?.usedJSHeapSize;
return typeof bytes === "number" && Number.isFinite(bytes) && bytes >= 0 ? bytes : undefined;
} catch {
return undefined;
}
}
function summary(samples: readonly number[]): string {
const sorted = [...samples].sort((a, b) => a - b);
return `median=${sorted[1].toFixed(2)} ms; range=${sorted[0].toFixed(2)}–${sorted[2].toFixed(2)} ms`;
}
export async function runReviewHarnessIdBenchmark(
operations: IdBenchmarkOperations,
clock: BenchmarkPerformance = performance,
yieldControl: () => Promise<void> = () => new Promise((resolve) => window.setTimeout(resolve, 0))
): Promise<ReviewHarnessScenarioResult> {
const before = readHeap(clock);
let highest = before;
const sampleHeap = () => {
const value = readHeap(clock);
if (value !== undefined) highest = Math.max(highest ?? value, value);
return value;
};
const observations = [
"Fixed synthetic inputs; 3 samples, alternating legacy/independent order; 32 warm-up IDs per sample. Legacy Chunk algorithm: xxhash64.",
"Compute timings include input construction and awaited ID generation. Initialisation, warm-up, and pauses between batches are excluded. This does not measure a Rebuild or remote transfer.",
];
const derivationSamples: number[] = [];
for (let sample = 0; sample < SAMPLES; sample++) {
await yieldControl();
const started = clock.now();
await operations.deriveKey();
derivationSamples.push(clock.now() - started);
sampleHeap();
}
observations.push(`ID key derivation at save time: ${summary(derivationSamples)} per derivation.`);
const cases = [
...[256, 4096, 32768].map((bytes) => {
const prefix = "r".repeat(bytes - 8);
return {
label: `Chunk IDs, ${bytes} B`,
run: (i: number, independent: boolean) =>
operations.chunkId(prefix + i.toString(36).padStart(8, "0"), independent),
};
}),
{
label: "Obfuscated document IDs",
run: (i: number, independent: boolean) => operations.documentId(`benchmark/path-${i}.md`, independent),
},
];
for (const scenario of cases) {
const samples: [number[], number[]] = [[], []];
for (let sample = 0; sample < SAMPLES; sample++) {
for (const independent of sample % 2 === 0 ? [false, true] : [true, false]) {
for (let i = 0; i < WARMUP_COUNT; i++) await scenario.run(i, independent);
let elapsed = 0;
for (let batch = 0; batch < ID_COUNT; batch += BATCH_SIZE) {
await yieldControl();
const started = clock.now();
for (let i = batch; i < batch + BATCH_SIZE; i++) await scenario.run(i, independent);
elapsed += clock.now() - started;
sampleHeap();
}
samples[independent ? 1 : 0].push(elapsed);
}
}
for (const [index, values] of samples.entries()) {
const median = [...values].sort((a, b) => a - b)[1];
observations.push(
`${scenario.label}, ${index === 0 ? "legacy" : "independent"}: ${ID_COUNT} IDs total ${summary(values)}; per ID=${(median / ID_COUNT).toFixed(4)} ms.`
);
}
}
const after = sampleHeap();
if (highest === undefined) {
observations.push("JavaScript heap: unavailable on this device.");
} else {
const mib = (bytes: number | undefined) =>
bytes === undefined ? "unavailable" : `${(bytes / 1048576).toFixed(2)} MiB`;
observations.push(
`JavaScript heap: before=${mib(before)}; highest sampled=${mib(highest)}; after=${mib(after)}.`
);
}
observations.push(
"Heap samples are approximate, may include other Obsidian work, and are affected by garbage collection. They are neither total app RAM nor a true peak."
);
return { status: "passed", detail: "ID generation measurements completed.", observations };
}
src/features/ReviewHarness/reviewHarnessIdBenchmark.unit.spec.ts
+99
View File
@@ -0,0 +1,99 @@
import { describe, expect, it } from "vitest";
import { runReviewHarnessIdBenchmark, type IdBenchmarkOperations } from "./reviewHarnessIdBenchmark";
function fixture() {
let elapsed = 0;
let derivations = 0;
const chunkCounts = [0, 0];
const documentCounts = [0, 0];
const chunkSizes = new Set<number>();
const operations: IdBenchmarkOperations = {
deriveKey: () => {
derivations++;
elapsed += 42;
return Promise.resolve("private-derived-key");
},
chunkId: (piece, independent) => {
chunkCounts[independent ? 1 : 0]++;
chunkSizes.add(piece.length);
elapsed += independent ? 2 : 1;
return Promise.resolve("private-chunk-id");
},
documentId: (_path, independent) => {
documentCounts[independent ? 1 : 0]++;
elapsed += independent ? 4 : 3;
return Promise.resolve("private-document-id");
},
};
return {
operations,
now: () => elapsed,
yieldControl: () => {
elapsed += 100;
return Promise.resolve();
},
counts: () => ({ derivations, chunkCounts, documentCounts, chunkSizes: [...chunkSizes] }),
};
}
describe("Review Harness ID measurements", () => {
it("reports totals and per-ID timings separately, excluding warm-up and cooperative pauses", async () => {
const f = fixture();
const result = await runReviewHarnessIdBenchmark(f.operations, { now: f.now }, f.yieldControl);
const report = result.observations.join("\n");
expect(result.status).toBe("passed");
expect(report).toContain("1000 IDs total median=1000.00 ms; range=1000.00–1000.00 ms; per ID=1.0000 ms");
expect(report).toContain("1000 IDs total median=2000.00 ms; range=2000.00–2000.00 ms; per ID=2.0000 ms");
expect(report).toContain("Obfuscated document IDs, legacy: 1000 IDs total median=3000.00 ms");
expect(report).toContain("Obfuscated document IDs, independent: 1000 IDs total median=4000.00 ms");
expect(report).toContain("ID key derivation at save time: median=42.00 ms");
expect(report).toContain("JavaScript heap: unavailable on this device.");
expect(report).not.toContain("private-");
expect(f.counts()).toEqual({
derivations: 3,
chunkCounts: [9288, 9288],
documentCounts: [3096, 3096],
chunkSizes: [256, 4096, 32768],
});
});
it("labels the highest sampled heap separately from total app RAM and allows a lower final sample", async () => {
const f = fixture();
let reads = 0;
const clock = {
now: f.now,
get memory() {
return { usedJSHeapSize: (reads++ === 0 ? 2 : reads === 2 ? 5 : 1) * 1048576 };
},
};
const result = await runReviewHarnessIdBenchmark(f.operations, clock, f.yieldControl);
expect(result.observations).toContain(
"JavaScript heap: before=2.00 MiB; highest sampled=5.00 MiB; after=1.00 MiB."
);
expect(result.observations.join("\n")).toContain("neither total app RAM nor a true peak");
});
it.each([Number.NaN, Number.POSITIVE_INFINITY, -1, "throws"])(
"keeps timings usable when the heap API returns %s",
async (value) => {
const f = fixture();
const result = await runReviewHarnessIdBenchmark(
f.operations,
{
now: f.now,
get memory() {
if (value === "throws") throw new Error("Heap API unavailable");
return { usedJSHeapSize: value as number };
},
},
f.yieldControl
);
expect(result.status).toBe("passed");
expect(result.observations).toContain("JavaScript heap: unavailable on this device.");
expect(result.observations.join("\n")).not.toMatch(/NaN|Infinity|private-/u);
}
);
});
src/features/ReviewHarness/reviewHarnessIdBenchmarkRuntime.ts
+35
View File
@@ -0,0 +1,35 @@
import { DEFAULT_SETTINGS, deriveIdKey } from "@vrtmrz/livesync-commonlib/settings";
import { path2id_base } from "@vrtmrz/livesync-commonlib/compat/string_and_binary/path";
import type { FilePath } from "@vrtmrz/livesync-commonlib/compat/common/types";
import { HashManager } from "@vrtmrz/livesync-commonlib/hashing";
import type { IdBenchmarkOperations } from "./reviewHarnessIdBenchmark";
const FIXTURE_PASSPHRASE = "Self-hosted LiveSync ID benchmark passphrase";
const FIXTURE_SOURCE = "Self-hosted LiveSync ID benchmark source";
const FIXTURE_KEY = "ab".repeat(32);
export async function createIdBenchmarkOperations(): Promise<IdBenchmarkOperations> {
const managers: HashManager[] = [];
for (const independent of [false, true]) {
const settings = Object.freeze({
...DEFAULT_SETTINGS,
encrypt: true,
passphrase: FIXTURE_PASSPHRASE,
hashAlg: "xxhash64" as const,
idDerivationVersion: independent ? (1 as const) : (0 as const),
idDerivationKey: independent ? FIXTURE_KEY : "",
});
// HashManager only reads currentSettings; this fixture has no storage or live service access.
const settingService = { currentSettings: () => settings } as HashManager["options"]["settingService"];
const manager = new HashManager({ settingService });
if (!(await manager.initialise())) throw new Error("The benchmark hash manager could not initialise.");
managers.push(manager);
}
return {
deriveKey: () => deriveIdKey(FIXTURE_SOURCE),
chunkId: (piece, independent) => managers[independent ? 1 : 0].computeHash(piece),
// Fixture paths are already normalised; use the same ID calculation as PathService.
documentId: (path, independent) =>
path2id_base(path as FilePath, FIXTURE_PASSPHRASE, false, independent ? FIXTURE_KEY : undefined),
};
}
src/features/ReviewHarness/reviewHarnessIdBenchmarkRuntime.unit.spec.ts
+38
View File
@@ -0,0 +1,38 @@
import { describe, expect, it, vi } from "vitest";
import { DEFAULT_SETTINGS } from "@vrtmrz/livesync-commonlib/settings";
import { createIdBenchmarkOperations } from "./reviewHarnessIdBenchmarkRuntime";
describe("Review Harness benchmark implementation", () => {
it("uses the packaged legacy and independent algorithms with isolated fixed settings", async () => {
const originalDefaults = structuredClone(DEFAULT_SETTINGS);
const fetch = vi.spyOn(globalThis, "fetch").mockRejectedValue(new Error("Network access is forbidden"));
try {
const operations = await createIdBenchmarkOperations();
const chunk = "r".repeat(256);
const legacy = await operations.chunkId(chunk, false);
const independent = await operations.chunkId(chunk, true);
expect(legacy).toMatch(/^\+[0-9a-z]{1,13}$/u);
expect(independent).toMatch(/^\+[0-9a-f]{64}$/u);
expect(independent).toBe("+9223e53d99e80c29effee9e95e38ed168d13c14f717054f9e996a1cd0a597000");
expect(await operations.chunkId(chunk, false)).toBe(legacy);
expect(await operations.chunkId(chunk, true)).toBe(independent);
expect(await operations.chunkId("s".repeat(256), true)).not.toBe(independent);
const legacyPath = await operations.documentId("benchmark/path-1.md", false);
const independentPath = await operations.documentId("benchmark/path-1.md", true);
expect(legacyPath).toMatch(/^f:[0-9a-f]{64}$/u);
expect(independentPath).toMatch(/^f:[0-9a-f]{64}$/u);
expect(legacyPath).not.toBe(independentPath);
expect(await operations.documentId("benchmark/path-1.md", true)).toBe(independentPath);
const second = await createIdBenchmarkOperations();
expect(await second.chunkId(chunk, true)).toBe(independent);
expect(await operations.deriveKey()).toMatch(/^[0-9a-f]{64}$/u);
expect(fetch).not.toHaveBeenCalled();
expect(DEFAULT_SETTINGS).toEqual(originalDefaults);
} finally {
fetch.mockRestore();
}
});
});
src/modules/coreFeatures/ModuleResolveMismatchedTweaks.ts
+13
View File
@@ -99,6 +99,17 @@ function resolutionSettingsSignature(settings: ObsidianLiveSyncSettings): string
}
export class ModuleResolvingMismatchedTweaks extends AbstractModule {
private requiresIdConfigurationReview(assessment: TweakAssessment): boolean {
if (!assessment.entries.some(({ key, relation }) => key === "idDerivationVersion" && relation === "different")) {
return false;
}
Logger(
"The document ID configurations differ. Import the correct Setup URI, or configure the matching ID key, before synchronising.",
LOG_LEVEL_NOTICE
);
return true;
}
private _selectNewerTweakSide(current: TweakValues, preferred: Partial<TweakValues>): "REMOTE" | "CURRENT" {
Logger(`Modified: ${current.tweakModified} (current) vs ${preferred.tweakModified} (preferred)`);
const currentModified = current.tweakModified;
@@ -196,6 +207,7 @@ export class ModuleResolvingMismatchedTweaks extends AbstractModule {
assessment = assessTweakCompatibility(this.settings, preferred)
): Promise<[TweakValues | boolean, boolean]> {
if (assessment.alignment === "matched") return [false, false];
if (this.requiresIdConfigurationReview(assessment)) return [false, false];
const acceptedSettings = settingsAfterAdoption(assessment, "adoptPreferred");
const autoAcceptSide = await this._shouldAutoAcceptCompatibleLossy(assessment);
if (autoAcceptSide === "REMOTE") return [acceptedSettings, false];
@@ -363,6 +375,7 @@ export class ModuleResolvingMismatchedTweaks extends AbstractModule {
const trialSignature = JSON.stringify(trialSetting);
const currentSignature = resolutionSettingsSignature(this.settings);
const assessment = assessTweakCompatibility(trialSetting, preferred);
if (this.requiresIdConfigurationReview(assessment)) return { result: false, requireFetch: false };
if (assessment.alignment === "matched") {
this._log("The settings in the remote database are the same as the local database.", LOG_LEVEL_NOTICE);
return { result: false, requireFetch: false };
src/modules/coreFeatures/ModuleResolveMismatchedTweaks.unit.spec.ts
+63 -1
View File
@@ -7,7 +7,7 @@ import {
type TweakValues,
} from "@vrtmrz/livesync-commonlib/compat/common/types";
import { extractObject } from "octagonal-wheels/object";
import { assessTweakCompatibility } from "@vrtmrz/livesync-commonlib/settings";
import { assessTweakCompatibility, configuredIdKey } from "@vrtmrz/livesync-commonlib/settings";
import { ModuleResolvingMismatchedTweaks } from "./ModuleResolveMismatchedTweaks";
import { setLang } from "@/common/translation";
import {
@@ -74,6 +74,68 @@ function createModule(settingsOverride: Partial<typeof DEFAULT_SETTINGS> = {}) {
}
describe("ModuleResolvingMismatchedTweaks", () => {
it.each([0, 1] as const)(
"keeps ID configuration %s when automatically aligning Chunk settings",
async (idDerivationVersion) => {
const idDerivationKey = idDerivationVersion === 1 ? "ab".repeat(32) : "";
const { module, core, askSelectStringDialogue } = createModule({
encrypt: true,
usePathObfuscation: false,
idDerivationVersion,
idDerivationKey,
autoAcceptCompatibleTweak: true,
hashAlg: "xxhash64",
tweakModified: 1,
});
const preferred: TweakValues = {
...extractObject(TweakValuesTemplate, core.settings),
idDerivationVersion: idDerivationVersion === 1 ? 0 : 1,
hashAlg: "xxhash32",
tweakModified: 2,
};
core._services.tweakValue = {
checkAndAskResolvingMismatched: module._checkAndAskResolvingMismatchedTweaks.bind(module),
};
core._services.setting.saveSettingData.mockImplementation(async () => {
configuredIdKey(core.settings);
});
await expect(module._askResolvingMismatchedTweaks(preferred, async () => true)).resolves.toBe("CHECKAGAIN");
expect(core.settings).toMatchObject({ idDerivationVersion, idDerivationKey, hashAlg: "xxhash32" });
expect(askSelectStringDialogue).not.toHaveBeenCalled();
}
);
it.each(["active", "trial"] as const)(
"withholds ordinary tweak adoption for different document ID modes (%s)",
async (route) => {
const { module, core, askSelectStringDialogue } = createModule({
encrypt: true,
usePathObfuscation: true,
idDerivationVersion: 0,
idDerivationKey: "",
});
const preferred: TweakValues = {
...extractObject(TweakValuesTemplate, core.settings),
idDerivationVersion: 1,
};
if (route === "active") {
await expect(module._checkAndAskResolvingMismatchedTweaks(preferred)).resolves.toEqual([false, false]);
} else {
await expect(module._askUseRemoteConfiguration(core.settings, preferred)).resolves.toEqual({
result: false,
requireFetch: false,
});
}
expect(askSelectStringDialogue).not.toHaveBeenCalled();
expect(core._services.setting.saveSettingData).not.toHaveBeenCalled();
expect(core.settings).toMatchObject({ idDerivationVersion: 0, idDerivationKey: "" });
}
);
it("compatibility: offers ordinary application for a missing legacy filename-case setting", async () => {
const { module, askSelectStringDialogue } = createModule({
autoAcceptCompatibleTweak: false,
src/modules/features/ModuleLog.ts
+7 -3
View File
@@ -10,7 +10,7 @@ import {
import { scheduleTask } from "octagonal-wheels/concurrency/task";
import { fireAndForget, isDirty, throttle } from "@vrtmrz/livesync-commonlib/compat/common/utils";
import {
collectingChunks,
chunkFetchCounts,
pluginScanningCount,
hiddenFilesEventCount,
hiddenFilesProcessingCount,
@@ -36,7 +36,11 @@ import {
formatRemoteActivityStatusLabel,
getTrackedRequestCount,
} from "./RemoteActivityStatus.ts";
import { createMinimumVisibleActivityCount, createPaddedCounterLabel } from "./StatusBarDisplay.ts";
import {
createChunkFetchCounterLabel,
createMinimumVisibleActivityCount,
createPaddedCounterLabel,
} from "./StatusBarDisplay.ts";
import type { LiveSyncCore } from "@/main.ts";
import { LiveSyncError } from "@vrtmrz/livesync-commonlib/compat/common/LSError";
import { isValidPath } from "@/common/utils.ts";
@@ -140,7 +144,7 @@ export class ModuleLog extends AbstractObsidianModule {
const labelStorageCount = registerDisplay(
createPaddedCounterLabel(this.services.replication.storageApplyingCount, `💾`)
);
const labelChunkCount = registerDisplay(createPaddedCounterLabel(collectingChunks, `🧩`));
const labelChunkCount = registerDisplay(createChunkFetchCounterLabel(chunkFetchCounts));
const labelPluginScanCount = registerDisplay(createPaddedCounterLabel(pluginScanningCount, `🔌`));
const labelConflictProcessCount = registerDisplay(
createPaddedCounterLabel(this.services.conflict.conflictProcessQueueCount, `🔩`)
src/modules/features/ModuleObsidianSettingAsMarkdown.ts
+4
View File
@@ -140,6 +140,8 @@ export class ModuleObsidianSettingsAsMarkdown extends AbstractModule {
settingToApply.couchDB_USER = this.settings.couchDB_USER;
settingToApply.couchDB_PASSWORD = this.settings.couchDB_PASSWORD;
settingToApply.passphrase = this.settings.passphrase;
settingToApply.idDerivationVersion = this.settings.idDerivationVersion;
settingToApply.idDerivationKey = this.settings.idDerivationKey;
}
const oldSetting = this.generateSettingForMarkdown(
this.settings,
@@ -203,11 +205,13 @@ export class ModuleObsidianSettingsAsMarkdown extends AbstractModule {
const saveData = { ...(settings ? settings : this.settings) } as Partial<ObsidianLiveSyncSettings>;
delete saveData.encryptedCouchDBConnection;
delete saveData.encryptedPassphrase;
delete saveData.encryptedIdDerivationKey;
delete saveData.additionalSuffixOfDatabaseName;
if (!saveData.writeCredentialsForSettingSync && !keepCredential) {
delete saveData.couchDB_USER;
delete saveData.couchDB_PASSWORD;
delete saveData.passphrase;
delete saveData.idDerivationKey;
delete saveData.jwtKey;
delete saveData.jwtKid;
delete saveData.jwtSub;
src/modules/features/SettingDialogue/ObsidianLiveSyncSettingTab.ts
+1
View File
@@ -569,6 +569,7 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
}
}
// Internal Metadata encryption affects future Metadata writes and is not a rebuild requirement.
isNeedRebuildLocal() {
return this.isSomeDirty([
"useIndexedDBAdapter",
src/modules/features/SettingDialogue/PaneRemoteConfig.ts
+33 -1
View File
@@ -47,6 +47,12 @@ function getSettingsFromEditingSettings(editingSettings: AllSettings): ObsidianL
}
return workObj;
}
function syncIdDerivationSettings(target: Partial<ObsidianLiveSyncSettings>, source: ObsidianLiveSyncSettings): void {
target.idDerivationVersion = source.idDerivationVersion;
target.idDerivationKey = source.idDerivationKey;
}
function createRemoteConfigurationId(): string {
return `remote-${Date.now().toString(36)}-${Math.random().toString(36).slice(2, 8)}`;
}
@@ -116,7 +122,28 @@ export function paneRemoteConfig(
.onClick(async () => {
const setupManager = this.core.getModule(SetupManager);
const originalSettings = getSettingsFromEditingSettings(this.editingSettings);
await setupManager.onlyE2EEConfiguration(UserMode.Update, originalSettings);
const originalIdDerivationVersion = this.core.settings.idDerivationVersion;
const originalIdDerivationKey = this.core.settings.idDerivationKey;
const applied = await setupManager.onlyE2EEConfiguration(UserMode.Update, originalSettings);
if (applied) {
this.editingSettings.encryptInternalMetadata =
this.core.settings.encryptInternalMetadata;
if (this.initialSettings) {
this.initialSettings.encryptInternalMetadata =
this.core.settings.encryptInternalMetadata;
}
this.requestUpdate();
}
if (
this.core.settings.idDerivationVersion !== originalIdDerivationVersion ||
this.core.settings.idDerivationKey !== originalIdDerivationKey
) {
syncIdDerivationSettings(this.editingSettings, this.core.settings);
if (this.initialSettings) {
syncIdDerivationSettings(this.initialSettings, this.core.settings);
}
this.requestUpdate();
}
updateE2EESummary();
})
.setButtonText("Configure")
@@ -155,9 +182,11 @@ export function paneRemoteConfig(
const currentConfigs = cloneRemoteConfigurations(this.core.settings.remoteConfigurations);
this.editingSettings.remoteConfigurations = currentConfigs;
this.editingSettings.activeConfigurationId = this.core.settings.activeConfigurationId;
syncIdDerivationSettings(this.editingSettings, this.core.settings);
if (this.initialSettings) {
this.initialSettings.remoteConfigurations = cloneRemoteConfigurations(currentConfigs);
this.initialSettings.activeConfigurationId = this.core.settings.activeConfigurationId;
syncIdDerivationSettings(this.initialSettings, this.core.settings);
}
};
const persistRemoteConfigurations = async (synchroniseActiveRemote: boolean = false) => {
@@ -243,7 +272,10 @@ export function paneRemoteConfig(
...DEFAULT_SETTINGS,
encrypt: this.editingSettings.encrypt,
usePathObfuscation: this.editingSettings.usePathObfuscation,
encryptInternalMetadata: this.editingSettings.encryptInternalMetadata,
passphrase: this.editingSettings.passphrase,
idDerivationVersion: this.editingSettings.idDerivationVersion,
idDerivationKey: this.editingSettings.idDerivationKey,
configPassphraseStore: this.editingSettings.configPassphraseStore,
});
const addRemoteConfiguration = async () => {
src/modules/features/SettingDialogue/PaneRemoteConfig.unit.spec.ts
+94 -1
View File
@@ -2,6 +2,7 @@ import { afterEach, describe, expect, it, vi } from "vitest";
const runtime = vi.hoisted(() => ({
buttonClasses: [] as string[],
clickHandlers: [] as Array<() => Promise<void> | void>,
panels: [] as Array<{ destroy: ReturnType<typeof vi.fn> }>,
settingClasses: [] as string[],
}));
@@ -51,7 +52,8 @@ vi.mock("./LiveSyncSetting.ts", () => ({
setDestructive() {
return this;
},
onClick() {
onClick(callback: () => Promise<void> | void) {
runtime.clickHandlers.push(callback);
return this;
},
setButtonText() {
@@ -97,6 +99,7 @@ vi.mock("@vrtmrz/livesync-commonlib/compat/common/ConnectionString", () => ({
},
}));
vi.mock("@/modules/features/SetupWizard/dialogs/SetupRemote.svelte", () => ({ default: {} }));
vi.mock("@/modules/features/SetupWizard/dialogs/SetupRemoteE2EE.svelte", () => ({ default: {} }));
vi.mock("@/modules/features/SetupWizard/dialogs/SetupRemoteCouchDB.svelte", () => ({ default: {} }));
vi.mock("@/modules/features/SetupWizard/dialogs/SetupRemoteBucket.svelte", () => ({ default: {} }));
vi.mock("@/modules/features/SetupWizard/dialogs/SetupRemoteP2P.svelte", () => ({ default: {} }));
@@ -114,6 +117,7 @@ function createPanelElement(): HTMLElement {
afterEach(() => {
runtime.buttonClasses.length = 0;
runtime.clickHandlers.length = 0;
runtime.panels.length = 0;
runtime.settingClasses.length = 0;
vi.clearAllMocks();
@@ -148,4 +152,93 @@ describe("paneRemoteConfig", () => {
expect(runtime.panels[0].destroy).toHaveBeenCalledOnce();
});
it("applies an internal Metadata preference change without scheduling setup initialisation", async () => {
const originalSettings = {
encrypt: true,
passphrase: "passphrase",
E2EEAlgorithm: "v2",
usePathObfuscation: true,
encryptInternalMetadata: false,
remoteConfigurations: {},
};
const setupManager = {
onlyE2EEConfiguration: vi.fn(async () => {
host.core.settings.encryptInternalMetadata = true;
return true;
}),
};
const host = {
editingSettings: { ...originalSettings },
initialSettings: { ...originalSettings },
core: {
settings: { ...originalSettings },
getModule: vi.fn(() => setupManager),
},
lifetimeComponent: { register: vi.fn() },
requestUpdate: vi.fn(),
};
const addPanel = vi.fn((_parent: HTMLElement, heading: string) => ({
then(callback: (paneEl: HTMLElement) => void) {
if (heading === "E2EE Configuration") {
callback(createPanelElement());
}
},
}));
paneRemoteConfig.call(host as never, {} as HTMLElement, { addPanel } as never);
await runtime.clickHandlers[0]();
expect(setupManager.onlyE2EEConfiguration).toHaveBeenCalledOnce();
expect(host.editingSettings.encryptInternalMetadata).toBe(true);
expect(host.initialSettings.encryptInternalMetadata).toBe(true);
expect(host.requestUpdate).toHaveBeenCalledOnce();
});
it("copies applied ID derivation settings into both dialogue buffers", async () => {
const nextIdKey = "ab".repeat(32);
const originalSettings = {
encrypt: true,
passphrase: "passphrase",
E2EEAlgorithm: "v2",
usePathObfuscation: true,
encryptInternalMetadata: false,
idDerivationVersion: 0,
idDerivationKey: "",
remoteConfigurations: {},
};
const setupManager = {
onlyE2EEConfiguration: vi.fn(() => {
host.core.settings.idDerivationVersion = 1;
host.core.settings.idDerivationKey = nextIdKey;
return Promise.resolve(false);
}),
};
const host = {
editingSettings: { ...originalSettings },
initialSettings: { ...originalSettings },
core: {
settings: { ...originalSettings },
getModule: vi.fn(() => setupManager),
},
lifetimeComponent: { register: vi.fn() },
requestUpdate: vi.fn(),
};
const addPanel = vi.fn((_parent: HTMLElement, heading: string) => ({
then(callback: (paneEl: HTMLElement) => void) {
if (heading === "E2EE Configuration") {
callback(createPanelElement());
}
},
}));
paneRemoteConfig.call(host as never, {} as HTMLElement, { addPanel } as never);
await runtime.clickHandlers[0]();
expect(host.editingSettings.idDerivationVersion).toBe(1);
expect(host.editingSettings.idDerivationKey).toBe(nextIdKey);
expect(host.initialSettings.idDerivationVersion).toBe(1);
expect(host.initialSettings.idDerivationKey).toBe(nextIdKey);
expect(host.requestUpdate).toHaveBeenCalledOnce();
});
});
src/modules/features/SettingDialogue/settingUtils.ts
+1
View File
@@ -68,6 +68,7 @@ export function getE2EEConfigSummary(setting: ObsidianLiveSyncSettings, showAdva
export function getSummaryFromPartialSettings(setting: Partial<ObsidianLiveSyncSettings>, showAdvanced = false) {
const outputSummary: Record<string, string> = {};
for (const key of Object.keys(setting) as (keyof ObsidianLiveSyncSettings)[]) {
if (key === "idDerivationKey" || key === "encryptedIdDerivationKey") continue;
const config = getConfig(key as AllSettingItemKey);
if (!config) continue;
if (config.isAdvanced && !showAdvanced) continue;
src/modules/features/SetupManager.ts
+60 -7
View File
@@ -1,6 +1,5 @@
import {
type BucketSyncSetting,
type EncryptionSettings,
type ObsidianLiveSyncSettings,
type P2PSyncSetting,
LOG_LEVEL_NOTICE,
@@ -36,6 +35,7 @@ import type {
SetupRemoteCouchDBResultType,
SetupRemoteCouchDBInitialData,
SetupRemoteE2EEResultType,
SetupRemoteE2EEInitialData,
SetupRemoteP2PInitialData,
SetupRemoteP2PResultType,
SetupRemoteResultType,
@@ -58,6 +58,20 @@ function copySettingsForRemoteProfileUpdate(settings: ObsidianLiveSyncSettings):
};
}
function normaliseImportedIdDerivationSettings(settings: ObsidianLiveSyncSettings): ObsidianLiveSyncSettings {
// Setup URIs are complete imports even when their encoder omitted default-valued fields.
// Fill each missing half so a receiving device cannot supply the unrelated saved key.
return {
...settings,
idDerivationVersion: Object.prototype.hasOwnProperty.call(settings, "idDerivationVersion")
? settings.idDerivationVersion
: 0,
idDerivationKey: Object.prototype.hasOwnProperty.call(settings, "idDerivationKey")
? settings.idDerivationKey
: "",
};
}
/**
* User modes for onboarding and setup
*/
@@ -219,7 +233,7 @@ export class SetupManager extends AbstractModule {
return false;
}
this._log("Setup URI dialog closed.", LOG_LEVEL_VERBOSE);
return await this.onConfirmApplySettingsFromWizard(newSetting, userMode);
return await this.onConfirmApplySettingsFromWizard(normaliseImportedIdDerivationSettings(newSetting), userMode);
}
/**
@@ -328,14 +342,44 @@ export class SetupManager extends AbstractModule {
* @returns
*/
async onlyE2EEConfiguration(userMode: UserMode, currentSetting: ObsidianLiveSyncSettings): Promise<boolean> {
const e2eeConf = await this.dialogManager.openWithExplicitCancel<SetupRemoteE2EEResultType, EncryptionSettings>(
const e2eeConf = await this.dialogManager.openWithExplicitCancel<
SetupRemoteE2EEResultType,
SetupRemoteE2EEInitialData
>(
SetupRemoteE2EE,
currentSetting
{ settings: currentSetting, newVault: userMode === UserMode.NewUser }
);
if (e2eeConf === "cancelled") {
this._log("E2EE configuration cancelled.", LOG_LEVEL_NOTICE);
return false;
}
const onlyInternalMetadataPreferenceChanged =
currentSetting.encryptInternalMetadata !== e2eeConf.encryptInternalMetadata &&
currentSetting.encrypt === e2eeConf.encrypt &&
currentSetting.passphrase === e2eeConf.passphrase &&
currentSetting.E2EEAlgorithm === e2eeConf.E2EEAlgorithm &&
currentSetting.usePathObfuscation === e2eeConf.usePathObfuscation &&
currentSetting.idDerivationVersion === e2eeConf.idDerivationVersion &&
currentSetting.idDerivationKey === e2eeConf.idDerivationKey;
if (userMode === UserMode.Update && onlyInternalMetadataPreferenceChanged) {
if (e2eeConf.encryptInternalMetadata && currentSetting.remoteType === REMOTE_COUCHDB) {
const proceed = "Enable without rebuilding — update every other device first";
const choice = await this.core.confirm.askSelectStringDialogue(
"A manual remote Rebuild is strongly recommended to protect existing file properties. " +
"Before continuing without rebuilding, update every other synchronising device to a version " +
"which supports this option, including devices currently running LiveSync. " +
"Existing properties remain unchanged until they are rewritten or rebuilt.",
[proceed, "Cancel"],
{ title: "Encrypt internal file Properties", defaultAction: "Cancel" }
);
if (choice !== proceed) return false;
}
await this.services.setting.applyPartial(
{ encryptInternalMetadata: e2eeConf.encryptInternalMetadata },
true
);
return true;
}
const newSetting = {
...currentSetting,
...e2eeConf,
@@ -350,9 +394,12 @@ export class SetupManager extends AbstractModule {
* @returns
*/
async onConfigureManually(originalSetting: ObsidianLiveSyncSettings, userMode: UserMode): Promise<boolean> {
const e2eeConf = await this.dialogManager.openWithExplicitCancel<SetupRemoteE2EEResultType, EncryptionSettings>(
const e2eeConf = await this.dialogManager.openWithExplicitCancel<
SetupRemoteE2EEResultType,
SetupRemoteE2EEInitialData
>(
SetupRemoteE2EE,
originalSetting
{ settings: originalSetting, newVault: userMode === UserMode.NewUser }
);
if (e2eeConf === "cancelled") {
this._log("Manual configuration cancelled.", LOG_LEVEL_NOTICE);
@@ -496,7 +543,13 @@ export class SetupManager extends AbstractModule {
* @returns Promise that resolves to true if settings applied successfully, false otherwise
*/
async decodeQR(qr: string) {
const newSettings = decodeSettingsFromQRCodeData(qr);
let newSettings: ObsidianLiveSyncSettings;
try {
newSettings = normaliseImportedIdDerivationSettings(decodeSettingsFromQRCodeData(qr));
} catch {
this._log("The QR configuration could not be decoded or contains unsupported settings.", LOG_LEVEL_NOTICE);
return false;
}
return await this.onConfirmApplySettingsFromWizard(newSettings, UserMode.Unknown);
}
src/modules/features/SetupManager.unit.spec.ts
+145
View File
@@ -193,6 +193,58 @@ describe("SetupManager", () => {
expect(setting.currentSettings().activeConfigurationId).toBe("legacy-couchdb");
});
it("compatibility: treats omitted ID derivation fields in a Setup URI as legacy defaults", async () => {
const { manager, setting, dialogManager } = createSetupManager();
const savedKey = "12".repeat(32);
setting.settings = {
...createLegacyRemoteSetting(),
isConfigured: true,
idDerivationVersion: 1,
idDerivationKey: savedKey,
};
const imported = {
...createLegacyRemoteSetting(),
isConfigured: true,
} as Partial<ObsidianLiveSyncSettings>;
delete imported.idDerivationVersion;
delete imported.idDerivationKey;
vi.spyOn(setting, "adjustSettings").mockImplementation((settings) => Promise.resolve(settings));
dialogManager.openWithExplicitCancel.mockResolvedValueOnce(imported).mockResolvedValueOnce("cancelled");
await manager.onUseSetupURI(UserMode.Unknown, "mock-config://legacy-settings");
const mergedSettings = vi.mocked(setting.adjustSettings).mock.calls[0][0];
expect(mergedSettings.idDerivationVersion).toBe(0);
expect(mergedSettings.idDerivationKey).toBe("");
expect(setting.currentSettings().idDerivationKey).toBe(savedKey);
});
it("does not inherit the missing half of a partially present Setup URI ID configuration", async () => {
const { manager, setting, dialogManager } = createSetupManager();
const savedKey = "34".repeat(32);
setting.settings = {
...createLegacyRemoteSetting(),
isConfigured: true,
idDerivationVersion: 1,
idDerivationKey: savedKey,
};
const imported = {
...createLegacyRemoteSetting(),
isConfigured: true,
idDerivationVersion: 1,
} as Partial<ObsidianLiveSyncSettings>;
delete imported.idDerivationKey;
vi.spyOn(setting, "adjustSettings").mockImplementation((settings) => Promise.resolve(settings));
dialogManager.openWithExplicitCancel.mockResolvedValueOnce(imported).mockResolvedValueOnce("cancelled");
await manager.onUseSetupURI(UserMode.Unknown, "mock-config://partial-settings");
const mergedSettings = vi.mocked(setting.adjustSettings).mock.calls[0][0];
expect(mergedSettings.idDerivationVersion).toBe(1);
expect(mergedSettings.idDerivationKey).toBe("");
expect(setting.currentSettings().idDerivationKey).toBe(savedKey);
});
it("compatibility: normalises imported flat remote settings from QR data before applying", async () => {
const { manager, setting, dialogManager } = createSetupManager();
vi.mocked(decodeSettingsFromQRCodeData).mockReturnValue(createLegacyRemoteSetting());
@@ -208,6 +260,79 @@ describe("SetupManager", () => {
expect(setting.currentSettings().activeConfigurationId).toBe("legacy-couchdb");
});
it("compatibility: applies legacy defaults when QR data omits ID derivation fields", async () => {
const { manager, setting, dialogManager } = createSetupManager();
const savedKey = "56".repeat(32);
setting.settings = {
...createLegacyRemoteSetting(),
isConfigured: true,
idDerivationVersion: 1,
idDerivationKey: savedKey,
};
const imported = { ...createLegacyRemoteSetting(), isConfigured: true } as Partial<ObsidianLiveSyncSettings>;
delete imported.idDerivationVersion;
delete imported.idDerivationKey;
vi.mocked(decodeSettingsFromQRCodeData).mockReturnValue(imported as ObsidianLiveSyncSettings);
vi.spyOn(setting, "adjustSettings").mockImplementation((settings) => Promise.resolve(settings));
dialogManager.openWithExplicitCancel.mockResolvedValueOnce("cancelled");
await manager.decodeQR("qr-data");
const mergedSettings = vi.mocked(setting.adjustSettings).mock.calls[0][0];
expect(mergedSettings.idDerivationVersion).toBe(0);
expect(mergedSettings.idDerivationKey).toBe("");
expect(setting.currentSettings().idDerivationKey).toBe(savedKey);
});
it("rejects invalid QR settings before applying them", async () => {
const { manager, setting } = createSetupManager();
vi.mocked(decodeSettingsFromQRCodeData).mockImplementationOnce(() => {
throw new Error("Invalid ID derivation key");
});
const applyExternalSettings = vi.spyOn(setting, "applyExternalSettings");
await expect(manager.decodeQR("invalid-qr")).resolves.toBe(false);
expect(applyExternalSettings).not.toHaveBeenCalled();
});
it("requires the normal Fetch choice when ID derivation changes with the Metadata preference", async () => {
const { manager, setting, dialogManager, core } = createSetupManager();
const currentSettings: ObsidianLiveSyncSettings = {
...createLegacyRemoteSetting(),
isConfigured: true,
encrypt: true,
passphrase: "e2ee-passphrase",
usePathObfuscation: true,
encryptInternalMetadata: false,
idDerivationVersion: 0,
idDerivationKey: "",
};
const nextIdKey = "78".repeat(32);
setting.settings = currentSettings;
const applyPartial = vi.spyOn(setting, "applyPartial");
core.confirm = {
askSelectStringDialogue: vi.fn(() =>
Promise.resolve("Enable without rebuilding — update every other device first")
),
};
dialogManager.openWithExplicitCancel
.mockResolvedValueOnce({
...currentSettings,
encryptInternalMetadata: true,
idDerivationVersion: 1,
idDerivationKey: nextIdKey,
})
.mockResolvedValueOnce("existing-user")
.mockResolvedValueOnce("apply");
await manager.onlyE2EEConfiguration(UserMode.Update, currentSettings);
expect(applyPartial).not.toHaveBeenCalled();
expect(core.rebuilder.scheduleFetch).toHaveBeenCalledWith(expect.any(Function));
expect(setting.currentSettings().idDerivationVersion).toBe(1);
expect(setting.currentSettings().idDerivationKey).toBe(nextIdKey);
});
it("reserves Rebuild before saving a new-user configuration", async () => {
const { manager, setting, dialogManager, core } = createSetupManager();
setting.settings = { ...setting.currentSettings(), isConfigured: false };
@@ -659,3 +784,23 @@ describe("SetupManager", () => {
expect(setting.currentSettings().P2P_ActiveRemoteConfigurationId).toBe("existing");
});
});
describe("internal Metadata configuration", () => {
it.each([true, false])(
"applies the preference only after accepting the no-Rebuild warning (%s)",
async (accept) => {
const { manager, setting, dialogManager, core } = createSetupManager();
const current = { ...setting.settings, encryptInternalMetadata: false, remoteType: REMOTE_COUCHDB };
dialogManager.openWithExplicitCancel.mockResolvedValue({ ...current, encryptInternalMetadata: true });
const ask = vi.fn(async (_message: string, choices: string[]) => (accept ? choices[0] : "Cancel"));
core.confirm = { askSelectStringDialogue: ask };
const apply = vi.spyOn(setting, "applyPartial").mockResolvedValue(undefined);
await expect(manager.onlyE2EEConfiguration(UserMode.Update, current)).resolves.toBe(accept);
expect(ask.mock.calls[0][1][0]).toContain("update every other device first");
expect(ask.mock.calls[0][0]).toContain("currently running LiveSync");
expect(apply).toHaveBeenCalledTimes(accept ? 1 : 0);
expect(core.rebuilder.scheduleRebuild).not.toHaveBeenCalled();
expect(core.rebuilder.scheduleFetch).not.toHaveBeenCalled();
}
);
});
src/modules/features/SetupWizard/dialogs/SetupRemoteE2EE.svelte
+330 -7
View File
@@ -13,38 +13,144 @@
E2EEAlgorithms,
type EncryptionSettings,
} from "@vrtmrz/livesync-commonlib/compat/common/types";
import {
deriveIdKey,
deriveOrImportIdKey,
formatIdRecoveryCode,
ID_DERIVATION_VERSION,
ID_RECOVERY_CODE_PREFIX,
} from "@vrtmrz/livesync-commonlib/settings";
import { onMount } from "svelte";
import type { GuestDialogProps } from "@/modules/services/LiveSyncUI/svelteDialog";
import { copyTo, pickEncryptionSettings } from "@vrtmrz/livesync-commonlib/compat/common/utils";
import { TYPE_CANCELLED, type SetupRemoteE2EEResultType } from "./setupDialogTypes";
import {
TYPE_CANCELLED,
type SetupRemoteE2EEInitialData,
type SetupRemoteE2EEResultType,
} from "./setupDialogTypes";
import { $msg as translateMessage } from "@/common/translation";
type Props = GuestDialogProps<SetupRemoteE2EEResultType, EncryptionSettings>;
type Props = GuestDialogProps<SetupRemoteE2EEResultType, SetupRemoteE2EEInitialData>;
type IdConfigurationChoice = "keep" | "random" | "custom";
type IdCustomChoice = "passphrase" | "source" | "recovery";
const { setResult, getInitialData }: Props = $props();
let default_encryption: EncryptionSettings = {
encrypt: true,
passphrase: "",
E2EEAlgorithm: DEFAULT_SETTINGS.E2EEAlgorithm,
usePathObfuscation: true,
} as EncryptionSettings;
encryptInternalMetadata: true,
idDerivationVersion: 0,
idDerivationKey: "",
};
let encryptionSettings = $state<EncryptionSettings>({ ...default_encryption });
let newVault = $state(false);
let idConfigurationChoice = $state<IdConfigurationChoice>("keep");
let idCustomChoice = $state<IdCustomChoice>("source");
let idDerivationSource = $state("");
let idDerivationError = $state("");
let recoveryCodeVisible = $state(false);
let recoveryCodeCopied = $state(false);
const idDerivationConfigured = $derived(
encryptionSettings.idDerivationVersion === ID_DERIVATION_VERSION &&
typeof encryptionSettings.idDerivationKey === "string" &&
encryptionSettings.idDerivationKey.length > 0
);
const recoveryCode = $derived.by(() =>
idDerivationConfigured ? formatIdRecoveryCode(encryptionSettings.idDerivationKey) : ""
);
onMount(() => {
if (getInitialData) {
const initialData = getInitialData();
if (initialData) {
copyTo(initialData, encryptionSettings);
copyTo(initialData.settings, encryptionSettings);
newVault = initialData.newVault;
}
}
idConfigurationChoice = !idDerivationConfigured && newVault ? "random" : "keep";
});
let e2eeValid = $derived.by(() => {
if (!encryptionSettings.encrypt) return true;
return encryptionSettings.passphrase.trim().length >= 1;
});
let canEncryptInternalMetadata = $derived(
encryptionSettings.encrypt &&
encryptionSettings.E2EEAlgorithm === E2EEAlgorithms.V2 &&
encryptionSettings.usePathObfuscation
);
function commit() {
setResult(pickEncryptionSettings(encryptionSettings));
function resetIdDerivationSource() {
idDerivationSource = "";
idDerivationError = "";
}
function toggleEncryption(enabled: boolean) {
encryptionSettings.encrypt = enabled;
if (!enabled) resetIdDerivationSource();
}
function selectIdConfiguration() {
recoveryCodeVisible = false;
recoveryCodeCopied = false;
resetIdDerivationSource();
}
function selectIdCustomSource() {
resetIdDerivationSource();
}
async function copyRecoveryCode() {
try {
await navigator.clipboard.writeText(recoveryCode);
recoveryCodeCopied = true;
} catch {
idDerivationError = translateMessage("The recovery code could not be copied. Select and copy the visible code instead.");
}
}
async function commit() {
idDerivationError = "";
const result = pickEncryptionSettings(encryptionSettings);
if (encryptionSettings.encrypt && idConfigurationChoice !== "keep") {
let source = idDerivationSource;
if (idConfigurationChoice === "random") {
const bytes = crypto.getRandomValues(new Uint8Array(32));
source = Array.from(bytes, (byte) => byte.toString(16).padStart(2, "0")).join("");
} else if (idCustomChoice === "passphrase") {
source = encryptionSettings.passphrase;
}
if (source.length === 0) {
if (!idDerivationConfigured) {
idDerivationError = translateMessage("An ID source is required to enable this option.");
return;
}
} else {
try {
result.idDerivationKey =
idConfigurationChoice === "custom" && idCustomChoice !== "passphrase"
? await importOrDeriveEnteredIdKey(source, idCustomChoice)
: await deriveIdKey(source);
result.idDerivationVersion = ID_DERIVATION_VERSION;
} catch {
idDerivationError = translateMessage("The ID source or recovery code is invalid. Check it and try again.");
return;
}
}
}
idDerivationSource = "";
setResult(result);
}
async function importOrDeriveEnteredIdKey(source: string, choice: IdCustomChoice): Promise<string> {
if (choice === "recovery" && !source.trim().startsWith(ID_RECOVERY_CODE_PREFIX)) {
throw new Error("An ID recovery code is required.");
}
return await deriveOrImportIdKey(source);
}
</script>
@@ -52,7 +158,11 @@
<DialogHeader title={translateMessage("End-to-End Encryption")} />
<Guidance>{translateMessage("Please configure your end-to-end encryption settings.")}</Guidance>
<InputRow label={translateMessage("End-to-End Encryption")}>
<input type="checkbox" bind:checked={encryptionSettings.encrypt} />
<input
type="checkbox"
checked={encryptionSettings.encrypt}
onchange={(event) => toggleEncryption(event.currentTarget.checked)}
/>
</InputRow>
<InfoNote title={translateMessage("Strongly Recommended")}>
{translateMessage(
@@ -87,6 +197,182 @@
</InfoNote>
{/if}
<fieldset class="sls-id-choices" disabled={!encryptionSettings.encrypt}>
<legend>{translateMessage("ID generation")}</legend>
<label class="sls-id-choice">
<input
type="radio"
name="id-derivation-choice"
value="keep"
bind:group={idConfigurationChoice}
onchange={selectIdConfiguration}
/>
<div class="sls-id-choice-text">
<span>{translateMessage("Keep current configuration")}</span>
<small class="sls-current-id-configuration">
{#if idDerivationConfigured}
{translateMessage(
encryptionSettings.encrypt
? "Current configuration: a saved ID key is used."
: "Current configuration: the saved ID key is retained while E2EE is off."
)}
{:else}
{translateMessage(
"Current configuration: no ID key is saved. With E2EE enabled, keeping it uses legacy IDs tied to the E2EE passphrase."
)}
{/if}
</small>
</div>
</label>
<label class="sls-id-choice">
<input
type="radio"
name="id-derivation-choice"
value="random"
bind:group={idConfigurationChoice}
onchange={selectIdConfiguration}
/>
<span>{translateMessage("Generate a random ID key")}</span>
</label>
<label class="sls-id-choice">
<input
type="radio"
name="id-derivation-choice"
value="custom"
bind:group={idConfigurationChoice}
onchange={selectIdConfiguration}
/>
<span>{translateMessage("Set an ID key")}</span>
</label>
</fieldset>
{#if encryptionSettings.encrypt && idConfigurationChoice === "keep" && !idDerivationConfigured}
<InfoNote warning>
{translateMessage("Changing the E2EE passphrase changes IDs generated by the legacy configuration.")}
</InfoNote>
{/if}
{#if (encryptionSettings.encrypt && idConfigurationChoice !== "keep") || idDerivationConfigured}
{#if encryptionSettings.encrypt}
<InfoNote>
{translateMessage(
"This uses a saved key for new Chunk IDs and obfuscated Metadata document IDs, so changing the E2EE passphrase does not derive a new key automatically."
)}
</InfoNote>
{/if}
{#if idDerivationConfigured}
<InfoNote title={translateMessage("Configured")}>
{translateMessage("The saved ID key is configured. Its source cannot be shown again.")}
</InfoNote>
<button type="button" onclick={() => (recoveryCodeVisible = !recoveryCodeVisible)}>
{translateMessage(recoveryCodeVisible ? "Hide current recovery code" : "Show current recovery code")}
</button>
{#if recoveryCodeVisible}
<InputRow label={translateMessage("Current ID recovery code")}>
<input type="text" readonly value={recoveryCode} aria-label={translateMessage("Current ID recovery code")} />
<button type="button" onclick={copyRecoveryCode}>{translateMessage("Copy recovery code")}</button>
</InputRow>
{#if recoveryCodeCopied}
<InfoNote>{translateMessage("Recovery code copied.")}</InfoNote>
{/if}
{/if}
{/if}
{#if encryptionSettings.encrypt}
{#if idConfigurationChoice === "custom"}
<fieldset class="sls-id-choices sls-id-custom-choices">
<legend>{translateMessage("How to set the ID key")}</legend>
<label class="sls-id-choice">
<input
type="radio"
name="id-custom-choice"
value="passphrase"
bind:group={idCustomChoice}
onchange={selectIdCustomSource}
/>
<span>{translateMessage("Derive from current E2EE passphrase")}</span>
</label>
<label class="sls-id-choice">
<input
type="radio"
name="id-custom-choice"
value="source"
bind:group={idCustomChoice}
onchange={selectIdCustomSource}
/>
<span>{translateMessage("Enter an ID source")}</span>
</label>
<label class="sls-id-choice">
<input
type="radio"
name="id-custom-choice"
value="recovery"
bind:group={idCustomChoice}
onchange={selectIdCustomSource}
/>
<span>{translateMessage("Import an ID recovery code")}</span>
</label>
</fieldset>
{#if idCustomChoice === "source" || idCustomChoice === "recovery"}
<InputRow
label={translateMessage(idCustomChoice === "source" ? "ID source" : "ID recovery code")}
>
<Password
name="id-derivation-source"
placeholder={translateMessage(
idCustomChoice === "source" ? "Enter an ID source" : "Enter an ID recovery code"
)}
bind:value={idDerivationSource}
/>
</InputRow>
{/if}
{/if}
{#if idDerivationConfigured && idConfigurationChoice !== "keep"}
<InfoNote>
{translateMessage("The displayed recovery code belongs to the current key. Reopen this dialogue after saving to copy the replacement key.")}
</InfoNote>
{/if}
{#if idConfigurationChoice === "custom" && idCustomChoice === "source"}
<InfoNote>
{translateMessage("Choose a long, unpredictable source. It is used once and cannot be shown again after saving. A recovery code can be displayed on this device later. This input also accepts a tagged recovery code.")}
</InfoNote>
{:else if idConfigurationChoice === "custom" && idCustomChoice === "recovery"}
<InfoNote>
{translateMessage("Paste a tagged recovery code from an existing device to restore the same ID key.")}
</InfoNote>
{:else if idConfigurationChoice === "random"}
<InfoNote warning>
{translateMessage("For recovery after losing every device, save the recovery code after setup or choose an ID source you can reproduce.")}
</InfoNote>
{:else if idConfigurationChoice === "custom" && idCustomChoice === "passphrase"}
<InfoNote warning>
{translateMessage(
"The ID key is derived from the current E2EE passphrase and saved separately. Changing that passphrase later does not change the saved ID key. To reduce the risk of guessing that passphrase from known IDs, use a separate, unpredictable ID source instead."
)}
</InfoNote>
{/if}
{#if idDerivationConfigured && idConfigurationChoice === "custom" && idCustomChoice !== "passphrase"}
<InfoNote>{translateMessage("Leave this input empty to keep the saved ID key.")}</InfoNote>
{/if}
{/if}
<InfoNote error visible={idDerivationError !== ""}>{idDerivationError}</InfoNote>
{/if}
<InputRow label="Encrypt internal file Properties">
<input
type="checkbox"
bind:checked={encryptionSettings.encryptInternalMetadata}
disabled={!canEncryptInternalMetadata}
/>
</InputRow>
<InfoNote>
This option encrypts file properties used by Hidden File Sync and Customisation Sync.
<br />
It applies only to CouchDB and requires End-to-End Encryption, the V2 algorithm, and Property Encryption
(Obfuscate Properties). The remote type is selected later in this setup wizard.
<br />
It protects properties written after the option is enabled; existing properties are not rewritten. A manual remote
Rebuild is strongly recommended to protect existing properties. Update every other synchronising device to a compatible
version before enabling this option, including devices currently running LiveSync.
</InfoNote>
<ExtraItems title={translateMessage("Advanced")}>
<InputRow label={translateMessage("Encryption Algorithm")}>
<select bind:value={encryptionSettings.E2EEAlgorithm} disabled={!encryptionSettings.encrypt}>
@@ -138,4 +424,41 @@
width: auto;
min-width: 8em;
}
.sls-id-choices {
border: 0;
display: flex;
flex-direction: column;
gap: 0.35em;
margin: 0;
min-width: 0;
padding: 0;
}
.sls-id-choices legend {
margin-bottom: 0.35em;
}
.sls-id-choices:disabled {
opacity: 0.6;
}
.sls-id-custom-choices {
margin-left: 1.5em;
}
.sls-id-choice {
align-items: flex-start;
display: flex;
gap: 0.5em;
}
.sls-id-choice input[type="radio"] {
flex: none;
margin-top: 0.25em;
}
.sls-id-choice-text {
display: flex;
flex-direction: column;
}
.sls-current-id-configuration {
color: var(--text-muted);
display: block;
font-size: var(--font-ui-smaller);
margin-top: 0.15em;
}
</style>
src/modules/features/SetupWizard/dialogs/setupDialogTypes.ts
+4
View File
@@ -110,6 +110,10 @@ export type SetupRemoteResultType = typeof TYPE_COUCHDB | typeof TYPE_BUCKET | t
export type UseSetupURIResultType = typeof TYPE_CANCELLED | ObsidianLiveSyncSettings;
export type SetupRemoteE2EEResultType = typeof TYPE_CANCELLED | EncryptionSettings;
export type SetupRemoteE2EEInitialData = {
settings: EncryptionSettings;
newVault: boolean;
};
export type SetupRemoteBucketResultType = typeof TYPE_CANCELLED | BucketSyncSetting;
src/modules/features/StatusBarDisplay.ts
+47
View File
@@ -131,3 +131,50 @@ export function createPaddedCounterLabel(
source.offChanged(update);
});
}
/**
* Displays the disjoint initial and retry chunk-fetch counts with the same
* padding and inactive linger behaviour as the other status counters.
*/
export function createChunkFetchCounterLabel(
source: ReactiveValue<{ initial: number; retrying: number }>
): DisposableReactiveValue<string> {
const initialCount = reactiveSource(0);
const retryingCount = reactiveSource(0);
const initialLabel = createPaddedCounterLabel(initialCount, "🛄");
const retryingLabel = createPaddedCounterLabel(retryingCount, "🔁");
const formatted = reactiveSource(`${initialLabel.value}${retryingLabel.value}`);
let updatingCounts = false;
let disposed = false;
const updateLabel = () => {
if (updatingCounts || disposed) return;
formatted.value = `${initialLabel.value}${retryingLabel.value}`;
};
initialLabel.onChanged(updateLabel);
retryingLabel.onChanged(updateLabel);
const updateCounts = () => {
if (disposed) return;
updatingCounts = true;
try {
initialCount.value = source.value.initial;
retryingCount.value = source.value.retrying;
} finally {
updatingCounts = false;
updateLabel();
}
};
source.onChanged(updateCounts);
updateCounts();
return asDisposableReactiveValue(formatted, () => {
if (disposed) return;
disposed = true;
source.offChanged(updateCounts);
initialLabel.offChanged(updateLabel);
retryingLabel.offChanged(updateLabel);
initialLabel.dispose();
retryingLabel.dispose();
});
}
src/modules/features/StatusBarDisplay.unit.spec.ts
+39
View File
@@ -3,6 +3,7 @@ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import {
STATUS_COUNTER_INACTIVE_LINGER_MS,
createChunkFetchCounterLabel,
createMinimumVisibleActivityCount,
createPaddedCounterLabel,
} from "./StatusBarDisplay.ts";
@@ -137,3 +138,41 @@ describe("createPaddedCounterLabel", () => {
expect(display.value).toBe(" 📄\u20070");
});
});
describe("createChunkFetchCounterLabel", () => {
beforeEach(() => {
vi.useFakeTimers();
});
afterEach(() => {
vi.useRealTimers();
});
it("keeps initial and retry counts separate across an unchanged-total handoff", () => {
const counts = reactiveSource({ initial: 2, retrying: 5 });
const display = createChunkFetchCounterLabel(counts);
const withoutPadding = () => display.value.replace(/\u2007/g, "");
const transitionSnapshots: string[] = [];
const observeTransitions = () => transitionSnapshots.push(withoutPadding());
expect(withoutPadding()).toBe(" 🛄2 🔁5");
display.onChanged(observeTransitions);
counts.value = { initial: 0, retrying: 7 };
expect(withoutPadding()).toBe(" 🛄0 🔁7");
expect(transitionSnapshots).toEqual([" 🛄0 🔁7"]);
display.offChanged(observeTransitions);
vi.advanceTimersByTime(STATUS_COUNTER_INACTIVE_LINGER_MS - 1);
expect(withoutPadding()).toBe(" 🛄0 🔁7");
vi.advanceTimersByTime(1);
expect(withoutPadding()).toBe(" 🔁7");
counts.value = { initial: 0, retrying: 0 };
expect(withoutPadding()).toBe(" 🔁0");
display.dispose();
vi.advanceTimersByTime(STATUS_COUNTER_INACTIVE_LINGER_MS);
counts.value = { initial: 1, retrying: 0 };
expect(withoutPadding()).toBe(" 🔁0");
});
});
src/serviceFeatures/replication/ReplicateResultProcessor.ts
+14 -5
View File
@@ -1,6 +1,7 @@
import { assessRemoteFeatureDocument, describeRemoteFeatureRejection } from "@vrtmrz/livesync-commonlib/replication";
import {
SYNCINFO_ID,
VER,
VERSIONING_DOCID,
type AnyEntry,
type EntryDoc,
type EntryLeaf,
@@ -107,8 +108,15 @@ export class ReplicateResultProcessor {
}
public resume() {
this._suspended = false;
this.continueHeldDocuments();
}
/**
* Continue the queued documents which were held, for example while the application was not ready.
* An explicit suspension, by `suspend()` or by the settings, remains in effect.
*/
public continueHeldDocuments() {
this.updateProcessingActivity();
fireAndForget(() => this.runProcessQueue());
this.triggerProcessQueue();
}
// Whether the processing is suspended
@@ -274,13 +282,14 @@ export class ReplicateResultProcessor {
this.log(`Processed chunk: ${shortenId(change._id)}`, LOG_LEVEL_DEBUG);
return true;
}
if (change.type == "versioninfo") {
if (change._id === VERSIONING_DOCID || change.type === "versioninfo") {
this.log(`Version info document received: ${change._id}`, LOG_LEVEL_VERBOSE);
if (change.version > VER) {
const assessment = assessRemoteFeatureDocument(change);
if (assessment.status !== "supported" && assessment.status !== "older-generation") {
// Fence and retire the active publication through its owner.
this.context.requestActiveReplicatorRetirement();
this.log(
`Remote database updated to incompatible version. update your Self-hosted LiveSync plugin.`,
`${describeRemoteFeatureRejection(assessment)} Update Self-hosted LiveSync before synchronising.`,
LOG_LEVEL_NOTICE
);
}
src/serviceFeatures/replication/ReplicateResultProcessor.unit.spec.ts
+146 -9
View File
@@ -1,7 +1,12 @@
import { promiseWithResolvers } from "octagonal-wheels/promises";
import { reactiveSource } from "octagonal-wheels/dataobject/reactive";
import { describe, expect, it, vi } from "vitest";
import { VER, type EntryDoc, type FilePathWithPrefix } from "@vrtmrz/livesync-commonlib/compat/common/types";
import {
VERSIONING_DOCID,
type EntryDoc,
type FilePathWithPrefix,
} from "@vrtmrz/livesync-commonlib/compat/common/types";
import { ENCRYPTED_INTERNAL_METADATA_FEATURE, REMOTE_FEATURE_GENERATION } from "@vrtmrz/livesync-commonlib/replication";
import {
isValidFilenameInAndroid,
isValidFilenameInWidows,
@@ -37,6 +42,10 @@ type SetupOptions = {
isValidPath?: (path: string) => boolean;
processSynchroniseResult?: (entry: unknown) => Promise<boolean>;
setSnapshot?: (key: string, value: unknown) => Promise<unknown>;
getSnapshot?: (key: string) => Promise<unknown>;
localVersionInfo?: unknown;
isTargetFile?: (path: string) => Promise<boolean>;
databaseId?: string;
};
function setup(options: SetupOptions = {}) {
@@ -47,6 +56,9 @@ function setup(options: SetupOptions = {}) {
const isReady = vi.fn(() => options.applicationReady ?? true);
const isValidPath = vi.fn(options.isValidPath ?? (() => true));
const getDBEntryFromMeta = vi.fn(async (entry: object) => ({ ...entry, data: "x" }));
const localPhysicalDatabase = {
...(options.databaseId ? { id: vi.fn(async () => options.databaseId) } : {}),
} as PouchDB.Database<EntryDoc>;
const core = {
services: {
appLifecycle: { isReady, isSuspended: () => false },
@@ -62,14 +74,21 @@ function setup(options: SetupOptions = {}) {
},
replicator: { onCloseActiveReplication, runBoundedLocalApplicationActivity },
vault: {
isTargetFile: vi.fn(async () => true),
isTargetFile: vi.fn(options.isTargetFile ?? (async () => true)),
isFileSizeTooLarge: vi.fn(() => false),
isValidPath,
},
},
kvDB: { set: setSnapshot },
kvDB: { set: setSnapshot, get: vi.fn(options.getSnapshot ?? (async () => undefined)) },
localDatabase: {
getRaw: vi.fn(async (id: string) => ({ _id: id, _rev: "1-test" })),
localDatabase: localPhysicalDatabase,
getRaw: vi.fn(async (id: string) => {
if (id === VERSIONING_DOCID) {
if (options.localVersionInfo === undefined) throw { status: 404 };
return options.localVersionInfo;
}
return { _id: id, _rev: "1-test" };
}),
getDBEntryFromMeta,
},
};
@@ -88,6 +107,9 @@ function setup(options: SetupOptions = {}) {
} as never);
return {
getDBEntryFromMeta,
isTargetFile: core.services.vault.isTargetFile,
localPhysicalDatabase,
localDatabase: core.localDatabase,
isReady,
isValidPath,
onCloseActiveReplication,
@@ -98,6 +120,34 @@ function setup(options: SetupOptions = {}) {
}
describe("ReplicateResultProcessor", () => {
it("does not add a permanent application block when snapshot recovery fails", async () => {
const { processor } = setup({
getSnapshot: async () => {
throw new Error("KV unavailable");
},
});
await expect(processor.restoreFromSnapshotOnce()).rejects.toThrow("KV unavailable");
expect(processor.isSuspended).toBe(false);
});
it("restores pending notes without retaining a past feature rejection in KV", async () => {
const { processor, processSynchroniseResult, onCloseActiveReplication } = setup({
databaseId: "same-database",
getSnapshot: async () => ({
databaseId: "same-database",
invalidControlObserved: true,
observedFeatures: ["future-format-v7"],
observedGeneration: 14,
queued: [note("recovered-note")],
processing: [],
}),
});
await processor.restoreFromSnapshotOnce();
expect(processor.isSuspended).toBe(false);
await vi.waitFor(() => expect(processSynchroniseResult).toHaveBeenCalledOnce());
expect(onCloseActiveReplication).not.toHaveBeenCalled();
});
it.each([
["Windows", isValidFilenameInWidows],
["Android", isValidFilenameInAndroid],
@@ -145,9 +195,9 @@ describe("ReplicateResultProcessor", () => {
});
}
expect(processSynchroniseResult).toHaveBeenCalledTimes(11);
expect(processSynchroniseResult.mock.calls.some(([entry]) =>
(entry as { _id: string })._id === "unrelated-queue"
)).toBe(true);
expect(
processSynchroniseResult.mock.calls.some(([entry]) => (entry as { _id: string })._id === "unrelated-queue")
).toBe(true);
});
it("suspends result application while the application is not ready", () => {
@@ -157,6 +207,35 @@ describe("ReplicateResultProcessor", () => {
expect(isReady).toHaveBeenCalledOnce();
});
it("applies documents held before readiness once the application becomes ready", async () => {
const { isReady, processor, processSynchroniseResult } = setup({ applicationReady: false });
processor.enqueueAll([note("held")]);
await new Promise((resolve) => setTimeout(resolve, 0));
expect(processSynchroniseResult).not.toHaveBeenCalled();
expect(processor["_queuedChanges"]).toHaveLength(1);
isReady.mockReturnValue(true);
processor.continueHeldDocuments();
await vi.waitFor(() => expect(processSynchroniseResult).toHaveBeenCalledOnce());
expect(processor["_queuedChanges"]).toHaveLength(0);
});
it("keeps an explicit suspension when the application becomes ready", async () => {
const { isReady, processor, processSynchroniseResult } = setup({ applicationReady: false });
processor.suspend();
processor.enqueueAll([note("held")]);
isReady.mockReturnValue(true);
processor.continueHeldDocuments();
await new Promise((resolve) => setTimeout(resolve, 0));
expect(processSynchroniseResult).not.toHaveBeenCalled();
expect(processor["_queuedChanges"]).toHaveLength(1);
processor.resume();
await vi.waitFor(() => expect(processSynchroniseResult).toHaveBeenCalledOnce());
});
it("applies results in remediation mode, which never reports readiness", () => {
const { processor } = setup({
applicationReady: false,
@@ -199,10 +278,10 @@ describe("ReplicateResultProcessor", () => {
it("retires active ownership when a newer remote version is observed", async () => {
const { onCloseActiveReplication, processor } = setup();
const versionInfo = {
_id: "versioninfo",
_id: VERSIONING_DOCID,
_rev: "1-test",
type: "versioninfo",
version: VER + 1,
version: REMOTE_FEATURE_GENERATION + 1,
} as unknown as PouchDB.Core.ExistingDocument<EntryDoc>;
processor.enqueueAll([versionInfo]);
@@ -210,6 +289,64 @@ describe("ReplicateResultProcessor", () => {
await vi.waitFor(() => expect(onCloseActiveReplication).toHaveBeenCalledOnce());
});
it("continues applying documents after restoring a legacy local version document", async () => {
const { onCloseActiveReplication, processor, processSynchroniseResult } = setup({
localVersionInfo: {
_id: VERSIONING_DOCID,
type: "versioninfo",
version: 11,
},
});
await processor.restoreFromSnapshotOnce();
processor.enqueueAll([note("legacy-database-note")]);
await vi.waitFor(() => expect(processSynchroniseResult).toHaveBeenCalledOnce());
expect(processor.isSuspended).toBe(false);
expect(onCloseActiveReplication).not.toHaveBeenCalled();
});
it("continues when a newly received feature is supported", async () => {
const { onCloseActiveReplication, processor, processSynchroniseResult } = setup();
const versionInfo = {
_id: VERSIONING_DOCID,
_rev: "2-supported",
type: "versioninfo",
version: REMOTE_FEATURE_GENERATION,
used_features: [ENCRYPTED_INTERNAL_METADATA_FEATURE],
} as PouchDB.Core.ExistingDocument<EntryDoc>;
processor.enqueueAll([versionInfo, note("supported-update")]);
await vi.waitFor(() => expect(processSynchroniseResult).toHaveBeenCalledOnce());
expect(onCloseActiveReplication).not.toHaveBeenCalled();
});
it("reports unknown feature identifiers and requests Replicator retirement", () => {
const logger = vi.fn();
setGlobalLogFunction(logger);
try {
const { processor, onCloseActiveReplication } = setup();
processor.enqueueAll([
{
_id: VERSIONING_DOCID,
_rev: "1-unknown",
type: "versioninfo",
version: REMOTE_FEATURE_GENERATION,
used_features: ["future-format-v7"],
} as PouchDB.Core.ExistingDocument<EntryDoc>,
]);
expect(onCloseActiveReplication).toHaveBeenCalledOnce();
expect(logger).toHaveBeenCalledWith(
expect.stringContaining("future-format-v7"),
LOG_LEVEL_NOTICE,
undefined
);
} finally {
setGlobalLogFunction(defaultLogger);
}
});
it("scans normal-file metadata without loading chunk documents and requeues it", async () => {
const documents = [
{ _id: "first", _rev: "1-a", type: "plain", path: "first.md" },
src/serviceFeatures/replication/centralCompatibilityRecovery.ts
+21 -1
View File
@@ -1,4 +1,8 @@
import type { ObsidianLiveSyncSettings } from "@vrtmrz/livesync-commonlib/compat/common/types";
import {
VERSIONING_DOCID,
type EntryDoc,
type ObsidianLiveSyncSettings,
} from "@vrtmrz/livesync-commonlib/compat/common/types";
import { assessTweakCompatibility } from "@vrtmrz/livesync-commonlib/settings";
import { LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, Logger } from "octagonal-wheels/common/logger";
import { skipIfDuplicated } from "octagonal-wheels/concurrency/lock";
@@ -7,12 +11,27 @@ import { LiveSyncCouchDBReplicator } from "@vrtmrz/livesync-commonlib/compat/rep
import {
CENTRAL_COMPATIBILITY_REJECTION_REASONS,
REPLICATION_PROGRESS_PRESENTATIONS,
assessRemoteFeatureDocument,
describeRemoteFeatureRejection,
type ReplicatorInstance,
type ReplicationFailureRequest,
} from "@vrtmrz/livesync-commonlib/replication";
import { $msg } from "@/common/translation";
import { usesLegacyIndexedDBAdapter } from "@/common/compatibilitySettings";
import type { LiveSyncBaseCore } from "@/LiveSyncBaseCore";
import type PouchDB from "pouchdb-core";
async function canInterpretCleanupDatabase(db: PouchDB.Database<EntryDoc>): Promise<boolean> {
try {
const assessment = assessRemoteFeatureDocument(await db.get(VERSIONING_DOCID));
if (assessment.status === "supported" || assessment.status === "older-generation") return true;
Logger(`Database cleanup cancelled: ${describeRemoteFeatureRejection(assessment)}`, LOG_LEVEL_NOTICE);
} catch (error) {
Logger("Database cleanup cancelled: feature compatibility could not be checked.", LOG_LEVEL_NOTICE);
Logger(error, LOG_LEVEL_INFO);
}
return false;
}
type CentralCompatibilityRecoveryServices = Pick<
LiveSyncBaseCore["services"],
@@ -60,6 +79,7 @@ export function createCentralCompatibilityRecovery(context: CentralCompatibility
) {
Logger("The remote database has been cleaned.", showProgress ? LOG_LEVEL_NOTICE : LOG_LEVEL_INFO);
await skipIfDuplicated("cleanup", async () => {
if (!(await canInterpretCleanupDatabase(context.getLocalDatabase().localDatabase))) return;
const count = await purgeUnreferencedChunks(context.getLocalDatabase().localDatabase, true);
const message = `The remote database has been cleaned up.
To synchronize, this device must be also cleaned up. ${count} chunk(s) will be erased from this device.
src/serviceFeatures/replication/centralCompatibilityRecovery.unit.spec.ts
+54 -4
View File
@@ -1,5 +1,5 @@
import { describe, expect, it, vi } from "vitest";
import type { ObsidianLiveSyncSettings } from "@vrtmrz/livesync-commonlib/compat/common/types";
import { VERSIONING_DOCID, type ObsidianLiveSyncSettings } from "@vrtmrz/livesync-commonlib/compat/common/types";
import { assessTweakCompatibility } from "@vrtmrz/livesync-commonlib/settings";
import { defaultLogger, LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, setGlobalLogFunction } from "octagonal-wheels/common/logger";
import {
@@ -24,6 +24,49 @@ import { LiveSyncCouchDBReplicator } from "@vrtmrz/livesync-commonlib/compat/rep
import { createCentralCompatibilityRecovery } from "./centralCompatibilityRecovery";
describe("central compatibility recovery", () => {
it("does not count chunks for cleanup when local feature requirements are unknown", async () => {
chunkMocks.purgeUnreferencedChunks.mockClear();
const confirmWithMessage = vi.fn(async () => "Dismiss");
const recovery = createCentralCompatibilityRecovery({
confirm: { confirmWithMessage },
getLocalDatabase: () => ({
localDatabase: {
get: vi.fn(async (id: string) => ({
_id: id,
type: "versioninfo",
version: 13,
used_features: ["future-format-v7"],
})),
},
}),
services: { replicator: {} },
} as never);
await recovery.reconcileCleanedRemote(true, {} as ObsidianLiveSyncSettings, {} as never);
expect(chunkMocks.purgeUnreferencedChunks).not.toHaveBeenCalled();
expect(confirmWithMessage).not.toHaveBeenCalled();
});
it("allows cleanup counting for a legacy local version document", async () => {
chunkMocks.purgeUnreferencedChunks.mockClear();
const confirmWithMessage = vi.fn(async () => "Dismiss");
const recovery = createCentralCompatibilityRecovery({
confirm: { confirmWithMessage },
getLocalDatabase: () => ({
localDatabase: {
get: vi.fn(async (id: string) => ({ _id: id, type: "versioninfo", version: 11 })),
},
}),
services: { replicator: {} },
} as never);
await recovery.reconcileCleanedRemote(true, {} as ObsidianLiveSyncSettings, {} as never);
expect(chunkMocks.purgeUnreferencedChunks).toHaveBeenCalledWith(expect.anything(), true);
expect(confirmWithMessage).toHaveBeenCalledOnce();
});
it("passes the failed attempt's exact tweak assessment to mismatch resolution", async () => {
const setting = { customChunkSize: 0 };
const preferredTweakValue = { customChunkSize: 60 };
@@ -292,7 +335,9 @@ describe("central compatibility recovery", () => {
});
const runFiniteReplicationActivity = vi.fn(async (task: () => unknown) => await task());
const openOneShotReplication = vi.fn(async () => true);
const remoteDatabase = { close: vi.fn(async () => undefined) };
const remoteDatabase = {
close: vi.fn(async () => undefined),
};
const close = vi.fn(async () => undefined);
const activeReplicator = Object.assign(new LiveSyncCouchDBReplicator({} as never), {
connectRemoteCouchDBWithSetting: vi.fn(async () => ({ db: remoteDatabase, close })),
@@ -303,7 +348,12 @@ describe("central compatibility recovery", () => {
const runWithActiveReplicatorContext = vi.fn(async (task: (context: unknown) => unknown) =>
task(expectedContext)
);
const localDatabase = { localDatabase: {}, clearCaches: vi.fn() };
const localDatabase = {
localDatabase: {
get: vi.fn(async () => ({ _id: VERSIONING_DOCID, type: "versioninfo", version: 12 })),
},
clearCaches: vi.fn(),
};
const getLocalDatabase = vi.fn(() => localDatabase);
const recovery = createCentralCompatibilityRecovery({
confirm: { confirmWithMessage: vi.fn(async () => "Cleanup") },
@@ -335,7 +385,7 @@ describe("central compatibility recovery", () => {
activityFinished.mock.invocationCallOrder[0]
);
expect(chunkMocks.balanceChunkPurgedDBs).toHaveBeenCalledOnce();
expect(getLocalDatabase).toHaveBeenCalledTimes(2);
expect(getLocalDatabase).toHaveBeenCalled();
expect(close).toHaveBeenCalledOnce();
expect(close.mock.invocationCallOrder[0]).toBeLessThan(activityFinished.mock.invocationCallOrder[0]);
});
src/serviceFeatures/replication/index.ts
+4
View File
@@ -4,6 +4,7 @@ import { UnresolvedErrorManager } from "@vrtmrz/livesync-commonlib/compat/servic
import type { ServiceContext } from "@vrtmrz/livesync-commonlib/context";
import { fireAndForget } from "octagonal-wheels/promises";
import type { IMinimumLiveSyncCommands, LiveSyncBaseCore } from "@/LiveSyncBaseCore";
import { EVENT_APPLICATION_READY } from "@/common/events";
import { createAutomaticReplicationTriggers } from "./automaticTriggers";
import { createCentralCompatibilityRecovery } from "./centralCompatibilityRecovery";
import { createOnlineReplicationPreflight, createSecuritySeedPreflight } from "./preflight";
@@ -102,6 +103,9 @@ export function useReplicationFeature<TContext extends ServiceContext, TCommands
fireAndForget(() => resultProcessor.restoreFromSnapshotOnce());
return Promise.resolve(true);
});
// Commonlib emits this each time it establishes readiness. Documents held until then, such as those restored
// from the snapshot or received during a fetch, continue from here.
services.context.events.onEvent(EVENT_APPLICATION_READY, () => resultProcessor.continueHeldDocuments());
services.appLifecycle.onSettingLoaded.addHandler(initialiseAutomaticReplicationTriggers);
services.replication.parseSynchroniseResult.addHandler((documents) => {
resultProcessor.enqueueAll(documents);
src/serviceFeatures/replication/receivedChangeReadiness.unit.spec.ts
+171
View File
@@ -0,0 +1,171 @@
import { createServiceContext } from "@vrtmrz/livesync-commonlib/context";
import { VERSIONING_DOCID, type EntryDoc } from "@vrtmrz/livesync-commonlib/compat/common/types";
import { EVENT_SETTING_SAVED } from "@vrtmrz/livesync-commonlib/compat/events/coreEvents";
import { describe, expect, it, vi } from "vitest";
import { EVENT_APPLICATION_READY, eventHub } from "@/common/events";
import { useReplicationFeature } from "./index";
type ParseHandler = (documents: PouchDB.Core.ExistingDocument<EntryDoc>[]) => Promise<boolean>;
function receivedNote(id: string): PouchDB.Core.ExistingDocument<EntryDoc> {
return {
_id: id,
_rev: "1-received",
path: `${id}.md`,
ctime: 1,
mtime: 2,
size: 1,
children: [],
datatype: "plain",
type: "plain",
eden: {},
} as unknown as PouchDB.Core.ExistingDocument<EntryDoc>;
}
function setup() {
let applicationReady = false;
const settings = {
handleFilenameCaseSensitive: false,
ignoreFiles: "",
maxMTimeForReflectEvents: 0,
suspendParseReplicationResult: false,
syncIgnoreRegEx: "",
syncInternalFiles: false,
syncMaxSizeInMB: 0,
syncOnlyRegEx: "",
useIgnoreFiles: false,
};
const processSynchroniseResult = vi.fn(async () => true);
const settingLoadedHandlers: (() => Promise<boolean>)[] = [];
const context = createServiceContext();
const keyValueDB = {
get: vi.fn(async () => undefined),
set: vi.fn(async () => undefined),
};
const localDatabase = {
getRaw: vi.fn(async (id: string) => {
if (id === VERSIONING_DOCID) throw { status: 404 };
return { _id: id, _rev: "1-received" };
}),
getDBEntryFromMeta: vi.fn(async (entry: object) => ({ ...entry, data: "received content" })),
};
let parseHandler: ParseHandler | undefined;
const services = {
API: { isMobile: vi.fn(() => false), isOnline: true },
appLifecycle: {
getUnresolvedMessages: { addHandler: vi.fn() },
isReady: () => applicationReady,
isSuspended: vi.fn(() => false),
onSettingLoaded: {
addHandler: vi.fn((handler: () => Promise<boolean>) => settingLoadedHandlers.push(handler)),
},
},
context,
database: { isDatabaseReady: vi.fn(() => true) },
databaseEvents: { onDatabaseInitialised: { addHandler: vi.fn() } },
keyValueDB: { kvDB: keyValueDB },
localDatabase,
path: { getPath: vi.fn((entry: { path: string }) => entry.path) },
replication: {
databaseQueueCount: { value: 0 },
storageApplyingCount: { value: 0 },
replicationResultCount: { value: 0 },
onBeforeReplicate: { addHandler: vi.fn() },
onPrepareCentralRemoteReplication: { addHandler: vi.fn() },
onReplicationFailed: { addHandler: vi.fn() },
parseSynchroniseResult: {
addHandler: vi.fn((handler: ParseHandler) => {
parseHandler = handler;
}),
},
processOptionalSynchroniseResult: vi.fn(async () => false),
processSynchroniseResult,
processVirtualDocument: vi.fn(async () => false),
replicateUnattendedByEvent: vi.fn(async () => ({ status: "completed" as const })),
},
replicator: {
createRemoteResource: vi.fn(async () => ({
read: vi.fn(async () => new Uint8Array([1])),
dispose: vi.fn(),
})),
onBeforeReplicatorPublication: { addHandler: vi.fn() },
onCloseActiveReplication: vi.fn(async () => true),
},
setting: { currentSettings: vi.fn(() => settings) },
tweakValue: {},
vault: {
isFileSizeTooLarge: vi.fn(() => false),
isTargetFile: vi.fn(async () => true),
isValidPath: vi.fn(() => true),
},
};
const core = {
confirm: {},
get localDatabase() {
return localDatabase;
},
rebuilder: {},
services,
};
useReplicationFeature(core as never);
return {
context,
get applicationReady() {
return applicationReady;
},
processSynchroniseResult,
get parseHandler() {
return parseHandler;
},
settingLoadedHandlers,
setApplicationReady(value: boolean) {
applicationReady = value;
},
settings,
};
}
describe("received change readiness composition", () => {
it("applies a queued received document once readiness is established and preserves explicit suspension", async () => {
eventHub.offAll();
const harness = setup();
try {
await harness.settingLoadedHandlers[0]();
await harness.parseHandler!([receivedNote("ready-note")]);
await new Promise((resolve) => setTimeout(resolve, 0));
expect(harness.processSynchroniseResult).not.toHaveBeenCalled();
harness.setApplicationReady(true);
harness.context.events.emitEvent(EVENT_APPLICATION_READY);
harness.context.events.emitEvent(EVENT_APPLICATION_READY);
await vi.waitFor(() => expect(harness.processSynchroniseResult).toHaveBeenCalledTimes(1));
harness.settings.suspendParseReplicationResult = true;
eventHub.emitEvent(EVENT_SETTING_SAVED, harness.settings as never);
harness.setApplicationReady(false);
await harness.parseHandler!([receivedNote("suspended-note")]);
await new Promise((resolve) => setTimeout(resolve, 0));
expect(harness.processSynchroniseResult).toHaveBeenCalledTimes(1);
harness.setApplicationReady(true);
harness.context.events.emitEvent(EVENT_APPLICATION_READY);
harness.context.events.emitEvent(EVENT_APPLICATION_READY);
await new Promise((resolve) => setTimeout(resolve, 0));
expect(harness.processSynchroniseResult).toHaveBeenCalledTimes(1);
harness.settings.suspendParseReplicationResult = false;
eventHub.emitEvent(EVENT_SETTING_SAVED, harness.settings as never);
await vi.waitFor(() => expect(harness.processSynchroniseResult).toHaveBeenCalledTimes(2));
expect(harness.processSynchroniseResult).toHaveBeenLastCalledWith(
expect.objectContaining({ _id: "suspended-note", path: "suspended-note.md" })
);
} finally {
eventHub.offAll();
}
});
});
src/serviceFeatures/replication/replicationFeature.unit.spec.ts
+32 -5
View File
@@ -1,8 +1,11 @@
import { describe, expect, it, vi } from "vitest";
import { createServiceContext } from "@vrtmrz/livesync-commonlib/context";
import { VER, type EntryDoc } from "@vrtmrz/livesync-commonlib/compat/common/types";
import { VERSIONING_DOCID, type EntryDoc } from "@vrtmrz/livesync-commonlib/compat/common/types";
import { REMOTE_FEATURE_GENERATION } from "@vrtmrz/livesync-commonlib/replication";
import { promiseWithResolvers } from "octagonal-wheels/promises";
import { EVENT_APPLICATION_READY } from "@/common/events";
import { useReplicationFeature } from "./index";
import { ReplicateResultProcessor } from "./ReplicateResultProcessor";
type BooleanHandler = (showMessage: boolean) => Promise<boolean>;
type ParseHandler = (documents: PouchDB.Core.ExistingDocument<EntryDoc>[]) => Promise<boolean>;
@@ -19,8 +22,14 @@ type SetupOptions = {
};
function setup(options: SetupOptions = {}) {
const defaultLocalDatabase = {
localDatabase: {},
getRaw: vi.fn(async () => {
throw { status: 404 };
}),
};
const {
getLocalDatabase = () => ({}),
getLocalDatabase = () => defaultLocalDatabase,
keyValueDB = {
kvDB: {
get: vi.fn(async () => undefined),
@@ -39,7 +48,7 @@ function setup(options: SetupOptions = {}) {
API: { isMobile: vi.fn(() => false), isOnline: true },
appLifecycle: {
getUnresolvedMessages: { addHandler: vi.fn() },
isReady: true,
isReady: vi.fn(() => true),
isSuspended: vi.fn(() => false),
onSettingLoaded: { addHandler: vi.fn() },
},
@@ -48,6 +57,7 @@ function setup(options: SetupOptions = {}) {
keyValueDB,
path: { getPath: vi.fn((entry: { path: string }) => entry.path) },
replication: {
replicationResultCount: { value: 0 },
onBeforeReplicate: {
addHandler: vi.fn((handler: BooleanHandler, priority = 0) => {
beforeReplicateHandlers.set(priority, handler);
@@ -87,6 +97,7 @@ function setup(options: SetupOptions = {}) {
return {
beforeReplicateHandlers,
centralRemoteHandlers,
context: services.context,
createRemoteResource,
dispose,
get parseHandler() {
@@ -160,15 +171,31 @@ describe("replication serviceFeature composition", () => {
expect(createRemoteResource).toHaveBeenCalledOnce();
});
it("continues held results when Commonlib establishes application readiness", () => {
const continueHeldDocuments = vi
.spyOn(ReplicateResultProcessor.prototype, "continueHeldDocuments")
.mockImplementation(() => undefined);
try {
const { context } = setup();
expect(continueHeldDocuments).not.toHaveBeenCalled();
context.events.emitEvent(EVENT_APPLICATION_READY);
expect(continueHeldDocuments).toHaveBeenCalledOnce();
} finally {
continueHeldDocuments.mockRestore();
}
});
it("requests owner retirement without awaiting the transition from result application", async () => {
const retirement = promiseWithResolvers<boolean>();
const onCloseActiveReplication = vi.fn(() => retirement.promise);
const harness = setup({ onCloseActiveReplication });
const versionInfo = {
_id: "versioninfo",
_id: VERSIONING_DOCID,
_rev: "1-test",
type: "versioninfo",
version: VER + 1,
version: REMOTE_FEATURE_GENERATION + 1,
} as unknown as PouchDB.Core.ExistingDocument<EntryDoc>;
expect(harness.parseHandler).toBeDefined();
src/serviceFeatures/useReviewHarness.ts
+4
View File
@@ -15,6 +15,8 @@ import {
runReviewHarnessVaultRoundTrip,
} from "@/features/ReviewHarness/reviewHarnessVaultFixture";
import type { CompatibilityReviewController } from "./compatibilityReview";
import { runReviewHarnessIdBenchmark } from "@/features/ReviewHarness/reviewHarnessIdBenchmark";
import { createIdBenchmarkOperations } from "@/features/ReviewHarness/reviewHarnessIdBenchmarkRuntime";
async function runVaultRoundTrip(plugin: ObsidianLiveSyncPlugin): Promise<ReviewHarnessScenarioResult> {
const vault = plugin.app.vault;
@@ -58,6 +60,8 @@ export function useReviewHarness(
getCompatibilityPause: () => compatibilityReview.pendingPause,
openCompatibilityReview: () => compatibilityReview.openReview(),
runVaultRoundTrip: () => runVaultRoundTrip(plugin),
runIdBenchmark: async () =>
runReviewHarnessIdBenchmark(await createIdBenchmarkOperations(), activeWindow.performance),
readContinuation: () => services.setting.getSmallConfig(REVIEW_HARNESS_STATE_KEY),
writeContinuation: (value) => services.setting.setSmallConfig(REVIEW_HARNESS_STATE_KEY, value),
deleteContinuation: () => services.setting.deleteSmallConfig(REVIEW_HARNESS_STATE_KEY),
test/e2e-obsidian/README.md
+27 -2
View File
@@ -129,6 +129,8 @@ The mobile pass uses Obsidian's `app.emulateMobile(true)`, a 390 by 844 CSS-pixe
`test:e2e:obsidian:review-harness` exercises only the boundaries owned by the opt-in maintainer Harness. It retains a real compatibility pause, uses the fixed Harness restart action to persist a device-local continuation and reload Obsidian, and requires the Harness to delete that state before reopening. It also runs the bounded settings-lifecycle observation, confirms the dedicated Vault fixture root is removed, captures the copied privacy-bounded Markdown report, and checks the Harness layout and touch targets in mobile test mode. Compatibility explanation and persistence details remain owned by `settings-ui`, real P2P transfer remains owned by the dedicated P2P suites, and general Vault reflection remains owned by `vault-reflection`; the Harness test does not duplicate those workflows.
The Harness also measures ID generation with fixed in-memory data on desktop and mobile displays, checks that live settings remain unchanged, and verifies that the copied report includes both per-1,000-ID and per-ID timings, key derivation, and JavaScript heap availability. Mobile test mode verifies the UI and execution path; measure native device performance by running the same Harness on that device.
`test:e2e:obsidian:p2p-pane` starts one configured CouchDB-only session with no P2P profile and separate configured P2P sessions for desktop and mobile. It proves that the command remains registered while the retired command, automatic pane, and ribbon entry without a P2P configuration are absent. For the configured P2P profiles, it verifies that the desktop ribbon is available, the current status command reaches the pane without it opening at start-up, checks its connection control and horizontal layout, and captures unobstructed desktop and mobile screenshots. The mobile session uses a fresh Vault, profile, and Obsidian process, enters `app.emulateMobile(true)` through `lifecycle.beforePluginStart`, and requires the P2P view to belong to the right drawer rather than inheriting desktop workspace state. It deliberately uses no relay or peer: replacement of the active replicator is covered by focused unit tests, the Deno and Compose CLI P2P lifecycle suite covers the headless transport, and `p2p-setup-uri-workflow` owns the visible transfer path between two real Obsidian sessions.
`test:e2e:obsidian:local-suite` builds the plug-in and, unless `LIVESYNC_CLI_COMMAND` selects an external CLI, the local LiveSync CLI. It then runs discovery, smoke, the onboarding invitation, Svelte dialogue mounting, revision repair, settings UI, the Review Harness, the P2P status pane, Vault reflection, CouchDB upload and manual setup, CLI-to-Obsidian synchronisation, Object Storage upload and Setup URI round-trip, P2P Setup URI round-trip, startup scan, provisioned CouchDB Setup URI, two-vault synchronisation, Hidden File Sync, Customisation Sync, and setting Markdown export in sequence. Start the local CouchDB, RustFS, and P2P relay fixtures before running it, or use `test:e2e:obsidian:local-suite:services` to let the wrapper stop leftover fixtures, start fresh fixtures, and stop them again after the run.
@@ -137,12 +139,26 @@ The mobile pass uses Obsidian's `app.emulateMobile(true)`, a 390 by 844 CSS-pixe
The same workflow checks the two remote-activity status boundaries. It first holds a real CouchDB request at the selected fetch implementation and confirms that `🌐N` is visible while `📲` is absent. It then holds the real one-shot replication immediately before its replicator call, confirms that `📲` is visible while no physical request is active, releases it, and requires the finite and bounded activity counts to return to zero, the request and response counts to balance, and both indicators to disappear. Finally, it creates a remote-only chunk, holds the real on-demand fetch immediately before its remote call, makes the same logical active and idle assertions, and verifies that the fetched chunk is written into the local database. These gates make the active states deterministic without replacing the remote request or operation.
`test:e2e:obsidian:couchdb-manual-setup-workflow` follows the visible onboarding path for the first device when no Setup URI is available. It enters end-to-end encryption and CouchDB details, runs the read-only `Check server requirements` step, requires the prepared fixture to pass without applying a server fix, and lets the onboarding connection test create the named database. After Rebuild completes on the first device, it creates an ordinary note, asks that working device to generate a Setup URI for a second device, completes Fetch there, and verifies a bidirectional note round-trip. The workflow captures each decision point and the expanded server-check result; password controls remain visually masked.
`npm run test:e2e:obsidian:focused -- chunk-fetch-retry` checks delayed Chunk availability through a real CouchDB service and Obsidian. It creates a Metadata-only remote fixture, starts ordinary one-shot replication with `readChunksOnline`, and inserts the missing Chunk only after a real fetch has returned an empty result. A pass-through observer records the replicator's call times and results without substituting responses or adding waits. The fixture sets the existing minimum request interval to 500 ms to keep the real retry status observable even if finite completion expedites the final probe. The actual status bar must show zero initial requests (`🛄`) and one retry (`🔁`), and both counts must return to zero after delivery ends. On-demand replication excludes Chunk documents from the ordinary pull, so the delayed Chunk must arrive through the observed fetch and produce the exact Vault content.
If finite replication was already inactive when the initial lookup began, the scenario requires a retry at least two seconds later and no physical request slot occupied during backoff. If finite replication ends during the initial lookup or the following backoff, the retry must instead be a post-completion final probe before the two-second delay would expire. This distinction is determined from the observed finite-count transitions, not an assumed ordering between replication and HTTP completion.
A second case never inserts the Chunk: it requires exactly one retry, a terminal missing notification, released activity, no Vault file, and no further fetch during another retry interval. A third starts a second genuine one-shot replication while the retry remains pending and passively observes its finite count. The final missing lookup must start after that finite operation ends and before the original backoff would expire. No test code changes the finite count. Deterministic Commonlib tests cover longer backoff stages and overlapping-completion races.
Each run uses an isolated Vault and remote database. This is a controlled availability-ordering reproduction, not a reproduction of a particular server's underlying delay or of mobile suspension. When comparing separately built pre-fix and fixed artefacts, retain the exact scenario, package, and bundle revisions: the original implementation ends delivery after the first missing response, while the interim single-retry implementation lacks the split status counts and finite-completion scheduling.
`test:e2e:obsidian:couchdb-manual-setup-workflow` follows the visible onboarding path for the first device when no Setup URI is available. It enters end-to-end encryption and CouchDB details, runs the read-only `Check server requirements` step, requires the prepared fixture to pass without applying a server fix, and lets the onboarding connection test create the named database. After Rebuild completes on the first device, it creates an ordinary note, asks that working device to generate a Setup URI for a second device, completes Fetch there, and verifies a bidirectional note round-trip. The workflow captures each decision point and the expanded server-check result; password controls remain visually masked. It uses an E2EE passphrase beginning with `%`, confirms that the saved settings do not contain it in plain text, and checks that Obsidian restores it after restarting with the first Vault.
The ordinary workflow now checks that all three ID-configuration radio choices are visible, disabled and dimmed while E2EE is off, and fully visible when it is enabled. It also checks that the random key is selected by default for a new Vault, **Keep current configuration** shows its legacy explanation, and the saved key is encrypted locally and transferred by Setup URI. A screenshot of the disabled group is saved as `guide-couchdb-manual-id-generation-disabled.png`. Set `E2E_OBSIDIAN_INDEPENDENT_IDS=true` for the same visible workflow with an explicitly entered, randomly generated source. That variant checks all three nested radio choices, requires a source when no key is saved, retains the saved key when a custom source is empty, rejects an ordinary string in the recovery-code input, restores the same key from a tagged code, verifies that the source is absent from local settings, and checks that both devices compute the same obfuscated document IDs after Setup URI import and Fast Fetch.
If this status workflow fails while Obsidian is running, it writes a full-page screenshot and a JSON snapshot of the status text and counters under `/tmp/obsidian-livesync-e2e`. The dialogue-mount workflow leaves desktop and mobile screenshots for both representative Svelte routes, the Hidden File Sync workflow captures the successfully displayed JSON Resolve dialogue before selecting an option, and the Security Seed reconnect workflow captures each significant application state. The suite therefore records representative evidence without capturing every interaction. Set `E2E_OBSIDIAN_DIAGNOSTICS_DIR` to use another directory.
The two-Vault workflow performs the missing-marker review once for each isolated Vault. Later process launches reuse the same profile-backed acknowledgement, rather than seeding a replacement or repeatedly applying a decision for the first device. The Hidden File Sync scenario is narrower: it starts from an explicitly acknowledged marker because it tests consumer-owned hidden-file behaviour, JSON resolution, target filtering, and grouped mobile Notices rather than duplicating the compatibility workflow. After `app.emulateMobile(true)`, its fixture operations use the active DevTools renderer because Obsidian can remove desktop-only CLI commands in mobile mode.
The two-Vault workflow also covers independent ID derivation with two real Obsidian sessions: a note travels in each direction, both devices retain the same obfuscated document IDs, and identical content reuses the same Chunk IDs. Fresh devices with a different ID key or legacy ID configuration must be rejected by ordinary CouchDB replication before any remote document or checkpoint changes. Set `E2E_OBSIDIAN_ONLY_INDEPENDENT_IDS=true` to run that case without the other two-Vault scenarios.
Set `E2E_OBSIDIAN_ONLY_DIFFERENT_CHUNK_ID_KEYS=true` to run the focused case where two devices use different saved ID keys with Path Obfuscation off. It verifies that each device can read the other's note, visible document IDs agree, and writing the same content produces different Chunk IDs.
`test:e2e:obsidian:cli-to-obsidian-sync` is the cross-runtime compatibility check for the official LiveSync CLI and the real Obsidian plug-in. Build the plug-in first, and build the local CLI too when no external CLI command is selected. The script uses E2EE, Path Obfuscation, and the current preferred chunk settings to create and synchronise a note through the CLI, starts real Obsidian with an isolated Vault and profile, synchronises the same CouchDB database, and verifies that the plug-in materialises identical note content. This covers the boundary that CLI-only and plug-in-only round trips do not exercise.
The isolated Obsidian session starts with its CouchDB settings and device-local compatibility acknowledgement already in place. This keeps the scenario focused on cross-runtime data compatibility; unconfigured start-up and visible CouchDB onboarding are covered by their dedicated workflows.
@@ -166,6 +182,9 @@ LIVESYNC_CLI_COMMAND="docker run --rm --network host --user $(id -u):$(id -g) --
`test:e2e:obsidian:minio-upload` reuses the Object Storage variables from `.test.env` or the process environment. It expects a reachable S3-compatible service and starts with isolated Object Storage settings and the device-local compatibility acknowledgement already in place, keeping the scenario focused on upload rather than unconfigured start-up or setup. It confirms those settings through `obsidian-cli eval`, creates a note in real Obsidian, runs one-shot Journal Sync, and verifies through the AWS SDK that objects were written under a unique bucket prefix. Adapter tests separately observe an in-progress SDK command, while this real-runtime workflow verifies the resulting request counters advance and rebalance.
Set `E2E_OBSIDIAN_INDEPENDENT_IDS=true` to run the same upload with E2EE, Path Obfuscation, and a separately derived ID key. The scenario verifies the local document and Chunk ID shapes before the Journal transfer.
Set `E2E_OBSIDIAN_CUSTOM_HTTP_HANDLER=true` when the local Object Storage fixture does not allow browser requests from Obsidian's renderer.
`test:e2e:obsidian:object-storage-setup-uri-workflow` uses the public Commonlib-backed tool to generate the initial Setup URI for a unique Object Storage prefix, completes visible initialisation on the first device, and then asks that working real Obsidian device to create a new Setup URI through the registered command. A second real Obsidian device imports only the device-generated URI. The workflow verifies the A-to-B note through explicit replication, then verifies that the B-to-A note arrives through `syncOnStart` after restarting the first device, without requesting manual replication. It captures the documented onboarding choices, and removes the Object Storage prefix only after both sessions have stopped.
`test:e2e:obsidian:p2p-setup-uri-workflow` runs two concurrent isolated real Obsidian sessions against the local Compose Nostr relay fixture. The first device imports a generated initial Setup URI and completes its signalling test with zero peers, creates a Setup URI for the second device through the registered command, and remains online while the second device imports it. The second device must select the expected online source before Fetch can rebuild its local database. The workflow accepts each connection request visibly on the receiving device, verifies the initial A-to-B fetch, checks that the menu for the three persistent per-peer actions remains within the viewport, reconnects both P2P sessions in join order, and verifies the B-to-A return journey. Every started session remains tracked until teardown completes.
@@ -200,7 +219,13 @@ This proves in real Obsidian the plug-in behaviour shared by supported platforms
`test:e2e:obsidian:customisation-sync` runs a two-vault Customisation Sync workflow. It scans a real snippet CSS file, config JSON file, and sample plug-in fixture into per-file Customisation Sync data, synchronises the entries through CouchDB, applies them on the second vault, verifies the resulting `.obsidian` files, propagates a snippet update, and verifies deletion of the source-vault snippet sync data without confusing it with the target vault's own applied copy.
`test:e2e:obsidian:setting-markdown-export` enables setting Markdown export, waits for the generated Markdown file in the vault, and verifies that credentials are omitted when `writeCredentialsForSettingSync=false`.
`test:e2e:obsidian:received-change-readiness` uses two sequential real Obsidian sessions and isolated CouchDB databases. The source creates ordinary notes and their Chunks; the target starts continuous replication, resets readiness through the public lifecycle service, and receives those documents while its Vault files remain absent. Marking the target ready twice must emit one readiness event and reflect the queued content. A second note remains absent after readiness while database reflecting is explicitly suspended, then appears when that setting is resumed. This focused scenario is outside `test:e2e:obsidian:local-suite`.
`test:e2e:obsidian:remote-feature-change` starts real Obsidian with continuous CouchDB replication, then changes the remote version document from generation 12 to generation 13 with an unknown feature. It waits for the control document to reach the local database and the active Replicator to retire, checks that another replication is refused, and verifies that an already accepted Vault note remains intact. After restarting the same Vault, the current remote declaration still blocks finite replication and the actual continuous connection attempt. No KV feature history is involved. It is a focused test outside `test:e2e:obsidian:local-suite`; recovery with a future compatible client remains a separate validation boundary.
`test:e2e:obsidian:internal-metadata-migration` enables internal Metadata encryption through the settings UI without Rebuild. It checks unchanged plaintext and rewritten encrypted Hidden File Sync and Customisation Sync Metadata in CouchDB, stable document IDs, mismatch rejection on a second device, and file restoration after aligning settings. It then turns the preference OFF, runs Fast Fetch, and compares content loaded from both Metadata representations and their Chunks while retaining the remote feature declaration. These focused tests use the local CouchDB fixture and are outside `test:e2e:obsidian:local-suite`.
`test:e2e:obsidian:setting-markdown-export` enables setting Markdown export, waits for the generated Markdown file in the vault, and verifies that credentials are omitted when `writeCredentialsForSettingSync=false`, including both the plaintext ID key and its encrypted local representation.
`test:e2e:obsidian:upgrade-from-stable` is the release-acceptance upgrade workflow. It installs the exact published 0.25.83 artefacts into an isolated Vault, verifies their pinned SHA-256 values, and then replaces only the plug-in artefacts with the current target while retaining the same Vault and isolated Obsidian profile. The first run downloads the old release into the ignored `_testdata/releases` cache; every later run verifies the cached bytes before use.
test/e2e-obsidian/scripts/chunk-fetch-retry.ts
+365
View File
@@ -0,0 +1,365 @@
import { mkdir } from "node:fs/promises";
import { join } from "node:path";
import type { Page } from "playwright";
import {
assertCouchDbReachable,
createCouchDbDatabase,
deleteCouchDbDatabase,
loadCouchDbConfig,
makeUniqueDatabaseName,
putCouchDbDocument,
type CouchDbConfig,
} from "../runner/couchdb.ts";
import { discoverObsidianCli, requireObsidianBinary } from "../runner/environment.ts";
import {
assertEqual,
createE2eCouchDbPluginData,
createE2eObsidianDeviceLocalState,
prepareRemote,
waitForLiveSyncCoreReady,
} from "../runner/liveSyncWorkflow.ts";
import { startObsidianLiveSyncSession, type ObsidianLiveSyncSession } from "../runner/session.ts";
import { withObsidianPage } from "../runner/ui.ts";
import { createTemporaryVault } from "../runner/vault.ts";
const observationKey = "__livesyncChunkFetchRetryE2E";
const observationSource = `globalThis[${JSON.stringify(observationKey)}]`;
const retryDelayMs = 2_000;
type FetchAttempt = {
startedAt: number;
finiteTransitionIndex: number;
completedAt?: number;
requestedIds: string[];
returnedIds?: string[];
unavailable?: boolean;
error?: string;
};
type Snapshot = {
attempts: FetchAttempt[];
missingEvents: number[];
replicationDone: boolean;
replicationSucceeded?: boolean;
replicationError?: string;
metadataPresent: boolean;
chunkPresent: boolean;
claimActive: boolean;
currentProcessing: number;
queued: number;
boundedActivity: number;
finiteActivity: number;
finiteTransitions: { at: number; count: number }[];
followupTransitionIndex?: number;
replicationResults: number;
databaseQueue: number;
storageApplying: number;
pendingChunkCount: number;
initialChunkCount: number;
retryChunkCount: number;
statusText: string;
content: string | null;
};
async function snapshot(page: Page): Promise<Snapshot> {
return await page.evaluate(`(async()=>{
const state=${observationSource};
const core=app.plugins.plugins['obsidian-livesync'].core;
const db=core.localDatabase;
const rows=await db.allDocsRaw({keys:[state.metadataId,state.chunkId],include_docs:true});
const present=(id)=>rows.rows.some((row)=>row.id===id&&row.doc&&!row.value?.deleted);
const file=app.vault.getAbstractFileByPath(state.path);
const statusText=document.querySelector('.syncstatusbar')?.textContent??'';
const initialChunkCount=Number(statusText.match(/🛄\\s*(\\d+)/u)?.[1]??0);
const retryChunkCount=Number(statusText.match(/🔁\\s*(\\d+)/u)?.[1]??0);
return {
attempts:state.attempts,missingEvents:state.missingEvents,
replicationDone:state.replicationDone,replicationSucceeded:state.replicationSucceeded,
replicationError:state.replicationError,metadataPresent:present(state.metadataId),
chunkPresent:present(state.chunkId),
claimActive:db.managers.chunkManager.deliveryCoordinator.isClaimActiveFor(state.chunkId),
currentProcessing:db.managers.chunkFetcher.currentProcessing,
queued:db.managers.chunkFetcher.queue.length,
boundedActivity:core.services.replicator.boundedRemoteActivityCount.value,
finiteActivity:core.services.replicator.finiteReplicationActivityCount.value,
finiteTransitions:state.finiteTransitions,followupTransitionIndex:state.followupTransitionIndex,
replicationResults:core.services.replication.replicationResultCount.value,
databaseQueue:core.services.replication.databaseQueueCount.value,
storageApplying:core.services.replication.storageApplyingCount.value,
initialChunkCount,retryChunkCount,pendingChunkCount:initialChunkCount+retryChunkCount,statusText,
content:file?await app.vault.read(file):null,
};
})()`);
}
async function waitForSnapshot(
page: Page,
predicate: (state: Snapshot) => boolean,
stage: string,
timeoutMs = 20_000
): Promise<Snapshot> {
const deadline = Date.now() + timeoutMs;
let state: Snapshot;
do {
state = await snapshot(page);
if (predicate(state)) return state;
if (state.replicationError || state.attempts.some((attempt) => attempt.error || attempt.unavailable)) {
throw new Error(`The real CouchDB operation failed during ${stage}: ${JSON.stringify(state)}`);
}
await new Promise((resolve) => setTimeout(resolve, 50));
} while (Date.now() < deadline);
throw new Error(`Timed out during ${stage}: ${JSON.stringify(state)}`);
}
function isIdle(state: Snapshot): boolean {
return (
state.replicationDone &&
!state.claimActive &&
state.currentProcessing === 0 &&
state.queued === 0 &&
state.boundedActivity === 0 &&
state.finiteActivity === 0 &&
state.replicationResults === 0 &&
state.databaseQueue === 0 &&
state.storageApplying === 0 &&
state.pendingChunkCount === 0
);
}
async function runScenario(
page: Page,
couchDb: CouchDbConfig,
dbName: string,
label: "delayed-arrival" | "permanently-missing" | "finite-completion"
): Promise<void> {
const delayedArrival = label === "delayed-arrival";
const expediteFinalProbe = label === "finite-completion";
const path = `chunk-fetch-${label}.md`;
const chunkId = `h:e2e-chunk-fetch-${label}`;
const content = `# Chunk fetch retry\n${label}\n`;
const metadataId = await page.evaluate<string>(
`app.plugins.plugins['obsidian-livesync'].core.services.path.path2id(${JSON.stringify(path)})`
);
const now = Date.now();
// Only Metadata is initially present. The Chunk is a separate real CouchDB document.
await putCouchDbDocument(couchDb, dbName, {
_id: metadataId,
path,
type: "plain",
ctime: now,
mtime: now,
size: Buffer.byteLength(content),
children: [chunkId],
eden: {},
});
await page.evaluate(`(()=>{
if(${observationSource}) throw new Error('A Chunk fetch observer is already installed.');
const core=app.plugins.plugins['obsidian-livesync'].core;
const settings=core.services.setting.currentSettings();
if(!settings.readChunksOnline||settings.useOnlyLocalChunk||settings.liveSync||settings.periodicReplication) {
throw new Error('The fixture requires one-shot replication with on-demand Chunk reads.');
}
const replicator=core.services.replicator.getActiveReplicator();
const original=replicator.fetchRemoteChunks;
if(typeof original!=='function') throw new Error('The real replicator has no Chunk fetch method.');
const manager=core.localDatabase.managers.chunkManager;
const observer=new AbortController();
const finiteCount=core.services.replicator.finiteReplicationActivityCount;
const state={path:${JSON.stringify(path)},metadataId:${JSON.stringify(metadataId)},
chunkId:${JSON.stringify(chunkId)},attempts:[],missingEvents:[],replicationDone:false,
finiteTransitions:[{at:Date.now(),count:finiteCount.value}]};
const observeFinite=()=>state.finiteTransitions.push({at:Date.now(),count:finiteCount.value});
finiteCount.onChanged(observeFinite);
${observationSource}=state;
state.restore=()=>{replicator.fetchRemoteChunks=original;observer.abort();finiteCount.offChanged(observeFinite);};
manager.addListener('missingChunkRemote',(id)=>{
if(id===state.chunkId) state.missingEvents.push(Date.now());
},{signal:observer.signal});
// Observe the real HTTP-backed method without changing its result or adding a wait.
replicator.fetchRemoteChunks=async function(...args){
if(!args[0].includes(state.chunkId)) return await original.apply(this,args);
const attempt={startedAt:Date.now(),finiteTransitionIndex:state.finiteTransitions.length,requestedIds:[...args[0]]};
state.attempts.push(attempt);
try{
const result=await original.apply(this,args);
attempt.unavailable=result===false;
attempt.returnedIds=Array.isArray(result)?result.map((chunk)=>chunk._id):[];
return result;
}catch(error){
attempt.error=String(error);
throw error;
}finally{
attempt.completedAt=Date.now();
}
};
state.startReplication=async()=>{
state.replicationDone=false;
try{state.replicationSucceeded=!!(await core.services.replication.replicate(true));}
catch(error){state.replicationError=String(error);}
finally{state.replicationDone=true;}
};
state.replication=state.startReplication();
})()`);
try {
const first = await waitForSnapshot(
page,
(state) => !!state.attempts[0]?.completedAt,
"initial missing response"
);
assertEqual(first.metadataPresent, true, "The Metadata did not arrive through real replication.");
assertEqual(first.chunkPresent, false, "The Chunk was already available locally before its delayed arrival.");
assertEqual(
first.attempts[0].unavailable,
false,
"The first fetch failed instead of returning a missing Chunk."
);
assertEqual(first.attempts[0].returnedIds?.length, 0, "The first fetch did not reproduce a missing Chunk.");
if (delayedArrival) {
await putCouchDbDocument(couchDb, dbName, { _id: chunkId, type: "leaf", data: content });
}
console.log(`${label}: first real response ${JSON.stringify(first)}`);
assertEqual(
first.missingEvents.length,
0,
"The first missing response ended delivery before the delayed retry."
);
assertEqual(first.claimActive, true, "The first missing response released the delivery claim.");
assertEqual(first.content, null, "The file was materialised before its missing Chunk arrived.");
const retryWaiting = await waitForSnapshot(
page,
(state) => state.initialChunkCount === 0 && state.retryChunkCount === 1,
"separate retry status during the retry delay",
1_000
);
assertEqual(retryWaiting.attempts.length, 1, "The pending count appeared only after the retry started.");
const completedDuringInitialLookup = retryWaiting.finiteTransitions
.slice(first.attempts[0].finiteTransitionIndex)
.some((transition) => transition.count === 0);
if (!completedDuringInitialLookup) {
assertEqual(retryWaiting.currentProcessing, 0, "Backoff retained a physical request slot.");
}
console.log(`${label}: retry waiting ${JSON.stringify(retryWaiting)}`);
if (expediteFinalProbe) {
await waitForSnapshot(
page,
(state) => state.replicationDone && state.finiteActivity === 0 && state.attempts.length === 1,
"first finite replication completion before the scheduled retry",
1_000
);
// A second genuine one-shot replication ends during backoff; counts are observed, never synthesised.
await page.evaluate(`(()=>{
const state=${observationSource};
state.followupTransitionIndex=state.finiteTransitions.length;
state.replication=state.startReplication();
})()`);
}
const completed = await waitForSnapshot(page, isIdle, "delivery and reflection quiescence");
assertEqual(completed.replicationSucceeded, true, "One-shot replication did not complete successfully.");
assertEqual(completed.attempts.length, 2, "The missing Chunk must be fetched exactly twice.");
const [initial, retry] = completed.attempts;
const retryDelay = retry.startedAt - initial.completedAt!;
if (expediteFinalProbe) {
const transitions = completed.finiteTransitions.slice(completed.followupTransitionIndex);
const ended = transitions.find(
(transition, index) => index > 0 && transition.count === 0 && transitions[index - 1].count > 0
);
if (!ended)
throw new Error(`The second real finite replication was not observed: ${JSON.stringify(transitions)}`);
if (retry.startedAt < ended.at) throw new Error("The final probe began before finite replication ended.");
if (retryDelay >= retryDelayMs)
throw new Error(`Finite completion did not interrupt backoff: ${retryDelay} ms.`);
} else {
const completion = completed.finiteTransitions
.slice(initial.finiteTransitionIndex)
.find((transition) => transition.count === 0);
if (completion) {
if (retry.startedAt < completion.at) throw new Error("The final probe preceded finite completion.");
if (retryDelay >= retryDelayMs) {
throw new Error(`Finite completion did not expedite the initial missing result: ${retryDelay} ms.`);
}
} else if (retryDelay < retryDelayMs) {
throw new Error(`The retry started too early: ${retryDelay} ms.`);
}
}
assertEqual(retry.requestedIds.join(","), chunkId, "The retry requested an unexpected Chunk.");
assertEqual(retry.unavailable, false, "The retry failed to contact the real remote.");
assertEqual(retry.returnedIds?.join(","), delayedArrival ? chunkId : "", "The retry returned unexpected data.");
assertEqual(
completed.missingEvents.length,
delayedArrival ? 0 : 1,
"Unexpected terminal missing notifications."
);
assertEqual(completed.chunkPresent, delayedArrival, "The local Chunk persistence result was unexpected.");
assertEqual(completed.content, delayedArrival ? content : null, "The Vault file content was unexpected.");
if (!delayedArrival) {
// Observe one more retry interval after quiescence to reject an unbounded retry loop.
await new Promise((resolve) => setTimeout(resolve, retryDelayMs + 100));
const settled = await snapshot(page);
assertEqual(isIdle(settled), true, "The permanently missing delivery became active again.");
assertEqual(settled.attempts.length, 2, "The permanently missing Chunk was retried again.");
}
console.log(`${label}: passed; retry after ${retryDelay} ms; ${JSON.stringify(completed)}`);
} catch (error) {
console.error(`${label}: ${JSON.stringify(await snapshot(page))}`);
const diagnostics = process.env.E2E_OBSIDIAN_DIAGNOSTICS_DIR ?? "/tmp/obsidian-livesync-e2e";
await mkdir(diagnostics, { recursive: true });
await page.screenshot({ path: join(diagnostics, `chunk-fetch-${label}.failure.png`), fullPage: true });
throw error;
} finally {
await page.evaluate(`(()=>{${observationSource}?.restore();delete ${observationSource};})()`);
}
}
async function main(): Promise<void> {
const binary = requireObsidianBinary();
const cli = discoverObsidianCli();
if (!cli.binary) throw new Error(`Could not find obsidian-cli. Checked: ${cli.checked.join(", ")}`);
const couchDb = await loadCouchDbConfig();
await assertCouchDbReachable(couchDb);
const dbName = makeUniqueDatabaseName(couchDb.dbPrefix, "chunk-fetch-retry");
const vault = await createTemporaryVault("obsidian-livesync-chunk-fetch-");
let session: ObsidianLiveSyncSession | undefined;
try {
await createCouchDbDatabase(couchDb, dbName);
session = await startObsidianLiveSyncSession({
binary,
cliBinary: cli.binary,
vault,
pluginData: createE2eCouchDbPluginData(
{ ...couchDb, dbName },
{
encrypt: false,
usePathObfuscation: false,
showStatusOnStatusbar: true,
// Keep the real retry status observable even when finite completion expedites the final probe.
minimumIntervalOfReadChunksOnline: 500,
periodicReplication: false,
syncOnFileOpen: false,
syncOnEditorSave: false,
syncAfterMerge: false,
}
),
localStorageEntries: createE2eObsidianDeviceLocalState(vault.name),
});
await waitForLiveSyncCoreReady(cli.binary, session.cliEnv);
await prepareRemote(cli.binary, session.cliEnv);
await withObsidianPage(session.remoteDebuggingPort, async (page) => {
await runScenario(page, couchDb, dbName, "delayed-arrival");
await runScenario(page, couchDb, dbName, "permanently-missing");
await runScenario(page, couchDb, dbName, "finite-completion");
});
} finally {
if (session) await session.app.stop();
await vault.dispose();
await deleteCouchDbDatabase(couchDb, dbName);
}
}
main().catch((error: unknown) => {
console.error(error instanceof Error ? error.stack : error);
process.exitCode = 1;
});
test/e2e-obsidian/scripts/cli-to-obsidian-sync.ts
+2
View File
@@ -210,6 +210,7 @@ async function configureLiveSyncCli(
encrypt: true,
passphrase: e2eePassphrase,
usePathObfuscation: true,
encryptInternalMetadata: true,
doctorProcessedVersion: "0.25.27",
isConfigured: true,
});
@@ -323,6 +324,7 @@ async function main(): Promise<void> {
encrypt: true,
passphrase: e2eePassphrase,
usePathObfuscation: true,
encryptInternalMetadata: true,
E2EEAlgorithm: "v2",
}
),
test/e2e-obsidian/scripts/couchdb-manual-setup-workflow.ts
+307 -17
View File
@@ -2,6 +2,7 @@ import { randomBytes } from "node:crypto";
import { readFile } from "node:fs/promises";
import { join } from "node:path";
import { DEVICE_ID_PREFERRED, MILESTONE_DOCID } from "@vrtmrz/livesync-commonlib/compat/common/types";
import { deriveIdKey, formatIdRecoveryCode } from "@vrtmrz/livesync-commonlib/settings";
import { evalObsidianJson } from "../runner/cli.ts";
import {
assertCouchDbReachable,
@@ -39,6 +40,7 @@ process.env.E2E_OBSIDIAN_CLI_TIMEOUT_MS ??= "90000";
process.env.E2E_OBSIDIAN_COUCHDB_TIMEOUT_MS ??= "30000";
const uiTimeoutMs = Number(process.env.E2E_OBSIDIAN_SETUP_URI_TIMEOUT_MS ?? 30000);
const e2eePassphrase = `%${randomBytes(24).toString("base64url")}`;
const notePath = "E2E/manual-couchdb/from-first-device.md";
const noteContent = "# Manual CouchDB setup\n\nThis note was sent by the manually configured first device.\n";
const returnNotePath = "E2E/manual-couchdb/from-second-device.md";
@@ -98,7 +100,12 @@ async function captureFailure(session: ObsidianLiveSyncSession, label: string):
}
}
async function enterManualCouchDBSettings(port: number, couchDb: CouchDbConfig, dbName: string): Promise<string[]> {
async function enterManualCouchDBSettings(
port: number,
couchDb: CouchDbConfig,
dbName: string,
independentIdSource?: string
): Promise<string[]> {
const screenshots: string[] = [];
await withObsidianPage(port, async (page) => {
const invitation = page.locator(".notice").filter({ hasText: "Welcome to Self-hosted LiveSync" });
@@ -133,12 +140,41 @@ async function enterManualCouchDBSettings(port: number, couchDb: CouchDbConfig,
0,
"The Obfuscate Properties row was present before end-to-end encryption was enabled."
);
const disabledIdChoices = encryption.locator("fieldset.sls-id-choices").first();
assertEqual(
await disabledIdChoices.evaluate((element) => element.hasAttribute("disabled")),
true,
"The ID configuration was enabled while E2EE was off."
);
for (const value of ["keep", "random", "custom"]) {
assertEqual(
await disabledIdChoices.locator(`input[value="${value}"]`).isDisabled(),
true,
`The ${value} ID configuration was enabled while E2EE was off.`
);
}
const disabledIdScreenshot = join(
process.env.E2E_OBSIDIAN_DIAGNOSTICS_DIR ?? "/tmp/obsidian-livesync-e2e",
"guide-couchdb-manual-id-generation-disabled.png"
);
await disabledIdChoices.screenshot({ path: disabledIdScreenshot });
screenshots.push(disabledIdScreenshot);
assertEqual(
await disabledIdChoices.evaluate((element) => Number(getComputedStyle(element).opacity) < 1),
true,
"The disabled ID configuration did not look disabled in the default theme."
);
await encryption
.locator("label.row")
.filter({ hasText: "End-to-End Encryption" })
.locator('input[type="checkbox"]')
.first()
.check({ timeout: uiTimeoutMs });
assertEqual(
await disabledIdChoices.evaluate((element) => Number(getComputedStyle(element).opacity)),
1,
"The ID configuration remained dimmed after E2EE was enabled."
);
const passphraseInput = encryption.locator('input[name="e2ee-passphrase"]');
await passphraseInput.waitFor({ state: "visible", timeout: uiTimeoutMs });
await encryption
@@ -147,9 +183,86 @@ async function enterManualCouchDBSettings(port: number, couchDb: CouchDbConfig,
.locator('input[type="checkbox"]')
.first()
.check({ timeout: uiTimeoutMs });
const passphraseValue = randomBytes(24).toString("base64url");
await passphraseInput.fill(passphraseValue);
const passwordToggle = encryption.locator("button.sls-password-toggle");
await passphraseInput.fill(e2eePassphrase);
const idChoices = encryption.locator('input[type="radio"][name="id-derivation-choice"]');
assertEqual(await idChoices.count(), 3, "The three ID configurations were not all shown.");
for (const value of ["keep", "random", "custom"]) {
assertEqual(
await encryption.locator(`input[name="id-derivation-choice"][value="${value}"]`).isVisible(),
true,
`The ${value} ID configuration was not visible.`
);
}
const keepChoice = encryption.locator('input[name="id-derivation-choice"][value="keep"]');
const randomChoice = encryption.locator('input[name="id-derivation-choice"][value="random"]');
const customChoice = encryption.locator('input[name="id-derivation-choice"][value="custom"]');
assertEqual(await randomChoice.isChecked(), true, "The default ID configuration was not random.");
assertEqual(
await encryption.getByText("Keep current configuration", { exact: true }).count(),
1,
"The current-configuration choice was not labelled consistently."
);
assertEqual(
await encryption.getByText("Current configuration: no ID key is saved.", { exact: false }).count(),
1,
"The current legacy configuration was not explained."
);
await keepChoice.check({ timeout: uiTimeoutMs });
assertEqual(
await encryption.getByText("Changing the E2EE passphrase changes IDs", { exact: false }).count(),
1,
"Keeping legacy IDs did not explain the effect of changing the E2EE passphrase."
);
await randomChoice.check({ timeout: uiTimeoutMs });
if (independentIdSource) {
await customChoice.check({ timeout: uiTimeoutMs });
const customChoices = encryption.locator('input[type="radio"][name="id-custom-choice"]');
assertEqual(await customChoices.count(), 3, "The three custom ID inputs were not all shown.");
for (const value of ["passphrase", "source", "recovery"]) {
assertEqual(
await encryption.locator(`input[name="id-custom-choice"][value="${value}"]`).isVisible(),
true,
`The ${value} custom ID input was not visible.`
);
}
const sourceChoice = encryption.locator('input[name="id-custom-choice"][value="source"]');
assertEqual(await sourceChoice.isChecked(), true, "The custom ID input was not selected by default.");
await encryption
.locator('input[name="id-custom-choice"][value="passphrase"]')
.check({ timeout: uiTimeoutMs });
assertEqual(
await encryption.locator('input[name="id-derivation-source"]').count(),
0,
"The E2EE passphrase choice exposed a second source input."
);
await encryption
.locator('input[name="id-custom-choice"][value="recovery"]')
.check({ timeout: uiTimeoutMs });
assertEqual(
await encryption.locator('input[name="id-derivation-source"]').getAttribute("placeholder"),
"Enter an ID recovery code",
"The recovery-code choice did not request a recovery code."
);
const recoveryChoice = encryption.locator('input[name="id-custom-choice"][value="recovery"]');
await sourceChoice.check({ timeout: uiTimeoutMs });
const sourceInput = encryption.locator('input[name="id-derivation-source"]');
await sourceInput.fill("");
await encryption.getByRole("button", { name: "Proceed", exact: true }).click({ timeout: uiTimeoutMs });
assertEqual(
await encryption.getByText("An ID source is required to enable this option.", { exact: false }).count(),
1,
"A first-time independent ID configuration did not require a source."
);
assertEqual(
await encryption.isVisible(),
true,
"The E2EE dialogue closed after a first-time ID source was omitted."
);
await recoveryChoice.check({ timeout: uiTimeoutMs });
await sourceChoice.check({ timeout: uiTimeoutMs });
await sourceInput.fill(independentIdSource);
}
const passwordToggle = passphraseInput.locator("..").locator("button.sls-password-toggle");
await passwordToggle.click({ timeout: uiTimeoutMs });
assertEqual(
await passphraseInput.getAttribute("type"),
@@ -158,7 +271,7 @@ async function enterManualCouchDBSettings(port: number, couchDb: CouchDbConfig,
);
assertEqual(
await passphraseInput.inputValue(),
passphraseValue,
e2eePassphrase,
"Toggling visibility changed the passphrase value."
);
await passwordToggle.click({ timeout: uiTimeoutMs });
@@ -167,16 +280,13 @@ async function enterManualCouchDBSettings(port: number, couchDb: CouchDbConfig,
"password",
"Toggling visibility again did not re-mask the passphrase."
);
assertEqual(
await passphraseInput.inputValue(),
passphraseValue,
"Re-masking the passphrase changed its value."
);
assertEqual(await passphraseInput.inputValue(), e2eePassphrase, "Re-masking the passphrase changed its value.");
});
screenshots.push(await captureGuideDialogue(port, "guide-couchdb-manual-encryption.png", "End-to-End Encryption"));
await withObsidianPage(port, async (page) => {
const encryption = modalByTitle(page, "End-to-End Encryption");
await encryption.getByRole("button", { name: "Proceed", exact: true }).click({ timeout: uiTimeoutMs });
await encryption.waitFor({ state: "hidden", timeout: uiTimeoutMs });
});
screenshots.push(
@@ -288,19 +398,169 @@ async function waitForRemoteEntry(context: RunnerContext, entry: { id: string; c
});
}
async function assertPersistedE2EE(vault: TemporaryVault): Promise<void> {
const persisted = JSON.parse(
await readFile(join(vault.path, ".obsidian", "plugins", "obsidian-livesync", "data.json"), "utf8")
) as {
async function assertPersistedE2EE(vault: TemporaryVault, independentIdSource?: string): Promise<void> {
const rawSettings = await readFile(
join(vault.path, ".obsidian", "plugins", "obsidian-livesync", "data.json"),
"utf8"
);
const persisted = JSON.parse(rawSettings) as {
encrypt?: unknown;
encryptedPassphrase?: unknown;
passphrase?: unknown;
idDerivationVersion?: unknown;
idDerivationKey?: unknown;
encryptedIdDerivationKey?: unknown;
};
assertEqual(persisted.encrypt, true, "Manual CouchDB setup did not persist E2EE as enabled.");
assertEqual(persisted.passphrase, "", "Manual CouchDB setup persisted the E2EE passphrase in plain text.");
if (typeof persisted.encryptedPassphrase !== "string" || persisted.encryptedPassphrase.length === 0) {
throw new Error("Manual CouchDB setup did not persist an encrypted E2EE passphrase.");
}
if (JSON.stringify(persisted).includes(e2eePassphrase)) {
throw new Error("Manual CouchDB setup persisted the E2EE passphrase in plain text.");
}
assertEqual(persisted.idDerivationVersion, 1, "The independent ID mode was not persisted.");
assertEqual(persisted.idDerivationKey, "", "The derived ID key was stored in plain text.");
if (typeof persisted.encryptedIdDerivationKey !== "string" || !persisted.encryptedIdDerivationKey) {
throw new Error("The derived ID key was not encrypted in local settings.");
}
if (independentIdSource) {
if (rawSettings.includes(independentIdSource)) throw new Error("The ID source was stored in local settings.");
}
}
async function assertRestoredE2EEPassphrase(session: ObsidianLiveSyncSession, cliBinary: string): Promise<void> {
const restored = await evalObsidianJson<boolean>(
cliBinary,
[
"(()=>{",
"const settings=app.plugins.plugins['obsidian-livesync'].core.services.setting.currentSettings();",
`return JSON.stringify(settings.passphrase === ${JSON.stringify(e2eePassphrase)});`,
"})()",
].join(""),
session.cliEnv
);
assertEqual(restored, true, "The E2EE passphrase was not restored after Obsidian restarted.");
}
async function assertCurrentIdDerivationKey(
session: ObsidianLiveSyncSession,
cliBinary: string,
expected: string,
context: string
): Promise<void> {
const settings = await evalObsidianJson<{ idDerivationVersion: number; idDerivationKey: string }>(
cliBinary,
[
"(()=>{",
"const settings=app.plugins.plugins['obsidian-livesync'].core.services.setting.currentSettings();",
"return JSON.stringify({idDerivationVersion:settings.idDerivationVersion,idDerivationKey:settings.idDerivationKey});",
"})()",
].join(""),
session.cliEnv
);
assertEqual(settings.idDerivationVersion, 1, `${context}: the independent ID version was not retained.`);
assertEqual(settings.idDerivationKey, expected, `${context}: the saved ID key changed.`);
}
async function assertRecoveryCodeCanBeRevealed(
session: ObsidianLiveSyncSession,
cliBinary: string,
source: string
): Promise<void> {
const port = session.remoteDebuggingPort;
const expected = formatIdRecoveryCode(await deriveIdKey(source));
await withObsidianPage(port, async (page) => {
const settingsNavigator = await openLiveSyncSettings(page, uiTimeoutMs);
const remotePage = await settingsNavigator.openPage("Remote Configuration");
await remotePage
.locator(".setting-item")
.filter({ hasText: "Configure E2EE" })
.getByRole("button", { name: "Configure", exact: true })
.click({ timeout: uiTimeoutMs });
const encryption = modalByTitle(page, "End-to-End Encryption");
await encryption.waitFor({ state: "visible", timeout: uiTimeoutMs });
assertEqual(
await encryption.locator('input[name="id-derivation-choice"][value="keep"]').isChecked(),
true,
"An existing ID key was not selected for reuse."
);
assertEqual(
await encryption.getByText("Current configuration: a saved ID key is used.").count(),
1,
"The saved ID key was not explained."
);
await encryption.getByRole("button", { name: "Show current recovery code" }).click({ timeout: uiTimeoutMs });
assertEqual(
await encryption.getByRole("textbox", { name: "Current ID recovery code" }).inputValue(),
expected,
"The displayed recovery code did not contain the saved ID key."
);
await encryption.locator('input[name="id-derivation-choice"][value="custom"]').check({ timeout: uiTimeoutMs });
await encryption.locator('input[name="id-custom-choice"][value="recovery"]').check({ timeout: uiTimeoutMs });
const recoveryInput = encryption.locator('input[name="id-derivation-source"]');
await recoveryInput.fill("not-a-recovery-code");
await encryption.getByRole("button", { name: "Proceed" }).click({ timeout: uiTimeoutMs });
assertEqual(
await encryption.getByText("The ID source or recovery code is invalid.", { exact: false }).count(),
1,
"The recovery-code choice accepted an ordinary source string."
);
await recoveryInput.fill(expected);
await encryption.getByRole("button", { name: "Proceed" }).click({ timeout: uiTimeoutMs });
await encryption.waitFor({ state: "hidden", timeout: uiTimeoutMs });
assertEqual(
await modalByTitle(page, "Mostly Complete: Decision Required").count(),
0,
"Recovering the existing ID key opened a new setup decision."
);
});
const expectedIdKey = await deriveIdKey(source);
await assertCurrentIdDerivationKey(session, cliBinary, expectedIdKey, "Recovering the saved ID key");
await withObsidianPage(port, async (page) => {
const settingsNavigator = await openLiveSyncSettings(page, uiTimeoutMs);
const remotePage = await settingsNavigator.openPage("Remote Configuration");
await remotePage
.locator(".setting-item")
.filter({ hasText: "Configure E2EE" })
.getByRole("button", { name: "Configure", exact: true })
.click({ timeout: uiTimeoutMs });
const encryption = modalByTitle(page, "End-to-End Encryption");
await encryption.waitFor({ state: "visible", timeout: uiTimeoutMs });
await encryption.locator('input[name="id-derivation-choice"][value="custom"]').check({ timeout: uiTimeoutMs });
await encryption.locator('input[name="id-custom-choice"][value="source"]').check({ timeout: uiTimeoutMs });
const sourceInput = encryption.locator('input[name="id-derivation-source"]');
await sourceInput.fill("");
assertEqual(await sourceInput.inputValue(), "", "The independent ID source input was not empty.");
assertEqual(
await encryption.getByText("Leave this input empty to keep the saved ID key.", { exact: true }).count(),
1,
"The configured ID source did not explain that an empty input keeps the saved ID key."
);
await encryption.getByRole("button", { name: "Proceed", exact: true }).click({ timeout: uiTimeoutMs });
await encryption.waitFor({ state: "hidden", timeout: uiTimeoutMs });
assertEqual(
await modalByTitle(page, "Mostly Complete: Decision Required").count(),
0,
"Keeping the saved ID key after an empty source opened a new setup decision."
);
});
await assertCurrentIdDerivationKey(session, cliBinary, expectedIdKey, "Saving an empty custom ID source");
}
async function readDocumentId(cliBinary: string, environment: NodeJS.ProcessEnv, path: string): Promise<string> {
return await evalObsidianJson<string>(
cliBinary,
[
"(async()=>{",
`const path=${JSON.stringify(path)};`,
"const core=app.plugins.plugins['obsidian-livesync'].core;",
"return JSON.stringify(await core.services.path.path2id(path));",
"})()",
].join(""),
environment
);
}
async function setRemotePreferredE2EEDisabled(context: RunnerContext): Promise<void> {
@@ -413,6 +673,10 @@ async function main(): Promise<void> {
};
const screenshots: string[] = [];
let secondDeviceArtifact: SetupArtifact | undefined;
const independentIdSource =
process.env.E2E_OBSIDIAN_INDEPENDENT_IDS === "true" ? randomBytes(32).toString("base64url") : undefined;
let firstEntryId: string | undefined;
let returnEntryId: string | undefined;
try {
await assertCouchDbReachable(couchDb);
@@ -423,7 +687,9 @@ async function main(): Promise<void> {
let session = await startUnconfiguredSession(context, vaultA);
try {
screenshots.push(...(await enterManualCouchDBSettings(session.remoteDebuggingPort, couchDb, dbName)));
screenshots.push(
...(await enterManualCouchDBSettings(session.remoteDebuggingPort, couchDb, dbName, independentIdSource))
);
screenshots.push(await captureAndStartInitialisation(session.remoteDebuggingPort, "new", captures));
screenshots.push(await confirmRebuild(session.remoteDebuggingPort, captures));
screenshots.push(await continueWithoutRemoteSettings(session.remoteDebuggingPort, captures));
@@ -436,10 +702,14 @@ async function main(): Promise<void> {
1,
"Manual CouchDB setup did not persist exactly one remote profile."
);
await assertPersistedE2EE(vaultA);
await assertPersistedE2EE(vaultA, independentIdSource);
if (independentIdSource) {
await assertRecoveryCodeCanBeRevealed(session, context.cliBinary, independentIdSource);
}
await writeNoteViaObsidian(context.cliBinary, session.cliEnv, notePath, noteContent);
const entry = await waitForLocalDatabaseEntry(context.cliBinary, session.cliEnv, notePath);
firstEntryId = entry.id;
await pushLocalChanges(context.cliBinary, session.cliEnv);
await waitForRemoteEntry(context, entry);
} catch (error) {
@@ -454,6 +724,7 @@ async function main(): Promise<void> {
session = await startUnconfiguredSession(context, vaultA);
try {
await assertRestoredE2EEPassphrase(session, context.cliBinary);
await scheduleRemoteOverwrite(session.remoteDebuggingPort);
screenshots.push(await confirmRebuild(session.remoteDebuggingPort, e2eeRebuildCaptures));
screenshots.push(
@@ -461,9 +732,12 @@ async function main(): Promise<void> {
);
await finishInitialisation(session.remoteDebuggingPort, context.cliBinary, session.cliEnv);
await resumeCompatibilityReviewIfShown(session.remoteDebuggingPort);
await assertPersistedE2EE(vaultA);
await assertPersistedE2EE(vaultA, independentIdSource);
const rebuiltEntry = await waitForLocalDatabaseEntry(context.cliBinary, session.cliEnv, notePath);
if (independentIdSource) {
assertEqual(rebuiltEntry.id, firstEntryId, "Rebuild changed the configured document ID.");
}
await waitForRemoteEntry(context, rebuiltEntry);
await assertRemoteEntryEncrypted(context, rebuiltEntry, notePath, noteContent);
await assertRemotePreferredE2EE(context, true);
@@ -494,11 +768,20 @@ async function main(): Promise<void> {
screenshots.push(...(await confirmFastFetch(session.remoteDebuggingPort, captures)));
await finishInitialisation(session.remoteDebuggingPort, context.cliBinary, session.cliEnv);
await resumeCompatibilityReviewIfShown(session.remoteDebuggingPort);
await assertPersistedE2EE(vaultB, independentIdSource);
await pushLocalChanges(context.cliBinary, session.cliEnv);
await waitForVaultFile(vaultB, notePath, noteContent);
if (independentIdSource) {
assertEqual(
await readDocumentId(context.cliBinary, session.cliEnv, notePath),
firstEntryId,
"The Setup URI did not restore the document ID key on the second device."
);
}
await writeNoteViaObsidian(context.cliBinary, session.cliEnv, returnNotePath, returnNoteContent);
const returnEntry = await waitForLocalDatabaseEntry(context.cliBinary, session.cliEnv, returnNotePath);
returnEntryId = returnEntry.id;
await pushLocalChanges(context.cliBinary, session.cliEnv);
await waitForRemoteEntry(context, returnEntry);
} catch (error) {
@@ -513,6 +796,13 @@ async function main(): Promise<void> {
await resumeCompatibilityReviewIfShown(session.remoteDebuggingPort);
await pushLocalChanges(context.cliBinary, session.cliEnv);
await waitForVaultFile(vaultA, returnNotePath, returnNoteContent);
if (independentIdSource) {
assertEqual(
await readDocumentId(context.cliBinary, session.cliEnv, returnNotePath),
returnEntryId,
"The first device did not retain the shared document ID key."
);
}
} catch (error) {
await captureFailure(session, "return-journey");
throw error;
test/e2e-obsidian/scripts/customisation-sync.ts
+19
View File
@@ -1,10 +1,13 @@
import { mkdir, readFile, rm, writeFile } from "node:fs/promises";
import { dirname, join } from "node:path";
import { VERSIONING_DOCID } from "@vrtmrz/livesync-commonlib/compat/common/types";
import { ENCRYPTED_INTERNAL_METADATA_FEATURE, REMOTE_FEATURE_GENERATION } from "@vrtmrz/livesync-commonlib/replication";
import { evalObsidianJson } from "../runner/cli.ts";
import {
assertCouchDbReachable,
createCouchDbDatabase,
deleteCouchDbDatabase,
fetchCouchDbDocument,
loadCouchDbConfig,
makeUniqueDatabaseName,
waitForCouchDbDocs,
@@ -159,6 +162,10 @@ async function startConfiguredSession(
dbName: context.dbName,
};
const customisationSettings = {
encrypt: true,
passphrase: "internal-metadata-e2e-secret",
usePathObfuscation: true,
encryptInternalMetadata: true,
deviceAndVaultName: deviceName,
usePluginSync: true,
usePluginSyncV2: true,
@@ -486,6 +493,18 @@ async function main(): Promise<void> {
(target) => ids.has(target.id) && target.children.every((childId) => ids.has(childId))
);
});
for (const target of [entry, configEntry, ...pluginEntries]) {
const remoteEntry = await fetchCouchDbDocument(context.couchDb, context.dbName, target.id);
if (!remoteEntry.path?.startsWith("/\\:") || remoteEntry.children?.length !== 0 ||
remoteEntry.ctime !== 0 || remoteEntry.mtime !== 0 || remoteEntry.size !== 0) {
throw new Error(`Customisation Sync Metadata was not encrypted for ${target.id}.`);
}
}
const versionInfo = await fetchCouchDbDocument(context.couchDb, context.dbName, VERSIONING_DOCID);
if (versionInfo.version !== REMOTE_FEATURE_GENERATION ||
!(versionInfo.used_features as unknown[] | undefined)?.includes(ENCRYPTED_INTERNAL_METADATA_FEATURE)) {
throw new Error("The remote feature list does not declare encrypted internal Metadata.");
}
await session.app.stop();
session = await startConfiguredSession(context, vaultB, targetDeviceName);
test/e2e-obsidian/scripts/hidden-file-snippet-sync.ts
+60 -12
View File
@@ -1,5 +1,7 @@
import { mkdir, readFile, rm, writeFile } from "node:fs/promises";
import { dirname, join } from "node:path";
import { VERSIONING_DOCID } from "@vrtmrz/livesync-commonlib/compat/common/types";
import { ENCRYPTED_INTERNAL_METADATA_FEATURE, REMOTE_FEATURE_GENERATION } from "@vrtmrz/livesync-commonlib/replication";
import {
assertLocatorHasMinimumTouchTarget,
assertLocatorWithinSafeArea,
@@ -11,6 +13,7 @@ import {
assertCouchDbReachable,
createCouchDbDatabase,
deleteCouchDbDatabase,
fetchCouchDbDocument,
loadCouchDbConfig,
makeUniqueDatabaseName,
waitForCouchDbDocs,
@@ -324,6 +327,10 @@ async function startConfiguredSession(
dbName: context.dbName,
};
const hiddenFileSettings = {
encrypt: true,
passphrase: "internal-metadata-e2e-secret",
usePathObfuscation: true,
encryptInternalMetadata: true,
syncInternalFiles: true,
syncInternalFilesBeforeReplication: true,
watchInternalFileChanges: false,
@@ -360,6 +367,23 @@ async function uploadHiddenFile(
const ids = new Set(docs.map((doc) => doc._id));
return ids.has(entry.id) && entry.children.every((childId) => ids.has(childId));
});
const remoteEntry = await fetchCouchDbDocument(context.couchDb, context.dbName, entry.id);
if (
!remoteEntry.path?.startsWith("/\\:") ||
remoteEntry.children?.length !== 0 ||
remoteEntry.ctime !== 0 ||
remoteEntry.mtime !== 0 ||
remoteEntry.size !== 0
) {
throw new Error(`Hidden File Sync Metadata was not encrypted for ${entry.id}.`);
}
const versionInfo = await fetchCouchDbDocument(context.couchDb, context.dbName, VERSIONING_DOCID);
if (
versionInfo.version !== REMOTE_FEATURE_GENERATION ||
!(versionInfo.used_features as unknown[] | undefined)?.includes(ENCRYPTED_INTERNAL_METADATA_FEATURE)
) {
throw new Error("The remote feature list does not declare encrypted internal Metadata.");
}
return entry;
}
@@ -383,6 +407,29 @@ async function runCreateRoundTrip(
await writeVaultFile(vaultA.path, snippetPath, snippetContent);
let session = await startConfiguredSession(context, vaultA);
const entry = await uploadHiddenFile(context, session, snippetPath);
await evalObsidianJson(
context.cliBinary,
[
"(async()=>{",
"const rebuilder=app.plugins.plugins['obsidian-livesync'].core.rebuilder;",
"const inform=rebuilder.informOptionalFeatures;",
"rebuilder.informOptionalFeatures=async()=>{};",
"try{await rebuilder.$rebuildRemote();}finally{rebuilder.informOptionalFeatures=inform;}",
"return JSON.stringify(true);",
"})()",
].join(""),
session.cliEnv
);
const rebuiltVersion = await fetchCouchDbDocument(context.couchDb, context.dbName, VERSIONING_DOCID);
const rebuiltEntry = await fetchCouchDbDocument(context.couchDb, context.dbName, entry.id);
if (
rebuiltVersion.version !== REMOTE_FEATURE_GENERATION ||
!(rebuiltVersion.used_features as unknown[] | undefined)?.includes(ENCRYPTED_INTERNAL_METADATA_FEATURE) ||
!rebuiltEntry.path?.startsWith("/\\:")
) {
throw new Error("Remote Rebuild did not declare and encrypt internal Metadata for its accepted writer.");
}
console.log("Remote Rebuild declared the feature and preserved encrypted Hidden File Sync Metadata.");
await session.app.stop();
session = await startConfiguredSession(context, vaultB);
@@ -571,11 +618,14 @@ async function runInitialisationNoticeGrouping(context: RunnerContext, vault: Te
await withObsidianPage(port, async (page) => {
const deadline = Date.now() + timeoutMs;
while ((await page.locator(".notice:visible").count()) > 0 && Date.now() < deadline) {
await page.locator(".notice:visible").first().click({
force: true,
position: { x: 2, y: 2 },
timeout: timeoutMs,
});
await page
.locator(".notice:visible")
.first()
.click({
force: true,
position: { x: 2, y: 2 },
timeout: timeoutMs,
});
}
assertEqual(
await page.locator(".notice:visible").count(),
@@ -707,17 +757,15 @@ async function runInitialisationNoticeGrouping(context: RunnerContext, vault: Te
const result = await withObsidianPage(port, async (page) => {
await page.evaluate((stateKey) => {
const state = (globalThis as unknown as Record<
string,
{ releasePreparation?: () => void } | undefined
>)[stateKey];
const state = (
globalThis as unknown as Record<string, { releasePreparation?: () => void } | undefined>
)[stateKey];
state?.releasePreparation?.();
}, hiddenFileInitialisationStateKey);
await page.waitForFunction(
(stateKey) =>
(globalThis as unknown as Record<string, { reachedInitialisation?: boolean } | undefined>)[
stateKey
]?.reachedInitialisation === true,
(globalThis as unknown as Record<string, { reachedInitialisation?: boolean } | undefined>)[stateKey]
?.reachedInitialisation === true,
hiddenFileInitialisationStateKey,
{ timeout: timeoutMs }
);
test/e2e-obsidian/scripts/internal-metadata-migration.ts
+285
View File
@@ -0,0 +1,285 @@
import { mkdir, readFile, writeFile } from "node:fs/promises";
import { dirname, join } from "node:path";
import { VERSIONING_DOCID, type LoadedEntry } from "@vrtmrz/livesync-commonlib/compat/common/types";
import { readContent } from "@vrtmrz/livesync-commonlib/compat/common/utils";
import { ENCRYPTED_INTERNAL_METADATA_FEATURE } from "@vrtmrz/livesync-commonlib/replication";
import { evalObsidianJson } from "../runner/cli.ts";
import {
assertCouchDbReachable,
createCouchDbDatabase,
deleteCouchDbDatabase,
fetchCouchDbDocument,
loadCouchDbConfig,
makeUniqueDatabaseName,
} from "../runner/couchdb.ts";
import { discoverObsidianCli, requireObsidianBinary } from "../runner/environment.ts";
import {
assertEqual,
configureCouchDb,
createE2eCouchDbPluginData,
createE2eObsidianDeviceLocalState,
prepareRemote,
pushLocalChanges,
waitForLiveSyncCoreReady,
} from "../runner/liveSyncWorkflow.ts";
import { startObsidianLiveSyncSession, type ObsidianLiveSyncSession } from "../runner/session.ts";
import { openLiveSyncSettings, waitForVisibleObsidianDialogue, withObsidianPage } from "../runner/ui.ts";
import { createTemporaryVault, type TemporaryVault } from "../runner/vault.ts";
process.env.E2E_OBSIDIAN_CLI_TIMEOUT_MS ??= "60000";
const hiddenPaths = [".metadata-migration/retained.json", ".metadata-migration/rewritten.json"];
const customPaths = [".obsidian/snippets/retained-metadata.css", ".obsidian/snippets/rewritten-metadata.css"];
const paths = [...hiddenPaths, ...customPaths];
const initialContent = "/* Metadata migration fixture */\n";
const updatedContent = "/* Updated after enabling internal Metadata encryption */\n";
const optionSettings = {
encrypt: true,
passphrase: "internal-metadata-migration-secret",
usePathObfuscation: true,
encryptInternalMetadata: false,
syncInternalFiles: true,
syncInternalFilesBeforeReplication: false,
watchInternalFileChanges: false,
syncInternalFilesTargetPatterns: "^\\.metadata-migration(?:/|$)",
usePluginSync: true,
usePluginSyncV2: true,
autoSweepPlugins: false,
autoSweepPluginsPeriodic: false,
autoAcceptCompatibleTweak: false,
};
type Entry = { id: string; path: string };
async function main(): Promise<void> {
const binary = requireObsidianBinary();
const cliBinary = discoverObsidianCli().binary;
if (!cliBinary) throw new Error("The Obsidian CLI is unavailable.");
const couchDb = await loadCouchDbConfig();
const dbName = makeUniqueDatabaseName(couchDb.dbPrefix, "internal-metadata-migration");
const connection = { ...couchDb, dbName };
const source = await createTemporaryVault();
const target = await createTemporaryVault();
let session: ObsidianLiveSyncSession | undefined;
const evaluate = async <T>(body: string): Promise<T> => {
if (!session) throw new Error("No active Obsidian session.");
return await evalObsidianJson<T>(
cliBinary,
`(async()=>{const core=app.plugins.plugins['obsidian-livesync'].core;${body}})()`,
session.cliEnv
);
};
const start = async (vault: TemporaryVault, device: string) => {
const settings = { ...optionSettings, deviceAndVaultName: device };
session = await startObsidianLiveSyncSession({
binary,
cliBinary,
vault,
pluginData: createE2eCouchDbPluginData(connection, settings),
localStorageEntries: createE2eObsidianDeviceLocalState(vault.name),
});
await waitForLiveSyncCoreReady(cliBinary, session.cliEnv);
await configureCouchDb(cliBinary, session.cliEnv, connection, settings);
await evaluate(`core.services.setting.setDeviceAndVaultName(${JSON.stringify(device)});
await core.services.setting.saveSettingData(); return JSON.stringify(true);`);
await prepareRemote(cliBinary, session.cliEnv);
};
const store = async (customisations: string[]) => {
return await evaluate<Entry[]>(`
await core.getAddOn('HiddenFileSync').scanAllStorageChanges(true);
const config=core.getAddOn('ConfigSync');
for(const path of ${JSON.stringify(customisations)}){
await config.storeCustomizationFiles(path,core.services.setting.getDeviceAndVaultName());
}
const rows=(await core.localDatabase.allDocsRaw({include_docs:true})).rows;
const entries=${JSON.stringify(paths)}.map(path=>rows.map(row=>row.doc).find(doc=>
doc?.path==='i:'+path || doc?.path?.startsWith('ix:migration-source/') && doc.path.endsWith('%'+path.split('/').pop())));
if(entries.some(entry=>!entry)) throw new Error('Missing internal Metadata fixtures: '+JSON.stringify({entries,paths:rows.map(row=>row.doc?.path).filter(Boolean)}));
return JSON.stringify(entries.map(doc=>({id:doc._id,path:doc.path})));`);
};
const preferCurrentSettings = async () => {
await evaluate(`await core.services.replicator.getActiveReplicator()
.setPreferredRemoteTweakSettings(core.services.setting.currentSettings()); return JSON.stringify(true);`);
};
const applyAndCheckFiles = async () => {
await evaluate(`
await core.getAddOn('HiddenFileSync').scanAllDatabaseChanges(true);
const config=core.getAddOn('ConfigSync');
const rows=(await core.localDatabase.allDocsRaw({include_docs:true})).rows;
for(const path of ${JSON.stringify(customPaths)}){
const entry=rows.map(row=>row.doc).find(doc=>doc?.path?.startsWith('ix:migration-source/') && doc.path.endsWith('%'+path.split('/').pop()));
if(!entry) throw new Error('Missing Customisation Sync Metadata');
const display=config.createPluginDataFromV2(entry.path);
await display.setFile(await config.createPluginDataExFileV2(entry.path));
if(!(await config.applyDataV2(display))) throw new Error('Could not apply Customisation Sync data');
}
return JSON.stringify(true);`);
for (const path of paths) {
assertEqual(
await readFile(join(target.path, path), "utf8"),
path.includes("rewritten") ? updatedContent : initialContent,
`Unexpected restored content: ${path}`
);
}
};
const assertDeclaration = async () => {
const version = await fetchCouchDbDocument(couchDb, dbName, VERSIONING_DOCID);
assertEqual(version.version, 13, "The feature generation was not retained.");
assertEqual(
(version.used_features as string[]).includes(ENCRYPTED_INTERNAL_METADATA_FEATURE),
true,
"The encrypted internal Metadata declaration was not retained."
);
};
try {
await assertCouchDbReachable(couchDb);
await createCouchDbDatabase(couchDb, dbName);
for (const path of paths) {
await mkdir(dirname(join(source.path, path)), { recursive: true });
await writeFile(join(source.path, path), initialContent);
}
await start(source, "migration-source");
const entries = await store(customPaths);
await pushLocalChanges(cliBinary, session!.cliEnv);
const originals = await Promise.all(entries.map((entry) => fetchCouchDbDocument(couchDb, dbName, entry.id)));
for (let index = 0; index < entries.length; index++) {
assertEqual(originals[index].path, entries[index].path, "OFF unexpectedly encrypted internal Metadata.");
}
assertEqual(
(await fetchCouchDbDocument(couchDb, dbName, VERSIONING_DOCID)).version,
12,
"The original database was not generation 12."
);
await withObsidianPage(session!.remoteDebuggingPort, async (page) => {
const navigator = await openLiveSyncSettings(page);
const remotePage = await navigator.openPage("Remote Configuration");
await remotePage
.locator(".setting-item")
.filter({
has: navigator.page.getByText("Configure E2EE", { exact: true }),
})
.getByRole("button", { name: "Configure", exact: true })
.click();
const dialog = await waitForVisibleObsidianDialogue(navigator.page, "End-to-End Encryption");
await dialog.getByLabel("Encrypt internal file Properties", { exact: true }).check();
await dialog.getByRole("button", { name: "Proceed", exact: true }).click();
const warning = await waitForVisibleObsidianDialogue(navigator.page, "Encrypt internal file Properties");
await warning
.getByRole("button", {
name: "Enable without rebuilding — update every other device first",
exact: true,
})
.click();
});
assertEqual(
await evaluate(`app.setting.close(); return JSON.stringify(core.settings.encryptInternalMetadata);`),
true,
"The setting dialogue did not enable encryption."
);
for (let index = 0; index < entries.length; index++) {
assertEqual(
(await fetchCouchDbDocument(couchDb, dbName, entries[index].id))._rev,
originals[index]._rev,
"Enabling without rebuilding rewrote an existing document."
);
}
await preferCurrentSettings();
for (const path of [hiddenPaths[1], customPaths[1]]) await writeFile(join(source.path, path), updatedContent);
const rewritten = await store([customPaths[1]]);
assertEqual(
JSON.stringify(rewritten),
JSON.stringify(entries),
"Enabling encryption changed document IDs or paths."
);
await pushLocalChanges(cliBinary, session!.cliEnv);
for (let index = 0; index < entries.length; index++) {
const raw = await fetchCouchDbDocument(couchDb, dbName, entries[index].id);
if (index % 2 === 0) {
assertEqual(raw._rev, originals[index]._rev, "An untouched document was rewritten.");
assertEqual(raw.path, entries[index].path, "An untouched document lost its plaintext Metadata.");
} else {
assertEqual(raw.path?.startsWith("/\\:"), true, "Updated Metadata was not encrypted.");
assertEqual(
JSON.stringify([raw.ctime, raw.mtime, raw.size, raw.children]),
"[0,0,0,[]]",
"Updated Metadata exposed file properties."
);
}
}
await assertDeclaration();
console.log(
"The settings UI enabled encryption without Rebuild; unchanged and encrypted Metadata coexist with stable IDs."
);
await session!.app.stop();
session = undefined;
await start(target, "migration-target");
const rejected = evaluate<boolean>(`return JSON.stringify(await core.services.replication.replicate(true));`);
await withObsidianPage(session!.remoteDebuggingPort, async (page) => {
const dialog = await waitForVisibleObsidianDialogue(page, "Configuration Mismatch Detected");
await dialog
.getByText("Encrypt internal file Properties", { exact: false })
.first()
.waitFor({ state: "visible" });
await dialog.getByRole("button", { name: "Dismiss", exact: true }).click();
});
assertEqual(await rejected, false, "Mismatched settings admitted replication.");
assertEqual(
await evaluate(`const rows=(await core.localDatabase.allDocsRaw({include_docs:true})).rows;
return JSON.stringify(rows.some(row=>${JSON.stringify(entries.map((entry) => entry.id))}.includes(row.id)));`),
false,
"The mismatched device received internal Metadata."
);
await evaluate(`await core.services.setting.applyPartial({encryptInternalMetadata:true},true);
return JSON.stringify(true);`);
await pushLocalChanges(cliBinary, session!.cliEnv);
await applyAndCheckFiles();
console.log("A second device rejected the mismatch, then restored both formats after setting alignment.");
await evaluate(`await core.services.setting.applyPartial({encryptInternalMetadata:false},true);
return JSON.stringify(true);`);
await preferCurrentSettings();
await evaluate(`await core.rebuilder.$fetchLocalDBFast(true);
await core.services.setting.applyPartial(${JSON.stringify(optionSettings)},true);
return JSON.stringify(true);`);
const fetchedEntries = await evaluate<LoadedEntry[]>(`
const entries=[];
for(const path of ${JSON.stringify(entries.map((entry) => entry.path))}){
const entry=await core.localDatabase.getDBEntry(path,undefined,false,true);
if(!entry || entry.deleted || entry._deleted) throw new Error('Could not read fetched Metadata: '+path);
const file=path.startsWith('ix:')
? await core.getAddOn('ConfigSync').createPluginDataExFileV2(path,entry) : entry;
if(!file) throw new Error('Could not decode fetched Customisation Sync content: '+path);
entries.push(file);
}
return JSON.stringify(entries);`);
for (let index = 0; index < fetchedEntries.length; index++) {
const expected = index % 2 === 0 ? initialContent : updatedContent;
const content = readContent(fetchedEntries[index]);
const text = typeof content === "string" ? content : new TextDecoder().decode(content);
assertEqual(
text,
expected,
`OFF did not read internal file content after Fast Fetch: ${entries[index].path}`
);
}
await applyAndCheckFiles();
await assertDeclaration();
console.log(
"Fast Fetch and database content reads accept both formats with the option OFF; the remote declaration remains."
);
} finally {
await session?.app.stop();
await source.dispose();
await target.dispose();
await deleteCouchDbDatabase(couchDb, dbName);
}
}
main().catch((error: unknown) => {
console.error(error instanceof Error ? error.stack : error);
process.exitCode = 1;
});
test/e2e-obsidian/scripts/minio-upload.ts
+41 -4
View File
@@ -15,6 +15,8 @@
* Separate successes would not prove that those observations belonged to the
* same upload.
*/
import { randomBytes } from "node:crypto";
import { deriveIdKey } from "@vrtmrz/livesync-commonlib/settings";
import { evalObsidianJson } from "../runner/cli.ts";
import { discoverObsidianCli, requireObsidianBinary } from "../runner/environment.ts";
import {
@@ -33,6 +35,7 @@ import {
listObjectStorageObjects,
loadObjectStorageConfig,
makeUniqueBucketPrefix,
readObjectStorageJson,
} from "../runner/objectStorage.ts";
import { startObsidianLiveSyncSession, type ObsidianLiveSyncSession } from "../runner/session.ts";
import { createTemporaryVault } from "../runner/vault.ts";
@@ -41,6 +44,8 @@ import { REMOTE_ACTIVITY_EXPECTED_STATE, waitForRemoteActivityState } from "../r
process.env.E2E_OBSIDIAN_CLI_TIMEOUT_MS ??= "30000";
const notePath = "E2E/minio-upload.md";
const useIndependentIds = process.env.E2E_OBSIDIAN_INDEPENDENT_IDS === "true";
const useCustomRequestHandler = process.env.E2E_OBSIDIAN_CUSTOM_HTTP_HANDLER === "true";
const noteContent = [
"# Object Storage upload from real Obsidian",
"",
@@ -127,10 +132,23 @@ async function main(): Promise<void> {
});
await waitForLiveSyncCoreReady(cli.binary, session.cliEnv);
const configured = await configureObjectStorage(cli.binary, session.cliEnv, {
...objectStorage,
bucketPrefix,
});
const configured = await configureObjectStorage(
cli.binary,
session.cliEnv,
{ ...objectStorage, bucketPrefix },
{
...(useIndependentIds
? {
encrypt: true,
usePathObfuscation: true,
passphrase: randomBytes(32).toString("base64url"),
idDerivationVersion: 1,
idDerivationKey: await deriveIdKey(randomBytes(32).toString("base64url")),
}
: {}),
...(useCustomRequestHandler ? { useCustomRequestHandler: true } : {}),
}
);
await waitForLiveSyncCoreReady(cli.binary, session.cliEnv);
assertEqual(configured.isConfigured, true, "Self-hosted LiveSync was not marked as configured.");
assertEqual(configured.remoteType, "MINIO", "Remote type was not Object Storage.");
@@ -145,6 +163,16 @@ async function main(): Promise<void> {
REMOTE_ACTIVITY_EXPECTED_STATE.idle
);
const localEntry = await createNoteAndWaitForLocalDb(cli.binary, session.cliEnv);
if (useIndependentIds) {
if (
!/^f:[0-9a-f]{64}$/u.test(localEntry.id) ||
localEntry.children.some((child) => !/^h:\+[0-9a-f]{64}$/u.test(child))
) {
throw new Error(
`The real Obsidian Journal upload did not use independent document and Chunk IDs (document length ${localEntry.id.length}, Chunk lengths ${localEntry.children.map((child) => child.length).join(",")}).`
);
}
}
await pushLocalChanges(cli.binary, session.cliEnv);
const activityAfterUpload = await waitForRemoteActivityState(
session.remoteDebuggingPort,
@@ -160,6 +188,15 @@ async function main(): Promise<void> {
);
const keys = await waitForObjectStorageObjects(bucketPrefix);
if (useIndependentIds) {
const milestone = await readObjectStorageJson<{ encrypted_id_derivation_proof?: string }>(
objectStorage,
`${bucketPrefix}_00000000-milestone.json`
);
if (!milestone.encrypted_id_derivation_proof) {
throw new Error("The Journal milestone did not retain an encrypted ID agreement proof.");
}
}
console.log(
`Uploaded ${localEntry.path} through Journal Sync to ${objectStorage.bucket}/${bucketPrefix} (${keys.length} object(s)); tracked requests: ${activityAfterUpload.requestCount - activityBeforeUpload.requestCount}`
test/e2e-obsidian/scripts/received-change-readiness.ts
+420
View File
@@ -0,0 +1,420 @@
import { readFile } from "node:fs/promises";
import { join } from "node:path";
import { EVENT_APPLICATION_READY } from "@vrtmrz/livesync-commonlib/compat/events/coreEvents";
import { evalObsidianJson } from "../runner/cli.ts";
import {
assertCouchDbReachable,
createCouchDbDatabase,
deleteCouchDbDatabase,
fetchCouchDbDocument,
loadCouchDbConfig,
makeUniqueDatabaseName,
putCouchDbDocument,
waitForCouchDbDocs,
type CouchDbConfig,
type CouchDbDocument,
} from "../runner/couchdb.ts";
import { discoverObsidianCli, requireObsidianBinary } from "../runner/environment.ts";
import {
assertEqual,
createE2eCouchDbPluginData,
createE2eObsidianDeviceLocalState,
prepareRemote,
pushLocalChanges,
waitForLiveSyncCoreReady,
waitForLocalDatabaseEntry,
type LocalDatabaseEntry,
} from "../runner/liveSyncWorkflow.ts";
import { startObsidianLiveSyncSession, type ObsidianLiveSyncSession } from "../runner/session.ts";
import { createTemporaryVault, type TemporaryVault } from "../runner/vault.ts";
process.env.E2E_OBSIDIAN_COUCHDB_TIMEOUT_MS ??= "20000";
const observerKey = "__livesyncE2eReceivedChangeReadiness";
const readyPath = "E2E/received-change-readiness/ready.md";
const suspendedPath = "E2E/received-change-readiness/suspended.md";
const readyContent = [
"# Received before readiness",
"",
"This note is replicated into the target database before the application readiness event.",
"Its content is longer than one configured chunk so the test uses the ordinary Chunk path.",
"",
].join("\n");
const suspendedContent = [
"# Received during explicit suspension",
"",
"This note remains queued while database reflecting is explicitly suspended.",
"Resuming result application must reflect the received metadata and its stored Chunks.",
"",
].join("\n");
type ReadinessObservation = {
readinessEvents: number;
content: string | null;
};
type CapturedNote = {
entry: LocalDatabaseEntry;
documents: CouchDbDocument[];
};
async function waitForVaultContent(
vault: TemporaryVault,
path: string,
expected: string,
timeoutMs = Number(process.env.E2E_OBSIDIAN_FILE_TIMEOUT_MS ?? 10000)
): Promise<void> {
const fullPath = join(vault.path, path);
const deadline = Date.now() + timeoutMs;
let lastContent: string | null = null;
while (Date.now() < deadline) {
try {
lastContent = await readFile(fullPath, "utf8");
if (lastContent === expected) return;
} catch (error) {
if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error;
}
await new Promise((resolve) => setTimeout(resolve, 100));
}
throw new Error(`Timed out waiting for ${path} in the target Vault. Last content: ${String(lastContent)}`);
}
async function assertVaultPathStaysMissing(vault: TemporaryVault, path: string, durationMs: number): Promise<void> {
const fullPath = join(vault.path, path);
const deadline = Date.now() + durationMs;
while (Date.now() < deadline) {
try {
const content = await readFile(fullPath, "utf8");
throw new Error(`The suspended or unready document was reflected early at ${path}: ${content}`);
} catch (error) {
if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error;
}
await new Promise((resolve) => setTimeout(resolve, 100));
}
}
async function createAndUploadNote(
cliBinary: string,
env: NodeJS.ProcessEnv,
couchDb: CouchDbConfig,
dbName: string,
path: string,
content: string
): Promise<CapturedNote> {
const created = await evalObsidianJson<{ path: string; content: string }>(
cliBinary,
`(async()=>{const path=${JSON.stringify(path)};const folders=path.split('/');for(let i=1;i<folders.length;i++){const folder=folders.slice(0,i).join('/');if(!app.vault.getAbstractFileByPath(folder)) await app.vault.createFolder(folder);}const file=await app.vault.create(path,${JSON.stringify(content)});return JSON.stringify({path:file.path,content:await app.vault.read(file)});})()`,
env
);
assertEqual(created.path, path, "Obsidian created the source note at an unexpected path.");
assertEqual(created.content, content, "Obsidian did not read back the source note content.");
const entry = await waitForLocalDatabaseEntry(cliBinary, env, path);
if (entry.children.length === 0) throw new Error(`The source note did not create Chunks: ${path}`);
await pushLocalChanges(cliBinary, env);
await waitForCouchDbDocs(couchDb, dbName, (documents) => {
const ids = new Set(documents.map((document) => document._id));
return ids.has(entry.id) && entry.children.every((childId) => ids.has(childId));
});
const documents = await Promise.all(
[...new Set([entry.id, ...entry.children])].map((documentId) =>
fetchCouchDbDocument(couchDb, dbName, documentId)
)
);
return { entry, documents };
}
async function injectCapturedNote(
couchDb: CouchDbConfig,
dbName: string,
note: CapturedNote,
publishedChunkIds: Set<string>
): Promise<void> {
const chunks = note.documents.filter((document) => document._id !== note.entry.id);
const metadata = note.documents.find((document) => document._id === note.entry.id);
if (!metadata) throw new Error(`The source note metadata was not captured: ${note.entry.path}`);
for (const document of chunks) {
if (publishedChunkIds.has(document._id)) continue;
const freshDocument = { ...document };
delete freshDocument._rev;
await putCouchDbDocument(couchDb, dbName, freshDocument);
publishedChunkIds.add(document._id);
}
const freshMetadata = { ...metadata };
delete freshMetadata._rev;
await putCouchDbDocument(couchDb, dbName, freshMetadata);
}
async function installObserver(cliBinary: string, env: NodeJS.ProcessEnv): Promise<void> {
await evalObsidianJson(
cliBinary,
[
"(()=>{",
`const key=${JSON.stringify(observerKey)};`,
`const readyEvent=${JSON.stringify(EVENT_APPLICATION_READY)};`,
"const previous=globalThis[key];",
"if(previous) previous.unsubscribeReady?.();",
"const state={readinessEvents:0,unsubscribeReady:null};",
"state.unsubscribeReady=app.plugins.plugins['obsidian-livesync'].core.services.context.events.onEvent(readyEvent,()=>state.readinessEvents++);",
"globalThis[key]=state;",
"return JSON.stringify(true);",
"})()",
].join(""),
env
);
}
async function readObservation(cliBinary: string, env: NodeJS.ProcessEnv, path: string): Promise<ReadinessObservation> {
return await evalObsidianJson<ReadinessObservation>(
cliBinary,
[
"(async()=>{",
`const key=${JSON.stringify(observerKey)};`,
`const path=${JSON.stringify(path)};`,
"const state=globalThis[key];",
"const file=app.vault.getAbstractFileByPath(path);",
"return JSON.stringify({readinessEvents:state?.readinessEvents??0,content:file?await app.vault.read(file):null});",
"})()",
].join(""),
env
);
}
async function resetReadiness(cliBinary: string, env: NodeJS.ProcessEnv, suspendReflecting: boolean): Promise<void> {
const result = await evalObsidianJson<{ ready: boolean }>(
cliBinary,
[
"(async()=>{",
"const core=app.plugins.plugins['obsidian-livesync'].core;",
...(suspendReflecting
? [
"await core.services.setting.applyPartial({suspendParseReplicationResult:true},true);",
"await core.services.control.applySettings();",
"await core.services.replication.startContinuous({trigger:'daemon',interaction:{kind:'forbidden'}});",
]
: []),
"core.services.appLifecycle.resetIsReady();",
"return JSON.stringify({ready:core.services.appLifecycle.isReady()});",
"})()",
].join(""),
env
);
assertEqual(result.ready, false, "The target application remained ready after resetIsReady().");
}
async function waitForReceivedWhileUnready(
cliBinary: string,
env: NodeJS.ProcessEnv,
target: TemporaryVault,
path: string
): Promise<void> {
const entry = await waitForLocalDatabaseEntry(cliBinary, env, path);
if (entry.children.length === 0) throw new Error(`The target received metadata without Chunks: ${path}`);
const readiness = await evalObsidianJson<{ ready: boolean }>(
cliBinary,
"(()=>JSON.stringify({ready:app.plugins.plugins['obsidian-livesync'].core.services.appLifecycle.isReady()}))()",
env
);
assertEqual(readiness.ready, false, `The target became ready before applying ${path}.`);
await assertVaultPathStaysMissing(target, path, 750);
}
async function startContinuousReplication(cliBinary: string, env: NodeJS.ProcessEnv): Promise<void> {
const result = await evalObsidianJson<{ status: string }>(
cliBinary,
[
"(async()=>{",
"const core=app.plugins.plugins['obsidian-livesync'].core;",
"await core.services.setting.applyExternalSettings({liveSync:true},true);",
"await core.services.control.applySettings();",
"const result=await core.services.replication.startContinuous({trigger:'daemon',interaction:{kind:'forbidden'}});",
"return JSON.stringify(result);",
"})()",
].join(""),
env
);
assertEqual(result.status, "completed", `Continuous replication did not start: ${JSON.stringify(result)}.`);
const deadline = Date.now() + Number(process.env.E2E_OBSIDIAN_REMOTE_ACTIVITY_TIMEOUT_MS ?? 30000);
let active = false;
while (!active && Date.now() < deadline) {
active = await evalObsidianJson<boolean>(
cliBinary,
"(()=>JSON.stringify(!!app.plugins.plugins['obsidian-livesync'].core.services.replicator.getActiveReplicator()))()",
env
);
if (!active) await new Promise((resolve) => setTimeout(resolve, 250));
}
if (!active) throw new Error("Timed out waiting for the target Replicator to become active.");
}
async function markReadyTwice(cliBinary: string, env: NodeJS.ProcessEnv): Promise<void> {
const result = await evalObsidianJson<{ ready: boolean }>(
cliBinary,
[
"(()=>{",
"const lifecycle=app.plugins.plugins['obsidian-livesync'].core.services.appLifecycle;",
"lifecycle.markIsReady();",
"lifecycle.markIsReady();",
"return JSON.stringify({ready:lifecycle.isReady()});",
"})()",
].join(""),
env
);
assertEqual(result.ready, true, "Commonlib did not establish application readiness.");
}
async function removeObserver(cliBinary: string, env: NodeJS.ProcessEnv): Promise<void> {
await evalObsidianJson(
cliBinary,
[
"(()=>{",
`const key=${JSON.stringify(observerKey)};`,
"const state=globalThis[key];",
"if(state){state.unsubscribeReady?.();delete globalThis[key];}",
"return JSON.stringify(true);",
"})()",
].join(""),
env
);
}
async function assertApplied(
cliBinary: string,
env: NodeJS.ProcessEnv,
vault: TemporaryVault,
path: string,
expectedContent: string,
expectedReadyEvents: number
): Promise<void> {
const beforeReflection = await readObservation(cliBinary, env, path);
assertEqual(
beforeReflection.readinessEvents,
expectedReadyEvents,
"The expected Commonlib readiness transition was not observed before reflection."
);
await waitForVaultContent(vault, path, expectedContent);
await new Promise((resolve) => setTimeout(resolve, 500));
const observation = await readObservation(cliBinary, env, path);
assertEqual(observation.content, expectedContent, `Obsidian Vault read-back was incorrect for ${path}.`);
assertEqual(
observation.readinessEvents,
expectedReadyEvents,
"markIsReady emitted an unexpected number of readiness transitions."
);
}
async function main(): Promise<void> {
const binary = requireObsidianBinary();
const cli = discoverObsidianCli();
if (!cli.binary) throw new Error(`Could not find obsidian-cli. Checked paths: ${cli.checked.join(", ")}`);
const couchDb = await loadCouchDbConfig();
const sourceDbName = makeUniqueDatabaseName(couchDb.dbPrefix, "received-change-readiness-source");
const targetDbName = makeUniqueDatabaseName(couchDb.dbPrefix, "received-change-readiness-target");
const couchDbSettings = {
uri: couchDb.uri,
username: couchDb.username,
password: couchDb.password,
};
const sourceCouchDbSettings = { ...couchDbSettings, dbName: sourceDbName };
const targetCouchDbSettings = { ...couchDbSettings, dbName: targetDbName };
const sourceVault = await createTemporaryVault("obsidian-livesync-readiness-source-");
const targetVault = await createTemporaryVault("obsidian-livesync-readiness-target-");
let source: ObsidianLiveSyncSession | undefined;
let target: ObsidianLiveSyncSession | undefined;
const publishedChunkIds = new Set<string>();
try {
await assertCouchDbReachable(couchDb);
await createCouchDbDatabase(couchDb, sourceDbName);
await createCouchDbDatabase(couchDb, targetDbName);
source = await startObsidianLiveSyncSession({
binary,
cliBinary: cli.binary,
vault: sourceVault,
startupGraceMs: Number(process.env.E2E_OBSIDIAN_STARTUP_GRACE_MS ?? 1000),
pluginData: createE2eCouchDbPluginData(sourceCouchDbSettings),
localStorageEntries: createE2eObsidianDeviceLocalState(sourceVault.name),
});
await waitForLiveSyncCoreReady(cli.binary, source.cliEnv);
await prepareRemote(cli.binary, source.cliEnv);
const readyNote = await createAndUploadNote(
cli.binary,
source.cliEnv,
couchDb,
sourceDbName,
readyPath,
readyContent
);
const suspendedNote = await createAndUploadNote(
cli.binary,
source.cliEnv,
couchDb,
sourceDbName,
suspendedPath,
suspendedContent
);
await source.app.stop();
source = undefined;
target = await startObsidianLiveSyncSession({
binary,
cliBinary: cli.binary,
vault: targetVault,
startupGraceMs: Number(process.env.E2E_OBSIDIAN_STARTUP_GRACE_MS ?? 1000),
pluginData: createE2eCouchDbPluginData(targetCouchDbSettings),
localStorageEntries: createE2eObsidianDeviceLocalState(targetVault.name),
});
await waitForLiveSyncCoreReady(cli.binary, target.cliEnv);
await prepareRemote(cli.binary, target.cliEnv);
await installObserver(cli.binary, target.cliEnv);
await startContinuousReplication(cli.binary, target.cliEnv);
await resetReadiness(cli.binary, target.cliEnv, false);
await injectCapturedNote(couchDb, targetDbName, readyNote, publishedChunkIds);
await waitForReceivedWhileUnready(cli.binary, target.cliEnv, targetVault, readyPath);
await markReadyTwice(cli.binary, target.cliEnv);
await assertApplied(cli.binary, target.cliEnv, targetVault, readyPath, readyContent, 1);
await resetReadiness(cli.binary, target.cliEnv, true);
await injectCapturedNote(couchDb, targetDbName, suspendedNote, publishedChunkIds);
await waitForReceivedWhileUnready(cli.binary, target.cliEnv, targetVault, suspendedPath);
await markReadyTwice(cli.binary, target.cliEnv);
await assertVaultPathStaysMissing(targetVault, suspendedPath, 1000);
const suspendedObservation = await readObservation(cli.binary, target.cliEnv, suspendedPath);
assertEqual(suspendedObservation.readinessEvents, 2, "The resumed-ready transition was not observed once.");
assertEqual(suspendedObservation.content, null, "Readiness bypassed explicit database-reflection suspension.");
await evalObsidianJson(
cli.binary,
"(async()=>{const setting=app.plugins.plugins['obsidian-livesync'].core.services.setting;await setting.applyPartial({suspendParseReplicationResult:false},true);return JSON.stringify(true);})()",
target.cliEnv
);
await assertApplied(cli.binary, target.cliEnv, targetVault, suspendedPath, suspendedContent, 2);
console.log(
"Received-change readiness: queued changes resumed once per readiness transition; explicit suspension held until settings resumed."
);
} finally {
if (target) {
await removeObserver(cli.binary, target.cliEnv).catch(() => undefined);
await target.app.stop();
}
if (source) await source.app.stop();
await sourceVault.dispose();
await targetVault.dispose();
if (process.env.E2E_OBSIDIAN_KEEP_COUCHDB !== "true") {
await Promise.all(
[sourceDbName, targetDbName].map((dbName) => deleteCouchDbDatabase(couchDb, dbName))
).catch((error: unknown) => {
console.warn(error instanceof Error ? error.message : error);
});
}
}
}
main().catch((error: unknown) => {
console.error(error instanceof Error ? error.stack : error);
process.exit(1);
});
test/e2e-obsidian/scripts/remote-feature-change.ts
+218
View File
@@ -0,0 +1,218 @@
import { mkdir, readFile, writeFile } from "node:fs/promises";
import { dirname, join } from "node:path";
import { VERSIONING_DOCID } from "@vrtmrz/livesync-commonlib/compat/common/types";
import { evalObsidianJson } from "../runner/cli.ts";
import {
assertCouchDbReachable,
createCouchDbDatabase,
deleteCouchDbDatabase,
fetchCouchDbDocument,
loadCouchDbConfig,
makeUniqueDatabaseName,
putCouchDbDocument,
} from "../runner/couchdb.ts";
import { discoverObsidianCli, requireObsidianBinary } from "../runner/environment.ts";
import {
assertE2eCompatibilityMarker,
configureCouchDb,
createE2eCouchDbPluginData,
createE2eObsidianDeviceLocalState,
prepareRemote,
pushLocalChanges,
waitForLiveSyncCoreReady,
waitForLocalDatabaseEntry,
} from "../runner/liveSyncWorkflow.ts";
import { startObsidianLiveSyncSession, type ObsidianLiveSyncSession } from "../runner/session.ts";
import { createTemporaryVault } from "../runner/vault.ts";
const acceptedPath = "E2E/remote-feature/accepted.md";
const acceptedContent = "Accepted before the remote feature changed.\n";
const unknownFeature = "future-format-v7";
type FeatureState = {
version: number | null;
features: string[];
hasActiveReplicator: boolean;
};
async function readFeatureState(cliBinary: string, env: NodeJS.ProcessEnv): Promise<FeatureState> {
return await evalObsidianJson<FeatureState>(
cliBinary,
[
"(async()=>{",
"const core=app.plugins.plugins['obsidian-livesync'].core;",
`const id=${JSON.stringify(VERSIONING_DOCID)};`,
"const info=await core.localDatabase.getRaw(id).catch(()=>null);",
"return JSON.stringify({",
"version:typeof info?.version==='number'?info.version:null,",
"features:Array.isArray(info?.used_features)?info.used_features:[],",
"hasActiveReplicator:!!core.services.replicator.getActiveReplicator(),",
"});",
"})()",
].join(""),
env
);
}
async function waitForState(
cliBinary: string,
env: NodeJS.ProcessEnv,
predicate: (state: FeatureState) => boolean,
description: string
): Promise<FeatureState> {
const deadline = Date.now() + 20_000;
let state = await readFeatureState(cliBinary, env);
while (!predicate(state) && Date.now() < deadline) {
await new Promise((resolve) => setTimeout(resolve, 250));
state = await readFeatureState(cliBinary, env);
}
if (!predicate(state)) throw new Error(`Timed out waiting for ${description}: ${JSON.stringify(state)}`);
return state;
}
async function main(): Promise<void> {
const binary = requireObsidianBinary();
const cli = discoverObsidianCli();
if (!cli.binary) throw new Error(`Could not find obsidian-cli. Checked paths: ${cli.checked.join(", ")}`);
const couchDb = await loadCouchDbConfig();
const dbName = makeUniqueDatabaseName(couchDb.dbPrefix, "remote-feature-change");
const vault = await createTemporaryVault();
let session: ObsidianLiveSyncSession | undefined;
try {
await assertCouchDbReachable(couchDb);
await createCouchDbDatabase(couchDb, dbName);
const couchDbSettings = {
uri: couchDb.uri,
username: couchDb.username,
password: couchDb.password,
dbName,
};
const settings = {
encrypt: false,
usePathObfuscation: false,
encryptInternalMetadata: false,
liveSync: false,
};
session = await startObsidianLiveSyncSession({
binary,
cliBinary: cli.binary,
vault,
startupGraceMs: Number(process.env.E2E_OBSIDIAN_STARTUP_GRACE_MS ?? 1000),
pluginData: createE2eCouchDbPluginData(couchDbSettings, settings),
localStorageEntries: createE2eObsidianDeviceLocalState(vault.name),
});
await waitForLiveSyncCoreReady(cli.binary, session.cliEnv);
await assertE2eCompatibilityMarker(cli.binary, session.cliEnv);
await configureCouchDb(cli.binary, session.cliEnv, couchDbSettings, settings);
await prepareRemote(cli.binary, session.cliEnv);
const fullPath = join(vault.path, acceptedPath);
await mkdir(dirname(fullPath), { recursive: true });
await writeFile(fullPath, acceptedContent, "utf-8");
await waitForLocalDatabaseEntry(cli.binary, session.cliEnv, acceptedPath);
await pushLocalChanges(cli.binary, session.cliEnv);
const initialVersion = await fetchCouchDbDocument(couchDb, dbName, VERSIONING_DOCID);
if (initialVersion.version !== 12 || "used_features" in initialVersion) {
throw new Error(
`An inactive feature unexpectedly changed the remote contract: ${JSON.stringify(initialVersion)}`
);
}
const start = await evalObsidianJson<{ status: string }>(
cli.binary,
[
"(async()=>{",
"const core=app.plugins.plugins['obsidian-livesync'].core;",
"await core.services.setting.applyExternalSettings({liveSync:true},true);",
"await core.services.control.applySettings();",
"const result=await core.services.replication.startContinuous({trigger:'daemon',interaction:{kind:'forbidden'}});",
"return JSON.stringify(result);",
"})()",
].join(""),
session.cliEnv
);
if (start.status !== "completed")
throw new Error(`Continuous replication did not start: ${JSON.stringify(start)}`);
await waitForState(cli.binary, session.cliEnv, (state) => state.hasActiveReplicator, "an active Replicator");
await putCouchDbDocument(couchDb, dbName, {
...initialVersion,
version: 13,
used_features: [unknownFeature],
});
const observed = await waitForState(
cli.binary,
session.cliEnv,
(state) => state.version === 13 && state.features.includes(unknownFeature) && !state.hasActiveReplicator,
"the live feature change and Replicator retirement"
);
const replicated = await evalObsidianJson<boolean>(
cli.binary,
"(async()=>JSON.stringify(!!(await app.plugins.plugins['obsidian-livesync'].core.services.replication.replicate(true))))()",
session.cliEnv
);
if (replicated) throw new Error("An unknown remote feature was admitted for another replication.");
const acceptedAfterStop = await readFile(fullPath, "utf-8");
if (acceptedAfterStop !== acceptedContent)
throw new Error("Previously accepted Vault content changed on stop.");
await session.app.stop();
session = undefined;
session = await startObsidianLiveSyncSession({ binary, cliBinary: cli.binary, vault });
await waitForLiveSyncCoreReady(cli.binary, session.cliEnv);
const afterRestart = await waitForState(
cli.binary,
session.cliEnv,
(state) => state.version === 13 && state.features.includes(unknownFeature),
"the remote feature requirement after restart"
);
const replicatedAfterRestart = await evalObsidianJson<boolean>(
cli.binary,
"(async()=>JSON.stringify(!!(await app.plugins.plugins['obsidian-livesync'].core.services.replication.replicate(true))))()",
session.cliEnv
);
const continuousAfterRestart = await evalObsidianJson<{ requestStatus: string; connected: boolean }>(
cli.binary,
[
"(async()=>{",
"const core=app.plugins.plugins['obsidian-livesync'].core;",
"const replicator=core.services.replicator.getActiveReplicator();",
"const original=replicator.openContinuousReplication;",
"let completion;",
"replicator.openContinuousReplication=function(...args){completion=original.apply(this,args);return completion;};",
"try{",
"const result=await core.services.replication.startContinuous({trigger:'daemon',interaction:{kind:'forbidden'}});",
"if(!completion) throw new Error('The continuous provider did not attempt its remote check');",
"return JSON.stringify({requestStatus:result.status,connected:await completion});",
"}finally{replicator.openContinuousReplication=original;}",
"})()",
].join(""),
session.cliEnv
);
if (replicatedAfterRestart || continuousAfterRestart.connected !== false)
throw new Error(
`Replication resumed after restart despite an unknown remote feature: ${JSON.stringify({ afterRestart, replicatedAfterRestart, continuousAfterRestart })}`
);
if ((await readFile(fullPath, "utf-8")) !== acceptedContent)
throw new Error("Previously accepted Vault content changed after restart.");
console.log(
`Active feature change retired the Replicator; the remote declaration refused synchronisation after restart: ${JSON.stringify({ observed, afterRestart })}`
);
} finally {
await session?.app.stop();
await vault.dispose();
if (process.env.E2E_OBSIDIAN_KEEP_COUCHDB !== "true") {
await deleteCouchDbDatabase(couchDb, dbName).catch((error: unknown) => {
console.warn(error instanceof Error ? error.message : error);
});
}
}
}
main().catch((error: unknown) => {
console.error(error instanceof Error ? error.stack : error);
process.exit(1);
});
test/e2e-obsidian/scripts/review-harness.ts
+114 -55
View File
@@ -6,6 +6,7 @@ import {
assertNoHorizontalOverflow,
} from "@vrtmrz/obsidian-test-session";
import { CURRENT_SETTING_VERSION } from "@vrtmrz/livesync-commonlib/compat/common/models/setting.const";
import { DoctorRegulation } from "@vrtmrz/livesync-commonlib/compat/common/configForDoc";
import { REVIEW_HARNESS_STATE_KEY } from "../../../src/features/ReviewHarness/reviewHarnessController.ts";
import { REVIEW_HARNESS_FIXTURE_ROOT } from "../../../src/features/ReviewHarness/reviewHarnessVaultFixture.ts";
import { evalObsidianJson } from "../runner/cli.ts";
@@ -166,7 +167,8 @@ async function captureReadinessFailure(
async function openHarness(): Promise<void> {
const opened = await withObsidianPage(obsidianRemoteDebuggingPort(), async (page) => {
return await page.evaluate(
(commandId) => (globalThis as ReviewHarnessTestGlobal).app?.commands?.executeCommandById(commandId) === true,
(commandId) =>
(globalThis as ReviewHarnessTestGlobal).app?.commands?.executeCommandById(commandId) === true,
"obsidian-livesync:open-review-harness"
);
});
@@ -205,11 +207,54 @@ async function runAutomaticScenarios(): Promise<void> {
});
}
async function runIdBenchmark(): Promise<void> {
await withObsidianPage(obsidianRemoteDebuggingPort(), async (page) => {
const snapshotSettings = () =>
page.evaluate(() => {
const plugin = (globalThis as ReviewHarnessTestGlobal).app?.plugins?.plugins["obsidian-livesync"] as {
core: { services: { setting: { currentSettings(): unknown } } };
};
return JSON.stringify(plugin.core.services.setting.currentSettings());
});
const before = await snapshotSettings();
const harness = page.locator('[data-testid="review-harness"]');
await harness
.locator('[data-testid="review-harness-run-id-generation-performance"]')
.click({ timeout: uiTimeoutMs });
const result = harness.locator('[data-testid="review-harness-result-id-generation-performance"]');
await result.getByText("Passed:", { exact: false }).waitFor({ state: "visible", timeout: uiTimeoutMs * 4 });
const observations = await result.locator("li").allTextContents();
for (const label of [
"Chunk IDs, 256 B",
"Chunk IDs, 4096 B",
"Chunk IDs, 32768 B",
"Obfuscated document IDs",
]) {
for (const mode of ["legacy", "independent"]) {
if (
!observations.some(
(line) =>
line.startsWith(`${label}, ${mode}: 1000 IDs total median=`) && line.includes("; per ID=")
)
) {
throw new Error(`Missing benchmark timing and units: ${label}, ${mode}`);
}
}
}
if (
!observations.some((line) => line.startsWith("ID key derivation at save time:")) ||
!observations.some((line) => line.startsWith("JavaScript heap:"))
) {
throw new Error("The benchmark did not report derivation and heap observations.");
}
if ((await snapshotSettings()) !== before) throw new Error("The benchmark changed the live settings.");
await assertNoHorizontalOverflow(page, harness, { label: "ID benchmark results" });
});
}
async function runVaultFixture(): Promise<string> {
await withObsidianPage(obsidianRemoteDebuggingPort(), async (page) => {
await page
.locator('[data-testid="review-harness-run-vault-round-trip"]')
.click({ timeout: uiTimeoutMs });
await page.locator('[data-testid="review-harness-run-vault-round-trip"]').click({ timeout: uiTimeoutMs });
const confirmation = page.locator(".modal-container").filter({
has: page.getByText("Review Harness: Vault fixture access", { exact: true }),
});
@@ -272,27 +317,22 @@ async function restartAndResumeHarness(): Promise<string> {
});
await keepCompatibilityPaused();
await waitForHarness();
return await captureObsidianDialogue(
obsidianRemoteDebuggingPort(),
"review-harness-resumed.png",
async (page) => {
const harness = page.locator('[data-testid="review-harness"]');
await harness
.locator('[data-testid="review-harness-resumed"]')
.waitFor({ state: "visible", timeout: uiTimeoutMs });
const continuationRemoved = await page.evaluate((stateKey) => {
const plugin = (globalThis as ReviewHarnessTestGlobal).app?.plugins?.plugins["obsidian-livesync"];
if (typeof plugin !== "object" || plugin === null || !("core" in plugin)) {
throw new Error("Self-hosted LiveSync is unavailable after restart.");
}
const core = (plugin as { core: { services: { setting: { getSmallConfig(key: string): string } } } })
.core;
return core.services.setting.getSmallConfig(stateKey) === "";
}, REVIEW_HARNESS_STATE_KEY);
if (!continuationRemoved) throw new Error("The one-shot continuation was not removed before use.");
await assertNoHorizontalOverflow(page, harness, { label: "resumed Review Harness" });
}
);
return await captureObsidianDialogue(obsidianRemoteDebuggingPort(), "review-harness-resumed.png", async (page) => {
const harness = page.locator('[data-testid="review-harness"]');
await harness
.locator('[data-testid="review-harness-resumed"]')
.waitFor({ state: "visible", timeout: uiTimeoutMs });
const continuationRemoved = await page.evaluate((stateKey) => {
const plugin = (globalThis as ReviewHarnessTestGlobal).app?.plugins?.plugins["obsidian-livesync"];
if (typeof plugin !== "object" || plugin === null || !("core" in plugin)) {
throw new Error("Self-hosted LiveSync is unavailable after restart.");
}
const core = (plugin as { core: { services: { setting: { getSmallConfig(key: string): string } } } }).core;
return core.services.setting.getSmallConfig(stateKey) === "";
}, REVIEW_HARNESS_STATE_KEY);
if (!continuationRemoved) throw new Error("The one-shot continuation was not removed before use.");
await assertNoHorizontalOverflow(page, harness, { label: "resumed Review Harness" });
});
}
async function completeResumedCompatibilityStep(): Promise<void> {
@@ -331,9 +371,7 @@ async function copyAndReadReport(): Promise<string> {
undefined,
{ timeout: uiTimeoutMs }
);
return await page.evaluate(
() => (globalThis as ReviewHarnessTestGlobal).reviewHarnessCopiedReport ?? ""
);
return await page.evaluate(() => (globalThis as ReviewHarnessTestGlobal).reviewHarnessCopiedReport ?? "");
});
}
@@ -347,35 +385,36 @@ async function verifyMobileHarness(): Promise<string> {
if (typeof plugin !== "object" || plugin === null || !("core" in plugin)) {
throw new Error("Self-hosted LiveSync is unavailable in mobile test mode.");
}
const core = (plugin as {
core: { services: { API: { showWindow(type: string): Promise<void> } } };
}).core;
const core = (
plugin as {
core: { services: { API: { showWindow(type: string): Promise<void> } } };
}
).core;
await core.services.API.showWindow(viewType);
}, "self-hosted-livesync-review-harness");
});
return await captureObsidianDialogue(
obsidianRemoteDebuggingPort(),
"review-harness-mobile.png",
async (page) => {
const harness = page.locator('[data-testid="review-harness"]');
await harness.waitFor({ state: "visible", timeout: uiTimeoutMs });
await assertNoHorizontalOverflow(page, harness, { label: "mobile Review Harness" });
const heading = harness.getByRole("heading", { name: "Self-hosted LiveSync review harness" });
await assertLocatorWithinSafeArea(page, heading, {
label: "mobile Review Harness heading",
safeAreaInsets: iPhoneSafeArea,
await runIdBenchmark();
return await captureObsidianDialogue(obsidianRemoteDebuggingPort(), "review-harness-mobile.png", async (page) => {
const harness = page.locator('[data-testid="review-harness"]');
await harness.waitFor({ state: "visible", timeout: uiTimeoutMs });
await harness.getByRole("heading", { name: "Self-hosted LiveSync review harness" }).scrollIntoViewIfNeeded();
await assertNoHorizontalOverflow(page, harness, { label: "mobile Review Harness" });
const heading = harness.getByRole("heading", { name: "Self-hosted LiveSync review harness" });
await assertLocatorWithinSafeArea(page, heading, {
label: "mobile Review Harness heading",
safeAreaInsets: iPhoneSafeArea,
});
for (const testId of [
"review-harness-run-automatic",
"review-harness-run-full",
"review-harness-copy-report",
"review-harness-run-id-generation-performance",
]) {
await assertLocatorHasMinimumTouchTarget(page, harness.locator(`[data-testid="${testId}"]`), {
label: testId,
});
for (const testId of [
"review-harness-run-automatic",
"review-harness-run-full",
"review-harness-copy-report",
]) {
await assertLocatorHasMinimumTouchTarget(page, harness.locator(`[data-testid="${testId}"]`), {
label: testId,
});
}
}
);
});
}
async function main(): Promise<void> {
@@ -391,7 +430,8 @@ async function main(): Promise<void> {
vault,
startupGraceMs: Number(process.env.E2E_OBSIDIAN_STARTUP_GRACE_MS ?? 1000),
pluginData: {
doctorProcessedVersion: "1.0.0",
// Config Doctor is covered by settings-ui; this fixture exercises the Harness.
doctorProcessedVersion: DoctorRegulation.version,
settingVersion: CURRENT_SETTING_VERSION,
isConfigured: true,
additionalSuffixOfDatabaseName: "",
@@ -440,15 +480,34 @@ async function main(): Promise<void> {
const vaultConfirmationScreenshot = await runVaultFixture();
const resumedScreenshot = await restartAndResumeHarness();
await completeResumedCompatibilityStep();
await runIdBenchmark();
const report = await copyAndReadReport();
if (!report.includes("## Self-hosted LiveSync Review Harness report")) {
throw new Error("The copied Review Harness report was not Markdown evidence.");
}
for (const forbidden of [vault.name, REVIEW_HARNESS_FIXTURE_ROOT]) {
if (report.includes(forbidden)) throw new Error(`The Review Harness report exposed local state: ${forbidden}`);
for (const expected of [
"1000 IDs total median=",
"; per ID=",
"ID key derivation at save time:",
"JavaScript heap:",
]) {
if (!report.includes(expected)) throw new Error(`Missing copied benchmark observation: ${expected}`);
}
for (const forbidden of [
vault.name,
REVIEW_HARNESS_FIXTURE_ROOT,
"ab".repeat(32),
"Self-hosted LiveSync ID benchmark passphrase",
"Self-hosted LiveSync ID benchmark source",
]) {
if (report.includes(forbidden))
throw new Error(`The Review Harness report exposed local state: ${forbidden}`);
}
const mobileScreenshot = await verifyMobileHarness();
const outputDirectory = process.env.E2E_OBSIDIAN_DIAGNOSTICS_DIR ?? "/tmp/obsidian-livesync-e2e";
await mkdir(outputDirectory, { recursive: true });
await writeFile(join(outputDirectory, "review-harness-report.md"), report, "utf8");
console.log(
`Review Harness passed one-shot, fixture, report, and mobile checks. Screenshots: ${[
initialScreenshot,
test/e2e-obsidian/scripts/run-focused.ts
+2
View File
@@ -17,6 +17,7 @@ const focusedScenarios = new Set([
"p2p-pane",
"vault-reflection",
"couchdb-upload",
"chunk-fetch-retry",
"couchdb-manual-setup-workflow",
"cli-to-obsidian-sync",
"minio-upload",
@@ -32,6 +33,7 @@ const focusedScenarios = new Set([
"security-seed-reconnect",
"hidden-file-snippet-sync",
"customisation-sync",
"received-change-readiness",
"setting-markdown-export",
"upgrade-from-stable",
]);
test/e2e-obsidian/scripts/setting-markdown-export.ts
+30 -2
View File
@@ -1,5 +1,6 @@
import { readFile } from "node:fs/promises";
import { join } from "node:path";
import { deriveIdKey } from "@vrtmrz/livesync-commonlib/settings";
import { evalObsidianJson } from "../runner/cli.ts";
import { discoverObsidianCli, requireObsidianBinary } from "../runner/environment.ts";
import { assertEqual } from "../runner/liveSyncWorkflow.ts";
@@ -34,7 +35,11 @@ async function waitForFileContaining(
throw new Error(`Timed out waiting for setting Markdown: ${fullPath}\nLast error: ${String(lastError)}`);
}
async function configureSettingMarkdown(cliBinary: string, env: NodeJS.ProcessEnv): Promise<void> {
async function configureSettingMarkdown(
cliBinary: string,
env: NodeJS.ProcessEnv,
idDerivationKey: string
): Promise<void> {
await evalObsidianJson<unknown>(
cliBinary,
[
@@ -46,6 +51,8 @@ async function configureSettingMarkdown(cliBinary: string, env: NodeJS.ProcessEn
"couchDB_USER:'e2e-user',",
"couchDB_PASSWORD:'e2e-password',",
"passphrase:'e2e-passphrase',",
"idDerivationVersion:1,",
`idDerivationKey:${JSON.stringify(idDerivationKey)},`,
"showVerboseLog:true,",
"},true);",
"await core.services.setting.saveSettingData();",
@@ -64,6 +71,7 @@ async function main(): Promise<void> {
}
const vault = await createTemporaryVault();
const idDerivationKey = await deriveIdKey("setting-markdown-export-independent-id-key-fixture");
let session: ObsidianLiveSyncSession | undefined;
try {
console.log(`Using Obsidian executable: ${binary}`);
@@ -77,19 +85,39 @@ async function main(): Promise<void> {
});
// The export is available while an unconfigured Vault remains outside
// application readiness; the session helper has already loaded the plug-in.
await configureSettingMarkdown(cli.binary, session.cliEnv);
await configureSettingMarkdown(cli.binary, session.cliEnv, idDerivationKey);
const content = await waitForFileContaining(vault.path, settingPath, [
(value) => value.includes("````yaml:livesync-setting"),
(value) => value.includes(`settingSyncFile: ${settingPath}`),
(value) => value.includes("showVerboseLog: true"),
]);
const persisted = JSON.parse(
await readFile(join(vault.path, ".obsidian", "plugins", "obsidian-livesync", "data.json"), "utf-8")
) as {
idDerivationVersion?: unknown;
idDerivationKey?: unknown;
encryptedIdDerivationKey?: unknown;
};
assertEqual(persisted.idDerivationVersion, 1, "The independent ID key fixture was not persisted.");
assertEqual(persisted.idDerivationKey, "", "The independent ID key was stored in plain text locally.");
const encryptedIdDerivationKey = persisted.encryptedIdDerivationKey;
if (typeof encryptedIdDerivationKey !== "string" || encryptedIdDerivationKey.length === 0) {
throw new Error("The independent ID key fixture was not saved in encrypted local settings.");
}
assertEqual(
content.includes("couchDB_PASSWORD: e2e-password"),
false,
"Credential leaked into setting Markdown."
);
assertEqual(content.includes("passphrase: e2e-passphrase"), false, "Passphrase leaked into setting Markdown.");
assertEqual(content.includes(idDerivationKey), false, "Plaintext ID key leaked into setting Markdown.");
assertEqual(
content.includes(encryptedIdDerivationKey),
false,
"Encrypted ID key leaked into setting Markdown."
);
console.log(`Generated setting Markdown without credentials: ${settingPath}`);
} finally {
test/e2e-obsidian/scripts/two-vault-sync.ts
+233 -21
View File
@@ -1,11 +1,16 @@
import { mkdir, readFile, rename as renameFilesystemPath, rm, writeFile } from "node:fs/promises";
import { dirname, join } from "node:path";
import { SALT_OF_PASSPHRASE } from "@vrtmrz/livesync-commonlib/compat/common/types";
import { encryptString } from "@vrtmrz/livesync-commonlib/compat/encryption/stringEncryption";
import { deriveIdKey } from "@vrtmrz/livesync-commonlib/settings";
import { CENTRAL_COMPATIBILITY_REJECTION_REASONS } from "@vrtmrz/livesync-commonlib/replication";
import { evalObsidianJson } from "../runner/cli.ts";
import {
assertCouchDbReachable,
createCouchDbDatabase,
deleteCouchDbDatabase,
fetchAllCouchDbDocs,
fetchCouchDbLocalDocs,
loadCouchDbConfig,
makeUniqueDatabaseName,
waitForCouchDbDocs,
@@ -19,6 +24,7 @@ import {
assertE2eCompatibilityReviewPending,
configureCouchDb,
createE2eCouchDbPluginData,
createE2eObsidianDeviceLocalState,
prepareRemote,
pushLocalChanges,
resumeCompatibilityReview,
@@ -442,7 +448,8 @@ async function renameNoteViaObsidian(cliBinary: string, env: NodeJS.ProcessEnv,
async function startConfiguredSession(
context: RunnerContext,
vault: TemporaryVault,
overrides: Record<string, unknown> = {}
overrides: Record<string, unknown> = {},
persistedOverrides: Record<string, unknown> = overrides
): Promise<ObsidianLiveSyncSession> {
const couchDbSettings = {
uri: context.couchDb.uri,
@@ -456,7 +463,7 @@ async function startConfiguredSession(
cliBinary: context.cliBinary,
vault,
startupGraceMs: Number(process.env.E2E_OBSIDIAN_STARTUP_GRACE_MS ?? 1000),
pluginData: createE2eCouchDbPluginData(couchDbSettings, overrides),
pluginData: createE2eCouchDbPluginData(couchDbSettings, persistedOverrides),
});
context.activeSessions.add(session);
try {
@@ -970,6 +977,193 @@ async function runEncryptedRoundTrip(
console.log("Two-vault encrypted note synchronisation round-tripped.");
}
async function runIndependentIdRoundTrip(
context: RunnerContext,
vaultA: TemporaryVault,
vaultB: TemporaryVault
): Promise<void> {
const source = "real-obsidian-e2e-independent-id-source";
const key = await deriveIdKey(source);
const content = "# Shared content with an independent ID key.\n";
const pathA = "E2E/independent-ids/from-a.md";
const pathB = "E2E/independent-ids/from-b.md";
const overrides = {
encrypt: true,
passphrase: "real-obsidian-e2e-independent-passphrase",
usePathObfuscation: true,
E2EEAlgorithm: "v2",
idDerivationVersion: 1,
idDerivationKey: key,
};
const persistedOverrides = {
...overrides,
idDerivationKey: "",
encryptedIdDerivationKey: await encryptString(key, `*${SALT_OF_PASSPHRASE}`),
};
let session = await startConfiguredSession(context, vaultA, overrides, persistedOverrides);
await writeNoteViaObsidian(context.cliBinary, session.cliEnv, pathA, content);
const entryA = await uploadNote(context, session, pathA);
if (entryA.children.length === 0) throw new Error("Independent ID mode produced no Chunks.");
await stopTrackedSession(context, session);
session = await startConfiguredSession(context, vaultB, overrides, persistedOverrides);
await syncAndApply(context, session);
await waitForPathContent(vaultB.path, pathA, (received) => received === content);
const receivedA = await waitForLocalDatabaseEntry(context.cliBinary, session.cliEnv, pathA);
assertEqual(receivedA.id, entryA.id, "The second device did not preserve the obfuscated document ID.");
await writeNoteViaObsidian(context.cliBinary, session.cliEnv, pathB, content);
const entryB = await uploadNote(context, session, pathB);
assertEqual(
JSON.stringify(entryB.children),
JSON.stringify(entryA.children),
"The second device did not reuse the same content-derived Chunk IDs."
);
await stopTrackedSession(context, session);
session = await startConfiguredSession(context, vaultA, overrides, persistedOverrides);
await syncAndApply(context, session);
await waitForPathContent(vaultA.path, pathB, (received) => received === content);
const receivedB = await waitForLocalDatabaseEntry(context.cliBinary, session.cliEnv, pathB);
assertEqual(receivedB.id, entryB.id, "The first device did not preserve the return document ID.");
await stopTrackedSession(context, session);
console.log("Two real Obsidian devices shared independent document and Chunk IDs in both directions.");
for (const [label, candidateKey] of [
["different key", "cd".repeat(32)],
["legacy IDs", ""],
] as const) {
const remoteBefore = await fetchAllCouchDbDocs(context.couchDb, context.dbName);
const checkpointsBefore = await fetchCouchDbLocalDocs(context.couchDb, context.dbName);
const rejectedVault = await createTemporaryVault();
let rejectedSession: ObsidianLiveSyncSession | undefined;
try {
rejectedSession = await startObsidianLiveSyncSession({
binary: context.binary,
cliBinary: context.cliBinary,
vault: rejectedVault,
localStorageEntries: createE2eObsidianDeviceLocalState(rejectedVault.name),
pluginData: createE2eCouchDbPluginData(
{ ...context.couchDb, dbName: context.dbName },
{
...overrides,
idDerivationVersion: candidateKey ? 1 : 0,
idDerivationKey: "",
encryptedIdDerivationKey: candidateKey
? await encryptString(candidateKey, `*${SALT_OF_PASSPHRASE}`)
: "",
}
),
});
context.activeSessions.add(rejectedSession);
await waitForLiveSyncCoreReady(context.cliBinary, rejectedSession.cliEnv);
const unsentPath = "E2E/independent-ids/rejected.md";
await writeNoteViaObsidian(context.cliBinary, rejectedSession.cliEnv, unsentPath, content);
await waitForLocalDatabaseEntry(context.cliBinary, rejectedSession.cliEnv, unsentPath);
const attempt = await evalObsidianJson<{ admitted: boolean; reason: string; replicated: boolean }>(
context.cliBinary,
[
"(async()=>{",
"const core=app.plugins.plugins['obsidian-livesync'].core;",
"const replicator=core.services.replicator.getActiveReplicator();",
"const settings=core.services.setting.currentSettings();",
"let reason='';",
"const connection=await replicator.checkReplicationConnectivity(settings,false,false,false,false,undefined,(decision)=>{reason=decision.reason??'';});",
"if(connection) await connection.close();",
"const replicated=await core.services.replication.replicate(true);",
"return JSON.stringify({admitted:!!connection,reason,replicated:!!replicated});",
"})()",
].join(""),
rejectedSession.cliEnv
);
assertEqual(attempt.admitted, false, `CouchDB admitted ${label} for obfuscated document IDs.`);
assertEqual(
attempt.reason,
CENTRAL_COMPATIBILITY_REJECTION_REASONS.ID_DERIVATION_MISMATCH,
`CouchDB rejected ${label} for an unrelated reason.`
);
assertEqual(attempt.replicated, false, `Ordinary replication accepted ${label}.`);
assertEqual(
await pathExists(rejectedVault.path, pathA),
false,
"A rejected device received a remote note."
);
await stopTrackedSession(context, rejectedSession);
rejectedSession = undefined;
assertEqual(
JSON.stringify(await fetchAllCouchDbDocs(context.couchDb, context.dbName)),
JSON.stringify(remoteBefore),
`A rejected ${label} connection changed remote documents.`
);
assertEqual(
JSON.stringify(await fetchCouchDbLocalDocs(context.couchDb, context.dbName)),
JSON.stringify(checkpointsBefore),
`A rejected ${label} connection changed remote checkpoints.`
);
} finally {
if (rejectedSession) await stopTrackedSession(context, rejectedSession);
await rejectedVault.dispose();
}
}
console.log("Ordinary CouchDB replication rejected different and legacy document ID keys without remote writes.");
}
async function runDifferentChunkIdKeysRoundTrip(
context: RunnerContext,
vaultA: TemporaryVault,
vaultB: TemporaryVault
): Promise<void> {
const keyA = await deriveIdKey("real-obsidian-e2e-chunk-source-a");
const keyB = await deriveIdKey("real-obsidian-e2e-chunk-source-b");
const content = "# Shared content with different Chunk ID keys.\n";
const pathA = "E2E/chunk-id-keys/from-a.md";
const pathB = "E2E/chunk-id-keys/from-b.md";
const commonSettings = {
encrypt: true,
passphrase: "real-obsidian-e2e-chunk-passphrase",
usePathObfuscation: false,
E2EEAlgorithm: "v2",
idDerivationVersion: 1,
};
const settingsFor = (key: string) => ({ ...commonSettings, idDerivationKey: key });
const persistedSettingsFor = async (key: string) => ({
...settingsFor(key),
idDerivationKey: "",
encryptedIdDerivationKey: await encryptString(key, `*${SALT_OF_PASSPHRASE}`),
});
const persistedA = await persistedSettingsFor(keyA);
const persistedB = await persistedSettingsFor(keyB);
let session = await startConfiguredSession(context, vaultA, settingsFor(keyA), persistedA);
await writeNoteViaObsidian(context.cliBinary, session.cliEnv, pathA, content);
const entryA = await uploadNote(context, session, pathA);
if (entryA.children.length === 0) throw new Error("The first device produced no Chunks.");
await stopTrackedSession(context, session);
session = await startConfiguredSession(context, vaultB, settingsFor(keyB), persistedB);
await syncAndApply(context, session);
await waitForPathContent(vaultB.path, pathA, (received) => received === content);
const receivedA = await waitForLocalDatabaseEntry(context.cliBinary, session.cliEnv, pathA);
assertEqual(receivedA.id, entryA.id, "The second device changed the visible document ID.");
await writeNoteViaObsidian(context.cliBinary, session.cliEnv, pathB, content);
const entryB = await uploadNote(context, session, pathB);
if (entryB.children.length === 0) throw new Error("The second device produced no Chunks.");
if (JSON.stringify(entryB.children) === JSON.stringify(entryA.children)) {
throw new Error("Different ID keys unexpectedly generated the same Chunk IDs.");
}
await stopTrackedSession(context, session);
session = await startConfiguredSession(context, vaultA, settingsFor(keyA), persistedA);
await syncAndApply(context, session);
await waitForPathContent(vaultA.path, pathB, (received) => received === content);
const receivedB = await waitForLocalDatabaseEntry(context.cliBinary, session.cliEnv, pathB);
assertEqual(receivedB.id, entryB.id, "The first device changed the return document ID.");
await stopTrackedSession(context, session);
console.log("Two real Obsidian devices exchanged notes with different Chunk ID keys and visible document paths.");
}
async function runMarkdownAutoMerge(
context: RunnerContext,
vaultA: TemporaryVault,
@@ -1094,10 +1288,9 @@ async function runConflictTimeStorageOperations(
showMergeDialogOnlyOnActive: true,
handleFilenameCaseSensitive: false,
};
const baseContent = Object.fromEntries(paths.map((path) => [path, `# Conflict operation\n\nBase for ${path}.\n`])) as Record<
(typeof paths)[number],
string
>;
const baseContent = Object.fromEntries(
paths.map((path) => [path, `# Conflict operation\n\nBase for ${path}.\n`])
) as Record<(typeof paths)[number], string>;
const leftContent = Object.fromEntries(
paths.map((path) => [path, `${baseContent[path]}\nEdit made on Vault A.\n`])
) as Record<(typeof paths)[number], string>;
@@ -1138,7 +1331,9 @@ async function runConflictTimeStorageOperations(
const initialBranchRevisions = new Map<string, Set<string>>();
for (const path of paths) {
const state = await waitForFileConflict(context.cliBinary, session.cliEnv, path);
const displayedBranch = state.branches.find((branch) => branch.content === rightContent[path] && !branch.deleted);
const displayedBranch = state.branches.find(
(branch) => branch.content === rightContent[path] && !branch.deleted
);
if (!displayedBranch) {
throw new Error(`Could not identify the branch displayed by Vault B: ${path}; ${JSON.stringify(state)}`);
}
@@ -1179,12 +1374,7 @@ async function runConflictTimeStorageOperations(
"A conflict-time deletion did not extend the displayed revision."
);
await renameNoteViaObsidian(
context.cliBinary,
session.cliEnv,
conflictCaseFromPath,
conflictCaseToPath
);
await renameNoteViaObsidian(context.cliBinary, session.cliEnv, conflictCaseFromPath, conflictCaseToPath);
const caseRenamedBranch = await waitForConflictBranch(
context.cliBinary,
session.cliEnv,
@@ -1225,12 +1415,7 @@ async function runConflictTimeStorageOperations(
"A conflict-time case-only rename did not record the new displayed revision."
);
await renameNoteViaObsidian(
context.cliBinary,
session.cliEnv,
conflictRenameFromPath,
conflictRenameToPath
);
await renameNoteViaObsidian(context.cliBinary, session.cliEnv, conflictRenameFromPath, conflictRenameToPath);
const renamedTarget = await waitForLocalDatabaseEntry(context.cliBinary, session.cliEnv, conflictRenameToPath);
const renamedSourceDeletion = await waitForConflictBranch(
context.cliBinary,
@@ -1376,10 +1561,13 @@ async function main(): Promise<void> {
const couchDb = await loadCouchDbConfig();
const dbName = makeUniqueDatabaseName(couchDb.dbPrefix, "two-vault-sync");
const encryptedDbName = makeUniqueDatabaseName(couchDb.dbPrefix, "two-vault-sync-e2ee");
const independentDbName = makeUniqueDatabaseName(couchDb.dbPrefix, "two-vault-sync-independent-ids");
const vaultA = await createTemporaryVault();
const vaultB = await createTemporaryVault();
const encryptedVaultA = await createTemporaryVault();
const encryptedVaultB = await createTemporaryVault();
const independentVaultA = await createTemporaryVault();
const independentVaultB = await createTemporaryVault();
const context: RunnerContext = {
binary,
cliBinary: cli.binary,
@@ -1396,11 +1584,20 @@ async function main(): Promise<void> {
reviewedVaults: new Set(),
activeSessions: new Set(),
};
const independentContext: RunnerContext = {
binary,
cliBinary: cli.binary,
couchDb,
dbName: independentDbName,
reviewedVaults: new Set(),
activeSessions: new Set(),
};
try {
await assertCouchDbReachable(couchDb);
await createCouchDbDatabase(couchDb, dbName);
await createCouchDbDatabase(couchDb, encryptedDbName);
await createCouchDbDatabase(couchDb, independentDbName);
console.log(`Using Obsidian executable: ${binary}`);
console.log(`Temporary vault A: ${vaultA.path}`);
@@ -1409,11 +1606,13 @@ async function main(): Promise<void> {
console.log(`Temporary encrypted CouchDB database: ${encryptedDbName}`);
const onlyParentCaseDeletion = process.env.E2E_OBSIDIAN_ONLY_PARENT_CASE_DELETION === "true";
const onlyIndependentIds = process.env.E2E_OBSIDIAN_ONLY_INDEPENDENT_IDS === "true";
const onlyDifferentChunkIdKeys = process.env.E2E_OBSIDIAN_ONLY_DIFFERENT_CHUNK_ID_KEYS === "true";
if (onlyParentCaseDeletion) {
await runParentCaseDeletionProtection(context, vaultA, vaultB);
}
const onlyConflictOperations = process.env.E2E_OBSIDIAN_ONLY_CONFLICT_OPERATIONS === "true";
if (!onlyParentCaseDeletion && !onlyConflictOperations) {
if (!onlyParentCaseDeletion && !onlyConflictOperations && !onlyIndependentIds && !onlyDifferentChunkIdKeys) {
await runCreateUpdateDelete(context, vaultA, vaultB);
await runRename(context, vaultA, vaultB);
await runCaseOnlyRename(context, vaultA, vaultB);
@@ -1427,17 +1626,27 @@ async function main(): Promise<void> {
) {
await runConflictTimeStorageOperations(context, vaultA, vaultB);
}
if (!onlyParentCaseDeletion && !onlyConflictOperations) {
if (!onlyParentCaseDeletion && !onlyConflictOperations && !onlyIndependentIds && !onlyDifferentChunkIdKeys) {
await runTargetMismatch(context, vaultA, vaultB);
await runEncryptedRoundTrip(encryptedContext, encryptedVaultA, encryptedVaultB);
}
if (!onlyParentCaseDeletion && !onlyConflictOperations) {
if (onlyDifferentChunkIdKeys) {
await runDifferentChunkIdKeysRoundTrip(independentContext, independentVaultA, independentVaultB);
} else {
await runIndependentIdRoundTrip(independentContext, independentVaultA, independentVaultB);
}
}
} finally {
await stopTrackedSessions(context);
await stopTrackedSessions(encryptedContext);
await stopTrackedSessions(independentContext);
await vaultA.dispose();
await vaultB.dispose();
await encryptedVaultA.dispose();
await encryptedVaultB.dispose();
await independentVaultA.dispose();
await independentVaultB.dispose();
if (process.env.E2E_OBSIDIAN_KEEP_COUCHDB !== "true") {
await deleteCouchDbDatabase(couchDb, dbName).catch((error: unknown) => {
console.warn(error instanceof Error ? error.message : error);
@@ -1445,6 +1654,9 @@ async function main(): Promise<void> {
await deleteCouchDbDatabase(couchDb, encryptedDbName).catch((error: unknown) => {
console.warn(error instanceof Error ? error.message : error);
});
await deleteCouchDbDatabase(couchDb, independentDbName).catch((error: unknown) => {
console.warn(error instanceof Error ? error.message : error);
});
}
}
}
updates.md
+54 -60
View File
@@ -12,14 +12,66 @@ Earlier releases remain available in the 1.0 release history, the 1.0 preview hi
## Unreleased
### Privacy and compatibility
#### New Feature
- An optional saved ID key can generate encrypted Chunk IDs and obfuscated Metadata document IDs independently of the current E2EE passphrase.
- New Vaults use a random key by default; existing Vaults keep their current ID configuration by default. You can also derive a key from the current E2EE passphrase, enter a separate source, or import a recovery code. The source is not retained; the saved key can be revealed locally as a recovery code.
- The saved key stays in place when the E2EE passphrase changes or E2EE is turned off. Share it with another device through a protected Setup URI. Changing document IDs on an existing remote requires the usual Rebuild and Fetch procedure.
- We can now keep the file properties used by Hidden File Sync and Customisation Sync private in CouchDB.
- **Encrypt internal file Properties** extends E2EE V2 and Property Encryption to their paths, times, sizes, and Chunk references.
- Existing configurations keep this preference disabled. New Vaults enable it for use when the required encryption settings are active.
- Update every synchronising device before enabling it. It protects future writes; a manual remote Rebuild is strongly recommended to protect existing properties.
- We can now see which unsupported feature prevents a client from synchronising with CouchDB.
- Clients check the features required by the remote before transferring data or resetting the local database for Fast Fetch. Receiving an unsupported requirement also stops active replication.
- We can now compare ID generation performance on a desktop or mobile device through **Open review harness**, available with the developers' debug tools enabled.
- The copied report includes legacy and independent ID timings and, where available, approximate JavaScript heap samples. The measurement uses fixed test data and keeps our Vault and settings unchanged.
#### Fixed
- We can now keep using an E2EE passphrase beginning with `%` after restarting Obsidian. (#1221)
- LiveSync encrypts it before saving the settings. If an earlier version saved it in plain text, re-enter the passphrase used to encrypt the existing data after updating. Treat that passphrase as exposed if the affected `data.json` was shared.
- A receiving device now retries an unavailable CouchDB Chunk when file Metadata arrives before that Chunk is visible, helping rapid edits reach the Vault after an initial on-demand lookup misses it. (#1224)
- Retries start after two seconds and continue with increasing delays while finite replication is active. When it ends, LiveSync checks locally and makes a final lookup if needed, without waiting out the remaining retry delay.
- We can now distinguish initial on-demand Chunk requests (`🛄`) from retries (`🔁`) in the status bar. These replace `🧩`; each pending Chunk appears in one category, including while a retry is waiting.
### Synchronisation and storage
#### Fixed
- Received changes held during start-up or a fetch are applied when LiveSync becomes ready, without waiting for another change or a settings save. **Suspend database reflecting** continues to hold changes (#1200).
## 1.0.32
27th September, 2026
The 1.0.31 pre-release was not promoted after validation found that a receiving device could reject encrypted CouchDB changes when Path Obfuscation was enabled. This release includes its changes and corrects that issue.
### Synchronisation and storage
#### Fixed
- The receiving device now accepts encrypted file information when both end-to-end encryption and Path Obfuscation are enabled. The 1.0.31 pre-release could reject this information, leaving files from another device absent from the Vault.
- Files with colons in their names now retain their full paths in synchronisation data instead of appearing as incorrectly named copies at the Vault root. (#1206)
- Obsidian may refuse to create a missing file with such a name. LiveSync also treats these names as invalid on Windows and Android, so the file may not appear in those devices' Vaults. Existing misplaced copies are left for you to review; this change does not remove them automatically.
- Received changes within the configured modification-time limit are applied to the Vault again while remediation mode is active. Changes newer than the limit remain blocked; changes arriving while a fetch makes the local database unavailable are kept for a later attempt.
- A scheduled fetch no longer offers Simple Fetch while remediation mode is active. This prevents the fetch from bypassing the modification-time limit; the detailed flow explains the restriction and offers to clear it first (#1202). Thank you to @kimjansheden for both fixes and the regression tests in PR #1208!
- On start-up, an unchanged file with a missing local revision record can be recognised before newer content arrives, avoiding an unnecessary conflict. Files with actual local edits still require conflict review. (#1207)
## 1.0.31
26th September, 2026
### Synchronisation and storage
#### Fixed
- Files with colons in their names now retain their full paths in synchronisation data instead of appearing as incorrectly named copies at the Vault root. (#1206)
- Obsidian may refuse to create a missing file with such a name. LiveSync also treats these names as invalid on Windows and Android, so the file may not appear in those devices' Vaults. Existing misplaced copies are left for you to review; this change does not remove them automatically.
- Received changes are applied again while remediation mode is active. That mode prevents the scan which readiness depends upon, so nothing had been applied since the plug-in began waiting for readiness, not even changes older than the configured modification-time limit. The limit itself is still enforced for every change, and application waits for a usable local database so that a change arriving during a fetch is not dropped.
- A scheduled fetch no longer offers Simple Fetch while remediation mode is active. Simple Fetch reconciles the Vault with the local database past the restriction, which could store the current files or apply changes newer than the limit; the detailed flow states the restriction and offers to clear it first (#1202). Thank you to @kimjansheden for both fixes and the regression tests in PR #1208!
- Received changes within the configured modification-time limit are applied to the Vault again while remediation mode is active. Changes newer than the limit remain blocked; changes arriving while a fetch makes the local database unavailable are kept for a later attempt.
- A scheduled fetch no longer offers Simple Fetch while remediation mode is active. This prevents the fetch from bypassing the modification-time limit; the detailed flow explains the restriction and offers to clear it first (#1202). Thank you to @kimjansheden for both fixes and the regression tests in PR #1208!
- On start-up, an unchanged file with a missing local revision record can be recognised before newer content arrives, avoiding an unnecessary conflict. Files with actual local edits still require conflict review. (#1207)
## 1.0.30
@@ -87,61 +139,3 @@ For now, I am addressing the issues I can resolve first. I hope this helps.
#### Fixed
- First-time Object Storage setup now completes when **Use Custom HTTP Handler** is enabled for an empty remote, including a new Cloudflare R2 bucket. LiveSync can now create the remote state required to begin synchronisation. (#1166)
## 1.0.26
~~1.0.25~~ was cancelled because pre-release validation found that LiveSync could appear to finish synchronising even though Android had not written a received file to the Vault; the warning appeared only after restart.
6th September, 2026
### Synchronisation and storage
#### Fixed
- Files inside a folder are no longer silently removed from synchronisation when an external tool changes only the letter case of that folder while Obsidian is running. This prevents the stale deletion from reaching other devices or later removing the local file. Moving files into ignored or otherwise excluded locations retains the existing behaviour, and the folder-name case itself may still differ between devices. (#1168)
- A problem processing one file during ordinary start-up no longer prevents every other file from synchronising. LiveSync warns about the affected files and can retry them later; Fetch and Rebuild still stop if they cannot finish safely. (#1164)
- When LiveSync cannot finish preparing this device for synchronisation, it now says that synchronisation is unavailable and directs you to generate a report, instead of remaining at 'Not ready'. (#1164)
#### Improved
- When LiveSync cannot write a received file to the Vault, it now warns immediately instead of appearing to have synchronised it successfully. The generated report identifies the affected path, and a later scan can try it again.
### Conflict handling and recovery
#### Improved
- Conflict resolution dialogues now close when the same file is resolved elsewhere or when the plug-in unloads. Requests for different files are shown one at a time, while a newer request for the same file replaces the older one.
### Setup and compatibility
#### Improved
- Unconfigured Vaults now stay focused on setup instead of running Config Doctor or incomplete-document checks before they can be used. Returning a configured Vault to an unconfigured state also stops those checks until the requested restart. (#1161)
- When the active file contains a file or folder name longer than 255 UTF-8 bytes, LiveSync now explains that the path may not work on some Android and Linux file systems. It does not rename or reject the file. (#1164)
## 1.0.24
3rd September, 2026
### Interface and translation
#### Fixed
- The Setup Wizard now correctly explains that the existing-device path adds this device to an existing synchronisation (PR #1118). Thank you to @nikhilmaddirala for the contribution!
- Spanish translations now resolve the **Display language** placeholder, cover previously untranslated Setup Wizard and CouchDB text, translate user-facing Config Doctor values and confirmation controls, and use Spanish sentence case (PR #1129). Thank you to @zeedif for the contribution!
#### Improved
- The Setup Wizard now shows the passphrase and **Obfuscate Properties** controls only after E2EE is enabled, provides a password-visibility button, allows longer translated labels to wrap, and keeps the invitation link compact on desktop while preserving its mobile touch target (PR #1130). Thank you to @zeedif for the contribution!
### Synchronisation and storage
#### Fixed
- **Overwrite Server Data with This Device's Files** now keeps this device's synchronisation settings instead of reapplying settings from the remote database which is about to be replaced. Enabling E2EE before a rebuild therefore remains enabled and uploads encrypted data. (#1146)
### Command-line tool
#### Fixed
- The systemd installer now finds the repository root correctly, installs every generated bundle chunk and required production dependency, checks the installed command before activation, and reports success only when the service remains active.
utils/couchdb/livesync-commonlib.ts
+2 -2
View File
@@ -1,5 +1,5 @@
// Keep CouchDB database-version negotiation isolated from Setup URI generation.
// The exact release must match utils/livesync-commonlib-version.ts; the setup
// tool suite checks every static specifier before release.
export { checkRemoteVersion } from "npm:@vrtmrz/livesync-commonlib@0.1.0-rc.4/compat/pouchdb/negotiation";
export { PouchDB } from "npm:@vrtmrz/livesync-commonlib@0.1.0-rc.4/compat/pouchdb/pouchdb-browser";
export { checkRemoteVersion } from "npm:@vrtmrz/livesync-commonlib@0.1.32/compat/pouchdb/negotiation";
export { PouchDB } from "npm:@vrtmrz/livesync-commonlib@0.1.32/compat/pouchdb/pouchdb-browser";
utils/flyio/deno.lock
Generated
+5 -5
View File
@@ -1,7 +1,7 @@
{
"version": "5",
"specifiers": {
"npm:@vrtmrz/livesync-commonlib@0.1.0-rc.4": "0.1.0-rc.4"
"npm:@vrtmrz/livesync-commonlib@0.1.32": "0.1.32"
},
"npm": {
"@aws-sdk/checksums@3.1000.18": {
@@ -284,8 +284,8 @@
"@trystero-p2p/core"
]
},
"@vrtmrz/livesync-commonlib@0.1.0-rc.4": {
"integrity": "sha512-u4FdbjnYg7lAf38z7eUv4eq4vxEdrl4rFMxiDZiJ7T701awKiflkXGIJQnaHdLaMa3zkRBU15qqEo7GtextmlA==",
"@vrtmrz/livesync-commonlib@0.1.32": {
"integrity": "sha512-gzjhd7bg+DKHhd/24WGxBM7557wsw3HJkc0YjKll1n8/8vuhqPnlLuZmM33fj+4bv9nGFUnDlnoVowpQrTns6w==",
"dependencies": [
"@aws-sdk/client-s3",
"@smithy/fetch-http-handler",
@@ -509,8 +509,8 @@
"whatwg-url"
]
},
"octagonal-wheels@0.1.51": {
"integrity": "sha512-KTlfqKPjobHJg/t3A539srnFf+VHr1aXkHSmsNDDpiI5UFC7FamZ95dWpJfGE2EI/HULR5hveQDgkazmz8SAcg==",
"octagonal-wheels@0.1.54": {
"integrity": "sha512-Je3ancYhjKX7UY2K19T/qTjG8C9nK8YVrACr5naIf78mN4bbjQkYyWmlj+ooifV/moWVsQrp4fEWz/7mv6It3A==",
"dependencies": [
"idb"
]
utils/flyio/generate_setupuri.test.ts
+7
View File
@@ -35,6 +35,13 @@ Deno.test("generates a current self-hosted Setup URI through the published Commo
const decoded = await decodeSettingsFromSetupURI(setupURI, "setup-secret");
assert(decoded, "Commonlib could not decode the generated Setup URI");
const effectiveSettings = { ...DEFAULT_SETTINGS, ...decoded };
const recoveryCode = stdout.match(/sls-id-v1:[0-9a-f]{64}/u)?.[0];
assert(recoveryCode, "the generator did not print an ID recovery code");
assert(
(effectiveSettings as typeof effectiveSettings & { idDerivationKey?: string }).idDerivationKey ===
recoveryCode.slice("sls-id-v1:".length),
"the CouchDB Setup URI did not contain the generated ID key",
);
assert(
effectiveSettings.isConfigured,
"the CouchDB Setup URI left the imported device unconfigured",
utils/livesync-commonlib-version.ts
+1 -1
View File
@@ -2,4 +2,4 @@
// Commonlib registry release. Static npm specifiers cannot interpolate this
// value, so livesync-commonlib-version.test.ts verifies the domain-specific
// facades against it.
export const LIVESYNC_COMMONLIB_VERSION = "0.1.0-rc.4";
export const LIVESYNC_COMMONLIB_VERSION = "0.1.32";
utils/readme.md
+2
View File
@@ -31,6 +31,8 @@ Authentication and other non-retryable HTTP failures stop immediately. Network a
The existing `flyio/generate_setupuri.ts` path remains a CouchDB-only compatibility wrapper for the Fly.io deployment script.
The generator creates a fresh random ID key by default and includes it in the encrypted Setup URI. It prints a tagged `sls-id-v1:` recovery code. Set `id_recovery_code` to that code when generating another URI for the same Vault; a new run without it creates a different key. This restores only the ID key: reuse the original connection details too. For P2P, provide the original `p2p_room_id` and `p2p_passphrase` because omitted values are generated afresh. Set `id_mode=legacy` to generate a URI with the previous ID behaviour. `id_mode=legacy` and `id_recovery_code` cannot be combined. Keep the recovery code private and retain it if every device might be lost.
### CouchDB
```sh
utils/setup/generate_setup_uri.test.ts
+62
View File
@@ -26,6 +26,20 @@ Deno.test("generates an Object Storage Setup URI with a selected S3 profile", as
);
assert(decoded, "Commonlib could not decode the Object Storage Setup URI");
const effective = { ...DEFAULT_SETTINGS, ...decoded };
const recoveryCode = generated.idRecoveryCode;
assert(
typeof recoveryCode === "string" && recoveryCode.startsWith("sls-id-v1:"),
"the generator did not return an ID recovery code",
);
assert(
(effective as typeof effective & { idDerivationVersion?: number }).idDerivationVersion === 1,
"the Setup URI did not enable independent IDs",
);
assert(
(effective as typeof effective & { idDerivationKey?: string }).idDerivationKey ===
recoveryCode.slice("sls-id-v1:".length),
"the Setup URI did not contain the generated ID key",
);
assert(
effective.isConfigured,
"the Setup URI left the imported device unconfigured",
@@ -74,6 +88,11 @@ Deno.test("generates a random-room P2P Setup URI without copying a device identi
);
assert(decoded, "Commonlib could not decode the P2P Setup URI");
const effective = { ...DEFAULT_SETTINGS, ...decoded };
assert(
(effective as typeof effective & { idDerivationKey?: string }).idDerivationKey ===
generated.idRecoveryCode?.slice("sls-id-v1:".length),
"the P2P Setup URI did not contain the generated ID key",
);
assert(
/^\d{3}-\d{3}-\d{3}-[a-z0-9]{3}$/.test(effective.P2P_roomID),
"Commonlib did not generate the expected random room ID",
@@ -121,3 +140,46 @@ Deno.test("generates a random-room P2P Setup URI without copying a device identi
"the selected profile was not a P2P connection URI",
);
});
Deno.test("reuses the ID key from a recovery code and permits explicit legacy IDs", async () => {
const environment = {
remote_type: "p2p",
passphrase: "vault-secret",
uri_passphrase: "setup-secret",
};
const first = await generateSetupURI(environment);
const second = await generateSetupURI({ ...environment, id_recovery_code: first.idRecoveryCode });
const independentlyGenerated = await generateSetupURI(environment);
assert(second.idRecoveryCode === first.idRecoveryCode, "the recovery code changed on repeat generation");
assert(independentlyGenerated.idRecoveryCode !== first.idRecoveryCode, "the default ID key was reused");
const repeatedSettings = await decodeSettingsFromSetupURI(second.setupURI, second.setupPassphrase);
assert(repeatedSettings, "the repeated Setup URI could not be decoded");
assert(
(repeatedSettings as typeof repeatedSettings & { idDerivationKey?: string }).idDerivationKey ===
first.idRecoveryCode?.slice("sls-id-v1:".length),
"the recovery code did not restore the original ID key",
);
const legacy = await generateSetupURI({ ...environment, id_mode: "legacy" });
const decoded = await decodeSettingsFromSetupURI(legacy.setupURI, legacy.setupPassphrase);
assert(decoded, "the legacy Setup URI could not be decoded");
assert(legacy.idRecoveryCode === undefined, "legacy mode returned an ID recovery code");
assert(
(decoded as typeof decoded & { idDerivationVersion?: number }).idDerivationVersion !== 1,
"legacy mode enabled independent IDs",
);
let rejected = false;
try {
await generateSetupURI({ ...environment, id_recovery_code: "sls-id-v1:wrong" });
} catch {
rejected = true;
}
assert(rejected, "an invalid recovery code was accepted");
rejected = false;
try {
await generateSetupURI({ ...environment, id_mode: "legacy", id_recovery_code: first.idRecoveryCode });
} catch {
rejected = true;
}
assert(rejected, "legacy mode silently ignored a recovery code");
});
utils/setup/generate_setup_uri.ts
+36 -1
View File
@@ -19,6 +19,36 @@ export interface GeneratedSetupURI {
remoteType: SetupRemoteType;
setupURI: string;
setupPassphrase: string;
idRecoveryCode?: string;
}
const ID_RECOVERY_CODE_PREFIX = "sls-id-v1:";
const ID_RECOVERY_CODE_PATTERN = /^sls-id-v1:([0-9a-f]{64})$/u;
function generateRandomIdKey(): string {
const bytes = crypto.getRandomValues(new Uint8Array(32));
return Array.from(bytes, (byte) => byte.toString(16).padStart(2, "0")).join("");
}
function configureIdDerivation(
settings: ObsidianLiveSyncSettings,
environment: SetupGeneratorEnvironment,
): string | undefined {
const mode = environment.id_mode?.trim().toLowerCase() || "random";
if (mode !== "random" && mode !== "legacy") {
throw new Error("id_mode must be random or legacy");
}
const suppliedCode = environment.id_recovery_code?.trim();
if (mode === "legacy") {
if (suppliedCode) throw new Error("id_recovery_code cannot be used with id_mode=legacy");
return undefined;
}
const key = suppliedCode
? ID_RECOVERY_CODE_PATTERN.exec(suppliedCode)?.[1]
: generateRandomIdKey();
if (!key) throw new Error("id_recovery_code must be a valid sls-id-v1 recovery code");
Object.assign(settings, { idDerivationVersion: 1, idDerivationKey: key });
return `${ID_RECOVERY_CODE_PREFIX}${key}`;
}
function requireValue(
@@ -166,11 +196,12 @@ export async function generateSetupURI(
const setupPassphrase = environment.uri_passphrase?.trim() ||
generateSecret();
const { remoteType, settings } = createSetupSettings(environment);
const idRecoveryCode = configureIdDerivation(settings, environment);
const setupURI = await encodeSettingsToSetupURI(settings, setupPassphrase, [
"pluginSyncExtendedSetting",
"doNotUseFixedRevisionForChunks",
], true);
return { remoteType, setupURI: setupURI.trim(), setupPassphrase };
return { remoteType, setupURI: setupURI.trim(), setupPassphrase, idRecoveryCode };
}
export async function runSetupURIGenerator(
@@ -183,6 +214,10 @@ export async function runSetupURIGenerator(
generated.setupPassphrase,
);
console.log("This passphrase is never shown again, so store it safely.");
if (generated.idRecoveryCode) {
console.log("ID recovery code:", generated.idRecoveryCode);
console.log("Use id_recovery_code with this value and reuse the same remote settings when generating another Setup URI for the same Vault.");
}
console.log(generated.setupURI);
}
utils/setup/livesync-commonlib.ts
+5 -5
View File
@@ -4,9 +4,9 @@
export {
decodeSettingsFromSetupURI,
encodeSettingsToSetupURI,
} from "npm:@vrtmrz/livesync-commonlib@0.1.0-rc.4/compat/API/processSetting";
export { generateP2PRoomId } from "npm:@vrtmrz/livesync-commonlib@0.1.0-rc.4/compat/common/utils";
export { upsertRemoteConfigurationInPlace } from "npm:@vrtmrz/livesync-commonlib@0.1.0-rc.4/remote-configurations";
} from "npm:@vrtmrz/livesync-commonlib@0.1.32/compat/API/processSetting";
export { generateP2PRoomId } from "npm:@vrtmrz/livesync-commonlib@0.1.32/compat/common/utils";
export { upsertRemoteConfigurationInPlace } from "npm:@vrtmrz/livesync-commonlib@0.1.32/remote-configurations";
export {
createNewVaultSettings,
DEFAULT_SETTINGS,
@@ -14,5 +14,5 @@ export {
PREFERRED_BASE,
PREFERRED_JOURNAL_SYNC,
PREFERRED_SETTING_SELF_HOSTED,
} from "npm:@vrtmrz/livesync-commonlib@0.1.0-rc.4/settings";
export type { ObsidianLiveSyncSettings } from "npm:@vrtmrz/livesync-commonlib@0.1.0-rc.4/settings";
} from "npm:@vrtmrz/livesync-commonlib@0.1.32/settings";
export type { ObsidianLiveSyncSettings } from "npm:@vrtmrz/livesync-commonlib@0.1.32/settings";
versions.json
+3 -1
View File
@@ -41,5 +41,7 @@
"1.0.27": "1.7.2",
"1.0.28": "1.7.2",
"1.0.29": "1.7.2",
"1.0.30": "1.7.2"
"1.0.30": "1.7.2",
"1.0.31": "1.7.2",
"1.0.32": "1.7.2"
}