diff --git a/.github/workflows/unit-ci.yml b/.github/workflows/unit-ci.yml index a2169d2d..f8b0f80c 100644 --- a/.github/workflows/unit-ci.yml +++ b/.github/workflows/unit-ci.yml @@ -121,6 +121,9 @@ jobs: node-version: '24.x' cache: 'npm' + - name: Verify clean installation with npm 10 + run: npx --yes npm@10.9.4 ci --ignore-scripts --no-audit --no-fund + - name: Install dependencies run: npm ci diff --git a/README.md b/README.md index 82c42bb3..f84b0267 100644 --- a/README.md +++ b/README.md @@ -98,7 +98,8 @@ Synchronisation status is shown in the status bar with the following icons. - πŸ›« Pending read storage processes - πŸ“¬ Batched read storage processes - βš™οΈ Working or pending storage processes for hidden files - - 🧩 Waiting chunks + - πŸ›„ Pending initial on-demand chunk requests + - πŸ” Pending chunk retries, including retry delays - πŸ”Œ Working customisation items (configuration, snippets, and plug-ins) To prevent file and database corruption, please avoid closing Obsidian until all progress indicators have disappeared as much as possible (although the plug-in will attempt to resume if interrupted). This is especially important if you have deleted or renamed files. diff --git a/devs.md b/devs.md index 21b1f2e4..713eb6cc 100644 --- a/devs.md +++ b/devs.md @@ -29,18 +29,7 @@ npm run build #### Community Review dependency installation -Community Review installs dependencies independently before applying type-aware source rules. A successful installation with the npm version bundled with the repository's current Node.js CI does not prove that the lockfile is accepted by the scanner's npm version. - -After changing `package.json`, a workspace manifest, or `package-lock.json`, verify both installation paths: - -```bash -npm ci --ignore-scripts -npx --yes npm@10.9.2 ci --ignore-scripts -``` - -The npm 10.9.2 command is the current project-side compatibility check for the Community Review installation path. Update this check when the scanner runtime changes. - -If Community Review reports widespread TypeScript `error` types across unrelated external packages, confirm that dependency installation completed successfully before changing source imports, declarations, or lint rules. An installation failure can make every unresolved external type appear as downstream unsafe-type findings. +After changing a dependency manifest or lockfile, follow the [npm 10 clean-installation check](test/README.md#npm-10-clean-installation-check) before the normal source and unit checks. The test guide records the command used by CI and the distinction between installation failures and source diagnostics. ### Commands @@ -77,6 +66,8 @@ To facilitate development and testing, the build process can automatically copy ### Testing Infrastructure +See the [test procedures](test/README.md) for clean-installation checks, local validation commands, and links to each runtime suite. + - **Vitest**: - **Unit Tests** (`vitest.config.unit.ts`): Unit tests run in Node.js (excluding harnesses and integration tests). Unit tests should be `*.unit.spec.ts` and placed alongside the implementation file (e.g., `ChunkFetcher.unit.spec.ts`). Executed via `npm run test:unit`. - **Integration Tests** (`vitest.config.integration.ts`): Tests run in Node.js against a real CouchDB instance. Integration tests should be `*.integration.spec.ts` or `*.integration.test.ts` and placed alongside the implementation file (e.g., `StreamingFetch.integration.spec.ts`). Executed via `npm run test:integration`. @@ -189,15 +180,17 @@ steps required to add a built-in provider. Commonlib owns one stable `LiveSyncP2PService`, its `P2PRoomSessionOwner`, and the replaceable Trystero room session. Host commands, event handlers, and views consume the focused transport, connection-probe admission, directory, peer-admission, transfer, change-relay, configuration, and diagnostic views returned by the service feature. They must not retain the deprecated compatibility Replicator as an ordinary service locator, close Trystero-owned raw peers, or install another Trystero transport generation at the application root. The exact implemented ownership and shutdown boundaries are recorded in Commonlib's [P2P transport lifecycle](https://github.com/vrtmrz/livesync-commonlib/blob/main/docs/p2p-transport-lifecycle.md) design document. +The [TURN connection settings design](docs/design_docs/renewable_turn_credentials.md) describes how the host prepares temporary ICE credentials in a connection-only settings copy. It covers room reuse and expiry, replication continuation, profile persistence and sharing, and report redaction. + ### Conflict Merge Policy Markdown conflict auto-merge should behave like a conservative three-way merge. The guiding rule is to merge changes when they touch non-overlapping regions, and to keep a manual conflict when the edits overlap semantically. When in doubt, prefer the safer outcome: preserve data, keep the conflict visible, and ask the user rather than silently discarding content or choosing one side. -The detailed contract is documented in [Conflict resolution and revision provenance](docs/specs_conflict_resolution.md). Determine the merge base by intersecting the exact `available` revision IDs from both leaf histories and selecting the nearest shared revision. Do not infer ancestry from revision generation numbers. When a remote resolution reaches a Vault which still contains the exact content of a deleted losing branch, treat that content as known synchronised history so the resolution can be reflected without recreating the conflict. +The detailed contract is documented in [Conflict resolution and revision provenance](docs/specs_conflict_resolution.md). Determine the merge base by intersecting the exact `available` revision IDs from both leaf histories and selecting the nearest shared revision. Do not infer ancestry from revision generation numbers. An unchanged file is recognised by comparing its bytes with its exact device-local file-reflection provenance, including when that revision belongs to a deleted losing branch. -File operations made while a conflict is active must use the device-local file-reflection provenance injected into `ServiceFileHandlerBase`. Treat its exact revision as authoritative; use byte equality only to reconstruct a missing record when exactly one available revision matches. If branch identity remains unknown, preserve data and leave the conflict visible. Do not hide key-value database readiness behind an implicit wait: maintained hosts open it through the sequential settings lifecycle before file events or replication begin. +Ordinary file saves and incoming reflection use that provenance even before a conflict exists. An unchanged stale file must not become a child of the current winner; a genuine edit extends the recorded revision. Without a readable recorded base, compare only current live leaves to avoid duplicate content. Otherwise, preserve the file as a fresh independent root under the same document ID, leaving ancestry unknown. Historical byte equality cannot distinguish an unchanged file from an intentional revert. Explicit reconciliation, deletion, and rename retain their separate contracts. Do not hide key-value database readiness behind an implicit wait: maintained hosts open it through the sequential settings lifecycle before file events or replication begin. - If one side deletes a line and the other side leaves that same line unchanged, treat it as a safe deletion. The deleted line must not be reintroduced into the merged result. - If one side inserts new content in a different region while the other side deletes an unchanged old region, preserve the insertion and the deletion. @@ -207,6 +200,8 @@ File operations made while a conflict is active must use the device-local file-r This policy is intentionally aligned with the conflict checkboxes and compatibility settings: automatic merge should remove avoidable prompts, but it must not silently choose between overlapping user intentions. +The [multiple-device conflict test procedure](test/README.md#multiple-device-conflict-regression-tests) documents the five CouchDB-backed cases, execution steps, expected results, and coverage boundaries. + ### File Structure Conventions - **Platform-specific code**: Use `.platform.ts` suffix (replaced with `.obsidian.ts` in production builds via esbuild) @@ -240,6 +235,18 @@ Commonlib owns the typed English fallback for messages requested by its services ### Logging & Debugging +#### ID generation measurements on a device + +Enable **Enable Developers' Debug Tools.**, restart Obsidian, and run **Open review harness** from the command palette. Choose **Run** beside **ID generation performance**, keep Obsidian in the foreground, and use **Copy Markdown report** to retain the results. The **Automatic** action does not run this measurement; **Full review** includes it. + +The measurement uses fixed in-memory inputs and keys, with no Vault, database, settings, or remote writes. It compares legacy `xxhash64` and independent Chunk IDs for 256-byte, 4-KiB, and 32-KiB inputs, and compares obfuscated document IDs. Each result reports the median and range of three 1,000-ID samples and the median time per ID. Key derivation at save time is measured separately. Warm-up and pauses between batches are excluded from the timings. These measurements do not represent a full Rebuild. + +Where `performance.memory` is available, the report includes approximate JavaScript heap samples before, during, and after measurement. These may include other Obsidian activity and garbage collection; they are neither total process RAM nor an exact peak. Unsupported devices explicitly report that heap measurements are unavailable. + +The developer-only adapter in `src/features/ReviewHarness/reviewHarnessIdBenchmarkRuntime.ts` imports `HashManager` from Commonlib's public `/hashing` entry. Compilation, packed-package checks, and runtime tests cover this boundary. The algorithms remain owned by Commonlib. + +#### Logs + - Use `this._log(msg, LOG_LEVEL_INFO)` in modules (automatically prefixes with module name) - Log levels: `LOG_LEVEL_DEBUG`, `LOG_LEVEL_VERBOSE`, `LOG_LEVEL_INFO`, `LOG_LEVEL_NOTICE`, `LOG_LEVEL_URGENT` - LOG_LEVEL_NOTICE and above are reported to the user via Obsidian notices diff --git a/docs/adr/2026_07_bounded_remote_activity.md b/docs/adr/2026_07_bounded_remote_activity.md index 07fcf60e..140136af 100644 --- a/docs/adr/2026_07_bounded_remote_activity.md +++ b/docs/adr/2026_07_bounded_remote_activity.md @@ -43,7 +43,7 @@ Applying downloaded document changes enters one local-application boundary from Manual P2P commands which bypass `ReplicationService` enter the broad boundary. Direct P2P pull and push entry points are therefore both protected as finite remote work, covering the Obsidian panes, CLI, and Webapp. A pull or bidirectional synchronisation also enters the narrower finite-replication boundary because it can place documents in the local database. A push-only request remains broad-only: it cannot satisfy a local missing-chunk read and must not present itself as a delivery source. Automatic synchronisation on peer discovery, a pull requested by a remote peer, and a watched pull following a peer progress notification enter both boundaries because each can deliver local documents. A normal P2P peer-selection dialogue represents one broad finite session: it remains inside the boundary while waiting for a peer and while the person may perform repeated synchronisations, then settles when the dialogue closes and any in-flight synchronisation has finished. Closing without synchronising returns a failed result and releases the boundary. The 'Start Sync & Close' action completes its synchronisation before closing. This deliberately protects peer discovery and selection, because display sleep can interrupt discovery or connection establishment and require the person to start detection again. It may therefore retain a Wake Lock longer than the network transfer alone. A transfer performed inside that session temporarily adds a nested activity; the count remains a logical-operation count rather than a connection total. -`ChunkFetcher` enters the broad boundary synchronously when it accepts newly missing chunk identifiers, but it does not increment the finite-replication count. A typed per-identifier claim keeps the broad boundary active through queue waiting, interval throttling, `fetchRemoteChunks`, validation, local persistence, and terminal event delivery. Duplicate requests share the existing claim. Explicit absence, failure, cancellation, or a conservative five-minute period without fetcher-observable progress settles the affected claim. The five-minute value is only a last-resort leak fuse: it prevents a never-settling integration from retaining the per-identifier claim and waiter indefinitely and, once the activity runner has entered the claim task, lets the associated Wake Lock, lifecycle deferral, and indicator finish. It is not an arrival estimate, proof of remote absence, or a transport deadline. This scope is defined in detail by the chunk-arrival-quiescence ADR. +`ChunkFetcher` enters the broad boundary synchronously when it accepts newly missing chunk identifiers, but it does not increment the finite-replication count. A typed per-identifier claim keeps the broad boundary active through queue waiting, interval throttling, `fetchRemoteChunks`, validation, local persistence, and terminal event delivery. Duplicate requests share the existing claim. A first successful omission schedules a two-second retry; further omissions increase the per-identifier delay up to ten seconds while finite replication remains active. Backoff releases physical concurrency without releasing the bounded task. Observed finite completion interrupts backoff for a local recheck and, if needed, a final remote probe. A current final successful omission settles the claim; failure, cancellation, or a conservative five-minute period without fetcher-observable progress also settles it. The five-minute value is only an inactivity leak fuse, not a total retry budget: it prevents a never-settling integration from retaining the per-identifier claim and waiter indefinitely and, once the activity runner has entered the claim task, lets the associated Wake Lock, lifecycle deferral, and indicator finish. It is not an arrival estimate, proof of remote absence, or a transport deadline. This scope is defined in detail by the chunk-arrival-quiescence ADR. Rebuild operations use the same boundary at their destructive or remote phase: @@ -105,7 +105,7 @@ Unit tests cover: - direct P2P pull and push entry points entering the broad boundary, while only pull and bidirectional operations enter the finite-delivery boundary; - automatic synchronisation on peer discovery, remote pull requests, and watched peer progress entering the boundary; - P2P peer-selection sessions settling on close, including cancellation, repeated synchronisation, and a close during in-flight work; -- remote chunk fetching remaining inside the shared boundary from synchronous queue acceptance through local persistence and terminal notification; +- remote chunk fetching remaining inside the shared boundary from synchronous queue acceptance through the delayed retry of omitted identifiers, local persistence, and terminal notification; - missing-chunk waiters rechecking local storage when observed per-identifier claims and finite replication have settled; - the finite-replication count excluding other bounded work; - replicated document application sharing one local boundary, settling after the final recovery snapshot, and releasing around processing suspension; diff --git a/docs/adr/2026_07_chunk_arrival_quiescence.md b/docs/adr/2026_07_chunk_arrival_quiescence.md index bbb77426..55e9937a 100644 --- a/docs/adr/2026_07_chunk_arrival_quiescence.md +++ b/docs/adr/2026_07_chunk_arrival_quiescence.md @@ -12,6 +12,8 @@ Before this decision, a missing-chunk waiter used a fixed 5-second or 30-second Finite replication provides a stronger boundary than elapsed wall-clock time. In the absence of an error, a finite replication does not complete until it has reached the latest sequence in its scope. Once it completes, that replication cannot deliver another chunk. An on-demand fetch is a separate finite delivery path and needs its own per-identifier boundary. +A successful CouchDB lookup which omits a requested chunk is a point-in-time observation. Metadata and Chunk documents can become visible separately, so the first such result immediately after Metadata arrives does not prove that a second lookup will return the same result. The on-demand fetch lifecycle therefore needs to distinguish an initial omission from its terminal result. + ## Decision Wait only for a delivery lifecycle which is observable when the local miss is handled. Do not guess how long an unobserved producer might take. @@ -29,7 +31,7 @@ The waiting layer follows these rules: 2. Dispatch `missingChunks` synchronously when direct fetch is permitted. `ChunkFetcher` must claim accepted identifiers before dispatch returns, closing the scheduling gap without a timer. 3. If a matching claim or finite replication is active, wait for that observable producer. 4. A valid chunk arrival resolves the waiter immediately. -5. An explicit remote-missing result resolves it as missing immediately. +5. A terminal explicit remote-missing result resolves it as missing immediately. 6. Once every observed producer completes, read the requested identifier from the local database once more, bypassing the cache. 7. Return the rechecked chunk, or return unavailable. Do not add a fixed grace period after the producer has stopped. 8. If no producer is observable after synchronous dispatch, return unavailable immediately. There is no operation for a duration to represent. @@ -46,17 +48,24 @@ A successful finite replication completion is the authoritative β€˜latest’ bou - configured interval throttling; - entry into the injected bounded-activity runner; - `fetchRemoteChunks`; +- scheduled retries for identifiers omitted from successful responses; - response validation; - local database persistence; and - fetched or missing event delivery. -The claim settles on every terminal path, including explicit absence, no active replicator, rejection, invalid results, destruction, and cancellation. Its completion Promise is the task passed to the bounded `chunk-fetch` activity, keeping Wake Lock, application lifecycle deferral, the remote-work indicator, and missing-chunk delivery aligned. +When the first successful response omits an identifier, `ChunkFetcher` keeps that identifier claimed and schedules a retry after two seconds, even if no finite replication is active. Identifiers present in a partial response are persisted and settled immediately. While finite replication remains active, subsequent omissions schedule per-identifier delays of four, six, eight, and then ten seconds, capped at ten seconds. Backoff returns the physical request slot to the queue without releasing the logical claim. Eligible retries can share a batch with new identifiers, and the queue wakes without requiring a new missing-chunk event. + +When the observed finite count falls to zero, the fetcher interrupts any remaining backoff. It rechecks local persistence and, if the identifier is still absent, makes one final remote probe, subject to the configured request interval and concurrency. An already-running lookup cannot overlap another lookup for that identifier. If it began before the latest finite completion, an omitted identifier still needs a post-completion probe. A new finite operation which ends during that probe similarly makes its negative result stale. A successful omission from a current final probe emits `missingChunkRemote` and settles the current read; it does not leave a retry pending for a future synchronisation. + +The retry policy observes only finite replication, not the claim itself or the broader bounded-activity count. It does not treat the continuous live channel as a finite producer. There is no fixed total duration or attempt limit while finite replication continues; the delay cap controls request frequency, not overall lifetime. No retry state is persisted, and transport errors do not enter this missing-result retry policy. + +The claim settles on every terminal path, including explicit absence after the retry, no active Replicator, rejection, invalid results, destruction, and cancellation. Its completion Promise is the task passed to the bounded `chunk-fetch` activity, keeping Wake Lock, application lifecycle deferral, the remote-work indicator, and missing-chunk delivery aligned. ### Five-minute leak fuse An accepted on-demand claim has a separate five-minute inactivity fuse. This is a last-resort leak safety valve, not a chunk-arrival budget or a remote-request timeout. -Its purpose is to prevent a faulty integration, a never-settling Promise, or a stalled transport from retaining logical ownership indefinitely. When it fires, the coordinator releases the per-identifier claim and its waiter. If the bounded activity callback has been entered, resolving the claim also allows the associated Wake Lock, application-lifecycle deferral, and remote-work indicator to be released. `ChunkFetcher` refreshes the fuse only at observable progress points, such as entering the activity boundary, beginning and completing throttling or transfer, and completing persistence. +Its purpose is to prevent a faulty integration, a never-settling Promise, or a stalled transport from retaining logical ownership indefinitely. When it fires, the coordinator releases the per-identifier claim and its waiter. If the bounded activity callback has been entered, resolving the claim also allows the associated Wake Lock, application-lifecycle deferral, and remote-work indicator to be released. `ChunkFetcher` refreshes the fuse at observable progress points, including entry into the activity boundary, request scheduling, transfer, and persistence. Repeated successful missing responses are progress for this fuse, so it is not a five-minute total limit on retries during active finite replication. Five minutes is deliberately a conservative operational limit, not a value derived from a network protocol, a benchmark, or evidence that a missing chunk will arrive within that period. Firing the fuse neither proves remote absence nor makes the underlying request safe to abort. The current `fetchRemoteChunks` contract has no `AbortSignal`, so a physical request may still complete after its logical claim has been released. A future cancellable transport contract should add transport-specific deadlines and explicit cancellation without changing the lifecycle-based wait rule. @@ -64,7 +73,7 @@ Five minutes is deliberately a conservative operational limit, not a value deriv The unbounded live channel is not a quiescence gate because it has no natural end. Its initial pull-only catch-up is finite, however, and must enter `runFiniteReplicationActivity`. This includes the one-shot parameter fallback chain: every retry remains within the catch-up boundary until it succeeds or stops. If continuous replication later restarts with adjusted parameters, the new initial catch-up enters a new finite boundary. -Once the live channel has begun, a chunk delivered through it still resolves an existing waiter immediately, but the channel itself does not keep a new waiter open. CouchDB on-demand fetching supplies its own per-identifier claim. If a future defect demonstrates a delivery race inside a live batch, that batch lifecycle should be exposed explicitly rather than approximated with another elapsed delay. +Once the live channel has begun, a chunk delivered through it still resolves an existing waiter immediately, but the channel itself does not keep a new waiter open. CouchDB on-demand fetching supplies its own per-identifier claim and at least one delayed follow-up lookup. Further retries require active finite replication; the live channel does not extend them. If another future defect requires waiting for a live batch itself, that batch lifecycle should be exposed explicitly rather than approximated with another elapsed delay. ## Ownership @@ -80,7 +89,7 @@ Once the live channel has begun, a chunk delivered through it still resolves an - Treat a positive deprecated `timeout` only as source-compatible opt-in to lifecycle waiting. Its numeric value no longer represents an arrival duration. - Preserve `preventRemoteRequest`: no on-demand request is dispatched, although an already-active finite replication may satisfy the waiter. - Preserve Promise sharing for concurrent reads of the same chunk identifier. -- Preserve immediate explicit remote-missing results. +- Preserve immediate waiter resolution when `ChunkFetcher` emits a terminal explicit remote-missing result. - Do not change which remote types support direct on-demand fetching. ## Historical Evidence and Scope @@ -90,6 +99,7 @@ This decision addresses the lifecycle-race class rather than treating every β€˜L - [Issue #166](https://github.com/vrtmrz/obsidian-livesync/issues/166) contained logs where chunk collection failed shortly before related chunk writes appeared. It is evidence for the timing class, although that issue's hidden-file start-up path was repaired separately and is not claimed as a direct regression test here. - The 2021 timing fixes in [commit `39e2eab0`](https://github.com/vrtmrz/obsidian-livesync/commit/39e2eab0238d9c37e3653cdec884cbeed543fc23) and the extended leaf timeout in [commit `9facb577`](https://github.com/vrtmrz/obsidian-livesync/commit/9facb577601d8aceff7df547cd2a6f9357fdaa29) show that elapsed timeout values have historically been used to absorb the same ordering uncertainty. They do not provide a protocol basis for retaining 5-second or 30-second delays. - Replication pacing introduced by [commit `8d66c372`](https://github.com/vrtmrz/obsidian-livesync/commit/8d66c372e15c43a2de84a223c6385077b7724eec) and commonlib [commit `051b50c`](https://github.com/vrtmrz/livesync-commonlib/commit/051b50ca38ec4c05a11e8216ac259b4488b825f0) is a direct precedent for preventing replication progress from outrunning chunk collection. The present design expresses that dependency as an explicit lifecycle and completion recheck. +- [Issue #1224](https://github.com/vrtmrz/obsidian-livesync/issues/1224) reports repeatable burst edits where the receiving device observes Metadata, finds its Chunk absent in the first direct lookup, and then receives the Chunk after the file read has already failed. Retrying addresses that transient ordering case. It does not claim to reconstruct a Chunk which was never uploaded. - [Issue #505](https://github.com/vrtmrz/obsidian-livesync/issues/505) was traced to chunks which were genuinely absent after the former bulk-send option broke the chunks-before-metadata guarantee. Waiting cannot recreate missing data, so this decision does not claim to fix it. - [Issue #771](https://github.com/vrtmrz/obsidian-livesync/issues/771) and [Issue #986](https://github.com/vrtmrz/obsidian-livesync/issues/986) contain ambiguous or version-dependent `Load failed` reports. They remain unclaimed until the original writer and database state can be reproduced. @@ -113,6 +123,8 @@ The broad count includes operations which cannot provide the requested chunk. Us An unobserved producer has no defined start, progress, or completion semantics. A timer would therefore be a guess rather than a safety property. Relevant delivery paths must claim their work synchronously or expose a finite replication boundary; otherwise the read returns unavailable. +The on-demand backoff is not such a fallback timer. `ChunkFetcher` retains the identifier claim throughout each delay and owns the lookup it schedules. Once finite activity ends, the fetcher cuts the delay short and uses a current final probe rather than waiting for an unobserved future producer. + ### Remove every timer The arrival wait has no elapsed timer, but an implementation fault can leave a delivery claim unresolved forever. The five-minute inactivity fuse bounds that leaked logical state without being used as a successful delivery condition. @@ -125,6 +137,15 @@ Unit tests use deterministic clocks and deferred Promises to cover: - successful finite completion causing a cache-bypassing local database recheck; - immediate unavailability when no producer is observable; - per-identifier claims covering queueing, throttling, remote fetch, validation, persistence, and event delivery; +- an autonomous two-second retry after a first successful omission, retaining the claim and bounded remote activity; +- per-identifier backoff capped at ten seconds, with physical concurrency released during each delay; +- mixed retry stages sharing batches without resetting their individual delays or starving eligible retries; +- finite completion interrupting backoff, local persistence avoiding the final request, and pre-completion lookups requiring a current final probe; +- partial results settling available identifiers and retrying only absent identifiers; +- partial-batch completion and expired-request responses preserving replacement claims; +- concurrent request starts respecting the configured interval after retry and throttle waits; +- a final successful omission producing the terminal remote-missing result; +- disjoint initial and retry counts, aggregate ownership across fetchers, and timer cleanup; - explicit missing, no-replicator, rejection, invalid response, cancellation, runner rejection, and teardown paths; - overlapping claims and finite replications; - a runner which never enters the task and a request which never settles; @@ -132,13 +153,16 @@ Unit tests use deterministic clocks and deferred Promises to cover: - continuous replication's finite initial catch-up, including its parameter fallback path; and - the setting and replicator decision matrix. -Integration-style unit tests exercise `LayeredChunkManager`, `ChunkFetcher`, a memory-backed PouchDB database, and a deferred fake replicator together. A real Obsidian test is not required because the change remains behind the existing database, service, and event boundaries and does not alter platform UI or an adapter contract. +Integration-style unit tests exercise `LayeredChunkManager`, `ChunkFetcher`, a memory-backed PouchDB database, and a deferred fake replicator together. The downstream `chunk-fetch-retry` scenario exercises real CouchDB lookup results, local persistence, Vault reflection, and the actual status bar in Obsidian. Deterministic clock tests remain responsible for the exact backoff schedule and overlapping completion races; the real-runtime scenario does not substitute HTTP responses or synthesise finite-activity counts. ## Consequences - A healthy finite replication or on-demand request no longer loses a race against an unrelated wall-clock estimate. +- A Chunk omitted from the first successful CouchDB lookup receives a follow-up lookup while the same delivery claim remains active, and further retries while finite replication continues. - Successful finite replication completion provides a precise latest boundary for missing-chunk reads. - The local recheck closes event-delivery and cache timing gaps without extending the wait after completion. -- Reads no longer pause for 5 or 30 seconds when no observable operation can deliver the chunk. +- Reads no longer pause for the former 5-second or 30-second arrival budgets. Backoff schedules observable requests rather than setting an elapsed arrival deadline. +- With no finite replication active, a genuinely absent remote Chunk takes one additional lookup after two seconds. Active finite replication can extend that lifetime; its completion expedites the final probe. +- The status bar separates initial pending identifiers (`πŸ›„`) from retrying identifiers (`πŸ”`). Their sum remains the internal pending count used for restart deferral. - Relevant producers must expose a lifecycle and must continue to prove cleanup on every exceptional path. - The five-minute fuse bounds leaked logical activity, but it neither establishes remote absence nor cancels a physical request. diff --git a/docs/adr/2026_07_release_notes_and_database_compatibility.md b/docs/adr/2026_07_release_notes_and_database_compatibility.md index 5a7d80aa..4985008d 100644 --- a/docs/adr/2026_07_release_notes_and_database_compatibility.md +++ b/docs/adr/2026_07_release_notes_and_database_compatibility.md @@ -57,6 +57,8 @@ Keep configured-state inference separate from new-Vault initialisation. If an ex - On resume, clear `versionUpFlash` and persist that fail-closed change before recording the current `VER` as acknowledged. If saving fails, restore the gate. Reapply settings only after the marker has advanced so that the previously configured synchronisation behaviour can resume without reconstruction. - Preserve the original legacy review message as a structured reason when no more specific database or settings-schema reason is available. Escape it before including it in Markdown UI. - Continue to reject a remote version document which is newer than the running implementation. That receiver-side check is independent of the local upgrade review. +- From remote generation 13, assess the `used_features` list in that document as a separate compatibility dimension. A client must recognise every listed feature before it interprets the database or runs maintenance which depends on Metadata. Declare a feature before writing its representation, and retain the declaration while older data may depend on it. An unknown identifier is reported as text without requiring a descriptive label in that client. +- Do not advance the device-local `VER` acknowledgement merely because a remote feature is introduced. The remote generation and its feature list govern remote admission; `VER` remains the local compatibility review gate. Connecting to a generation-12 database does not promote it solely because the client understands generation 13. ### Onboarding activation and initialisation @@ -90,7 +92,7 @@ Keep configured-state inference separate from new-Vault initialisation. If an ex - Accepted new-device and existing-device setup cannot enable ordinary processing before the selected Rebuild or Fetch has been reserved. - An older installation cannot dismiss evidence that a newer implementation or settings schema has already been used on the device. - The Obsidian-specific dialogue depends only on a host-neutral compatibility result and the injected confirmation capability. Commonlib remains responsible for settings migration, device-local storage, and the replication gate. -- A future incompatible database change must increment `VER`, provide an actionable review message, verify the remote version negotiation, and test both the pending and acknowledged states. A major SemVer increase without those changes has no database-compatibility effect. +- A future local database change which requires compatibility review must increment `VER`, provide an actionable review message, and test both the pending and acknowledged states. A new remote representation must declare its feature before use and verify remote admission independently. A major SemVer increase alone has no database-compatibility effect. ## Verification diff --git a/docs/adr/2026_08_p2p_transport_compatibility.md b/docs/adr/2026_08_p2p_transport_compatibility.md index c9691c28..48ef540b 100644 --- a/docs/adr/2026_08_p2p_transport_compatibility.md +++ b/docs/adr/2026_08_p2p_transport_compatibility.md @@ -46,7 +46,7 @@ LiveSync will expose a separate `Connection path` choice: - `Automatic` retains normal ICE selection and is the default. - `TURN relay only` supplies `iceTransportPolicy: 'relay'` and prevents direct or server-reflexive candidates from being selected. -`TURN relay only` is enabled only when at least one syntactically valid `turn:` or `turns:` URL is configured. If the last valid TURN URL is removed while relay-only mode is selected, the dialogue restores `Automatic` and displays a concise explanation. +`TURN relay only` is enabled when a managed TURN provider is selected or at least one syntactically valid manual `turn:` or `turns:` URL is configured. If neither is available while relay-only mode is selected, the dialogue restores `Automatic` and displays a concise explanation. Selecting a managed provider does not itself force relay use; `Automatic` retains normal ICE selection. The route policy is an ordinary P2P profile property. It is retained in P2P connection strings and encrypted Setup URIs so that an imported compatibility profile has reproducible transport behaviour. @@ -60,7 +60,15 @@ The first settings revision retains the existing storage and dialogue contract o A future interface may present the existing comma-separated value as ordered `turn:` and `turns:` URL rows without changing its serialised representation. A structured list of multiple credential profiles is deferred until a provider or self-hosted use case requires different credentials in the same P2P profile. -Static long-term credentials are the supported first stage. Managed providers may return short-lived credentials, but LiveSync must not store a provider API token or a Coturn shared authentication secret. A future managed-credential design needs a separately trusted HTTPS endpoint, expiry handling, refresh behaviour, failure reporting, and a clear Setup URI policy. It is not represented as another static password field. +Static long-term credentials remain supported. For managed credentials, a host preparation hook requests ICE settings and places them on a connection-only copy of `P2PSyncSetting`. Service-specific requests and validation belong under `src/integrations/`; Commonlib consumes that copy and owns room reuse, expiry checks, and replacement. It has no provider catalogue or source factory. + +A user-supplied provider API token is persisted as a sensitive P2P profile setting and included in encrypted Setup URI sharing, so that participating devices can use the same configuration without repeated token entry. Existing profile-URI encryption covers the saved token; its flat runtime projection is omitted from persistence. Reports and logs redact the complete provider configuration and issued credentials, including inactive profiles and settings projections. Coturn's server-side shared authentication secret remains outside client settings. + +Issued short-lived TURN credentials and their expiry remain in memory. The existing room reuse decision checks both the effective connection settings and credential validity. When reconciliation finds expired credentials, it uses the normal room retirement and replacement path with newly acquired credentials. Replacement may cancel an in-progress transfer; the next replication attempt uses stored checkpoints and revision comparison to retain received progress. Whether that next attempt starts automatically follows the existing synchronisation policy. + +Time passing alone does not trigger acquisition or disconnection. This design adds no renewal timer, per-peer acquisition hook, configuration update on raw peers, or credential-driven ICE restart. Internal peer reconnection within an unchanged room does not guarantee fresh issuance. Acquisition failure is reported without changing the selected provider or route policy. See [TURN connection settings](../design_docs/renewable_turn_credentials.md) for the preparation hook, persistence and sharing formats, room replacement, and verified replication continuation behaviour. + +Relay-only validation accepts a valid managed TURN configuration as well as the existing manual URL list. Failure to acquire usable TURN entries keeps relay-only mode selected and reports the connection failure; it does not restore `Automatic` silently. ### TURN allocation check and route diagnostics diff --git a/docs/design_docs/chunk_retrieval_and_waiting.md b/docs/design_docs/chunk_retrieval_and_waiting.md index 81038d92..17d93c71 100644 --- a/docs/design_docs/chunk_retrieval_and_waiting.md +++ b/docs/design_docs/chunk_retrieval_and_waiting.md @@ -53,7 +53,9 @@ For CouchDB, `readChunksOnline` changes what normal replication includes, not th | `true` | No | Primary chunk delivery after metadata arrives. | | `false` | Yes | Recovery fallback for a chunk which is unexpectedly unavailable locally. | -`concurrencyOfReadChunksOnline` and `minimumIntervalOfReadChunksOnline` affect only the scheduling of CouchDB on-demand requests. They do not change whether a request may be dispatched or which lifecycle a reader observes. Accepted identifiers remain claimed while they wait for a concurrency slot and while the configured interval is applied. A minimum interval of five minutes or more is an exceptional value: the inactivity fuse may release the logical claim before that deliberate pause completes. This safety precedence does not abort the delayed physical request. +`concurrencyOfReadChunksOnline` and `minimumIntervalOfReadChunksOnline` affect only the scheduling of CouchDB on-demand requests. They do not change whether a request may be dispatched or which lifecycle a reader observes. Accepted identifiers remain claimed while they wait for a concurrency slot and while the configured interval is applied. Backoff releases the physical concurrency slot but retains logical ownership. Eligible retries and new identifiers can share a batch of up to 100 identifiers. An owned timer wakes the queue at the earliest eligible retry without requiring a new event. New arrivals do not reset existing retry times, and eligible older work precedes newer queued work. + +After every wait or asynchronous local recheck, the fetcher checks the configured interval against the latest shared request time and reserves its start synchronously. A minimum interval of five minutes or more is an exceptional value: the inactivity fuse may release the logical claim before that deliberate pause completes. An expired claim is not dispatched after the pause; an already-running physical request is not aborted by the fuse. ## Wait State Machine @@ -62,11 +64,12 @@ For CouchDB, `readChunksOnline` changes what normal replication includes, not th 3. Register one shared waiter per missing identifier. 4. If policy permits direct fetch, emit `missingChunks`. `ChunkFetcher` synchronously creates the per-identifier claim before the event dispatch returns. 5. Observe both the matching claim and `finiteReplicationActivityCount`. -6. Resolve immediately if a valid chunk or explicit remote-missing event arrives. -7. If an observed producer remains active, do not charge elapsed time against an arrival budget. -8. When all observed producers end, bypass the cache and read the identifiers from the local database once. -9. Return the rechecked chunk, or return unavailable. Do not add another fixed grace after the authoritative boundary. -10. If no producer is observable after synchronous dispatch, return unavailable immediately. +6. Resolve immediately if a valid chunk or terminal explicit remote-missing event arrives. +7. If a successful direct-fetch response omits an identifier, apply the per-identifier retry policy below. Settle identifiers present in a partial result and retry only those which remain absent. +8. If an observed producer remains active, do not charge elapsed time against a general arrival budget. +9. When all observed producers end, bypass the cache and read the identifiers from the local database once. +10. Return the rechecked chunk, or return unavailable. Do not add another fixed grace after the authoritative boundary. +11. If no producer is observable after synchronous dispatch, return unavailable immediately. If new relevant activity starts while the final database recheck is pending, that result becomes stale. The waiter remains active until the newer producer completes and a current recheck finishes. @@ -84,18 +87,30 @@ The continuous live channel is intentionally excluded because it has no completi ## Meaning of an On-demand Claim -An accepted identifier remains claimed from synchronous queue acceptance through throttling, physical fetch, validation, local persistence, and terminal event delivery. The claim is identifier-scoped because a global remote-work count cannot say whether unrelated work can provide this chunk. +An accepted identifier remains claimed from synchronous queue acceptance through throttling, physical fetch, validation, local persistence, retry delays, and terminal event delivery. The claim is identifier-scoped because a global remote-work count cannot say whether unrelated work can provide this chunk. -The claim finishes when the fetcher has recorded an outcome for the identifier. A transport error, missing active replicator, or invalid result releases the claim without emitting an explicit remote-missing result unless the remote actually supplied that information. +The claim finishes when the fetcher has recorded an outcome for the identifier. A successful first omission schedules a two-second retry even if finite replication is already inactive. While finite replication remains active, each subsequent omission increases that identifier's delay by two seconds, up to ten seconds: `2, 4, 6, 8, 10, 10, ...`. Joining a different batch does not reset its retry stage. Each retry first bypasses the local cache and disables further remote dispatch and delivery waiting for its local recheck. + +When the observed finite count falls to zero, remaining backoff is interrupted. The fetcher rechecks local persistence and makes a final remote probe only for still-absent identifiers, respecting concurrency and minimum request spacing. A pre-completion in-flight lookup cannot count as that final probe: if it omits the identifier, a subsequent post-completion lookup is needed, without overlapping requests for the same identifier. If another finite operation ends during the final probe, its negative result is stale too. The current final successful omission emits the terminal explicit remote-missing result and settles the current read. No retry remains for a future synchronisation. + +A transport error, missing active Replicator, or invalid result releases the claim according to its existing terminal path rather than entering this missing-result retry. The retry gate is finite replication alone: neither the fetcher's own claim nor broader bounded remote work keeps it alive. There is no attempt limit or absolute elapsed limit while finite replication continues. All retry state is in memory; destruction and inactivity expiry remove queued work and its timer. + +Each request retains the identity of the claims it accepted. If another read claims the same identifier after an earlier claim settles or expires, the earlier request cannot release the replacement claim, refresh its fuse, or report unavailability for it. This also applies when a partial batch has already settled one identifier but is still retrying another. + +The status indicators count two disjoint sets of pending on-demand Chunk identifiers. `πŸ›„` shows identifiers with no successful missing response yet; `πŸ”` shows identifiers omitted at least once, including backoff, retry requests, and the final probe. Thus `πŸ›„3 πŸ”2` means five pending identifiers. The atomic `chunkFetchCounts` snapshot provides this classification, while `collectingChunks` retains their total for the existing restart-deferral check. Moving between categories does not change that total. + +Each fetcher contributes its unique accepted identifiers from queueing through terminal delivery. Repeated requests and retry attempts do not increase the count. Settled, expired, and destroyed claims leave the count; one fetcher's teardown preserves another fetcher's contribution. These are not counts of replication connections or all missing Chunks. Zero means that no on-demand claims remain, not that every Chunk was retrieved successfully. ## Meaning of the Five-minute Value -The five-minute value is an inactivity leak fuse for an accepted on-demand claim. It is the only elapsed duration in this state machine, and it is not a normal terminal condition. +The five-minute value is an inactivity leak fuse for an accepted on-demand claim. It is not a normal terminal condition and is distinct from the backoff which schedules identified follow-up lookups. The fuse bounds retention if a faulty activity runner never enters its task, a Promise never settles, or a transport stops making observable progress. It prevents the per-identifier claim and waiter from remaining live forever. Once the bounded activity callback has entered, releasing the claim also allows Wake Lock, application-lifecycle deferral, and the remote-work indicator associated with that callback to finish. Observable progress rearms the fuse. Five minutes is a conservative operational ceiling rather than a measured chunk-arrival expectation. It must not be used to infer that the remote lacks a chunk, and it does not abort the physical request. `fetchRemoteChunks` does not yet accept an `AbortSignal`, so the request may complete after the logical state has been released. Transport cancellation and transport-specific deadlines are separate future work. +Backoff does not resolve the waiter by elapsed time or prove that another producer will deliver the Chunk. Successful missing responses refresh the inactivity fuse, so retries can continue beyond five minutes while finite replication remains active. This is an intentional distinction between inactivity protection and a total lifetime limit. + The old 5-second and 30-second constants remain exported for source compatibility only. A positive deprecated `ChunkReadOptions.timeout` opts into lifecycle waiting, but its numeric value is ignored. Zero or a negative value still requests an immediate result. New code uses `waitForDelivery` explicitly. ## Test Obligations @@ -110,6 +125,17 @@ Changes to this behaviour must keep automated coverage for: - overlapping finite operations and overlapping per-identifier claims; - activity restarting while a local recheck is pending; - direct fetch queueing, throttling, persistence, and terminal notification; +- an autonomous two-second retry after a first successful omission, with bounded remote activity retained throughout; +- per-identifier `2, 4, 6, 8, 10, 10, ...` backoff while finite replication is active; +- backoff releasing physical concurrency, mixed-stage batching, and eligible retries not being starved by new work; +- expiry of the earliest queued claim preserving the scheduled retry for later identifiers; +- finite completion interrupting backoff and pre-completion in-flight requests requiring a current final probe; +- local persistence avoiding an unnecessary retry, including completion during request-interval throttling; +- partial fetch results settling available identifiers and retrying only absent identifiers; +- partial-batch completion and expired-request responses preserving replacement claims; +- disjoint initial and retry counts retaining queued identifiers without duplicates and releasing only settled ownership; +- concurrent request starts respecting the configured interval after retry and throttle waits; +- terminal unavailability after a current final successful omission; - explicit remote absence versus transport or replicator failure; - runner rejection, cancellation, teardown, and an operation which never enters its task; - leak-fuse refresh at observable progress points; and diff --git a/docs/design_docs/configurable_id_derivation.md b/docs/design_docs/configurable_id_derivation.md new file mode 100644 index 00000000..8a88ce5f --- /dev/null +++ b/docs/design_docs/configurable_id_derivation.md @@ -0,0 +1,515 @@ +--- +date: 2026-09-29 +commonlib-version: "0.1.33" +self-hosted-livesync-version: "1.0.32" +status: unreleased +--- + +# Configurable ID derivation + +## Purpose and baseline + +Introduce an optional, saved secret for deterministic Chunk IDs and obfuscated +Metadata document IDs. This allows an E2EE passphrase to change without also +changing those IDs, and allows their derivation to use an independent secret. +Identical inputs must produce identical IDs on participating devices so that +Chunks can be reused and edits to the same path share one document identity. + +The baseline is [PR #1222](https://github.com/vrtmrz/obsidian-livesync/pull/1222), +including its passphrase-persistence correction at commit +`126d6eadb858a79a08ad7f600061e54fc8d31196`. Commonlib `0.1.33`, published with +the `next` tag, provides the construction described here. It replaces the +independent Chunk algorithm from the `0.1.32` prerelease without a +compatibility branch or a new settings version. Legacy ID generation remains +unchanged. LiveSync pins the published `0.1.33` package and its registry +integrity in the lockfile. +Its [Internal Metadata encryption design](https://github.com/vrtmrz/obsidian-livesync/blob/126d6eadb858a79a08ad7f600061e54fc8d31196/docs/design_docs/internal_metadata_encryption.md) +remains the basis for Properties encryption and CouchDB feature admission. +This document records an unreleased LiveSync feature. It does not select a +plug-in release version. + +## Feasibility + +The change is feasible within the existing architecture. Commonlib already +centralises Chunk hashing, path-to-ID conversion, settings persistence, and +Setup URI encoding. Chunk reads follow stored IDs, so changing the generator +does not require a new Chunk reader or content representation. + +The work spans Commonlib and its consumers. The principal constraints are +agreement on document IDs, complete propagation of the saved secret, and cache +behaviour after a setting change. The construction and transport-specific +agreement checks are described below. No database migration framework is +required. + +## Scope + +| Value or operation | Proposed behaviour | +| -------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Encrypted Chunk IDs | Use the saved ID secret in the new mode. Keep legacy generation when the option is absent. | +| Obfuscated Metadata document IDs | Use the same saved secret with a separate derivation purpose. Preserve existing path normalisation and namespace handling, including ordinary files, `i:`, `ix:`, and supported legacy `ps:` entries. | +| Unobfuscated document IDs | Retain the current path-based identity. | +| Content and Properties encryption | Continue using the E2EE passphrase and existing encryption format. | +| CouchDB/PouchDB `_rev` | Retain existing revision generation and replication behaviour. | +| Internal content digests and transport bookkeeping | Retain existing behaviour unless they directly construct one of the IDs above. | + +Document IDs are already assigned in the local database. Properties encryption +protects the path and other fields during transfer, while preserving `_id`. +Consequently, the saved ID secret must reach local path conversion as well as +remote-facing code. Path Obfuscation continues to control whether this +conversion is used; this proposal does not enable it automatically. + +Journal keys which incorporate document IDs inherit the resulting IDs. They do +not need another secret. Content digests inside encrypted Customisation Sync +content also do not need a separate setting. + +Automatic migration or enablement of existing or already migrated users, +Setup URI expiry or revocation, QR format security changes, revision redesign, +remote-only E2EE passphrase rotation, and a new remote configuration management +protocol are outside this change. + +## Input and saved state + +New Vault setup selects independent ID derivation and a random source by +default when E2EE is enabled. An existing Vault with no saved key retains +legacy mode until the user selects a new key explicitly. The setup dialogue +shows three radio choices: + +1. Keep current configuration, selected by default for an existing Vault. It + retains the saved key when present and otherwise retains legacy ID + generation. A small description below this choice shows which configuration + is currently saved. In legacy mode, changing the E2EE passphrase still + changes IDs. +2. Generate a random ID key, selected by default for a new Vault. +3. Set an ID key. Three nested radio choices derive it once from the current + E2EE passphrase, accept a source string, or import a tagged recovery code. + Only the latter two show the text input. + +The action is not persisted. An ordinary source is converted to a key when the +settings are applied, and then discarded. A tagged `sls-id-v1:` recovery code +imports its exact 256-bit key without deriving it again, including when pasted +into the source-string input. The recovery-code choice accepts only tagged +codes. A malformed tagged code is rejected. If either input is empty, a saved +key is kept; without a saved key, the dialogue requests input. Changing the +E2EE passphrase later preserves the saved ID key. Cancelling or failing to +save preserves the previous settings. + +The source itself cannot be recovered from its key. A user can explicitly +display and copy the saved key as a tagged recovery code on the local device; +it is hidden when the dialogue opens. The dialogue warns that anyone who needs +recovery after losing every device should save that code or choose a source +they can reproduce. +Turning E2EE off retains the saved value but suspends its use for ID generation. +The setup dialogue disables new ID-key configuration while E2EE is off. Turning +E2EE on again reactivates the same value. Existing E2EE re-encryption and +Rebuild requirements still apply when the passphrase changes. + +The source-input warning concerns only that input. The E2EE passphrase retains +its separate, existing storage behaviour. The recovery code contains the actual +saved ID key and must be handled as a secret. + +Settings need to represent legacy mode or a supported version plus a derived +secret. Final field names belong in Commonlib. A declared new version with a +missing, malformed, or unavailable secret is an error; it must not silently +fall back to legacy generation. Loading or exporting an already derived value +must not derive it again. + +## Deterministic derivation + +The required contract is: + +```text +source string --versioned derivation at save--> saved ID secret +saved ID secret + Chunk content ------------> Chunk ID +saved ID secret + canonical path -----------> obfuscated document ID +E2EE passphrase ----------------------------> content and Properties encryption +``` + +Derivation is offline and deterministic across supported runtimes. Its version +fixes the text encoding, treatment of Unicode and whitespace, salt, parameters, +and saved representation. It must not depend on server state, the E2EE Security +Seed, a device identifier, time, or device-specific iteration calibration. +Repeated saving of the same source under the same version produces the same +value. Reusing that source in another Vault consequently also reuses the value. + +Version 1 uses PBKDF2-HMAC-SHA-256 with 310,000 iterations, the UTF-8 bytes of +the source after NFC normalisation, the fixed salt +`self-hosted-livesync:id-source:v1`, and a 256-bit output encoded as 64 lowercase +hexadecimal characters. Whitespace is preserved. The existing +`idDerivationVersion: 1` setting, saved-settings fields, and recovery-code format +remain unchanged; this implementation change does not add an ID format version +or migration path. + +Obfuscated document IDs continue to use the saved 256-bit value directly as the +key for full HMAC-SHA-256. Their message remains UTF-8 encoding of +`self-hosted-livesync:id-v1:document`, a NUL byte, and the canonical path. +Agreement proofs likewise retain their existing full-HMAC messages. Neither +path uses the new Chunk-specific cache. + +For encrypted Chunk IDs, first compute xxHash64 over the UTF-8 bytes of the +exact Chunk text with seed 0. Encode its result as a fixed 16-character +lowercase hexadecimal prehash. Derive a Chunk-specific subkey from the saved +32-byte value, then HMAC the domain-separated prehash: + +```text +Kchunk = HMAC-SHA-256( + saved 32-byte key, + UTF8('self-hosted-livesync:id-v1:chunk-key:xxhash64') +) +prehash = fixed16lowerhex(xxHash64(UTF8(exact Chunk text), seed 0)) +Chunk ID = full64lowerhex(HMAC-SHA-256( + Kchunk, + UTF8('self-hosted-livesync:id-v1:chunk:xxhash64' + NUL + prehash) +)) +``` + +The resulting Chunk ID is the full 64-character lowercase hexadecimal HMAC +output. Existing namespace prefixes remain outside the digest. Keyed Chunk IDs +use this fixed prehash regardless of `hashAlg`; legacy mode and its existing +hash selection remain unchanged. + +The Chunk-specific HMAC subkey and imported key, along with the WASM xxHash64 +generator, are cached per `HashManager` and active saved key. Concurrent +preparation is shared. Replacing the manager or key, turning E2EE off, or +returning to legacy mode clears the cache; failed preparation can be retried. +This cache adds no persistent state. The current E2EE setting also selects the +legacy encrypted or plain Chunk route when a manager remains alive while E2EE is +turned off. + +### Security properties and limits + +A derived value remains a secret capable of generating IDs. Hashing does not +increase the entropy of its source. A password KDF adds guessing cost; it does +not make a weak source strong. HKDF alone does not provide that password +stretching. See [RFC 8018](https://www.rfc-editor.org/rfc/rfc8018.html#section-8) +and [RFC 5869](https://www.rfc-editor.org/rfc/rfc5869.html#section-4). + +The random default separates ID generation from the E2EE passphrase. Deriving +both secrets from the same source retains a relationship with the original +passphrase, even after that passphrase changes. An independent source with +sufficient entropy provides the intended separation. HMAC with purpose +separation is the construction for using that secret; see +[RFC 2104](https://www.rfc-editor.org/rfc/rfc2104.html). +The fixed derivation salt means that reusing a source across Vaults reuses the +ID key; use separate sources when independent Vault identities are required. + +CouchDB authentication and database access control remain the first access +boundary. This design also considers exposure through database credentials, +server administration, or backups. It does not promise to conceal equality, +document counts, revision history, or ciphertext lengths from database readers. + +For Chunk IDs, xxHash64 is a public, non-cryptographic prehash. Distinct Chunk +texts which produce the same 64-bit prehash produce the same ID under the same +saved key. The final 256-bit HMAC does not restore distinctions lost at that +stage, so collision resistance for Chunk IDs is bounded by xxHash64 rather than +by the HMAC output width. This limit is an accepted trade-off for bounding the +content processed by HMAC. + +The independent ID key preserves existing file contents, Chunk representation, +and `_rev` behaviour. It does not change the privacy properties of those +formats. Payload and virtual file padding remain outside this change. + +## Sharing, import, and storage + +Include the saved derived value and its version in Setup URIs, protected by the +existing, separate Setup URI passphrase. Import the saved value directly. +Additional devices therefore need neither the original source nor another +derivation step. Manual setup can reproduce it by entering the same source and +version, or by importing the tagged recovery code. After an E2EE passphrase +change, the current passphrase cannot be assumed to reproduce the old ID secret. + +The standalone Setup URI generator uses a fresh random 256-bit ID key by +default, prints its tagged recovery code, and accepts that code for repeatable +generation for the same Vault. `id_mode=legacy` selects the old ID behaviour. +Running it again without the code produces a different key, so the generated +URI must not be treated as an update for an existing remote. + +QR sharing includes the same fields through the existing QR representation and +warnings. Its current payload is not encrypted like a Setup URI. The agreed +scope accepts that existing sharing model and user responsibility for keeping +QR material private; it adds no QR storage or expiry mechanism. + +| Boundary | Required handling | +| ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Existing settings and old complete Setup URI/QR/P2P imports | Missing fields select legacy behaviour. Do not inherit an unrelated value already present on the receiving device. | +| Ordinary partial setting updates | Preserve the current secret and version when neither is supplied. | +| New-format imports | Validate version and value together before applying or starting database work. | +| Local persistence | Integrate the secret explicitly with sensitive-configuration encryption and loading. Adding an arbitrary field does not currently provide this protection. | +| Reports, logs, and Markdown settings | Redact the secret in reports and logs; treat it as a credential in the existing Markdown export/import policy. An export which omits credentials must omit this secret. | +| Remote profiles, CLI, WebApp, WebPeer, and direct writers | Carry the effective value and version through every supported configuration path. A selected new mode must never degrade silently to legacy mode. | + +Setup URI JSON encoding can carry ordinary new settings, but QR encoding uses +an explicit key-index table. Append stable QR entries without reordering old +ones. Complete imports and partial edits must have distinct missing-value +semantics even where the current implementation merges settings objects. +In particular, `SetupManager` currently merges decoded URI settings over the +receiving device's settings. Complete imports must normalise the new fields +before that merge to prevent accidental inheritance. + +## Compatibility and changes to existing data + +| Difference or change | Consequence | +| ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | +| Different Chunk derivation only | Existing content remains readable through Metadata `children`; new writes can duplicate Chunks and reduce reuse. | +| Different obfuscated document ID derivation | The same path can become separate documents. Treat this as an incompatible configuration requiring resolution. | +| New E2EE passphrase, unchanged saved ID secret | IDs remain stable for unchanged content, paths, and other ID settings. Re-encryption still requires the existing E2EE workflow. | +| Enabling, replacing, or disabling the option with Path Obfuscation active | Document identity changes. Use the established authoritative Rebuild and secondary-device Fetch workflow. | +| Existing installation with no new option | Preserve its exact legacy behaviour; do not derive or copy a value during upgrade. | + +Copying the legacy passphrase into the new derivation does not preserve legacy +IDs, because the derivation itself changes. This proposal therefore makes no +automatic or seamless migration promise. Before an explicit transition, update +and stop the participating devices, select the authoritative data, and use the +existing [Rebuild and Fetch procedures](../recovery.md). Share the resulting +configuration before other devices rejoin. One-entry +[Metadata ID repair](metadata_document_id_validation_and_repair.md) does not +perform this transition. + +Commonlib's Chunk cache includes content-to-ID lookup before hashing. Changing +the hash function alone can keep producing old IDs, and reading old Chunks can +populate that lookup again. The implementation must distinguish read reuse +from the ID selected for a new write, including after manager replacement and +restart. Readers continue accepting referenced legacy Chunks. + +## Agreement checks and older clients + +Keep checks focused on preventing incompatible document identities. Reuse the +existing configuration review and replication admission paths. A mismatch +must not be treated as an automatically alignable Chunk setting when document +IDs depend on it. Show a mismatch or unsupported version without exposing the +saved secret. + +The advertised ID version is used for comparison only. Ordinary Tweak alignment +preserves each device's ID version and key together, including when only Chunk +IDs differ. A document ID mode mismatch requires explicit configuration through +a Setup URI or the matching key, rather than adopting a version without its key. + +For CouchDB, extend the supported feature set in the remote feature contract +introduced by PR #1222, then declare the new requirement before writing data +under it. That mechanism rejects unsupported features at admission and provides +a best-effort stop when an unsupported requirement arrives later. It checks +format support, not equality of saved secrets, and does not make a live +migration atomic. Journal can extend its existing milestone compatibility path; +P2P needs its separate admission handling. The CouchDB feature contract alone +cannot protect those transports. + +The implementation checks up to two remote documents in each ordinary and +internal obfuscated-ID namespace before CouchDB replication or direct writes. +This includes a legacy-mode caller connecting to a remote which uses keyed IDs. +For each available sample it recomputes the ID from the decrypted path; any +mismatch rejects the connection. An empty database, or one with no usable +sample, is reported as unverified and may proceed because there is no observed +document identity to conflict with. A sampled match is evidence, not a proof +that every document has the same identity; an unsampled mixture remains a +limitation of this bounded check. + +When E2EE and Path Obfuscation are both active, Journal stores an +E2EE-encrypted, domain-separated proof in its existing milestone. It is +encrypted before the milestone is uploaded and compared on later connections. +An established milestone without this proof requires a Rebuild before the new +document IDs can be used. Journal advertises a new compatibility range for +keyed document IDs so older clients reject it. P2P compares a +purpose-separated HMAC over a fresh challenge during peer admission; the proof +is not stored. When Path Obfuscation is off, different keys affect only Chunk +IDs, so Journal keeps its legacy compatibility range and P2P does not require +key agreement. Neither transport publishes the key or a plaintext verifier in +Tweak values. CouchDB and direct writers use the document sample check above +rather than a stored verifier. The sample check uses the host's path service +with the attempted settings snapshot so stored non-canonical paths are treated +the same way as ID generation. +The existing `_rev` behaviour for ordinary content remains unchanged. + +## Implementation responsibilities + +The following are the confirmed integration points in the reviewed baseline. +Commonlib paths refer to its package implementation, not a source mirror in +this repository. + +| Owner and entry points | Work | +| -------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Commonlib `HashManagerCore`, concrete hash managers, `PathService`, and `path2id_base` | Select legacy or new derivation consistently; preserve path and namespace semantics. | +| Commonlib `EntryManagerImpls`, `LayeredChunkManager`, and `LiveSyncManagers` | Keep referenced Chunks readable and invalidate or partition generation-dependent caches. | +| Commonlib settings definitions/lifecycle, `SettingService`, `pickEncryptionSettings`, and `API/processSetting` | Own version validation, persistence, copying, imports, Setup URI encoding, and QR slots. | +| Commonlib compatibility assessment and replication implementations | Classify identity differences, protect any comparison data, and enforce supported formats at each transport boundary. | +| Commonlib `API/DirectFileManipulatorV2` | Carry the option through its explicit settings and path-obfuscation configuration. | +| LiveSync `SetupRemoteE2EE.svelte`, `PaneRemoteConfig.ts`, and `SetupManager.ts` | Implement the three configuration actions and three nested ID-key inputs, configured state, local recovery-code reveal, and existing Apply/Rebuild/Fetch choices. | +| LiveSync `replicatorConfigurationIdentity.ts`, `reportTool.ts`, and `ModuleObsidianSettingAsMarkdown.ts` | Replace connections when effective settings change, redact the secret, and apply credential-sharing rules. | +| CLI, browser applications, and setup tools | Use the same Commonlib contract in manual setup and imports; generate new Setup URIs with a reusable random ID key by default. | + +Implement Commonlib changes in its own repository, validate its packed artefact, +and validate LiveSync against that exact dependency before adopting a released +version. Translations remain outside this implementation scope. + +## Validation + +The fixed-vector and cache tests first failed against unchanged Commonlib +`0.1.32`, then passed after the implementation change. Commonlib's 2,036 Unit +tests, type check, package boundary, and isolated packed-package checks pass. +Three Integration tests against real CouchDB and Object Storage verify direct +access, Journal agreement, and rejection before control-document changes. +The same-manager E2EE-off regression was reproduced and fixed. + +LiveSync's 1,049 Unit tests, type and lint checks, production build, and iOS 15 +bundle syntax check pass after installing the exact published `0.1.33` +package. Its tarball matches the validated publication candidate, and all 559 +installed package files match the registry artefact. The resulting bundle is +identical to the one checked before publication. + +Real Obsidian two-Vault checks with the accepted Chunk construction cover +matching-key synchronisation, incompatible document key rejection, and +differing Chunk keys with visible paths. The Review Harness also passes with +the published package, verifying actual ID calculations and report copying +without changing live settings. Its adapter imports `HashManager` through +Commonlib's focused `/hashing` entry, whose package checks cover public types, +Node execution, and browser bundling. + +### Current Chunk calculation performance + +The actual Commonlib `HashManager` implementations were compared in Obsidian +1.12.7 on ARM64 Linux. Six samples rotate all three variants through each +execution position twice. The table reports median total ID calculation time; +1,000 means the total for 1,000 IDs, not the time per ID. Inputs are synthetic. + +| Input | IDs per sample | Legacy xxHash64 | Previous independent HMAC | Updated independent ID | +| --------------------------------- | -------------: | --------------: | ------------------------: | ---------------------: | +| 256-byte text | 1,000 | 4.50 ms | 41.40 ms | 24.45 ms | +| 4 KiB text | 1,000 | 10.00 ms | 96.25 ms | 29.70 ms | +| 32 KiB text | 1,000 | 48.40 ms | 494.40 ms | 68.95 ms | +| 10 MiB binary, default splitting | 103 | 19.60 ms | 192.50 ms | 21.45 ms | +| 50 MiB binary, default splitting | 512 | 99.85 ms | 972.75 ms | 106.65 ms | +| 10 MiB binary, Self-hosted preset | 5 | 24.10 ms | 216.90 ms | 18.50 ms | +| 50 MiB binary, Self-hosted preset | 35 | 118.40 ms | 1,067.00 ms | 91.75 ms | + +The first updated ID, including Chunk-key preparation, took 1.3 ms in this +run. Repeated measurements exclude preparation, warm-up, and pauses. Binary +cases use the actual splitter and Base64 representation; all decoded bytes +and repeated IDs were checked. Splitting, Base64 conversion, database work, +payload encryption, and transfer are outside the measured interval. These +results establish lower ID calculation cost on this host, not a complete +Rebuild speedup or native mobile performance. + +### Native-device ID measurements + +User-supplied Review Harness reports compare the previous and updated builds +on Android 13 and iOS 18.7. Each value is the median total time for 1,000 +independent Chunk IDs, using three samples in each run. + +| Input | Android, previous | Android, updated | iOS, previous | iOS, updated | +| ------------- | ----------------: | ---------------: | ------------: | -----------: | +| 256-byte text | 53.6 ms | 37.8 ms | 21 ms | 20 ms | +| 4 KiB text | 70.9 ms | 43.4 ms | 22 ms | 22 ms | +| 32 KiB text | 151.1 ms | 66.5 ms | 36 ms | 39 ms | + +Android's 32-KiB result takes about 56% less time. The corresponding iOS +result increases by 3 ms per 1,000 IDs; separate runs with three samples do +not establish the cause of that difference. Across these sizes, the updated +independent calculation adds approximately 18–25 ms per 1,000 IDs over each +device's legacy xxHash64 calculation. Save-time key derivation has medians of +46.6 ms on Android and 51 ms on iOS. + +These reports measure synthetic ID calculations, excluding database work, +payload encryption, and transfer. Android's heap samples remain constant, +and iOS does not expose them, so the reports do not establish memory usage or +improvement. The updated build has not been measured on Windows. + +The checks below are historical reference evidence for the predecessor +independent-ID implementation, which used full-content HMAC-SHA-256 for Chunk +IDs. They do not validate the current xxHash64-prehash construction. + +### Historical predecessor checks + +Earlier consumer validation used a local Commonlib `0.1.32` candidate. Clean +installations with npm 10 and npm 11, type checking, lint, Svelte checks, the +production build, and the iOS 15 bundle compatibility check passed. LiveSync +had 1,039 passing Unit tests, six passing Integration tests against real +CouchDB, and seven passing Setup URI utility tests with the frozen Deno +lockfile. + +Predecessor Commonlib candidate checks covered deterministic vectors, Unicode +normalisation, legacy behaviour, encrypted settings persistence, imports, +cache transitions, and transport admission. Its Integration tests against real +Object Storage accept a matching Journal key, reject a different document ID +key before changing the milestone, and allow different Chunk keys when paths +remain visible. A direct-access Integration test against real CouchDB reads +with the same key and rejects a different key before changing the version +document. These library tests complemented the consumer checks for that +predecessor; they were not additional LiveSync Unit tests. + +Real Obsidian 1.12.7 on ARM64 Linux verifies the following consumer boundaries: + +| Boundary | Verified behaviour | +| ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| CouchDB synchronisation | Two Vaults exchange notes in both directions with matching document and Chunk IDs. Different Chunk keys also work with Path Obfuscation off. | +| CouchDB rejection | Ordinary replication rejects different or legacy document ID keys before downloading files or changing remote documents and checkpoints. | +| Visible onboarding | Both a separate source and the random default persist an encrypted key, transfer it through a Setup URI, complete Fast Fetch, and synchronise in both directions. The `%`-prefixed E2EE passphrase survives restart. | +| Input and recovery | The radio controls and disabled styling are exercised. An empty first source keeps the dialogue open with an error; empty input keeps an existing key. A recovery code restores the same key. | +| Journal upload | The uploaded documents and Chunks have keyed IDs, and the first Object Storage milestone contains the encrypted agreement proof. | +| Credential-free Markdown | Neither the saved ID key nor its encrypted representation appears in exported settings Markdown. | + +A CLI P2P E2E run with a local relay imports encrypted Setup URIs, transfers a +note with matching keys, and rejects a peer with a different document ID key. +A separate check loads the published `0.1.31` DirectFileManipulator in another +process: it reads a legacy remote, and rejects an independent-ID remote with +or without Path Obfuscation, leaving remote documents and checkpoints +unchanged. This checks the previous library API, rather than an older +Obsidian installation. Bounded sampling tests cover empty and mixed document +collections; they do not establish that every document in a remote is +compatible. + +### Performance reference for the predecessor + +The measurements below are historical results for a local predecessor +Commonlib `0.1.32` candidate which used full-content HMAC-SHA-256 for +independent Chunk IDs. They are reference evidence only, not performance +results for the accepted xxHash64-prehash construction. Both modes enable E2EE +and Path Obfuscation; the legacy baseline uses `xxhash64`. Three trials per +mode alternate their order. These are synthetic corpora and serial local +writes, excluding Vault enumeration, remote payload encryption, and transfer; +they are not timings of the complete Rebuild action. + +Saving an ordinary ID source takes 52–57 ms in Node.js 24, with a median of +56 ms. This PBKDF2 operation happens once when saving the source. Per-Chunk +and per-document IDs use the saved key and do not repeat PBKDF2. + +The actual Obsidian renderer gives these median times for 1,000 serial calls +to the Commonlib hash manager or Path Service, after warm-up: + +| Input | Legacy | Independent ID | +| ------------------------ | ------: | -------------: | +| Distinct 256-byte Chunks | 5.4 ms | 43.3 ms | +| Distinct 4 KiB Chunks | 15.0 ms | 104.8 ms | +| Distinct 32 KiB Chunks | 52.8 ms | 581.7 ms | +| Distinct document paths | 36.5 ms | 50.0 ms | + +These direct calls include no Chunk-content cache hits. The predecessor hash +had a measurable cost, especially when there were many small Chunks. The +local-write experiments exercise splitting, ID generation, Chunk reuse, and +PouchDB writes: + +| Workload and adapter | Legacy median (range) | Independent median (range) | +| ------------------------------------------------------------- | --------------------: | -------------------------: | +| 5,000 text/binary files, 100 MiB, Node PouchDB memory adapter | 102.6 s (87.4–109.3) | 86.6 s (75.4–93.1) | +| 1,000 text files, 19.5 MiB, actual Obsidian local database | 78.7 s (52.0–82.8) | 63.2 s (62.8–63.8) | + +The measurements do not show a large overall slowdown for these workloads, +but the variation does not support a general speedup claim. Both experiments +checked document counts, Chunk-reference counts, and sample content readback. +The Node experiment also checked that every referenced Chunk was present. The +100 MiB corpus includes 250 duplicate files and produces 213,788 Chunk +references to 199,468 distinct Chunks in both modes, preserving reuse. + +For that corpus, stored document JSON grows from 133,157,581 to 154,342,743 +UTF-8 bytes, an increase of 15.9%. The text-only Obsidian corpus produces many +small Chunks and grows from 27,893,656 to 33,594,994 bytes, or 20.4%, including +32 warm-up documents. Longer Chunk IDs occur in both Chunk documents and +Metadata references. Ordinary obfuscated document IDs remain 66 characters. +These totals measure serialised document JSON; physical database and index +growth depend on the adapter and have not been measured. + +### Remaining validation + +Larger binary workloads on mobile, a representative user's Vault, physical +storage growth, and the complete Rebuild wall time remain unmeasured. The +native-device reports above verify the synthetic ID calculation scenario; +desktop E2E and mobile viewport checks do not establish other mobile +operating-system behaviour. URI revocation, QR redesign, and automatic +migration remain outside this change. diff --git a/docs/design_docs/internal_metadata_encryption.md b/docs/design_docs/internal_metadata_encryption.md new file mode 100644 index 00000000..5c3c9cf6 --- /dev/null +++ b/docs/design_docs/internal_metadata_encryption.md @@ -0,0 +1,163 @@ +--- +date: 2026-09-27 +commonlib-version: "0.1.30" +self-hosted-livesync-version: "1.0.32" +status: unreleased +--- + +# Internal Metadata encryption and remote feature changes + +This document defines the LiveSync integration of Commonlib's remote feature +contract and encrypted Metadata for Hidden File Sync and Customisation Sync. +It describes unreleased behaviour being implemented in this branch. + +Commonlib's companion `docs/remote-feature-compatibility.md` is the +source of truth for the wire document, identifiers, validation, and shared +assessment. This document owns the application behaviour, settings, Doctor +recommendation, and verification of the Obsidian and CLI integrations. + +## Scope and settings + +Add `encryptInternalMetadata` to the shared encryption settings. A genuinely new +Vault or CLI configuration defaults to true. Existing stored settings and old +Setup URI or QR imports complete an absent value as false. Ordinary partial +setting updates retain the current value. + +The preference applies to CouchDB with E2EE V2 and Property Encryption enabled. +Show the preference as unavailable and explain its prerequisites when they are +absent. Keep Journal and P2P's existing +transport protection and avoid unrelated setting mismatches for those remotes. + +Use the existing HKDF Metadata representation to protect path, creation and +modification times, size, and Chunk references for obfuscated internal entries. +Keep the `i:`, `ix:`, and supported legacy `ps:` document IDs, path conversion, +and content Chunk representation. Read encrypted Metadata independently of the +write preference, including after that preference is disabled. + +The protection leaves document IDs, namespaces, revisions, deletion state, +document counts, and ciphertext lengths visible. It does not encrypt device or +Vault names stored in separate participant records. + +## Enabling the preference + +Changing the preference does not automatically reconstruct a database or gather +all devices' data. It affects subsequent Metadata writes. Unchanged documents +and old revisions can retain plaintext; mixed plaintext and encrypted Metadata +are a supported transition state. + +Strongly recommend the existing manual remote Rebuild workflow when the person +wants existing Metadata protected as well. The person prepares the authoritative +data for that workflow. Describe this distinction in the setting, Doctor reason, +and operational documentation. Do not advertise complete historical protection +merely because the preference is enabled. + +Copy the preference with the other encryption settings when preparing a remote +profile. Recreate a connection when its effective encryption settings change. +Use the existing Tweak assessment and manual mismatch resolution; do not change +the remote's shared policy silently when importing or loading settings. + +## Doctor + +Use Commonlib's existing conditional recommendation rules. Recommend true when +the selected CouchDB settings have E2EE V2 and Property Encryption enabled and +the new preference is false. Do not require Hidden File Sync or Customisation +Sync to be active before offering the recommendation. + +Retain the existing E2EE V2 recommendation for a legacy algorithm. After that +change, ensure the newly applicable Metadata recommendation is not hidden by a +premature `doctorProcessedVersion` update. Advance the Doctor rule revision so +an older completed consultation does not suppress this new recommendation. + +Apply the preference only when the person accepts the recommendation. Include +the existing-data limitation, the manual Rebuild recommendation, and the need +for compatible clients in the explanation. Do not set `requireRebuild` or +`requireRebuildLocal` for this rule: the current host wrapper can schedule those +operations and restart. `recommendRebuild` currently exists only as an unused +rule field, so setting it alone does not display an explanation. + +## Admission and received version documents + +The remote version document is the source of feature requirements. Commonlib +checks it before replication, Fast Fetch, and direct access. The milestone keeps +the existing Tweak comparison and Rebuild lock. An accepted writer declares the +feature before using it, including the writer admitted to a locked rebuilt +remote; an unaccepted device remains blocked by that lock. + +Retain the existing received-version path through `parseSynchroniseResult`, +`enqueueAll`, and `processIfNonDocumentChange`. Replace its numeric comparison +with the shared assessment so unknown names at the same generation are also +reported. Known features, reordered lists, and ordinary revision updates do not +retire the connection. Unsupported or malformed control documents request +retirement through the existing Replicator owner and display the reason. +The callback must not await retirement of the operation which delivered it. + +This is an admission check and a best-effort stop for exceptional changes during +an active connection. It does not fence every queued file application, roll back +accepted writes, or guarantee an atomic change across live devices. Feature +changes are an infrequent administrative operation: update all devices first, +then enable the preference and use the recommended manual Rebuild. Rebuild +locks the remote using the existing workflow; changing this preference alone +does not lock it. The action to proceed without rebuilding explicitly reminds +the person to update every other device, including currently connected devices. + +## Persistence and recovery boundaries + +Do not retain a second feature list, highest generation, or rejection flag in +KV storage. Do not add compatibility checks to pending-work snapshot recovery +or make that recovery a new prerequisite for application readiness. Preserve +the existing queue and startup behaviour. A later attempt checks the current +remote declaration, including after restart. Declared features remain on the +remote when the write preference is disabled because older data can still use +them; manually shortening that declaration is not a supported migration. + +After updating clients, use normal reconnection and the existing Hatch +inspection or Fetch workflow if reconciliation is needed. This feature does not +repair unrelated KV inconsistencies or the existing readiness queue behaviour. + +Garbage Collection V3 is a beta manual operation which begins with an ordinary +bidirectional synchronisation. That admission checks the remote feature +contract; no additional per-step GC checks are introduced. The separate +cleaned-remote recovery path checks the local version document before its +first Chunk-reference count because it does not start with that synchronisation. + +Use the same Commonlib assessment at the CLI, Fast Fetch, and direct-access +boundaries. The Obsidian result processor is one consumer, not the only place +which determines compatibility. Keep unrelated Vaults and databases operational. +Fast Fetch checks the remote declaration before opening or resetting the local +database, both for a fresh Fetch and for checkpoint resumption. + +## Verification and documentation + +Keep focused tests for settings defaults and imports, the Doctor condition +matrix, acceptance and dismissal, connection replacement, and absence of an +automatic Rebuild, Fetch, or restart for this rule. + +Keep unit tests for known and unknown feature notifications, generic identifier +presentation, retirement without a circular wait, and the unchanged snapshot +behaviour after KV failure or obsolete snapshot fields. The previous batch +fences, physical-database tracking, and persistent rejection tests are outside +this design; they must not imply an atomic live migration guarantee. + +Use real Obsidian Hidden File Sync and Customisation Sync scenarios to inspect +raw CouchDB Metadata and restore content in another Vault. Check the admitted +writer on a locked remote, unknown-feature rejection before and during +replication, and remote-based rejection after restart. Retain the encrypted +CLI-to-Obsidian interoperability scenario. A future client upgrade that adds +support for an unknown feature is a separate validation boundary. + +Also exercise enabling the preference through the settings UI without Rebuild: +retain unchanged plaintext Metadata, encrypt rewritten entries with stable IDs, +reject a second device's mismatched preference, and restore both representations +after alignment. With the preference subsequently OFF, verify that Fast Fetch +still decodes encrypted Metadata and preserves the remote feature declaration. + +Keep the primary-language settings and troubleshooting guides, the +database-compatibility ADR, and Unreleased notes aligned with this behaviour. +Keep the detailed shared protocol in Commonlib and link to it after publication; +do not maintain another copy of its wire schema here. Translations are a separate +change. Update tested-version evidence when the implementation and its +validation have been accepted. + +Related application contracts: [Replicator architecture](replicator_architecture.md), +[Tweak compatibility](tweak_compatibility.md), and +[database compatibility](../adr/2026_07_release_notes_and_database_compatibility.md). diff --git a/docs/design_docs/renewable_turn_credentials.md b/docs/design_docs/renewable_turn_credentials.md new file mode 100644 index 00000000..0f903ba6 --- /dev/null +++ b/docs/design_docs/renewable_turn_credentials.md @@ -0,0 +1,177 @@ +--- +date: 2026-09-16 +commonlib-version: "0.1.25" +self-hosted-livesync-version: "1.0.28" +status: unreleased +--- + +# TURN credentials in P2P connection settings + +## Purpose + +This design addresses [Issue #1182](https://github.com/vrtmrz/obsidian-livesync/issues/1182) +by acquiring temporary TURN credentials on the device before opening a P2P room. +The [P2P transport compatibility ADR](../adr/2026_08_p2p_transport_compatibility.md) +records the connection and persistence policy. + +LiveSync prepares a connection copy of `P2PSyncSetting`. Commonlib owns the room +lifecycle and consumes the resulting ICE settings. Service-specific HTTP and +validation remain under `src/integrations/`; Commonlib has no provider catalogue +or versioned acquisition descriptor. Cloudflare is the first optional integration. +Manual TURN configuration remains available without a provider account. + +## Settings and ownership + +| Setting | Meaning | Lifetime | +| --- | --- | --- | +| `P2P_managedType` | Provider identifier; `CF` selects Cloudflare | P2P profile | +| `P2P_managedId` | Provider key identifier; Cloudflare TURN Key ID | P2P profile | +| `P2P_managedToken` | Provider API token used to request credentials | P2P profile | +| `P2P_iceServers` | Prepared `RTCIceServer[]` | One room connection | +| `P2P_iceServersExpiresAt` | Absolute expiry in Unix milliseconds | One room connection | + +The first three values use ordinary ConnStr query parameters `managedType`, +`managedId`, and `token`. The existing `appId` parameter continues to identify the +P2P application. Commonlib reads and writes the three scalar values so profile +editing and activation preserve them. The host interprets the provider identifier. +An absent identifier selects the existing manual fields; an unsupported identifier +produces an explicit error when a connection is requested. + +Keep `P2P_turnServers`, `P2P_turnUsername`, and `P2P_turnCredential` for manual +configuration. Issuance does not overwrite them. Retain the complete ICE array: +individual entries can contain different credentials or STUN-only URLs. + +## Host preparation + +The optional `prepareP2PSettings(settings, signal)` composition hook receives a +snapshot of requested P2P settings. LiveSync supplies the same preparation function +to Obsidian, CLI, WebApp, and WebPeer using each host's HTTP adapter. + +For a managed selection, the function validates the provider inputs, requests +credentials, and returns a connection copy: + +```typescript +return { + ...settings, + P2P_iceServers: iceServers, + P2P_iceServersExpiresAt: expiresAt, +}; +``` + +Commonlib takes the prepared ICE fields into its session snapshot and passes that +snapshot through `ReplicatorHostEnv.settings`. The hook does not change the +requested room identity, persist settings, own replication, or schedule renewal. +Its HTTP request must settle on cancellation and has a bounded deadline. The room +owner also stops waiting for preparation when the connection request is retired. +An explicitly managed configuration requires a preparation hook and usable ICE +credentials; acquisition failure does not select a fallback provider or route. +Managed credential acquisition is independent of the connection path. `Automatic` +retains normal ICE selection, including direct candidates; only `TURN relay only` +forces relay use. Acquisition must still succeed before opening a managed room +when `Automatic` is selected. + +## Room reuse and expiry + +The active connection settings hold the issued credentials. They are the only +credential cache. The existing room reuse decision checks: + +1. whether the requested database and connection settings still match; and +2. whether the active connection's credentials have enough remaining lifetime. + +The static connection signature includes the provider type, key ID, and token. +It excludes the generated ICE array and expiry. Comparing the prepared and stored +settings directly would incorrectly trigger issuance on every reconciliation. + +When reuse is unavailable, the owner retires the existing room, obtains a fresh +connection copy, and opens its replacement. It checks settings, room demand, +cancellation, and expiry again before publishing the replacement. A late result +cannot reopen a closed room or apply credentials requested for different settings. +Explicit reconnection acquires fresh credentials. Closing the room releases its +credential references. Preserve a 30-second connection-establishment margin. + +Reconciliation runs at existing connection, settings, and lifecycle boundaries. +Time passing alone does not trigger acquisition or disconnection. There is no +renewal timer, per-peer acquisition, raw WebRTC configuration update, ICE restart, +or general retry mechanism. Trystero's internal peer reconnection within an +unchanged room uses that room's existing configuration. + +Normal retirement may cancel an in-progress transfer. A later replication attempt +uses stored checkpoints and revision comparison to retain received progress. +An unfinished network message may be sent again. Whether another attempt starts +automatically continues to follow the existing synchronisation policy. + +## Persistence, sharing, and privacy + +Persist provider values only inside the selected P2P profile URI. Flat values in +runtime settings are a projection restored by profile activation. Profile edits +update that URI explicitly. General settings saves do not rebuild a P2P profile +from unrelated flat settings. Flat-settings migration creates and selects its +P2P profile once, independently of the selected main remote. + +Existing whole-profile encryption covers the saved API token. The default mode +uses the existing built-in key; a user-supplied configuration passphrase has its +existing protection semantics. Failure to encrypt a managed profile leaves the +previous saved data intact. No separate encrypted-token field is added. A draft +containing provider credentials but no Group ID remains unsaved. + +Setup URIs and ordinary settings QR codes already contain `remoteConfigurations`. +The provider values travel inside that profile URI, including inactive profiles. +Omit their duplicate flat projections from sharing. No new URI scheme, encoded QR +slot, or encryption envelope is needed. Setup URIs retain passphrase encryption; +QR codes retain their unencrypted format and 'FOR YOUR EYES ONLY' display. + +Issued ICE credentials and expiry appear only in connection copies. Remove both +runtime fields at save, import, and sharing boundaries, including +`TrysteroReplicator.getAllConfig`, which starts from the session settings. +Incoming settings cannot install an issued credential override. Reports omit +runtime ICE fields, redact provider values, and retain scheme-only profile URIs. +Logs use safe errors and omit request headers, raw responses, and connection +signatures. Ordinary plaintext in process memory is permitted. + +Markdown settings omit managed provider values and the profile collection with +its selections. If that group is omitted during import, preserve the corresponding +local P2P connection values as well as the profiles. This prevents combining an +imported room with the local provider token or overwriting the saved profile. + +## Cloudflare integration + +The UI presents `Manual` and `Managed (Cloudflare)`, with `TURN Key ID` and a masked +`TURN Key API Token` input for Cloudflare. It requires no account ID, custom +endpoint, SDK, credential broker, or renewal interval setting. + +The provider function uses Cloudflare's +[credential-generation endpoint](https://developers.cloudflare.com/realtime/turn/generate-credentials/) +and converts its response into ICE servers. The implementation requests a fixed +24-hour lifetime and derives local expiry from the clock before the request starts. +This lifetime applies to the issued TURN credentials, not the provider API token. +There is currently no setting to change it. + +The HTTP boundary uses the injected standard fetch adapter with cancellation, +a 15-second deadline, refused redirects, omitted cookies, and disabled caching. +It bounds the response to 32 KiB, 16 ICE entries, and 32 URLs, and validates URLs +and complete TURN credentials. These are local implementation limits. Keep this +validation at the provider boundary instead of repeating it in Commonlib. + +The token is supplied and shared by the user on their devices. The provider +function sends the key ID, API token, and requested lifetime; it has no need for +Vault data, the Group ID, or the Vault passphrase. + +## Setup and verification + +The Setup connection test remains a signalling check. A separately owned trial +uses signalling-only settings and performs no managed TURN issuance. The existing +active-relay admission rule still applies. Success does not verify the API token, +TURN allocation, or document transfer. Actual room connections use the preparation +hook and preserve the selected route policy on failure. + +Focused tests cover provider validation and cancellation, room reuse and expiry, +late results after configuration changes or closure, migration without duplicate +profiles, Markdown import through save/reload, safe acquisition failures, and +exclusion of runtime credentials from storage and sharing. + +Validate Commonlib as an exact packed artefact before testing its LiveSync +consumer. Verify the changed settings and restart boundary in real Obsidian. +Previously observed provider issuance and relay synchronisation do not establish +expiry-driven reconnection for a revised build. Fresh TURN allocation after +expiry, mobile runtimes, and cross-network behaviour require their own runtime +verification; a surviving Trystero shared peer is not evidence of new allocation. diff --git a/docs/glossary.md b/docs/glossary.md index 8af12013..9bc5ed31 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -92,6 +92,12 @@ recovery guidance, or diagnostics intended for users. reports, and advanced edge-case settings. - **Hidden File Sync:** The feature which synchronises files in hidden directories, such as `.obsidian`. +- **ID key:** A saved secret used to generate encrypted Chunk IDs and + obfuscated Metadata document IDs when independent ID derivation is enabled. + It is separate from the current E2EE passphrase. +- **ID recovery code:** A versioned text form of the saved ID key which can be + shown on the current device and imported without deriving a different key. + Treat it as a secret. - **JWT Authentication:** An experimental CouchDB authentication option which uses a JSON Web Token instead of standard credentials. It requires a private key or secret, algorithm, expiry duration, subject, and key ID. diff --git a/docs/p2p.md b/docs/p2p.md index 23253771..75160f48 100644 --- a/docs/p2p.md +++ b/docs/p2p.md @@ -18,7 +18,7 @@ flowchart LR The signalling relay and TURN server have different roles: - The **signalling relay** is required for peer discovery and connection negotiation. LiveSync uses Nostr-compatible WebSocket relays for this role. The relay does not store or transfer Vault contents. -- A **TURN server** is an optional fallback. WebRTC uses it to relay the encrypted peer connection only when the devices cannot establish a direct path through their networks. +- A **TURN server** is an optional fallback. WebRTC uses it to relay the encrypted peer connection when the devices cannot establish a direct path through their networks, or whenever **TURN relay only** is selected. ## The project's public signalling relay @@ -40,16 +40,49 @@ Both settings contain server addresses, but they are not interchangeable. | Setting | Required | Carries Vault contents | Purpose | | --- | --- | --- | --- | | **Signalling relay URLs** | Yes | No | Finds peers and exchanges the information needed to establish WebRTC connections. | -| **TURN server URLs** | Only when direct WebRTC connectivity fails | Encrypted WebRTC traffic | Relays traffic between peers when NAT or firewall rules prevent a direct path. | +| **TURN server URLs** | When direct WebRTC connectivity fails or **TURN relay only** is selected | Encrypted WebRTC traffic | Relays traffic between peers when NAT or firewall rules prevent a direct path. | -A TURN provider cannot read LiveSync's encrypted Vault contents, but it can observe connection metadata and traffic volume. Use a provider you trust. The project does not operate an official TURN service. +WebRTC encrypts data between the devices, including when it passes through TURN. The TURN provider cannot read the transferred data, but it can observe network addresses and traffic volume. This transport encryption also applies when LiveSync's optional database encryption is disabled. The project does not operate an official TURN service. + +## TURN credentials + +In **TURN configuration**, select **Manual** to enter your own TURN server URLs, +username, and credential, or select **Managed (Cloudflare)** to enter a **TURN Key ID** and +**TURN Key API Token**. Cloudflare is optional; the project does not require a +particular TURN provider or operate a credential broker. See Cloudflare's +[credential instructions](https://developers.cloudflare.com/realtime/turn/generate-credentials/) +for creating a TURN key and its API token. + +The API token is saved with the P2P profile and included when sharing settings +through an existing Setup URI or QR code. Setup URIs retain their existing +passphrase encryption. QR codes retain their existing unencrypted format and +'FOR YOUR EYES ONLY' display. Missing provider settings use the ordinary manual +configuration defaults. Receiving clients need support for the selected provider +to acquire its temporary TURN credentials. +Markdown settings omit the connection profile group when it contains a managed +TURN provider, including inactive profiles, and importing those omitted settings +preserves this device's existing profiles. Diagnostic reports redact provider settings. The existing profile-URI +encryption also covers the saved token. + +Each device requests temporary TURN credentials when opening a new room. +An existing room reuses its credentials while they remain valid. Cloudflare credentials have a requested lifetime +of 24 hours and remain in memory only. Expiry is checked when LiveSync next +reconciles the room connection. If necessary, it replaces the room and obtains +new credentials. There is no periodic renewal: if a long-lived room cannot +reconnect after credentials expire, disconnect and open the connection again. + +Room replacement may interrupt replication. The next synchronisation keeps +received Metadata and Chunks, resumes from its saved checkpoint, and compares +revisions to fetch missing data. An unfinished network message can be sent +again. Automatic synchronisation follows the existing peer rules; after an +interrupted manual operation, use **Replicate now** again. ## Connection compatibility profiles `P2P Configuration` includes a separate `Connection compatibility` section. Its defaults preserve the existing transport behaviour: - **P2P message size** defaults to **Standard**. **Reduced**, **Conservative**, and **Maximum compatibility** progressively limit outgoing P2P messages when a network path appears to drop larger WebRTC messages. This is not a Vault Chunk size or an IP MTU. Smaller values add framing and processing overhead. -- **Connection path** defaults to **Automatic**, which lets WebRTC select a viable direct or TURN-relayed path. **TURN relay only** forces the encrypted connection through TURN and is available only when the profile contains at least one valid `turn:` or `turns:` URL. +- **Connection path** defaults to **Automatic**, which lets WebRTC select a viable direct or TURN-relayed path. **TURN relay only** forces the encrypted connection through TURN and is available when the profile contains a valid manual TURN URL or a configured TURN provider. The sending device controls its outgoing message size. Select the same conservative preset on every device which may send across the constrained path. Existing devices do not receive the choice retrospectively merely because another device changed it. diff --git a/docs/releases/1.0.md b/docs/releases/1.0.md index 61ed6ffe..46584f8b 100644 --- a/docs/releases/1.0.md +++ b/docs/releases/1.0.md @@ -2,6 +2,76 @@ This document contains earlier published releases from the 1.0 line of the [current Self-hosted LiveSync release history](../../updates.md). Beta and release-candidate builds published before 1.0.0 are recorded in the [1.0 preview history](1.0-previews.md). Earlier release lines continue in the [0.25 history](0.25.md) and the [legacy history](legacy.md). +## 1.0.26 + +~~1.0.25~~ was cancelled because pre-release validation found that LiveSync could appear to finish synchronising even though Android had not written a received file to the Vault; the warning appeared only after restart. + +6th September, 2026 + +### Synchronisation and storage + +#### Fixed + +- Files inside a folder are no longer silently removed from synchronisation when an external tool changes only the letter case of that folder while Obsidian is running. This prevents the stale deletion from reaching other devices or later removing the local file. Moving files into ignored or otherwise excluded locations retains the existing behaviour, and the folder-name case itself may still differ between devices. (#1168) +- A problem processing one file during ordinary start-up no longer prevents every other file from synchronising. LiveSync warns about the affected files and can retry them later; Fetch and Rebuild still stop if they cannot finish safely. (#1164) +- When LiveSync cannot finish preparing this device for synchronisation, it now says that synchronisation is unavailable and directs you to generate a report, instead of remaining at 'Not ready'. (#1164) + +#### Improved + +- When LiveSync cannot write a received file to the Vault, it now warns immediately instead of appearing to have synchronised it successfully. The generated report identifies the affected path, and a later scan can try it again. + +### Conflict handling and recovery + +#### Improved + +- Conflict resolution dialogues now close when the same file is resolved elsewhere or when the plug-in unloads. Requests for different files are shown one at a time, while a newer request for the same file replaces the older one. + +### Setup and compatibility + +#### Improved + +- Unconfigured Vaults now stay focused on setup instead of running Config Doctor or incomplete-document checks before they can be used. Returning a configured Vault to an unconfigured state also stops those checks until the requested restart. (#1161) +- When the active file contains a file or folder name longer than 255 UTF-8 bytes, LiveSync now explains that the path may not work on some Android and Linux file systems. It does not rename or reject the file. (#1164) + +## 1.0.24 + +3rd September, 2026 + +### Interface and translation + +#### Fixed + +- The Setup Wizard now correctly explains that the existing-device path adds this device to an existing synchronisation (PR #1118). Thank you to @nikhilmaddirala for the contribution! +- Spanish translations now resolve the **Display language** placeholder, cover previously untranslated Setup Wizard and CouchDB text, translate user-facing Config Doctor values and confirmation controls, and use Spanish sentence case (PR #1129). Thank you to @zeedif for the contribution! + +#### Improved + +- The Setup Wizard now shows the passphrase and **Obfuscate Properties** controls only after E2EE is enabled, provides a password-visibility button, allows longer translated labels to wrap, and keeps the invitation link compact on desktop while preserving its mobile touch target (PR #1130). Thank you to @zeedif for the contribution! + +### Synchronisation and storage + +#### Fixed + +- **Overwrite Server Data with This Device's Files** now keeps this device's synchronisation settings instead of reapplying settings from the remote database which is about to be replaced. Enabling E2EE before a rebuild therefore remains enabled and uploads encrypted data. (#1146) + +### Command-line tool + +#### Fixed + +- The systemd installer now finds the repository root correctly, installs every generated bundle chunk and required production dependency, checks the installed command before activation, and reports success only when the service remains active. + +## 1.0.23 + +2nd September, 2026 + +I am sorry to make this release while several pull requests are still awaiting merge, but I believe that the safeguards provided by this work are significant, so I have decided to release it. I will merge the remaining pull requests in turn. Thank you for bearing with me while I have been less active recently. + +### Synchronisation and storage + +#### Fixed + +- **Sync now** once again keeps routine progress quiet, while still opening recovery dialogues when a decision is required. Repeated OneShot Sync requests received while an earlier attempt is running are now ignored instead of starting overlapping work. + ## 1.0.22 1st September, 2026 diff --git a/docs/settings.md b/docs/settings.md index feddbb73..44b21a4b 100644 --- a/docs/settings.md +++ b/docs/settings.md @@ -239,12 +239,34 @@ Setting key: passphrase Encrypting passphrase. If you change the passphrase, you need to rebuild databases (You will be informed). +#### Independent ID derivation + +Setting keys: `idDerivationVersion`, `idDerivationKey` + +This setting saves a separate key for encrypted Chunk IDs and obfuscated Metadata document IDs. New Vault setup selects **Generate a random ID key** by default when E2EE is enabled. Existing Vaults select **Keep current configuration** by default. The radio choices show the available configurations together. A small description under **Keep current configuration** identifies the saved configuration: an existing ID key, or legacy IDs linked to the E2EE passphrase. That choice retains either one; on a new Vault, choosing it explicitly uses legacy IDs. If the configuration is legacy, changing the E2EE passphrase also changes IDs. + +To set a key yourself, choose **Set an ID key**. Three further radio choices then appear: **Derive from current E2EE passphrase**, **Enter an ID source**, and **Import an ID recovery code**. The last two choices show a text input. The source input also recognises a tagged recovery code. An empty input keeps an existing key; a first key requires input. An ordinary source is converted to a key when you apply the settings and cannot be shown again. A recovery code imports the saved key directly. + +Use **Show current recovery code** to display and copy the saved key on this device. The code starts with `sls-id-v1:` and can be pasted into the manual input on another device without deriving a different key. A Setup URI carries the same saved key under its separate passphrase. If you need to restore the configuration after losing every device, save the recovery code or choose a source you can reproduce before relying on the random default. Keep the code private. + +Using the E2EE passphrase as the source keeps IDs stable after later passphrase changes, but it does not separate the original passphrase from guesses based on known IDs. Use a long, unpredictable, separate source when that separation matters. Hashing a weak source does not make it strong. + +Changing the E2EE passphrase later does not change the saved ID key, although the existing re-encryption and Rebuild procedure still applies to the encrypted data. While E2EE is off, the saved ID key is retained but is not used; existing legacy ID generation applies until E2EE is enabled again. Devices with different ID keys can synchronise when Path Obfuscation is off, although identical content may produce duplicate Chunks. Enabling, replacing, or disabling the ID key can change document IDs when Path Obfuscation is active. Update participating devices, Rebuild from the authoritative Vault, and Fetch on other devices before resuming ordinary synchronisation. A QR code includes the saved key under the existing QR sharing rules, so keep the QR code private. + #### Path Obfuscation Setting key: usePathObfuscation In default, the path of the file is not obfuscated to improve the performance. If you enable this, the path of the file will be obfuscated. This is useful when you want to hide the path of the file. +#### Encrypt internal file Properties + +Setting key: encryptInternalMetadata + +For CouchDB, this encrypts paths, times, sizes, and Chunk references in the Metadata used by Hidden File Sync and Customisation Sync. It requires E2EE V2 and **Property Encryption**. New Vaults enable the preference by default, but it has no effect until those prerequisites are enabled. Existing Vaults and older Setup URIs and QR codes keep it disabled unless you enable it. + +Enabling the preference protects future Metadata writes. Existing Metadata and earlier revisions can remain readable in the remote database. If you want to protect existing Metadata too, prepare the authoritative data, update every device to a compatible version, and manually Rebuild the remote database. LiveSync does not gather data or start a Rebuild when you change this preference. The action to enable it without rebuilding explicitly reminds you to update every other synchronising device first, including devices currently running LiveSync. Plaintext and encrypted Metadata can coexist during the transition. Document IDs, revision information, document counts, and ciphertext lengths remain visible. + #### Encryption Algorithm Setting key: E2EEAlgorithm @@ -485,6 +507,25 @@ Setting key: P2P_AutoBroadcast When enabled, this device notifies connected peers after a local change. The notification contains no Vault data. A receiving peer fetches the change only when it follows this device. +#### TURN configuration + +Setting key: P2P_managedType + +Select **Manual** for the existing TURN server fields, or **Managed (Cloudflare)** for a +TURN Key ID and TURN Key API Token. The API token is persisted with the profile +and included in Setup URI and QR code sharing. Issued temporary credentials are +kept in memory only. Reports redact the provider settings. See +[TURN credentials](p2p.md#turn-credentials) for sharing, expiry, and reconnect +behaviour. + +#### TURN Key ID and TURN Key API Token + +Setting keys: P2P_managedId, P2P_managedToken + +These fields appear when **Managed (Cloudflare)** is selected. Enter the TURN key's ID and +its dedicated API token. The token field is masked. No account ID, custom +endpoint, or renewal interval is required. + #### TURN Server URLs (comma-separated) Setting key: P2P_turnServers @@ -515,7 +556,7 @@ The sender controls the size of its outgoing messages. Select the same conservat Setting key: P2P_connectionPath -**Automatic** lets WebRTC select a viable direct or TURN-relayed path and is the default. **TURN relay only** forces `iceTransportPolicy: 'relay'` and is available only when the profile contains at least one valid `turn:` or `turns:` URL. Removing the last valid TURN URL while relay-only mode is selected restores **Automatic** and displays a Notice. +**Automatic** lets WebRTC select a viable direct or TURN-relayed path and is the default. **TURN relay only** forces `iceTransportPolicy: 'relay'` and is available when the profile contains a valid manual TURN URL or a configured TURN credential source. Removing the manual TURN configuration while relay-only mode is selected restores **Automatic** and displays a Notice. A selected credential source which cannot supply credentials prevents the connection from opening; it does not change the connection path. This choice belongs to the P2P profile and is retained in P2P connection strings and encrypted Setup URIs. Separate profiles may use the same Group ID and credentials with different compatibility choices; only the selected P2P profile is active. @@ -1006,6 +1047,8 @@ Setting key: hashAlg `xxhash64` is the supported current value. Older algorithms remain selectable only as an edge-case compatibility path for existing databases. Changing the algorithm can reduce chunk reuse between devices and requires the normal tweak review. +When independent ID derivation is enabled, encrypted Chunk IDs use its versioned HMAC construction instead of `hashAlg`. The selected `hashAlg` continues to apply to legacy IDs. + ### 6. Edge case addressing (Behaviour) #### Fetch database with previous behaviour diff --git a/docs/setup_object_storage.md b/docs/setup_object_storage.md index 0244a766..4c72b716 100644 --- a/docs/setup_object_storage.md +++ b/docs/setup_object_storage.md @@ -31,7 +31,7 @@ deno run --minimum-dependency-age=0 --allow-env https://raw.githubusercontent.co For providers which require them, set `force_path_style`, `use_custom_request_handler`, or `bucket_custom_headers` as described in the [setup utility reference](../utils/readme.md#object-storage). -Store the generated Setup URI and Setup URI passphrase separately. The URI is encrypted, but it contains the Object Storage credentials. +Store the generated Setup URI and Setup URI passphrase separately. The URI is encrypted, but it contains the Object Storage credentials. The generator also prints an ID recovery code; reuse it through `id_recovery_code` if you regenerate a URI for the same Vault. A new run without it creates a different ID key. The [setup utility reference](../utils/readme.md#setup-uri-generation) describes the `id_mode=legacy` option for existing Vaults. ## Set up the first device diff --git a/docs/setup_own_server.md b/docs/setup_own_server.md index 0eac36b2..1177b820 100644 --- a/docs/setup_own_server.md +++ b/docs/setup_own_server.md @@ -190,6 +190,8 @@ deno run --minimum-dependency-age=0 --allow-env https://raw.githubusercontent.co > > If `uri_passphrase` is omitted, the generator creates a cryptographically random value and prints it once. +The generator also prints an ID recovery code for its random ID key. Save that code if you may need to regenerate a Setup URI for the same Vault. Pass it back as `id_recovery_code`; otherwise a later run creates a different key. Set `id_mode=legacy` only when connecting to a Vault which uses the previous ID behaviour. See the [setup utility reference](../utils/readme.md#setup-uri-generation). + The generator consumes the exact registry-pinned Commonlib release used by the provisioning utility. It creates a configured CouchDB remote profile, applies the current defaults for a new Vault, and encodes them with Commonlib's Setup URI contract. You will then get the following output: @@ -198,6 +200,8 @@ You will then get the following output: Generated couchdb Setup URI. Your passphrase for the Setup URI is: H7vX...a-random-32-character-value This passphrase is never shown again, so store it safely. +ID recovery code: sls-id-v1:<64 lowercase hexadecimal characters> +Use id_recovery_code with this value and reuse the same remote settings when generating another Setup URI for the same Vault. obsidian://setuplivesync?settings=%5B%22tm2DpsOE74nJAryprZO2M93wF%2Fvg.......4b26ed33230729%22%5D ``` diff --git a/docs/setup_p2p.md b/docs/setup_p2p.md index fb13a9c1..82b74ba6 100644 --- a/docs/setup_p2p.md +++ b/docs/setup_p2p.md @@ -135,4 +135,4 @@ export uri_passphrase= deno run --minimum-dependency-age=0 --allow-env https://raw.githubusercontent.com/vrtmrz/obsidian-livesync/main/utils/setup/generate_setup_uri.ts ``` -The generated Setup URI contains the encrypted room, relay, and Vault settings. It deliberately omits the device-specific name. Store the URI and its passphrase separately. After importing it on the first device, continue from the initialisation step above, then generate a fresh Setup URI for an additional device from that working device. +The generated Setup URI contains the encrypted room, relay, and Vault settings. It deliberately omits the device-specific name. Store the URI and its passphrase separately. The generator prints an ID recovery code; pass it as `id_recovery_code` if you regenerate a URI for the same Vault. A run without it creates a different ID key. Also reuse the original `p2p_room_id` and `p2p_passphrase`, since omitted values are generated afresh. The [setup utility reference](../utils/readme.md#setup-uri-generation) describes the `id_mode=legacy` option for existing Vaults. After importing the URI on the first device, continue from the initialisation step above, then generate a fresh Setup URI for an additional device from that working device. diff --git a/docs/specs_conflict_resolution.md b/docs/specs_conflict_resolution.md index c247ef3f..8f62b992 100644 --- a/docs/specs_conflict_resolution.md +++ b/docs/specs_conflict_resolution.md @@ -28,24 +28,27 @@ The modifiers defined under [Revision](glossary.md#revision) describe independen | The Vault displays conflict leaf `C` | `W` | `C` | `C` | | The database advances before Vault reflection | new winner `W2` | previous revision `R`, while the Vault is unchanged | `R` | | A local edit of displayed revision `R` is pending | independent | none, or a coincidental content match | `R`, as the branch which the edit must extend | -| Provenance is missing and exactly one revision fits | independent | `M` | none, then `M` after safe reconstruction | +| Provenance is missing and exactly one current non-deleted leaf fits | independent | `M` | none, then `M` after safe reconstruction | | Provenance is missing and several revisions fit | independent | every matching revision | none | | A logical-deletion winner agrees with an absent file | deleted winner `D` | `D`, and possibly other logical-deletion revisions | none; an absent file retains no displayed provenance | At most one revision is the winner, more than one revision can be Vault-matching, and at most one revision can be displayed for a path on one device. A displayed revision may stop matching the Vault while a local edit is pending, but its branch identity remains authoritative until that edit is stored or the relationship is safely reconstructed. -## Implemented 1.0 guarantees +## File saving and reflection guarantees - Automatic text and structured-data merge uses the nearest `available` revision ID which is present in both leaf histories. - Missing or compacted history stops conservative automatic merge instead of guessing a base. -- A receiving Vault file which exactly matches any available revision in the document tree is treated as previously synchronised content. This includes an ancestor below a deleted losing leaf. -- A receiving Vault file whose bytes do not match any available revision is preserved as an unsynchronised local change. +- A Vault file which still matches its exact recorded revision is unchanged. An ordinary save does not append those stale bytes to a newer database revision; a newer, unconflicted database result is reflected through the existing file-reflection path. +- A file which differs from its readable recorded revision is an edit of that revision, even if its bytes match another historical revision. Saving and incoming overwrite protection use the same rule. +- Without a readable recorded revision, current non-deleted leaves are checked for duplicate content. If none matches, the file is preserved as a fresh independent root under the same document ID. Its unknown ancestry cannot supply a three-way merge base. - File bytes, rather than path, size, modification time, or revision generation, determine whether content is known. - Three or more current versions are reviewed one pair at a time in a deterministic order, with each completed pair committed before the next pair is read. -- Each device records the exact revision most recently reflected in each Vault file. An edit, deletion, or case-only rename made while a conflict is active extends that displayed branch rather than the deterministic database winner. +- Each device records the exact revision most recently reflected in each Vault file. An ordinary edit extends that displayed branch even before a conflict exists. Conflict-time deletion and case-only rename retain their separate displayed-branch contracts. - A cross-path rename stores the target before logically deleting only the displayed source branch. -The all-branch history check prevents a resolved conflict from being recreated merely because the receiving Vault still contains the known losing version. If the user has edited that version again, its bytes differ and the overwrite guard preserves it. +The recorded revision can belong to a deleted losing branch. If its readable body still matches the Vault, the propagated resolution can be reflected without recreating the conflict. A historical byte match without that record does not establish that the file is unchanged: it may be an intentional revert. Existing Vaults can lack records, so an upgrade, reset, or unavailable old body can expose additional conflicts requiring review. + +The explicit **Always overwrite with a newer file** option retains its existing modification-time policy. An independent branch prevents an inferred three-way merge; it does not disable the user's selected conflict-resolution option. Metadata and Chunks retain their existing format, and matching chunks can be shared between branches. ## Resolution patterns @@ -55,8 +58,10 @@ The all-branch history check prevents a resolved conflict from being recreated m | Text or structured data has an available shared base and non-overlapping changes | Perform a conservative three-way merge. | | One side deletes content which the other leaves unchanged | Preserve the deletion. | | One side deletes content which the other modifies | Ask the user. | -| A receiving file matches a revision available anywhere in the tree | Apply the propagated database result. | -| A receiving file matches no available revision | Preserve it and ask the user. | +| A receiving file matches its exact readable recorded revision | Apply the propagated database result under the existing conflict policy. | +| A receiving file differs from its readable recorded revision | Preserve the edit as a child of that exact revision. | +| Provenance is unknown and no current non-deleted leaf matches the file | Preserve a fresh independent branch for conflict resolution. | +| Provenance is unknown and current non-deleted leaves already hold the file bytes | Avoid duplicate storage; infer provenance only for a unique match. | | A required body or shared ancestor is missing or compacted | Ask the user. | | Binary contents differ | Prefer an explicit user selection; semantic merge is unavailable. | @@ -138,13 +143,17 @@ LiveSync composes Commonlib's injected `FileReflectionProvenance` with its local path -> { revision, observedStorageMtime? } ``` -`revision` identifies the exact database revision which most recently produced the displayed Vault file. `observedStorageMtime` is the raw local modification time observed after reflection. It is not rounded, combined with another device's value, or used as proof of branch identity. No content hash is persisted. +`revision` identifies the exact database revision most recently saved from or reflected in this device's Vault. It is the base for subsequent local edits, rather than a certificate that the current file still contains those bytes. `observedStorageMtime` is the raw local modification time of the saved snapshot or the file observed after reflection. It is not rounded, combined with another device's value, or used as proof of branch identity. No content hash is persisted. -The record changes only after a successful database-to-Vault reflection or Vault-to-database write. Reading a file does not change it. The recorded revision remains authoritative even if the user edits the file to bytes which equal another branch; otherwise content equality could silently move the edit to a branch which was not displayed. +The record changes after a successful database-to-Vault reflection or Vault-to-database write. An equal-time scan may also fill a missing record while the current database revision uniquely matches the file's bytes and no conflict leaf exists. It rereads storage and the current database revision before recording; it does not change file bytes or create a database revision. Ordinary file reads do not change the record. An existing recorded revision remains authoritative even if the user edits the file to bytes which equal another branch; otherwise content equality could silently move the edit to a branch which was not displayed. + +Saving and reflection for the same Metadata document run one at a time, including the final provenance update. An ordinary save holds one captured file body and its base until the database write completes. An edit made while that save is running belongs to the next operation; the save does not reread the file to prove that it remained unchanged. Different files retain their existing concurrency limits, and the handler acquires the lock before loading a file body from storage. Conflict checking runs after the lock is released so that an immediate resolution can safely call the file handler again. The host queues count document-lock waiters against their concurrency limits, so a burst for one document can temporarily delay unrelated files. + +The common lock does not stop Obsidian edits, external filesystem writes, or replication into the database. Incoming overwrite and deletion protection still checks current storage. Pending events restored at startup retain bounded rechecks because they run before file watching begins and cannot rely on another change notification. LiveSync creates the namespaced store handle during service composition, before the key-value database is open. The sequential `onSettingLoaded` lifecycle opens that database before Vault scanning, watching, or replication starts. Store operations do not wait for implicit readiness: a lifecycle violation fails promptly, avoiding an indefinite or self-referential initialisation wait. Local database reset is a transient unavailable boundary, after which scanning reconstructs derived state. -When no record exists, LiveSync may reconstruct the displayed revision only if the current Vault bytes match exactly one available revision body. No match, or identical content in multiple revisions, cannot prove branch identity. +For ordinary saves and incoming reflection, a missing or unreadable recorded base permits reconstruction only from exactly one matching current non-deleted leaf. Matching several current leaves avoids duplicate storage but does not identify a displayed branch. No current match creates an independent branch, even when an older ancestor has the same bytes. Deletion and rename retain their existing provenance-recovery contracts. ## Operations while a conflict exists @@ -231,7 +240,7 @@ If the user renames `draft.md` to `published.md`, LiveSync stores `published.md` ### A remote resolution reaches a device which still shows the losing content -Android may resolve a conflict and continue editing while Mac still shows the losing revision. When Mac receives the resolved tree, LiveSync searches every available branch and recognises Mac's unchanged bytes as content which was already synchronised below the deleted losing leaf. It can apply Android's resolution without asking Mac to resolve the same unchanged conflict again. +Android may resolve a conflict and continue editing while Mac still shows the losing revision. When Mac receives the resolved tree, LiveSync compares Mac's bytes with the exact revision recorded for its Vault. If that body remains readable and matches, it can apply Android's resolution without asking Mac to resolve the same unchanged conflict again. If the user edited the file on Mac before the resolution arrived, the bytes no longer match that historical revision. LiveSync preserves the Mac edit as an unsynchronised conflict instead of overwriting it. @@ -243,15 +252,15 @@ The first decision has already changed the ordinary revision tree. On restart, L ### The device-local record is missing -A local-database reset removes revision provenance. On the next scan, if the Vault file matches exactly one available revision, LiveSync can reconstruct which branch was displayed and continue from it. If the bytes match multiple revisions, or no available revision, the branch remains unproved. +A local-database reset removes revision provenance. When an ordinary save or incoming reflection examines the file, exactly one matching current non-deleted leaf can reconstruct the record. An equal-time scan can fill a missing record only while the file bytes match the sole current non-deleted revision, the revision tree has no conflict, and the file and database remain unchanged during verification. A failed provenance read is not treated as a missing record. Multiple current matches prevent duplicate storage but leave branch identity unproved. A match only in past history is insufficient; differing current content is preserved as an independent branch. Once the local database has advanced beyond an unrecorded file, the scan cannot infer its earlier origin from historical byte equality. -In that unproved state, an edit is retained as another manual-resolution branch. A deletion leaves all existing branches intact. A cross-path rename stores the target but leaves every source branch for review. The result can require an extra decision, but it does not discard data by guessing the winner. +If no current non-deleted leaf contains the file bytes, an ordinary save retains them as another independent branch. An unproven deletion leaves all existing branches intact. A cross-path rename stores the target but leaves every unproven source branch for review. The result can require an extra decision, but it does not discard data by guessing the winner. ### Start-up or reset overlaps a provenance operation LiveSync creates the provenance handle during composition, then opens its backing store during the sequential settings lifecycle before starting scans, watchers, or replication. If the store cannot open, start-up stops rather than leaving file processing waiting indefinitely. -During reset, the store can be temporarily unavailable. A racing provenance lookup fails promptly and follows the same conservative missing-record behaviour. After reopen, scanning can reconstruct a record when one exact revision body matches the Vault file. +During reset, the store can be temporarily unavailable. A racing provenance lookup fails promptly and follows the same conservative missing-record behaviour. After reopen, ordinary saving or reflection can reconstruct a record from a unique matching current non-deleted leaf. ## Unsafe shortcuts @@ -260,6 +269,7 @@ Do not: - infer a common ancestor from generation numbers alone; - assume that the PouchDB winner is the version currently displayed in the Vault; - replace recorded displayed provenance merely because current bytes match another branch; +- classify a file as unchanged solely because it matches an ancestor somewhere in history; - discard local content when revision-history lookup fails; - infer revision identity from path, size, modification time, or content hash without a revision ID; - select the newest modification time unless the user has explicitly chosen that destructive policy; or @@ -267,7 +277,9 @@ Do not: ## Verification -Commonlib's real-PouchDB and injected-boundary unit tests cover unequal branch lengths, exact shared ancestry, deterministic ordering of multiple current leaves, a sensible stage followed by reconstruction of a manual pair, content below a deleted losing leaf, recorded and reconstructed branch identity, ambiguous matches, conflict-time editing, missing-body preservation when parent metadata is available, refusal to invent a parent for a generation-one revision, logical deletion, case-only rename, cross-path rename, and safe unproven fallbacks. +LiveSync also exercises three and four independently editing devices through real CouchDB, using the installed Commonlib package and the CLI conflict-resolution command dispatcher. These tests check unchanged losing files before and after resolution, genuine edits on a losing branch, missing provenance, compacted bases, independent-root deduplication, and propagation of the selected result. See the [multiple-device regression procedure and coverage boundaries](../test/README.md#multiple-device-conflict-regression-tests). + +Commonlib owns the real-PouchDB and injected-boundary tests for revision ancestry, content preservation, provenance, independent branches, and repeated file events. LiveSync owns persistent host composition and actual Obsidian restart coverage. The focused `test:e2e:obsidian:stale-file-restart` scenario advances the local DB while old Vault bytes remain, persists pending file events, and restarts the same isolated profile. It requires an unchanged recorded file to reflect the DB without a new revision, an unknown file to remain on an independent branch alongside the DB content, and repeated processing after provenance loss to leave those branches unchanged. It also removes the record for a file whose bytes still match the current database revision, checks that start-up restores the record without a new database revision, and then checks that a later incoming revision reflects without a conflict. It uses real local storage and startup processing; transport replication and mobile lifecycle coverage are separate. LiveSync's optional real-Obsidian two-Vault checks have two scopes. `E2E_OBSIDIAN_INCLUDE_MARKDOWN_CONFLICT=true` resolves and edits a Markdown conflict, propagates it to a Vault which still displays the deleted losing content, and requires one current result to remain. `E2E_OBSIDIAN_INCLUDE_CONFLICT_OPERATIONS=true` edits, deletes, case-renames, and cross-path-renames files while conflicts remain active; it verifies the parent revision of each resulting branch, replicates those exact trees, and confirms that the other conflict branches remain intact. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 0119ded6..0d19c90a 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -47,6 +47,12 @@ Do not switch to P2P or reset the database as the first response. Check: If the remote is healthy but one device's local database is not, use [Reset Synchronisation on This Device](recovery.md#reset-synchronisation-on-this-device) only after backing up unsynchronised local files. +## An unchanged file appears as a conflict after restart + +If the log says `Preserved unsynchronised local changes as a conflict`, keep both versions available until you have checked their contents. The message means that LiveSync could not prove that the Vault file was unchanged; it does not establish that you edited the file. Use **Inspect conflicts and file/database differences** in Hatch to review the current branches before selecting a version. + +At start-up, LiveSync can restore a missing device-local revision record when the file still exactly matches the current local database revision. If a newer revision has already arrived, that match may no longer exist. LiveSync then preserves the old file for review because it cannot distinguish an unchanged copy from an intentional edit back to older content. A new release cannot resolve a conflict which was already created merely by recognising a historical content match. If the problem recurs, include the first relevant log entries, plug-in versions on both devices, and a redacted full report with the issue steps. + ## Synchronisation is paused for compatibility review A compatibility review is separate from the Change Log. It can appear after an internal database or settings-format change, or when a configured Vault is copied, restored, or opened in a new Obsidian profile without its device-local acknowledgement. @@ -90,6 +96,8 @@ Current releases automatically align compatible settings which control how new c A missing legacy file-name case setting means case-insensitive handling. It matches an explicit disabled setting and does not require a rebuild for that difference. An explicitly enabled setting can use different document IDs and still requires a compatibility decision against either value. Other configuration differences shown in the dialogue must still be resolved. +If the mismatch names **Encrypt internal file Properties**, update every device before accepting that preference. It affects subsequent Metadata writes for Hidden File Sync and Customisation Sync; it does not automatically protect existing Metadata. A manual remote Rebuild is strongly recommended if you need to protect existing paths, times, sizes, and Chunk references. + The `Sync now` command keeps routine replication progress quiet so that it is convenient to assign to a keyboard shortcut; assign one in Obsidian if that suits your workflow. A quiet command may still open this dialogue when a mismatch or another decision requires your attention. The available actions depend on when the mismatch is found: @@ -103,6 +111,10 @@ The available actions depend on when the mismatch is found: Historic defect notices and renamed controls are retained in the [0.25 release history](releases/0.25.md) and [legacy release history](releases/legacy.md), rather than in the current troubleshooting path. +## The remote database uses an unknown feature + +When a notice identifies an unknown feature, update this device and every other client of the same CouchDB database, including the CLI. The notice includes the feature identifier even if this version has no descriptive name for it. New synchronisation is refused, and receiving an unsupported requirement stops active replication, because an older client may not interpret the Metadata and its Chunk references correctly. Already queued file changes are not rolled back. The cleaned-remote recovery path also checks compatibility before counting Chunk references. Do not remove the feature name from the remote version document to bypass the check. After updating, reconnect and review any pending file changes before running Garbage Collection. + ## Setup and settings questions ### Share a configuration with another device diff --git a/eslint.community.config.mjs b/eslint.community.config.mjs index 4944a5f8..e81aeb31 100644 --- a/eslint.community.config.mjs +++ b/eslint.community.config.mjs @@ -52,6 +52,8 @@ export default defineConfig( "obsidianmd/rule-custom-message": "off", "no-console": "warn", "obsidianmd/no-unsupported-api": "error", + // Treat direct globalThis access as an error so the CI gate rejects it. + "obsidianmd/no-global-this": "error", // Keep legacy type-safety debt visible while reserving errors for directory-review blockers. "@typescript-eslint/no-unsafe-argument": "warn", "@typescript-eslint/no-unsafe-assignment": "warn", diff --git a/eslint.config.mjs b/eslint.config.mjs index b436c9e3..84cbbfbd 100644 --- a/eslint.config.mjs +++ b/eslint.config.mjs @@ -99,6 +99,13 @@ export default defineConfig([ ...ImportAliasRules("."), }, }, + { + files: ["src/integrations/**/*.ts"], + rules: { + // External-service integrations also run in Node and do not own window UI. + "obsidianmd/no-global-this": "off", + }, + }, { files: ["src/apps/**/*.ts"], rules: { diff --git a/manifest.json b/manifest.json index 7cae12be..93b3ff90 100644 --- a/manifest.json +++ b/manifest.json @@ -1,7 +1,7 @@ { "id": "obsidian-livesync", "name": "Self-hosted LiveSync", - "version": "1.0.28", + "version": "1.0.32", "minAppVersion": "1.7.2", "description": "Community implementation of self-hosted livesync. Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.", "author": "vorotamoroz", diff --git a/package-lock.json b/package-lock.json index 6e70af04..f4dc1a02 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "obsidian-livesync", - "version": "1.0.28", + "version": "1.0.32", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "obsidian-livesync", - "version": "1.0.28", + "version": "1.0.32", "license": "MIT", "workspaces": [ "src/apps/cli", @@ -23,7 +23,7 @@ "@smithy/types": "^4.14.3", "@smithy/util-retry": "^4.4.5", "@vrtmrz/browser-ui-kit": "0.1.0", - "@vrtmrz/livesync-commonlib": "0.1.24", + "@vrtmrz/livesync-commonlib": "0.1.34", "@vrtmrz/obsidian-plugin-kit": "0.1.4", "@vrtmrz/ui-interactions": "0.1.2", "diff-match-patch": "^1.0.5", @@ -4567,9 +4567,9 @@ } }, "node_modules/@vrtmrz/livesync-commonlib": { - "version": "0.1.24", - "resolved": "https://registry.npmjs.org/@vrtmrz/livesync-commonlib/-/livesync-commonlib-0.1.24.tgz", - "integrity": "sha512-gOXKo3ptEUYDkjLd5PGq2vAhOSvxOqW6xE7YWo9Y8ienglfYBz8R3eZ0I/JruvwZltH2B7Bmi41pHMjmRJQe3Q==", + "version": "0.1.34", + "resolved": "https://registry.npmjs.org/@vrtmrz/livesync-commonlib/-/livesync-commonlib-0.1.34.tgz", + "integrity": "sha512-EeVpeFg3W43cpN+x0BZvFj9fN+eQFil9vohvmLLV1z/8bCTT0Hsq/bfRV59oL0qFNggPetkwyCZUusdfjajNOw==", "license": "MIT", "dependencies": { "@aws-sdk/client-s3": "^3.808.0", @@ -5629,9 +5629,9 @@ } }, "node_modules/devalue": { - "version": "5.8.1", - "resolved": "https://registry.npmjs.org/devalue/-/devalue-5.8.1.tgz", - "integrity": "sha512-4CXDYRBGqN+57wVJkuXBYmpAVUSg3L6JAQa/DFqm238G73E1wuyc/JhGQJzN7vUf/CMphYau2zXbfWzDR5aTEw==", + "version": "5.9.2", + "resolved": "https://registry.npmjs.org/devalue/-/devalue-5.9.2.tgz", + "integrity": "sha512-po4PAY5c53tw5XMocSnf8A/5OHhbbUftpr93aEN6BBoAdntUmK7vu7wOATqvt7cXO7m1Cl4gMVn6p7n6n4mj0w==", "devOptional": true, "license": "MIT" }, @@ -12665,7 +12665,7 @@ }, "src/apps/cli": { "name": "self-hosted-livesync-cli", - "version": "1.0.28-cli", + "version": "1.0.32-cli", "dependencies": { "chokidar": "^4.0.0", "minimatch": "^10.2.5", @@ -12690,7 +12690,7 @@ }, "src/apps/webapp": { "name": "livesync-webapp", - "version": "1.0.28-webapp", + "version": "1.0.32-webapp", "dependencies": { "octagonal-wheels": "^0.1.54" }, @@ -12702,7 +12702,7 @@ } }, "src/apps/webpeer": { - "version": "1.0.28-webpeer", + "version": "1.0.32-webpeer", "dependencies": { "octagonal-wheels": "^0.1.54" }, diff --git a/package.json b/package.json index abacfc2b..df277592 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "obsidian-livesync", - "version": "1.0.28", + "version": "1.0.32", "description": "Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.", "main": "main.js", "type": "module", @@ -23,7 +23,7 @@ "prettyNoWrite": "prettier --config ./.prettierrc.mjs \"**/*.js\" \"**/*.ts\" \"**/*.json\" ", "precheck:compatibility": "npm run build", "check:compatibility": "node utils/check-compatibility.js --file main.js --ios 15", - "check": "npm run tsc-check && npm run tsc-check:apps && npm run lint && npm run lint:community -- --quiet && npm run lint:community:tools && npm run svelte-check && npm run check:compatibility", + "check": "npm run tsc-check && npm run tsc-check:apps && npm run lint && npm run lint:community && npm run lint:community:tools && npm run svelte-check && npm run check:compatibility", "i18n:bake": "npm run i18n:yaml2json && npm run i18n:bakejson && npm run i18n:format", "i18n:bakejson": "tsx _tools/bakei18n.ts", "i18n:format": "prettier --config .prettierrc.mjs --write --log-level error 'src/common/messagesJson/*.json' 'src/common/messages/*.ts'", @@ -66,6 +66,7 @@ "test:e2e:obsidian:p2p-pane": "tsx test/e2e-obsidian/scripts/p2p-pane.ts", "test:e2e:obsidian:vault-reflection": "tsx test/e2e-obsidian/scripts/vault-reflection.ts", "test:e2e:obsidian:couchdb-upload": "tsx test/e2e-obsidian/scripts/couchdb-upload.ts", + "test:e2e:obsidian:chunk-fetch-retry": "tsx test/e2e-obsidian/scripts/chunk-fetch-retry.ts", "test:e2e:obsidian:tweak-compatibility": "tsx test/e2e-obsidian/scripts/tweak-compatibility.ts", "test:e2e:obsidian:couchdb-manual-setup-workflow": "tsx test/e2e-obsidian/scripts/couchdb-manual-setup-workflow.ts", "test:e2e:obsidian:cli-to-obsidian-sync": "tsx test/e2e-obsidian/scripts/cli-to-obsidian-sync.ts", @@ -78,11 +79,15 @@ "test:e2e:obsidian:p2p-connection-check:services": "npm run test:e2e:obsidian:p2p-connection-check -- --manage-p2p", "test:e2e:obsidian:partial-startup-file-failure": "tsx test/e2e-obsidian/scripts/partial-startup-file-failure.ts", "test:e2e:obsidian:startup-scan": "tsx test/e2e-obsidian/scripts/startup-scan.ts", + "test:e2e:obsidian:stale-file-restart": "tsx test/e2e-obsidian/scripts/stale-file-restart.ts", + "test:e2e:obsidian:folder-batch": "tsx test/e2e-obsidian/scripts/folder-batch.ts", "test:e2e:obsidian:setup-uri-workflow": "tsx test/e2e-obsidian/scripts/setup-uri-workflow.ts", "test:e2e:obsidian:two-vault-sync": "tsx test/e2e-obsidian/scripts/two-vault-sync.ts", "test:e2e:obsidian:security-seed-reconnect": "tsx test/e2e-obsidian/scripts/security-seed-reconnect.ts", "test:e2e:obsidian:hidden-file-snippet-sync": "tsx test/e2e-obsidian/scripts/hidden-file-snippet-sync.ts", "test:e2e:obsidian:customisation-sync": "tsx test/e2e-obsidian/scripts/customisation-sync.ts", + "test:e2e:obsidian:remote-feature-change": "tsx test/e2e-obsidian/scripts/remote-feature-change.ts", + "test:e2e:obsidian:internal-metadata-migration": "tsx test/e2e-obsidian/scripts/internal-metadata-migration.ts", "test:e2e:obsidian:setting-markdown-export": "tsx test/e2e-obsidian/scripts/setting-markdown-export.ts", "test:e2e:obsidian:upgrade-from-stable": "tsx test/e2e-obsidian/scripts/upgrade-from-stable.ts", "test:e2e:obsidian:local-suite": "tsx test/e2e-obsidian/scripts/local-suite.ts", @@ -181,7 +186,7 @@ "@smithy/types": "^4.14.3", "@smithy/util-retry": "^4.4.5", "@vrtmrz/browser-ui-kit": "0.1.0", - "@vrtmrz/livesync-commonlib": "0.1.24", + "@vrtmrz/livesync-commonlib": "0.1.34", "@vrtmrz/obsidian-plugin-kit": "0.1.4", "@vrtmrz/ui-interactions": "0.1.2", "diff-match-patch": "^1.0.5", diff --git a/src/apps/browser/BrowserP2PTransportSettings.svelte b/src/apps/browser/BrowserP2PTransportSettings.svelte index eed3bc52..56bd5831 100644 --- a/src/apps/browser/BrowserP2PTransportSettings.svelte +++ b/src/apps/browser/BrowserP2PTransportSettings.svelte @@ -1,105 +1,67 @@
Optional TURN server settings -

- Configure TURN only when a direct peer-to-peer connection cannot be established. -

- - - +

Configure TURN only when a direct peer-to-peer connection cannot be established.

+
- -
@@ -107,27 +69,7 @@
diff --git a/src/apps/cli/commands/runCommand.unit.spec.ts b/src/apps/cli/commands/runCommand.unit.spec.ts index 651e47d1..f4d3a45a 100644 --- a/src/apps/cli/commands/runCommand.unit.spec.ts +++ b/src/apps/cli/commands/runCommand.unit.spec.ts @@ -419,6 +419,30 @@ describe("runCommand abnormal cases", () => { expect(appliedSettings.useIndexedDBAdapter).toBe(false); }); + it("setup imports managed TURN through the existing encrypted URI", async () => { + const core = createCoreMock(); + const profiles = { + turn: { id: "turn", name: "TURN", isEncrypted: false, + uri: "sls+p2p://room?managedType=CF&managedId=turn-key&token=private-token" }, + }; + const passphrase = "setup-passphrase"; + const setupURI = await processSetting.encodeSettingsToSetupURI( + { + ...DEFAULT_SETTINGS, + remoteConfigurations: profiles, + }, + passphrase + ); + expect(setupURI.startsWith(configURIBase)).toBe(true); + expect(setupURI).not.toContain("private-token"); + core.services.context.standardIo.prompt.mockResolvedValue(passphrase); + await runCommand(makeOptions("setup", [setupURI]), { ...context, core }); + expect(core.services.setting.applyExternalSettings).toHaveBeenCalledWith( + expect.objectContaining({ remoteConfigurations: profiles }), + true + ); + }); + it("setup rejects encoded URI when passphrase is wrong", async () => { const core = createCoreMock(); const setupURI = await createSetupURI("correct-passphrase"); diff --git a/src/apps/cli/main.ts b/src/apps/cli/main.ts index 1b2bff53..3fd1ccb9 100644 --- a/src/apps/cli/main.ts +++ b/src/apps/cli/main.ts @@ -1,3 +1,4 @@ +import { useP2PSettingsPreparation } from "@/serviceFeatures/useP2PSettingsPreparation"; import { NodeServiceContext, NodeServiceHub } from "./services/NodeServiceHub"; import { configureNodeLocalStorage, ensureGlobalNodeLocalStorage } from "./services/NodeLocalStorage"; import { LiveSyncBaseCore, type StartupDatabaseOptions } from "@/LiveSyncBaseCore"; @@ -524,7 +525,9 @@ export async function main( useOfflineScanner(core); } // Register P2P replicator feature. - p2pReplicator = useP2PReplicatorFeature(core); + p2pReplicator = useP2PReplicatorFeature(core, undefined, undefined, { + prepareP2PSettings: useP2PSettingsPreparation(core.services.API.webCompatFetch.bind(core.services.API)), + }); // Add target filter to prevent internal files are handled core.services.vault.isTargetFile.addHandler(async (target) => { const targetPath = stripAllPrefixes(getPathFromUXFileInfo(target)); diff --git a/src/apps/cli/package.json b/src/apps/cli/package.json index e82ae069..f85b56a9 100644 --- a/src/apps/cli/package.json +++ b/src/apps/cli/package.json @@ -1,7 +1,7 @@ { "name": "self-hosted-livesync-cli", "private": true, - "version": "1.0.28-cli", + "version": "1.0.32-cli", "main": "dist/index.cjs", "type": "module", "scripts": { diff --git a/src/apps/cli/test/test-mirror-linux.sh b/src/apps/cli/test/test-mirror-linux.sh index 415548eb..70ccb687 100755 --- a/src/apps/cli/test/test-mirror-linux.sh +++ b/src/apps/cli/test/test-mirror-linux.sh @@ -7,6 +7,8 @@ # 3. DB-deleted file β†’ NOT restored to storage (UPDATE STORAGE skip) # 4. Both, storage newer β†’ DB updated (SYNC: STORAGE β†’ DB) # 5. Both, DB newer β†’ storage updated (SYNC: DB β†’ STORAGE) +# 6. Compatibility mode β†’ omitted vault-path works +# 7. Unknown local origin β†’ conflict preserved, deduplicated, and resolved # # Not covered (require precise mtime control or artificial conflict injection): # - Both, equal mtime β†’ no-op (EVEN) @@ -43,7 +45,8 @@ cli_test_init_settings_file "$SETTINGS_FILE" # isConfigured=true is required for mirror (canProceedScan checks this) cli_test_mark_settings_configured "$SETTINGS_FILE" -# Enable writeDocumentsIfConflicted to resolve unsynced conflicts during mirror +# Allow incoming DB content to be reflected when conflicts exist (Case 5). +# This does not resolve conflicts or authorise overwriting DB content. node -e ' const fs = require("fs"); const file = process.argv[1]; @@ -181,6 +184,11 @@ echo "=== Case 4: storage newer β†’ DB updated (Separated Paths) ===" # Seed DB with old content (mtime β‰ˆ now) printf 'old content\n' | run_cli "$DB_DIR" --settings "$DB_SETTINGS" put test/sync-storage-newer.md +# Establish the file's recorded base before making an ordinary local edit. +# A direct put followed by unrelated local content has unknown provenance. +run_mirror_test +cli_test_assert_equal "old content" "$(cat "$VAULT_DIR/test/sync-storage-newer.md")" "Case 4 base was not reflected" + # Write new content to storage with a timestamp 1 hour in the future printf 'new content\n' > "$VAULT_DIR/test/sync-storage-newer.md" touch -t "$(portable_touch_timestamp '+1 hour')" "$VAULT_DIR/test/sync-storage-newer.md" @@ -188,6 +196,8 @@ touch -t "$(portable_touch_timestamp '+1 hour')" "$VAULT_DIR/test/sync-storage-n run_mirror_test DB_RESULT_FILE="$WORK_DIR/case4-pull.txt" +CASE4_INFO="$(run_cli "$DB_DIR" --settings "$DB_SETTINGS" info test/sync-storage-newer.md)" +cli_test_assert_equal "N/A" "$(printf '%s' "$CASE4_INFO" | cli_test_json_string_field_from_stdin conflicts)" "Ordinary local edit unexpectedly created a conflict" run_cli "$DB_DIR" --settings "$DB_SETTINGS" pull test/sync-storage-newer.md "$DB_RESULT_FILE" if cmp -s "$VAULT_DIR/test/sync-storage-newer.md" "$DB_RESULT_FILE"; then assert_pass "DB updated to match newer storage file" @@ -238,6 +248,55 @@ else assert_fail "Compatibility mode failed to sync file into DB" fi +# ───────────────────────────────────────────────────────────────────────────── +# Case 7: Unknown local origin must preserve both contents, regardless of mtime +# ───────────────────────────────────────────────────────────────────────────── +echo "" +echo "=== Case 7: unknown local origin β†’ preserve and resolve conflict ===" + +UNKNOWN_PATH="test/unknown-origin.md" +printf 'original DB content\n' | run_cli "$DB_DIR" --settings "$DB_SETTINGS" put "$UNKNOWN_PATH" +printf 'unrelated local content\n' > "$VAULT_DIR/$UNKNOWN_PATH" +touch -t "$(portable_touch_timestamp '+1 hour')" "$VAULT_DIR/$UNKNOWN_PATH" +run_mirror_test + +UNKNOWN_INFO="$(run_cli "$DB_DIR" --settings "$DB_SETTINGS" info "$UNKNOWN_PATH")" +WINNER="$(printf '%s' "$UNKNOWN_INFO" | cli_test_json_string_field_from_stdin revision)" +CONFLICT="$(printf '%s' "$UNKNOWN_INFO" | cli_test_json_string_field_from_stdin conflicts)" +if [[ ! "$WINNER" =~ ^1-[[:xdigit:]]+$ || ! "$CONFLICT" =~ ^1-[[:xdigit:]]+$ || "$WINNER" == "$CONFLICT" ]]; then + echo "[FAIL] Expected two independent non-deleted root revisions: $UNKNOWN_INFO" >&2 + exit 1 +fi + +# Do not assume which randomly identified root PouchDB selects as the winner. +LOCAL_REV="" +DB_REV="" +for revision in "$WINNER" "$CONFLICT"; do + CONTENT="$(run_cli "$DB_DIR" --settings "$DB_SETTINGS" cat-rev "$UNKNOWN_PATH" "$revision" | cli_test_sanitise_cat_stdout)" + case "$CONTENT" in + 'unrelated local content') LOCAL_REV="$revision" ;; + 'original DB content') DB_REV="$revision" ;; + *) echo "[FAIL] Unexpected content for $revision: $CONTENT" >&2; exit 1 ;; + esac +done +[[ -n "$LOCAL_REV" && -n "$DB_REV" ]] || { echo "[FAIL] Both contents must remain readable" >&2; exit 1; } + +# Force another ordinary save of the same unknown bytes, even if incoming +# reflection replaced the file under writeDocumentsIfConflicted. +printf 'unrelated local content\n' > "$VAULT_DIR/$UNKNOWN_PATH" +touch -t "$(portable_touch_timestamp '+1 hour')" "$VAULT_DIR/$UNKNOWN_PATH" +run_mirror_test +REPEATED_INFO="$(run_cli "$DB_DIR" --settings "$DB_SETTINGS" info "$UNKNOWN_PATH")" +cli_test_assert_equal "$WINNER" "$(printf '%s' "$REPEATED_INFO" | cli_test_json_string_field_from_stdin revision)" "Repeated mirror changed the winning revision" +cli_test_assert_equal "$CONFLICT" "$(printf '%s' "$REPEATED_INFO" | cli_test_json_string_field_from_stdin conflicts)" "Repeated mirror created another conflict" + +run_cli "$DB_DIR" --vault "$VAULT_DIR" --settings "$DB_SETTINGS" resolve "$UNKNOWN_PATH" "$LOCAL_REV" +RESOLVED_INFO="$(run_cli "$DB_DIR" --settings "$DB_SETTINGS" info "$UNKNOWN_PATH")" +cli_test_assert_equal "N/A" "$(printf '%s' "$RESOLVED_INFO" | cli_test_json_string_field_from_stdin conflicts)" "CLI resolve left a conflict" +cli_test_assert_equal "$LOCAL_REV" "$(printf '%s' "$RESOLVED_INFO" | cli_test_json_string_field_from_stdin revision)" "CLI resolve selected the wrong revision" +cli_test_assert_equal "unrelated local content" "$(cat "$VAULT_DIR/$UNKNOWN_PATH")" "CLI resolve did not reflect the selected content" +assert_pass "Unknown local content was preserved, deduplicated, and resolved through the CLI" + # ───────────────────────────────────────────────────────────────────────────── # Summary # ───────────────────────────────────────────────────────────────────────────── diff --git a/src/apps/cli/testdeno/helpers/docker.ts b/src/apps/cli/testdeno/helpers/docker.ts index 159d6e60..822f0211 100644 --- a/src/apps/cli/testdeno/helpers/docker.ts +++ b/src/apps/cli/testdeno/helpers/docker.ts @@ -628,7 +628,7 @@ export async function startP2pRelay(): Promise { //TODO: port mapping should be configurable. "4000:7777", "--tmpfs", - "/app/strfry-db:rw,size=256m", + "/app/strfry-db:rw,size=256m,mode=1777", "--entrypoint", "sh", P2P_RELAY_IMAGE, diff --git a/src/apps/cli/testdeno/helpers/settings.ts b/src/apps/cli/testdeno/helpers/settings.ts index 524622c0..bb999b13 100644 --- a/src/apps/cli/testdeno/helpers/settings.ts +++ b/src/apps/cli/testdeno/helpers/settings.ts @@ -13,7 +13,11 @@ export async function initSettingsFile(settingsFile: string): Promise { * Generate a full setup URI from a settings file via the Commonlib package API. * Mirrors the bash flow in test-setup-put-cat-linux.sh. */ -export async function generateSetupUriFromSettings(settingsFile: string, setupPassphrase: string): Promise { +export async function generateSetupUriFromSettings( + settingsFile: string, + setupPassphrase: string, + preserveRemoteSettings = false +): Promise { const script = [ "import { fs } from '@vrtmrz/livesync-commonlib/node';", "import { encodeSettingsToSetupURI } from '@vrtmrz/livesync-commonlib/compat/API/processSetting';", @@ -21,13 +25,17 @@ export async function generateSetupUriFromSettings(settingsFile: string, setupPa " const settingsPath = process.env.SETTINGS_FILE;", " const passphrase = process.env.SETUP_PASSPHRASE;", " const settings = JSON.parse(fs.readFileSync(settingsPath, 'utf-8'));", - " settings.couchDB_DBNAME = 'setup-put-cat-db';", - " settings.couchDB_URI = 'http://127.0.0.1:5999';", - " settings.couchDB_USER = 'dummy';", - " settings.couchDB_PASSWORD = 'dummy';", - " settings.liveSync = false;", - " settings.syncOnStart = false;", - " settings.syncOnSave = false;", + ...(preserveRemoteSettings + ? [] + : [ + " settings.couchDB_DBNAME = 'setup-put-cat-db';", + " settings.couchDB_URI = 'http://127.0.0.1:5999';", + " settings.couchDB_USER = 'dummy';", + " settings.couchDB_PASSWORD = 'dummy';", + " settings.liveSync = false;", + " settings.syncOnStart = false;", + " settings.syncOnSave = false;", + ]), " const uri = await encodeSettingsToSetupURI(settings, passphrase);", " process.stdout.write(uri.trim());", "})();", diff --git a/src/apps/cli/testdeno/test-mirror.ts b/src/apps/cli/testdeno/test-mirror.ts index b4dc899a..1b2bcc22 100644 --- a/src/apps/cli/testdeno/test-mirror.ts +++ b/src/apps/cli/testdeno/test-mirror.ts @@ -11,6 +11,7 @@ * 4. Both, storage newer -> DB updated (SYNC: STORAGE -> DB) * 5. Both, DB newer -> storage updated (SYNC: DB -> STORAGE) * 6. Compatibility mode -> omitted vault-path works (same DB + vault path) + * 7. Unknown local origin -> conflict preserved, deduplicated, and resolved * * No external services are required. * @@ -18,9 +19,9 @@ * deno test -A test-mirror.ts */ -import { assert } from "@std/assert"; +import { assert, assertEquals } from "@std/assert"; import { TempDir } from "./helpers/temp.ts"; -import { runCliOrFail } from "./helpers/cli.ts"; +import { runCliOrFail, runCliWithInputOrFail } from "./helpers/cli.ts"; import { initSettingsFile, markSettingsConfigured } from "./helpers/settings.ts"; Deno.test("mirror: storage <-> DB synchronisation", async (t) => { @@ -130,6 +131,10 @@ Deno.test("mirror: storage <-> DB synchronisation", async (t) => { await Deno.writeTextFile(seedFile, "old content\n"); await dbRun("push", seedFile, "test/sync-storage-newer.md"); + // Reflect the shared base into the actual Vault before editing it. + await runMirror(); + assertEquals(await Deno.readTextFile(workDir.join("vault", "test", "sync-storage-newer.md")), "old content\n"); + // Write new content to storage with a timestamp 1 hour in the future const storageFile = workDir.join("vault", "test", "sync-storage-newer.md"); await Deno.writeTextFile(storageFile, "new content\n"); @@ -138,6 +143,8 @@ Deno.test("mirror: storage <-> DB synchronisation", async (t) => { await runMirror(); const resultFile = workDir.join("case4-pull.txt"); + const info = JSON.parse(await dbRun("info", "test/sync-storage-newer.md")); + assertEquals(info.conflicts, "N/A", "An ordinary local edit must not create a conflict"); await dbRun("pull", "test/sync-storage-newer.md", resultFile); const storageContent = await Deno.readTextFile(storageFile); const pulledContent = await Deno.readTextFile(resultFile); @@ -184,6 +191,47 @@ Deno.test("mirror: storage <-> DB synchronisation", async (t) => { assert(pulled === "compat-content\n", `Compatibility mode failed to sync file into DB (got: '${pulled}')`); console.log("[PASS] case 6: compatibility mode works"); }); + + // ------------------------------------------------------------------- + // Case 7: unknown local origin must preserve both contents regardless of mtime. + // This deliberately uses put: push would record a file provenance entry. + // ------------------------------------------------------------------- + await t.step("case 7: unknown local content is preserved, deduplicated, and resolved", async () => { + const path = "test/unknown-origin.md"; + const storageFile = workDir.join("vault", "test", "unknown-origin.md"); + await runCliWithInputOrFail("original DB content\n", dbDir, "--settings", dbSettings, "put", path); + const writeUnknownFile = async () => { + await Deno.writeTextFile(storageFile, "unrelated local content\n"); + await Deno.utime(storageFile, new Date(), new Date(Date.now() + 3600_000)); + }; + await writeUnknownFile(); + await runMirror(); + + const info = JSON.parse(await dbRun("info", path)); + assert(/^1-[\da-f]+$/.test(info.revision), "Expected an independent winning root"); + assert(/^1-[\da-f]+$/.test(info.conflicts), "Expected exactly one independent conflicting root"); + assert(info.revision !== info.conflicts, "Expected two distinct revisions"); + const contents = new Map(); + for (const revision of [info.revision, info.conflicts]) { + contents.set(await dbRun("cat-rev", path, revision), revision); + } + assertEquals([...contents.keys()].sort(), ["original DB content\n", "unrelated local content\n"]); + + // Re-submit identical local bytes even if incoming reflection replaced + // the file under writeDocumentsIfConflicted; no third branch is needed. + await writeUnknownFile(); + await runMirror(); + const repeated = JSON.parse(await dbRun("info", path)); + assertEquals(repeated.revision, info.revision); + assertEquals(repeated.conflicts, info.conflicts); + + const localRevision = contents.get("unrelated local content\n")!; + await runCliOrFail(dbDir, "--vault", vaultDir, "--settings", dbSettings, "resolve", path, localRevision); + const resolved = JSON.parse(await dbRun("info", path)); + assertEquals(resolved.conflicts, "N/A"); + assertEquals(resolved.revision, localRevision); + assertEquals(await Deno.readTextFile(storageFile), "unrelated local content\n"); + }); }); // --------------------------------------------------------------------------- diff --git a/src/apps/cli/testdeno/test-p2p-sync.ts b/src/apps/cli/testdeno/test-p2p-sync.ts index e05c6efe..32f3d1ad 100644 --- a/src/apps/cli/testdeno/test-p2p-sync.ts +++ b/src/apps/cli/testdeno/test-p2p-sync.ts @@ -1,6 +1,11 @@ import { assert } from "@std/assert"; import { TempDir } from "./helpers/temp.ts"; -import { initSettingsFile, applyP2pSettings, applyP2pTestTweaks } from "./helpers/settings.ts"; +import { + initSettingsFile, + applyP2pSettings, + applyP2pTestTweaks, + generateSetupUriFromSettings, +} from "./helpers/settings.ts"; import { startCliInBackground } from "./helpers/backgroundCli.ts"; import { discoverPeer, @@ -9,10 +14,10 @@ import { maybeStartCoturn, stopCoturnIfStarted, } from "./helpers/p2p.ts"; -import { runCli } from "./helpers/cli.ts"; +import { runCli, runCliOrFail, runCliWithInputOrFail, sanitiseCatStdout } from "./helpers/cli.ts"; import { getOptimalLoopbackIp } from "./helpers/net.ts"; -Deno.test("p2p-sync: discovers peer and completes sync", async () => { +Deno.test("p2p-sync: transfers with the same ID key and rejects a different document ID key", async () => { const loopbackIp = await getOptimalLoopbackIp(); const loopbackHost = loopbackIp === "::1" ? "[::1]" : loopbackIp; @@ -32,14 +37,18 @@ Deno.test("p2p-sync: discovers peer and completes sync", async () => { const hostSettings = workDir.join("settings-host.json"); const clientVault = workDir.join("vault-sync"); const clientSettings = workDir.join("settings-sync.json"); + const rejectedVault = workDir.join("vault-rejected"); + const rejectedSettings = workDir.join("settings-rejected.json"); await Deno.mkdir(hostVault, { recursive: true }); await Deno.mkdir(clientVault, { recursive: true }); + await Deno.mkdir(rejectedVault, { recursive: true }); const relayStarted = await maybeStartLocalRelay(relay); const coturnStarted = await maybeStartCoturn(turnServers); try { await initSettingsFile(hostSettings); await initSettingsFile(clientSettings); + await initSettingsFile(rejectedSettings); await applyP2pSettings( hostSettings, roomId, @@ -58,8 +67,52 @@ Deno.test("p2p-sync: discovers peer and completes sync", async () => { "~.*", turnServers ); + await applyP2pSettings( + rejectedSettings, + roomId, + passphrase, + "self-hosted-livesync-cli-tests", + relay, + "~.*", + turnServers + ); await applyP2pTestTweaks(hostSettings, hostPeerName, passphrase); await applyP2pTestTweaks(clientSettings, clientPeerName, passphrase); + await applyP2pTestTweaks(rejectedSettings, "p2p-rejected-" + nonce, passphrase); + for (const [vault, path, key, label] of [ + [hostVault, hostSettings, "ab".repeat(32), "host"], + [clientVault, clientSettings, "ab".repeat(32), "client"], + [rejectedVault, rejectedSettings, "cd".repeat(32), "rejected"], + ]) { + const settings = JSON.parse(await Deno.readTextFile(path)); + settings.idDerivationVersion = 1; + settings.idDerivationKey = key; + const sourcePath = workDir.join("setup-source-" + label + ".json"); + await Deno.writeTextFile(sourcePath, JSON.stringify(settings)); + const setupPassphrase = "independent-id-setup-passphrase"; + const setupUri = await generateSetupUriFromSettings(sourcePath, setupPassphrase, true); + await runCliWithInputOrFail(setupPassphrase + "\n", vault, "--settings", path, "setup", setupUri); + const persisted = JSON.parse(await Deno.readTextFile(path)); + assert(persisted.idDerivationVersion === 1, "The Setup URI lost the ID derivation version."); + assert(persisted.idDerivationKey === "", "The CLI stored the ID key in plain text."); + assert( + typeof persisted.encryptedIdDerivationKey === "string" && persisted.encryptedIdDerivationKey.length > 0, + "The CLI did not encrypt the saved ID key." + ); + assert(persisted.P2P_Enabled === true, "The Setup URI disabled P2P."); + assert(persisted.P2P_roomID === roomId, "The Setup URI changed the P2P room."); + assert(persisted.P2P_relays === relay, "The Setup URI changed the P2P relay."); + assert(persisted.remoteType === "ONLY_P2P", "The Setup URI changed the remote type."); + } + const notePath = "p2p/independent-id-note.md"; + await runCliWithInputOrFail( + "A note transferred with the saved ID key.\n", + clientVault, + "--settings", + clientSettings, + "put", + notePath + ); const host = startCliInBackground(hostVault, "--settings", hostSettings, "p2p-host"); try { @@ -82,9 +135,32 @@ Deno.test("p2p-sync: discovers peer and completes sync", async () => { syncResult.code === 0, `p2p-sync failed\nstdout: ${syncResult.stdout}\nstderr: ${syncResult.stderr}` ); + const rejectedPeer = await discoverPeer(rejectedVault, rejectedSettings, peersTimeout, hostPeerName); + const rejectedSync = await runCli( + rejectedVault, + "--settings", + rejectedSettings, + "p2p-sync", + rejectedPeer.id, + String(syncTimeout) + ); + assert( + rejectedSync.code !== 0, + `P2P accepted a different key for obfuscated document IDs.\nstdout: ${rejectedSync.stdout}\nstderr: ${rejectedSync.stderr}` + ); + assert( + rejectedSync.combined.includes("Tweak values are not matched"), + `P2P failed before checking peer settings.\nstdout: ${rejectedSync.stdout}\nstderr: ${rejectedSync.stderr}` + ); } finally { await host.stop(); } + const received = sanitiseCatStdout( + await runCliOrFail(hostVault, "--settings", hostSettings, "cat", notePath) + ).trimEnd(); + assert(received === "A note transferred with the saved ID key.", "The host did not receive the keyed note."); + const rejectedRead = await runCli(rejectedVault, "--settings", rejectedSettings, "cat", notePath); + assert(rejectedRead.code !== 0, "The rejected device received the keyed note."); } finally { await stopLocalRelayIfStarted(relayStarted); await stopCoturnIfStarted(coturnStarted); diff --git a/src/apps/webapp/WebAppRuntime.ts b/src/apps/webapp/WebAppRuntime.ts index 63940081..48405a4c 100644 --- a/src/apps/webapp/WebAppRuntime.ts +++ b/src/apps/webapp/WebAppRuntime.ts @@ -1,3 +1,4 @@ +import { useP2PSettingsPreparation } from "@/serviceFeatures/useP2PSettingsPreparation"; /** Browser runtime for Self-hosted LiveSync over the File System Access API. */ import { LiveSyncBaseCore } from "@/LiveSyncBaseCore"; @@ -217,7 +218,9 @@ export class WebAppRuntime { useRedFlagFeatures(core); useCheckRemoteSize(core); useRemoteConfiguration(core); - this.p2p = useP2PReplicatorFeature(core); + this.p2p = useP2PReplicatorFeature(core, undefined, undefined, { + prepareP2PSettings: useP2PSettingsPreparation(core.services.API.webCompatFetch.bind(core.services.API)), + }); this.paneHost = { services: core.services, p2p: this.p2p, diff --git a/src/apps/webapp/package.json b/src/apps/webapp/package.json index 4c77ed2b..f8764ab7 100644 --- a/src/apps/webapp/package.json +++ b/src/apps/webapp/package.json @@ -1,7 +1,7 @@ { "name": "livesync-webapp", "private": true, - "version": "1.0.28-webapp", + "version": "1.0.32-webapp", "type": "module", "description": "Browser-based Self-hosted LiveSync using FileSystem API", "scripts": { diff --git a/src/apps/webpeer/package.json b/src/apps/webpeer/package.json index c11c37af..adbda057 100644 --- a/src/apps/webpeer/package.json +++ b/src/apps/webpeer/package.json @@ -1,7 +1,7 @@ { "name": "webpeer", "private": true, - "version": "1.0.28-webpeer", + "version": "1.0.32-webpeer", "type": "module", "scripts": { "dev": "vite", diff --git a/src/apps/webpeer/src/WebPeerRuntime.ts b/src/apps/webpeer/src/WebPeerRuntime.ts index c6e7d433..b4f69758 100644 --- a/src/apps/webpeer/src/WebPeerRuntime.ts +++ b/src/apps/webpeer/src/WebPeerRuntime.ts @@ -1,3 +1,4 @@ +import { useP2PSettingsPreparation } from "@/serviceFeatures/useP2PSettingsPreparation"; import { type P2PSyncSetting, SETTING_KEY_P2P_DEVICE_NAME } from "@vrtmrz/livesync-commonlib/compat/common/types"; import { compatGlobal } from "@vrtmrz/livesync-commonlib/compat/common/coreEnvFunctions"; import { EVENT_LAYOUT_READY } from "@vrtmrz/livesync-commonlib/compat/events/coreEvents"; @@ -70,9 +71,8 @@ export class WebPeerRuntime { isScheduled: () => this.restartScheduled, }, }); - this.p2p = useP2PReplicatorFeature({ - services: this.services, - serviceModules: {}, + this.p2p = useP2PReplicatorFeature({ services: this.services, serviceModules: {} }, undefined, undefined, { + prepareP2PSettings: useP2PSettingsPreparation(this.services.API.webCompatFetch.bind(this.services.API)), }); this.p2pLogCollector = new P2PLogCollector(this.events); this.paneHost = { diff --git a/src/common/messages/LiveSyncProvisionalMessages.ts b/src/common/messages/LiveSyncProvisionalMessages.ts index 37f0a7b3..4825633f 100644 --- a/src/common/messages/LiveSyncProvisionalMessages.ts +++ b/src/common/messages/LiveSyncProvisionalMessages.ts @@ -7,6 +7,26 @@ * remove it from this map in the same change. */ export const liveSyncProvisionalEnglishMessages = { + "Configure TURN when a direct connection cannot be established or when you select TURN relay only.": + "Configure TURN when a direct connection cannot be established or when you select TURN relay only.", + "TURN configuration": "TURN configuration", + Manual: "Manual", + "Managed (Cloudflare)": "Managed (Cloudflare)", + "TURN Key ID": "TURN Key ID", + "TURN Key API Token": "TURN Key API Token", + "Unsupported TURN configuration": "Unsupported TURN configuration", + "The API token is saved with this profile and included in Setup URI and QR code sharing. Temporary TURN credentials are kept in memory only.": + "The API token is saved with this profile and included in Setup URI and QR code sharing. Temporary TURN credentials are kept in memory only.", + "TURN relay only requires a TURN server or a configured credential source under Advanced Settings.": + "TURN relay only requires a TURN server or a configured credential source under Advanced Settings.", + "TURN relay only requires TURN configuration. Connection path has been restored to Automatic.": + "TURN relay only requires TURN configuration. Connection path has been restored to Automatic.", + "Enter a TURN Key ID.": "Enter a TURN Key ID.", + "TURN Key ID contains unsupported characters.": "TURN Key ID contains unsupported characters.", + "Enter a TURN Key API Token.": "Enter a TURN Key API Token.", + "TURN Key API Token must use Bearer token syntax.": "TURN Key API Token must use Bearer token syntax.", + "The selected TURN configuration is not supported.": "The selected TURN configuration is not supported.", + "Setup Complete: Preparing to Fetch from Another Device": "Setup Complete: Preparing to Fetch from Another Device", "The P2P connection has been configured successfully. The initial synchronisation data must now be fetched from an online source device.": "The P2P connection has been configured successfully. The initial synchronisation data must now be fetched from an online source device.", @@ -28,8 +48,8 @@ export const liveSyncProvisionalEnglishMessages = { "The project's public signalling relay is a best-effort convenience operated by the project author. It does not store Vault contents, but signalling metadata may be visible to the relay. Availability and log retention are not guaranteed. You can replace it with your own Nostr-compatible relay.", "Learn more about P2P connections": "Learn more about P2P connections", "Learn more about signalling and TURN": "Learn more about signalling and TURN", - "TURN relays the encrypted WebRTC connection only when a direct path cannot be established. A TURN provider cannot read encrypted Vault contents, but it can observe connection metadata and traffic volume. Use a provider you trust.": - "TURN relays the encrypted WebRTC connection only when a direct path cannot be established. A TURN provider cannot read encrypted Vault contents, but it can observe connection metadata and traffic volume. Use a provider you trust.", + "WebRTC encrypts data between your devices, including when it passes through TURN. The TURN provider cannot read the transferred data. It can see network addresses and traffic volume.": + "WebRTC encrypts data between your devices, including when it passes through TURN. The TURN provider cannot read the transferred data. It can see network addresses and traffic volume.", "Connection compatibility": "Connection compatibility", "P2P message size": "P2P message size", Standard: "Standard", @@ -184,6 +204,50 @@ export const liveSyncProvisionalEnglishMessages = { "Repair failed before the source was removed. Run inspection again before retrying.", "Connection settings": "Connection settings", "Saved connections": "Saved connections", + "ID generation": "ID generation", + "Keep current configuration": "Keep current configuration", + "Set an ID key": "Set an ID key", + "Current configuration: a saved ID key is used.": "Current configuration: a saved ID key is used.", + "Current configuration: the saved ID key is retained while E2EE is off.": + "Current configuration: the saved ID key is retained while E2EE is off.", + "Current configuration: no ID key is saved. With E2EE enabled, keeping it uses legacy IDs tied to the E2EE passphrase.": + "Current configuration: no ID key is saved. With E2EE enabled, keeping it uses legacy IDs tied to the E2EE passphrase.", + "Changing the E2EE passphrase changes IDs generated by the legacy configuration.": + "Changing the E2EE passphrase changes IDs generated by the legacy configuration.", + "This uses a saved key for new Chunk IDs and obfuscated Metadata document IDs, so changing the E2EE passphrase does not derive a new key automatically.": + "This uses a saved key for new Chunk IDs and obfuscated Metadata document IDs, so changing the E2EE passphrase does not derive a new key automatically.", + Configured: "Configured", + "The saved ID key is configured. Its source cannot be shown again.": + "The saved ID key is configured. Its source cannot be shown again.", + "Leave this input empty to keep the saved ID key.": "Leave this input empty to keep the saved ID key.", + "Generate a random ID key": "Generate a random ID key", + "How to set the ID key": "How to set the ID key", + "Derive from current E2EE passphrase": "Derive from current E2EE passphrase", + "Enter an ID source": "Enter an ID source", + "Import an ID recovery code": "Import an ID recovery code", + "ID source": "ID source", + "ID recovery code": "ID recovery code", + "Enter an ID recovery code": "Enter an ID recovery code", + "Choose a long, unpredictable source. It is used once and cannot be shown again after saving. A recovery code can be displayed on this device later. This input also accepts a tagged recovery code.": + "Choose a long, unpredictable source. It is used once and cannot be shown again after saving. A recovery code can be displayed on this device later. This input also accepts a tagged recovery code.", + "Paste a tagged recovery code from an existing device to restore the same ID key.": + "Paste a tagged recovery code from an existing device to restore the same ID key.", + "For recovery after losing every device, save the recovery code after setup or choose an ID source you can reproduce.": + "For recovery after losing every device, save the recovery code after setup or choose an ID source you can reproduce.", + "Show current recovery code": "Show current recovery code", + "Hide current recovery code": "Hide current recovery code", + "Current ID recovery code": "Current ID recovery code", + "Copy recovery code": "Copy recovery code", + "Recovery code copied.": "Recovery code copied.", + "The displayed recovery code belongs to the current key. Reopen this dialogue after saving to copy the replacement key.": + "The displayed recovery code belongs to the current key. Reopen this dialogue after saving to copy the replacement key.", + "The recovery code could not be copied. Select and copy the visible code instead.": + "The recovery code could not be copied. Select and copy the visible code instead.", + "The ID key is derived from the current E2EE passphrase and saved separately. Changing that passphrase later does not change the saved ID key. To reduce the risk of guessing that passphrase from known IDs, use a separate, unpredictable ID source instead.": + "The ID key is derived from the current E2EE passphrase and saved separately. Changing that passphrase later does not change the saved ID key. To reduce the risk of guessing that passphrase from known IDs, use a separate, unpredictable ID source instead.", + "An ID source is required to enable this option.": "An ID source is required to enable this option.", + "The ID source or recovery code is invalid. Check it and try again.": + "The ID source or recovery code is invalid. Check it and try again.", } as const; export type LiveSyncProvisionalMessageKey = keyof typeof liveSyncProvisionalEnglishMessages; diff --git a/src/common/replicatorConfigurationIdentity.ts b/src/common/replicatorConfigurationIdentity.ts index ff6221d8..03f10306 100644 --- a/src/common/replicatorConfigurationIdentity.ts +++ b/src/common/replicatorConfigurationIdentity.ts @@ -1,4 +1,5 @@ import type { RemoteDBSettings } from "@vrtmrz/livesync-commonlib/compat/common/types"; +import { usesEncryptedInternalMetadata } from "@vrtmrz/livesync-commonlib/replication"; type EndpointProjection = readonly [kind: "url" | "invalid-url", value: string]; @@ -42,7 +43,10 @@ function projectHeaders(value: string): readonly (readonly [name: string, value: } function projectRemoteSecurity(settings: RemoteDBSettings) { - return settings.encrypt + return [ + settings.idDerivationVersion, + settings.idDerivationKey, + settings.encrypt ? ([ "encrypted", settings.passphrase, @@ -50,7 +54,8 @@ function projectRemoteSecurity(settings: RemoteDBSettings) { settings.E2EEAlgorithm, settings.permitEmptyPassphrase, ] as const) - : (["plain"] as const); + : (["plain"] as const), + ] as const; } /** @@ -77,6 +82,7 @@ export function getCouchDBReplicatorConfigurationIdentity(settings: RemoteDBSett settings.useRequestAPI, settings.disableRequestURI, projectRemoteSecurity(settings), + usesEncryptedInternalMetadata(settings), settings.enableCompression, ]); } diff --git a/src/common/replicatorConfigurationIdentity.unit.spec.ts b/src/common/replicatorConfigurationIdentity.unit.spec.ts index 719db130..f5a540b4 100644 --- a/src/common/replicatorConfigurationIdentity.unit.spec.ts +++ b/src/common/replicatorConfigurationIdentity.unit.spec.ts @@ -31,6 +31,17 @@ describe("active Replicator configuration identity", () => { }); } + it("replaces a connection when the independent ID key changes", () => { + const first = configuredSettings({ idDerivationVersion: 1, idDerivationKey: "a".repeat(64) }); + const second = { ...first, idDerivationKey: "b".repeat(64) }; + expect(getCouchDBReplicatorConfigurationIdentity(second)).not.toBe( + getCouchDBReplicatorConfigurationIdentity(first) + ); + expect(getObjectStorageReplicatorConfigurationIdentity(second)).not.toBe( + getObjectStorageReplicatorConfigurationIdentity(first) + ); + }); + it.each([ ["couchDB_URI", "https://other.example.test/base"], ["couchDB_DBNAME", "other-vault"], @@ -67,6 +78,22 @@ describe("active Replicator configuration identity", () => { ); }); + it("recreates the CouchDB connection when internal Metadata encryption becomes effective", () => { + const active = configuredSettings({ usePathObfuscation: true, encryptInternalMetadata: false }); + const enabled = { ...active, encryptInternalMetadata: true }; + + expect(getCouchDBReplicatorConfigurationIdentity(enabled)).not.toBe( + getCouchDBReplicatorConfigurationIdentity(active) + ); + const inactive = { ...active, usePathObfuscation: false }; + expect(getCouchDBReplicatorConfigurationIdentity({ ...inactive, encryptInternalMetadata: true })).toBe( + getCouchDBReplicatorConfigurationIdentity(inactive) + ); + expect(getObjectStorageReplicatorConfigurationIdentity(enabled)).toBe( + getObjectStorageReplicatorConfigurationIdentity(active) + ); + }); + it("projects only the active CouchDB authentication mode", () => { const basic = configuredSettings({ useJWT: false, jwtKey: "inactive-a" }); expect(getCouchDBReplicatorConfigurationIdentity({ ...basic, jwtKey: "inactive-b" })).toBe( diff --git a/src/common/replicatorProviders.ts b/src/common/replicatorProviders.ts index 527215e6..845d8c2e 100644 --- a/src/common/replicatorProviders.ts +++ b/src/common/replicatorProviders.ts @@ -3,6 +3,7 @@ import { CAPABILITY_NOT_APPLICABLE, CENTRAL_REMOTE_REPLICATION_READINESS, NO_INTERACTION, + PROVIDER_OWNED_CENTRAL_REMOTE_REPLICATION_READINESS, REPLICATION_PROGRESS_PRESENTATIONS, REMOTE_RESOURCE_KINDS, defineReplicatorProviderDefinitions, @@ -134,7 +135,7 @@ export function createCentralReplicatorProviderDefinitions( [REMOTE_MINIO]: { kind: REMOTE_MINIO, diagnosticName: "Object Storage", - readiness: CENTRAL_REMOTE_REPLICATION_READINESS, + readiness: PROVIDER_OWNED_CENTRAL_REMOTE_REPLICATION_READINESS, isConfigured: (settings) => settings.remoteType === REMOTE_MINIO && !!settings.endpoint?.trim() && !!settings.bucket?.trim(), configurationIdentity: getObjectStorageReplicatorConfigurationIdentity, diff --git a/src/common/replicatorProviders.unit.spec.ts b/src/common/replicatorProviders.unit.spec.ts index fecfc4f3..54a2c441 100644 --- a/src/common/replicatorProviders.unit.spec.ts +++ b/src/common/replicatorProviders.unit.spec.ts @@ -47,6 +47,12 @@ describe("central Replicator provider definitions", () => { .toEqual(["connection", "preferred-tweak", "security-seed", "synchronisation-information"].sort()); }); + it("lets Journal prepare its own fresh Security Seed while CouchDB uses central preparation", () => { + const definitions = createCentralReplicatorProviderDefinitions({} as never); + expect(definitions.get(REMOTE_COUCHDB)?.readiness.centralRemotePreparation).toBe("required"); + expect(definitions.get(REMOTE_MINIO)?.readiness.centralRemotePreparation).toBe("provider-owned"); + }); + it("composes CouchDB and Object Storage policies outside LiveSyncBaseCore", async () => { const host = {} as Parameters[0]; const definitions = createCentralReplicatorProviderDefinitions(host); diff --git a/src/common/replicatorResources.unit.spec.ts b/src/common/replicatorResources.unit.spec.ts index fa0bba62..2215a48a 100644 --- a/src/common/replicatorResources.unit.spec.ts +++ b/src/common/replicatorResources.unit.spec.ts @@ -285,6 +285,46 @@ describe("replicator probe factories", () => { expect(objectReplicator.closeReplication).toHaveBeenCalledOnce(); }); + it("reads the Security Seed once per resource, including concurrent reads, and refreshes for a new resource", async () => { + const settings = createSettings(); + const factory = createCouchDBSecuritySeedResourceFactory({} as never); + const firstResource = await factory(settings); + const firstReplicator = mocks.couchDB[0]; + const firstSeed = new Uint8Array([1]); + firstReplicator.getReplicationPBKDF2Salt.mockResolvedValue(firstSeed); + + const [first, concurrent] = await Promise.all([firstResource.read(), firstResource.read()]); + expect(first).toBe(firstSeed); + expect(concurrent).toBe(firstSeed); + await expect(firstResource.read()).resolves.toBe(firstSeed); + expect(firstReplicator.getReplicationPBKDF2Salt).toHaveBeenCalledOnce(); + expect(firstReplicator.getReplicationPBKDF2Salt).toHaveBeenCalledWith({ ...settings }, true); + await firstResource.dispose(); + + const nextResource = await factory(settings); + const nextReplicator = mocks.couchDB[1]; + const nextSeed = new Uint8Array([2]); + nextReplicator.getReplicationPBKDF2Salt.mockResolvedValue(nextSeed); + await expect(nextResource.read()).resolves.toBe(nextSeed); + expect(nextReplicator.getReplicationPBKDF2Salt).toHaveBeenCalledOnce(); + expect(nextReplicator.getReplicationPBKDF2Salt).toHaveBeenCalledWith({ ...settings }, true); + await nextResource.dispose(); + }); + + it("retries a failed Security Seed read within the same resource", async () => { + const resource = await createCouchDBSecuritySeedResourceFactory({} as never)(createSettings()); + const replicator = mocks.couchDB[0]; + const failure = new Error("connection interrupted"); + const seed = new Uint8Array([1]); + replicator.getReplicationPBKDF2Salt.mockRejectedValueOnce(failure).mockResolvedValueOnce(seed); + + await expect(resource.read()).rejects.toBe(failure); + await expect(resource.read()).resolves.toBe(seed); + await expect(resource.read()).resolves.toBe(seed); + expect(replicator.getReplicationPBKDF2Salt).toHaveBeenCalledTimes(2); + await resource.dispose(); + }); + it("checks synchronisation information through an owned connection and disposes the private Replicator", async () => { const settings = createSettings(); const snapshot = { ...settings }; diff --git a/src/common/replicatorResources/securitySeed.ts b/src/common/replicatorResources/securitySeed.ts index 6abf91b9..e8adafbd 100644 --- a/src/common/replicatorResources/securitySeed.ts +++ b/src/common/replicatorResources/securitySeed.ts @@ -21,8 +21,19 @@ function createSecuritySeedResourceFactory( return (setting) => { const snapshot = snapshotRemoteSettings(setting); const replicator = createReplicator(); + let readPromise: Promise> | undefined; + const read = () => { + if (readPromise) return readPromise; + const pending = Promise.resolve().then(() => replicator.getReplicationPBKDF2Salt(snapshot, true)); + readPromise = pending; + // A failed read must not poison a later retry within the same resource. + void pending.catch(() => { + if (readPromise === pending) readPromise = undefined; + }); + return pending; + }; return Promise.resolve({ - read: () => replicator.getReplicationPBKDF2Salt(snapshot, true), + read, dispose: createReplicatorDisposer(replicator), }); }; diff --git a/src/common/reportTool.ts b/src/common/reportTool.ts index 5942dfa8..5c10cbf6 100644 --- a/src/common/reportTool.ts +++ b/src/common/reportTool.ts @@ -1,3 +1,4 @@ +import { redactTurnSettingsForReport } from "./turnSettingsPrivacy"; import { REMOTE_COUCHDB, REMOTE_MINIO } from "@vrtmrz/livesync-commonlib/compat/common/models/setting.const"; import { DEFAULT_SETTINGS, type ObsidianLiveSyncSettings } from "@vrtmrz/livesync-commonlib/settings"; import { generateCredentialObject } from "@vrtmrz/livesync-commonlib/compat/replication/httplib"; @@ -67,6 +68,7 @@ export async function generateReport(settings: ObsidianLiveSyncSettings, core: L delete pluginConfig[key as keyof ObsidianLiveSyncSettings]; } + redactTurnSettingsForReport(pluginConfig); pluginConfig.couchDB_DBNAME = REDACTED; pluginConfig.couchDB_PASSWORD = REDACTED; const scheme = pluginConfig.couchDB_URI.startsWith("http:") @@ -78,6 +80,8 @@ export async function generateReport(settings: ObsidianLiveSyncSettings, core: L pluginConfig.couchDB_USER = REDACTED; pluginConfig.passphrase = REDACTED; pluginConfig.encryptedPassphrase = REDACTED; + pluginConfig.idDerivationKey = REDACTED; + pluginConfig.encryptedIdDerivationKey = REDACTED; pluginConfig.encryptedCouchDBConnection = REDACTED; pluginConfig.accessKey = REDACTED; pluginConfig.secretKey = REDACTED; diff --git a/src/common/reportTool.unit.spec.ts b/src/common/reportTool.unit.spec.ts new file mode 100644 index 00000000..8436b313 --- /dev/null +++ b/src/common/reportTool.unit.spec.ts @@ -0,0 +1,57 @@ +import { describe, expect, it, vi } from "vitest"; +import { DEFAULT_SETTINGS } from "@vrtmrz/livesync-commonlib/settings"; +import { REMOTE_P2P } from "@vrtmrz/livesync-commonlib/compat/common/types"; +import type { LiveSyncBaseCore } from "@/LiveSyncBaseCore"; +import { generateReport } from "./reportTool"; + +vi.mock("./utils", () => ({ requestToCouchDBWithCredentials: vi.fn() })); +vi.mock("@vrtmrz/livesync-commonlib/compat/common/coreEnvFunctions", () => ({ + compatGlobal: { origin: "test", navigator: { userAgent: "test" } }, +})); + +describe("TURN credentials in diagnostic reports", () => { + it("redacts the derived ID key and its encrypted local wrapper", async () => { + const key = "f3205cc41d24116d8c2484993c9d9a2e667373af338ba02f2ee71199adb82f2e"; + const wrapper = "encrypted-id-key-test-wrapper"; + const settings = { + ...DEFAULT_SETTINGS, + idDerivationVersion: 1 as const, + idDerivationKey: key, + encryptedIdDerivationKey: wrapper, + }; + const core = { services: { vault: { isStorageInsensitive: () => false } } } as unknown as LiveSyncBaseCore; + const report = await generateReport(settings, core); + const text = JSON.stringify(report); + expect(text).not.toContain(key); + expect(text).not.toContain(wrapper); + }); + + it("redacts provider tokens in all profiles and runtime credentials", async () => { + const token = "private+token/with=symbols"; + const provider = { P2P_managedType: "CF", P2P_managedId: "private-key", P2P_managedToken: token }; + const settings = { + ...DEFAULT_SETTINGS, + remoteType: REMOTE_P2P, + ...provider, + P2P_iceServers: [{ urls: "turn:example.test", username: "issued-user", credential: "issued-password" }], + P2P_iceServersExpiresAt: 123456789, + remoteConfigurations: { + inactive: { + id: "inactive", + name: "Inactive TURN", + isEncrypted: false, + uri: `sls+p2p://room?managedType=CF&managedId=private-key&token=${encodeURIComponent(token)}`, + }, + }, + }; + const core = { services: { vault: { isStorageInsensitive: () => false } } } as unknown as LiveSyncBaseCore; + const report = await generateReport(settings, core); + const text = JSON.stringify(report); + expect(text).not.toContain(token); + expect(text).not.toContain(encodeURIComponent(token)); + expect(text).not.toContain("private-key"); + expect(report.pluginConfig.remoteConfigurations.inactive.uri).toBe("sls+p2p://"); + expect(settings.P2P_managedToken).toBe(token); + expect(text).not.toMatch(/issued-user|issued-password|P2P_iceServers/); + }); +}); diff --git a/src/common/turnSettingsPrivacy.ts b/src/common/turnSettingsPrivacy.ts new file mode 100644 index 00000000..6a67ddcb --- /dev/null +++ b/src/common/turnSettingsPrivacy.ts @@ -0,0 +1,63 @@ +import { + hasManagedP2PTurnConfiguration, + type ObsidianLiveSyncSettings, +} from "@vrtmrz/livesync-commonlib/compat/common/types"; +import { pickP2PSyncSettings } from "@vrtmrz/livesync-commonlib/compat/common/utils"; +import { CLOUDFLARE_TURN_TYPE } from "@/integrations/cloudflare/settings"; + +/** Include inactive profiles when deciding whether Markdown would disclose provider settings. */ +export function hasManagedTurnSettings(settings: Partial): boolean { + return ( + hasManagedP2PTurnConfiguration(settings) || + Object.values(settings.remoteConfigurations ?? {}).some(({ uri }) => { + if (!uri.startsWith("sls+p2p://")) return false; + const queryStart = uri.indexOf("?"); + return ( + queryStart >= 0 && new URLSearchParams(uri.slice(queryStart + 1).split("#", 1)[0]).has("managedType") + ); + }) + ); +} + +/** Reports retain a recognised provider label and omit issued credentials. */ +export function redactTurnSettingsForReport(settings: Partial): void { + if (settings.P2P_managedType) { + settings.P2P_managedType = + settings.P2P_managedType === CLOUDFLARE_TURN_TYPE ? CLOUDFLARE_TURN_TYPE : "redacted"; + } + if (settings.P2P_managedId !== undefined) settings.P2P_managedId = "redacted"; + if (settings.P2P_managedToken !== undefined) settings.P2P_managedToken = "redacted"; + delete settings.P2P_iceServers; + delete settings.P2P_iceServersExpiresAt; +} + +/** Managed connection profiles are shared through Setup URIs and QR codes. */ +export function omitManagedTurnProfilesFromMarkdown(settings: Partial): void { + delete settings.P2P_iceServers; + delete settings.P2P_iceServersExpiresAt; + if (!hasManagedTurnSettings(settings)) return; + delete settings.P2P_managedType; + delete settings.P2P_managedId; + delete settings.P2P_managedToken; + delete settings.remoteConfigurations; + delete settings.activeConfigurationId; + delete settings.P2P_ActiveRemoteConfigurationId; +} + +/** Preserve the complete connection when Markdown omits its profile group. */ +export function preserveManagedTurnProfilesOnMarkdownImport( + incoming: Partial, + current: ObsidianLiveSyncSettings, + merged: ObsidianLiveSyncSettings +): void { + if ( + !hasManagedTurnSettings(current) || + incoming.remoteConfigurations !== undefined || + incoming.P2P_managedType !== undefined + ) + return; + merged.remoteConfigurations = structuredClone(current.remoteConfigurations); + merged.activeConfigurationId = current.activeConfigurationId; + merged.P2P_ActiveRemoteConfigurationId = current.P2P_ActiveRemoteConfigurationId; + Object.assign(merged, pickP2PSyncSettings(current)); +} diff --git a/src/common/turnSettingsPrivacy.unit.spec.ts b/src/common/turnSettingsPrivacy.unit.spec.ts new file mode 100644 index 00000000..c0243c79 --- /dev/null +++ b/src/common/turnSettingsPrivacy.unit.spec.ts @@ -0,0 +1,139 @@ +import { describe, expect, it } from "vitest"; +import { + DEFAULT_SETTINGS, + REMOTE_P2P, + type ObsidianLiveSyncSettings, +} from "@vrtmrz/livesync-commonlib/compat/common/types"; +import { + SettingService, + type SettingServiceDependencies, +} from "@vrtmrz/livesync-commonlib/compat/services/base/SettingService"; +import { ServiceContext } from "@vrtmrz/livesync-commonlib/compat/services/base/ServiceBase"; +import { ConnectionStringParser } from "@vrtmrz/livesync-commonlib/compat/common/ConnectionString"; +import { + hasManagedTurnSettings, + omitManagedTurnProfilesFromMarkdown, + preserveManagedTurnProfilesOnMarkdownImport, + redactTurnSettingsForReport, +} from "./turnSettingsPrivacy"; + +class MemorySettingService extends SettingService { + readonly items = new Map(); + saved?: ObsidianLiveSyncSettings; + protected setItem(key: string, value: string) { + this.items.set(key, value); + } + protected getItem(key: string) { + return this.items.get(key) ?? ""; + } + protected deleteItem(key: string) { + this.items.delete(key); + } + protected saveData(settings: ObsidianLiveSyncSettings) { + this.saved = structuredClone(settings); + return Promise.resolve(); + } + protected loadData() { + return Promise.resolve(this.saved); + } +} + +function configuredSettings() { + return { + ...DEFAULT_SETTINGS, + P2P_managedType: "CF", + P2P_managedId: "private-key-id", + P2P_managedToken: "private-token", + remoteConfigurations: { + managed: { + id: "managed", + name: "Managed TURN", + isEncrypted: false, + uri: "sls+p2p://room?managedType=CF&managedId=private-key-id&token=private-token", + }, + }, + activeConfigurationId: "central", + P2P_ActiveRemoteConfigurationId: "managed", + }; +} + +describe("managed TURN settings privacy", () => { + it("preserves the active managed room through Markdown import, save, and reload", async () => { + const current = { + ...configuredSettings(), + remoteType: REMOTE_P2P, + activeConfigurationId: "managed", + P2P_roomID: "local-room", + P2P_relays: "wss://local-relay.example.test", + P2P_passphrase: "local-passphrase", + }; + const originalURI = ConnectionStringParser.serialize({ type: "p2p", settings: current }); + current.remoteConfigurations.managed.uri = originalURI; + const service = new MemorySettingService(new ServiceContext(), { + APIService: { + getSystemVaultName: () => "test-vault", + getAppID: () => "test-app", + addLog: () => undefined, + confirm: { askString: async () => "" }, + } as unknown as SettingServiceDependencies["APIService"], + }); + service.settings = structuredClone(current); + const incoming: Partial = { + P2P_roomID: "imported-room", + P2P_relays: "wss://imported-relay.example.test", + P2P_passphrase: "imported-passphrase", + }; + const merged = { ...structuredClone(DEFAULT_SETTINGS), ...incoming }; + preserveManagedTurnProfilesOnMarkdownImport(incoming, current, merged); + await service.applyExternalSettings(merged, true); + const saved = service.saved!.remoteConfigurations.managed; + const uri = saved.isEncrypted ? await service.decryptConfigurationItem(saved.uri, "*") : saved.uri; + expect(uri).toBe(originalURI); + expect(service.settings.P2P_roomID).toBe("local-room"); + await service.loadSettings(); + expect(service.settings.P2P_roomID).toBe("local-room"); + }); + + it("redacts provider fields and issued credentials, including unknown integrations", () => { + const settings = configuredSettings(); + settings.P2P_managedType = "private-token"; + redactTurnSettingsForReport(settings); + expect([settings.P2P_managedType, settings.P2P_managedId, settings.P2P_managedToken]).toEqual([ + "redacted", + "redacted", + "redacted", + ]); + }); + + it("omits the whole managed profile group from Markdown, including inactive sources", () => { + const settings = configuredSettings(); + settings.P2P_managedType = ""; + expect(hasManagedTurnSettings(settings)).toBe(true); + omitManagedTurnProfilesFromMarkdown(settings); + expect(JSON.stringify(settings)).not.toMatch(/private-token|private-key-id|sls\+p2p/); + expect(settings).not.toHaveProperty("remoteConfigurations"); + expect(settings).not.toHaveProperty("activeConfigurationId"); + expect(settings).not.toHaveProperty("P2P_ActiveRemoteConfigurationId"); + }); + + it("preserves existing profiles and both selections when Markdown omits the group", () => { + const current = configuredSettings(); + const incoming = { ...DEFAULT_SETTINGS }; + delete (incoming as Partial).remoteConfigurations; + delete (incoming as Partial).P2P_managedType; + const merged = { ...DEFAULT_SETTINGS, ...incoming }; + preserveManagedTurnProfilesOnMarkdownImport(incoming, current, merged); + expect(merged.remoteConfigurations).toEqual(current.remoteConfigurations); + expect(merged.remoteConfigurations).not.toBe(current.remoteConfigurations); + expect(merged.P2P_managedToken).toEqual(current.P2P_managedToken); + expect(merged.activeConfigurationId).toBe("central"); + expect(merged.P2P_ActiveRemoteConfigurationId).toBe("managed"); + }); + + it("retains the manual-only Markdown contract", () => { + const settings = { ...DEFAULT_SETTINGS }; + const before = structuredClone(settings); + omitManagedTurnProfilesFromMarkdown(settings); + expect(settings).toEqual(before); + }); +}); diff --git a/src/common/utils.path.unit.spec.ts b/src/common/utils.path.unit.spec.ts new file mode 100644 index 00000000..2f008b86 --- /dev/null +++ b/src/common/utils.path.unit.spec.ts @@ -0,0 +1,64 @@ +import { beforeEach, describe, expect, it, vi } from "vitest"; + +const mocks = vi.hoisted(() => ({ + normalizePath: vi.fn((path: string) => `normalised(${path})`), + path2idBase: vi.fn(async (path: string) => path), + id2pathBase: vi.fn((path: string) => path), + expandFilePathPrefix: vi.fn((path: string): [string, string] => { + if (path.startsWith("i:")) return ["i:", path.substring(2)]; + return ["", path]; + }), +})); + +vi.mock("@/deps.ts", () => ({ + normalizePath: mocks.normalizePath, + Platform: {}, + requestUrl: vi.fn(), +})); + +vi.mock("@vrtmrz/livesync-commonlib/compat/string_and_binary/path", () => ({ + path2id_base: mocks.path2idBase, + id2path_base: mocks.id2pathBase, + expandFilePathPrefix: mocks.expandFilePathPrefix, + isValidFilenameInLinux: vi.fn(), + isValidFilenameInDarwin: vi.fn(), + isValidFilenameInWidows: vi.fn(), + isValidFilenameInAndroid: vi.fn(), + stripAllPrefixes: vi.fn(), +})); + +describe("path ID normalisation", () => { + beforeEach(() => { + vi.clearAllMocks(); + }); + + it.each([ + ["Folder/Note.md", "", "Folder/Note.md"], + ["Folder/Poem: Example.md", "", "Folder/Poem: Example.md"], + ["Folder/Poem: Example: Final Draft.md", "", "Folder/Poem: Example: Final Draft.md"], + ["i:Folder/Poem: Example.md", "i:", "Folder/Poem: Example.md"], + ])("normalises the complete path body for %s", async (filename, prefix, body) => { + const { path2id } = await import("./utils.ts"); + + const result = await path2id(filename as never, false, false); + + expect(mocks.normalizePath).toHaveBeenCalledWith(body); + expect(mocks.path2idBase).toHaveBeenCalledWith(`${prefix}normalised(${body})`, false, false); + expect(result).toBe(`${prefix}normalised(${body})`); + }); + + it.each([ + ["Folder/Note.md", "", "Folder/Note.md"], + ["Folder/Poem: Example.md", "", "Folder/Poem: Example.md"], + ["Folder/Poem: Example: Final Draft.md", "", "Folder/Poem: Example: Final Draft.md"], + ["i:Folder/Poem: Example.md", "i:", "Folder/Poem: Example.md"], + ])("preserves the path namespace while normalising %s", async (filename, prefix, body) => { + mocks.id2pathBase.mockReturnValue(filename); + const { id2path } = await import("./utils.ts"); + + const result = id2path(filename as never); + + expect(mocks.normalizePath).toHaveBeenCalledWith(body); + expect(result).toBe(`${prefix}normalised(${body})`); + }); +}); diff --git a/src/common/utils.ts b/src/common/utils.ts index f9d60799..f4bc621d 100644 --- a/src/common/utils.ts +++ b/src/common/utils.ts @@ -7,6 +7,7 @@ import { isValidFilenameInWidows, isValidFilenameInAndroid, stripAllPrefixes, + expandFilePathPrefix, } from "@vrtmrz/livesync-commonlib/compat/string_and_binary/path"; import { Logger } from "@vrtmrz/livesync-commonlib/compat/common/logger"; @@ -41,22 +42,18 @@ export async function path2id( obfuscatePassphrase: string | false, caseInsensitive: boolean ): Promise { - const temp = filename.split(":"); - const path = temp.pop(); + const [prefix, path] = expandFilePathPrefix(filename); const normalizedPath = normalizePath(path as FilePath); - temp.push(normalizedPath); - const fixedPath = temp.join(":") as FilePathWithPrefix; + const fixedPath = `${prefix}${normalizedPath}` as FilePathWithPrefix; const out = await path2id_base(fixedPath, obfuscatePassphrase, caseInsensitive); return out; } export function id2path(id: DocumentID, entry?: EntryHasPath): FilePathWithPrefix { const filename = id2path_base(id, entry); - const temp = filename.split(":"); - const path = temp.pop(); + const [prefix, path] = expandFilePathPrefix(filename); const normalizedPath = normalizePath(path as FilePath); - temp.push(normalizedPath); - const fixedPath = temp.join(":") as FilePathWithPrefix; + const fixedPath = `${prefix}${normalizedPath}` as FilePathWithPrefix; return fixedPath; } diff --git a/src/features/P2PSync/TurnConfiguration.svelte b/src/features/P2PSync/TurnConfiguration.svelte new file mode 100644 index 00000000..049cd752 --- /dev/null +++ b/src/features/P2PSync/TurnConfiguration.svelte @@ -0,0 +1,69 @@ + + +
+ + {#if managedType === ""} + + + + {:else if managedType === CLOUDFLARE_TURN_TYPE} + + +

{translate("The API token is saved with this profile and included in Setup URI and QR code sharing. Temporary TURN credentials are kept in memory only.")}

+ {/if} + {#if error} +

{translateIfAvailable(error)}

+ {/if} +
+ + diff --git a/src/features/ReviewHarness/reviewHarnessContract.ts b/src/features/ReviewHarness/reviewHarnessContract.ts index 29a76a8c..3925857d 100644 --- a/src/features/ReviewHarness/reviewHarnessContract.ts +++ b/src/features/ReviewHarness/reviewHarnessContract.ts @@ -1,9 +1,6 @@ import type { ObsidianLiveSyncSettings, SettingsMigrationState } from "@vrtmrz/livesync-commonlib/settings"; import type { CompatibilityPause } from "@/common/databaseCompatibility.ts"; -import type { - ReviewHarnessScenarioResult, - ReviewHarnessScenarioStatus, -} from "./reviewHarnessTypes"; +import type { ReviewHarnessScenarioResult, ReviewHarnessScenarioStatus } from "./reviewHarnessTypes"; export type { ReviewHarnessScenarioResult, ReviewHarnessScenarioStatus } from "./reviewHarnessTypes"; @@ -32,6 +29,14 @@ export const REVIEW_HARNESS_SCENARIOS = [ mode: "automatic", access: "dedicated-vault-fixtures", }, + { + id: "id-generation-performance", + title: "ID generation performance", + description: + "Measures legacy and independent IDs with fixed in-memory inputs. Reports time per 1,000 IDs and per ID, key derivation time, and JavaScript heap samples where available. Keep Obsidian in the foreground.", + mode: "automatic", + access: "read-only", + }, ] as const; export const REVIEW_HARNESS_SCENARIO_IDS = REVIEW_HARNESS_SCENARIOS.map(({ id }) => id); @@ -114,7 +119,9 @@ const NEW_VAULT_RECOMMENDATION_KEYS = [ "E2EEAlgorithm", ] as const; -type LifecycleSettingKey = (typeof PRESERVED_SYNC_SETTING_KEYS)[number] | (typeof NEW_VAULT_RECOMMENDATION_KEYS)[number]; +type LifecycleSettingKey = + | (typeof PRESERVED_SYNC_SETTING_KEYS)[number] + | (typeof NEW_VAULT_RECOMMENDATION_KEYS)[number]; type SettingsForLifecycleInspection = Partial>; export function inspectSettingsLifecycle(input: { @@ -130,9 +137,7 @@ export function inspectSettingsLifecycle(input: { }; } - const invalidSyncSettings = PRESERVED_SYNC_SETTING_KEYS.filter( - (key) => typeof input.settings[key] !== "boolean" - ); + const invalidSyncSettings = PRESERVED_SYNC_SETTING_KEYS.filter((key) => typeof input.settings[key] !== "boolean"); if (invalidSyncSettings.length > 0) { return { status: "failed", @@ -205,6 +210,7 @@ export interface ReviewHarnessReportScenario { readonly mode: ReviewHarnessScenarioMode; readonly status: ReviewHarnessScenarioStatus; readonly detail: string; + readonly observations?: readonly string[]; } export interface ReviewHarnessReportInput { @@ -248,13 +254,15 @@ export function formatReviewHarnessReport(input: ReviewHarnessReportInput): stri ); const scenarios = table( ["Scenario", "Mode", "Status", "Detail"], - input.scenarios.map(({ id, title, mode, status, detail }) => [ - `${title} (${id})`, - mode, - status, - detail, - ]) + input.scenarios.map(({ id, title, mode, status, detail }) => [`${title} (${id})`, mode, status, detail]) ); + const observations = input.scenarios + .filter((scenario) => scenario.observations?.length) + .map( + ({ title, observations }) => + `### ${title}\n\n${observations!.map((value) => `- ${tableCell(value)}`).join("\n")}` + ) + .join("\n\n"); return `## Self-hosted LiveSync Review Harness report Generated at \`${tableCell(input.generatedAt)}\`. @@ -267,6 +275,8 @@ ${environment} ${scenarios} +${observations} +
Event transcript diff --git a/src/features/ReviewHarness/reviewHarnessContract.unit.spec.ts b/src/features/ReviewHarness/reviewHarnessContract.unit.spec.ts index a24b53e1..bc33c775 100644 --- a/src/features/ReviewHarness/reviewHarnessContract.unit.spec.ts +++ b/src/features/ReviewHarness/reviewHarnessContract.unit.spec.ts @@ -75,6 +75,7 @@ describe("Review Harness contract", () => { "settings-lifecycle", "compatibility-review", "vault-round-trip", + "id-generation-performance", ]); }); diff --git a/src/features/ReviewHarness/reviewHarnessController.ts b/src/features/ReviewHarness/reviewHarnessController.ts index 7bfc755c..12c0e370 100644 --- a/src/features/ReviewHarness/reviewHarnessController.ts +++ b/src/features/ReviewHarness/reviewHarnessController.ts @@ -21,6 +21,7 @@ export interface ReviewHarnessRuntime { getCompatibilityPause(): CompatibilityPause | undefined; openCompatibilityReview(): Promise; runVaultRoundTrip(): Promise; + runIdBenchmark(): Promise; readContinuation(): string | null; writeContinuation(value: string): void; deleteContinuation(): void; @@ -159,6 +160,8 @@ export class ReviewHarnessController { }); } else if (id === "vault-round-trip") { result = await this.runtime.runVaultRoundTrip(); + } else if (id === "id-generation-performance") { + result = await this.runtime.runIdBenchmark(); } else { const inspection = this.inspectCompatibilityReview(); result = @@ -206,10 +209,7 @@ export class ReviewHarnessController { detail: "The device-local compatibility review remains pending.", observations: inspection.observations, }; - this.record( - "compatibility-review-updated", - this.results["compatibility-review"].status - ); + this.record("compatibility-review-updated", this.results["compatibility-review"].status); } catch (error) { this.setUnexpectedFailure("compatibility-review", error); } finally { @@ -259,6 +259,7 @@ export class ReviewHarnessController { mode, status: this.results[id].status, detail: this.results[id].detail, + observations: this.results[id].observations, })), transcript: this.transcript, }); diff --git a/src/features/ReviewHarness/reviewHarnessController.unit.spec.ts b/src/features/ReviewHarness/reviewHarnessController.unit.spec.ts index 04b07bee..dee43186 100644 --- a/src/features/ReviewHarness/reviewHarnessController.unit.spec.ts +++ b/src/features/ReviewHarness/reviewHarnessController.unit.spec.ts @@ -80,6 +80,11 @@ function createRuntime(): ReviewHarnessRuntime & { detail: "The owned fixture tree was exercised and removed.", observations: [], })), + runIdBenchmark: vi.fn(async () => ({ + status: "passed" as const, + detail: "ID generation measurements completed.", + observations: ["Chunk 256 B: 1000 IDs total=43.00 ms; per ID=0.0430 ms"], + })), readContinuation() { return this.continuation; }, @@ -150,6 +155,60 @@ describe("ReviewHarnessController", () => { expect(runtime.reportError).toHaveBeenCalledOnce(); }); + it("runs ID measurements on request and includes their units in the copied report", async () => { + const runtime = createRuntime(); + const controller = new ReviewHarnessController(runtime); + + await controller.runAutomaticScenarios(); + expect(runtime.runIdBenchmark).not.toHaveBeenCalled(); + + await controller.runScenario("id-generation-performance"); + await controller.copyReport(); + + expect(runtime.runIdBenchmark).toHaveBeenCalledOnce(); + expect(controller.snapshot().results["id-generation-performance"].status).toBe("passed"); + expect(vi.mocked(runtime.copyText).mock.calls[0][0]).toContain("1000 IDs total=43.00 ms; per ID=0.0430 ms"); + expect(runtime.runVaultRoundTrip).not.toHaveBeenCalled(); + expect(runtime.events).toEqual([]); + expect(runtime.continuation).toBeNull(); + }); + + it("excludes an unexpected measurement error from the copied report", async () => { + const runtime = createRuntime(); + runtime.runIdBenchmark = vi.fn().mockRejectedValue(new Error("private measurement error")); + const controller = new ReviewHarnessController(runtime); + + await controller.runScenario("id-generation-performance"); + + expect(controller.snapshot().results["id-generation-performance"].status).toBe("failed"); + expect(controller.createReport()).not.toContain("private measurement error"); + expect(runtime.reportError).toHaveBeenCalledOnce(); + }); + + it("does not overlap an ID measurement with another scenario", async () => { + const runtime = createRuntime(); + let finish!: () => void; + const pending = new Promise((resolve) => { + finish = resolve; + }); + runtime.runIdBenchmark = vi.fn(async () => { + await pending; + return { status: "passed" as const, detail: "Measured", observations: [] }; + }); + const controller = new ReviewHarnessController(runtime); + + const running = controller.runScenario("id-generation-performance"); + await controller.runScenario("id-generation-performance"); + await controller.runScenario("vault-round-trip"); + + expect(runtime.runIdBenchmark).toHaveBeenCalledOnce(); + expect(runtime.runVaultRoundTrip).not.toHaveBeenCalled(); + expect(controller.snapshot().running).toBe(true); + finish(); + await running; + expect(controller.snapshot().running).toBe(false); + }); + it("deletes a one-shot continuation before exposing the resumed guided step", () => { const runtime = createRuntime(); runtime.continuation = JSON.stringify({ @@ -167,9 +226,7 @@ describe("ReviewHarnessController", () => { expect(controller.snapshot().results["compatibility-review"]).toMatchObject({ status: "waiting-for-user", }); - expect(controller.snapshot().resumedRequestId).toBe( - "compatibility-review-2026-07-18T11:59:00.000Z" - ); + expect(controller.snapshot().resumedRequestId).toBe("compatibility-review-2026-07-18T11:59:00.000Z"); }); it("does not copy rejected continuation values into the report", () => { diff --git a/src/features/ReviewHarness/reviewHarnessIdBenchmark.ts b/src/features/ReviewHarness/reviewHarnessIdBenchmark.ts new file mode 100644 index 00000000..f9212724 --- /dev/null +++ b/src/features/ReviewHarness/reviewHarnessIdBenchmark.ts @@ -0,0 +1,109 @@ +import type { ReviewHarnessScenarioResult } from "./reviewHarnessTypes"; + +export interface IdBenchmarkOperations { + deriveKey(): Promise; + chunkId(piece: string, independent: boolean): Promise; + documentId(path: string, independent: boolean): Promise; +} + +type BenchmarkPerformance = Pick & { + readonly memory?: { readonly usedJSHeapSize: number }; +}; + +const ID_COUNT = 1000; +const SAMPLES = 3; +const BATCH_SIZE = 100; +const WARMUP_COUNT = 32; + +function readHeap(clock: BenchmarkPerformance): number | undefined { + try { + const bytes = clock.memory?.usedJSHeapSize; + return typeof bytes === "number" && Number.isFinite(bytes) && bytes >= 0 ? bytes : undefined; + } catch { + return undefined; + } +} + +function summary(samples: readonly number[]): string { + const sorted = [...samples].sort((a, b) => a - b); + return `median=${sorted[1].toFixed(2)} ms; range=${sorted[0].toFixed(2)}–${sorted[2].toFixed(2)} ms`; +} + +export async function runReviewHarnessIdBenchmark( + operations: IdBenchmarkOperations, + clock: BenchmarkPerformance = performance, + yieldControl: () => Promise = () => new Promise((resolve) => window.setTimeout(resolve, 0)) +): Promise { + const before = readHeap(clock); + let highest = before; + const sampleHeap = () => { + const value = readHeap(clock); + if (value !== undefined) highest = Math.max(highest ?? value, value); + return value; + }; + const observations = [ + "Fixed synthetic inputs; 3 samples, alternating legacy/independent order; 32 warm-up IDs per sample. Legacy Chunk algorithm: xxhash64.", + "Compute timings include input construction and awaited ID generation. Initialisation, warm-up, and pauses between batches are excluded. This does not measure a Rebuild or remote transfer.", + ]; + const derivationSamples: number[] = []; + for (let sample = 0; sample < SAMPLES; sample++) { + await yieldControl(); + const started = clock.now(); + await operations.deriveKey(); + derivationSamples.push(clock.now() - started); + sampleHeap(); + } + observations.push(`ID key derivation at save time: ${summary(derivationSamples)} per derivation.`); + + const cases = [ + ...[256, 4096, 32768].map((bytes) => { + const prefix = "r".repeat(bytes - 8); + return { + label: `Chunk IDs, ${bytes} B`, + run: (i: number, independent: boolean) => + operations.chunkId(prefix + i.toString(36).padStart(8, "0"), independent), + }; + }), + { + label: "Obfuscated document IDs", + run: (i: number, independent: boolean) => operations.documentId(`benchmark/path-${i}.md`, independent), + }, + ]; + for (const scenario of cases) { + const samples: [number[], number[]] = [[], []]; + for (let sample = 0; sample < SAMPLES; sample++) { + for (const independent of sample % 2 === 0 ? [false, true] : [true, false]) { + for (let i = 0; i < WARMUP_COUNT; i++) await scenario.run(i, independent); + let elapsed = 0; + for (let batch = 0; batch < ID_COUNT; batch += BATCH_SIZE) { + await yieldControl(); + const started = clock.now(); + for (let i = batch; i < batch + BATCH_SIZE; i++) await scenario.run(i, independent); + elapsed += clock.now() - started; + sampleHeap(); + } + samples[independent ? 1 : 0].push(elapsed); + } + } + for (const [index, values] of samples.entries()) { + const median = [...values].sort((a, b) => a - b)[1]; + observations.push( + `${scenario.label}, ${index === 0 ? "legacy" : "independent"}: ${ID_COUNT} IDs total ${summary(values)}; per ID=${(median / ID_COUNT).toFixed(4)} ms.` + ); + } + } + const after = sampleHeap(); + if (highest === undefined) { + observations.push("JavaScript heap: unavailable on this device."); + } else { + const mib = (bytes: number | undefined) => + bytes === undefined ? "unavailable" : `${(bytes / 1048576).toFixed(2)} MiB`; + observations.push( + `JavaScript heap: before=${mib(before)}; highest sampled=${mib(highest)}; after=${mib(after)}.` + ); + } + observations.push( + "Heap samples are approximate, may include other Obsidian work, and are affected by garbage collection. They are neither total app RAM nor a true peak." + ); + return { status: "passed", detail: "ID generation measurements completed.", observations }; +} diff --git a/src/features/ReviewHarness/reviewHarnessIdBenchmark.unit.spec.ts b/src/features/ReviewHarness/reviewHarnessIdBenchmark.unit.spec.ts new file mode 100644 index 00000000..2e68837a --- /dev/null +++ b/src/features/ReviewHarness/reviewHarnessIdBenchmark.unit.spec.ts @@ -0,0 +1,99 @@ +import { describe, expect, it } from "vitest"; +import { runReviewHarnessIdBenchmark, type IdBenchmarkOperations } from "./reviewHarnessIdBenchmark"; + +function fixture() { + let elapsed = 0; + let derivations = 0; + const chunkCounts = [0, 0]; + const documentCounts = [0, 0]; + const chunkSizes = new Set(); + const operations: IdBenchmarkOperations = { + deriveKey: () => { + derivations++; + elapsed += 42; + return Promise.resolve("private-derived-key"); + }, + chunkId: (piece, independent) => { + chunkCounts[independent ? 1 : 0]++; + chunkSizes.add(piece.length); + elapsed += independent ? 2 : 1; + return Promise.resolve("private-chunk-id"); + }, + documentId: (_path, independent) => { + documentCounts[independent ? 1 : 0]++; + elapsed += independent ? 4 : 3; + return Promise.resolve("private-document-id"); + }, + }; + return { + operations, + now: () => elapsed, + yieldControl: () => { + elapsed += 100; + return Promise.resolve(); + }, + counts: () => ({ derivations, chunkCounts, documentCounts, chunkSizes: [...chunkSizes] }), + }; +} + +describe("Review Harness ID measurements", () => { + it("reports totals and per-ID timings separately, excluding warm-up and cooperative pauses", async () => { + const f = fixture(); + const result = await runReviewHarnessIdBenchmark(f.operations, { now: f.now }, f.yieldControl); + const report = result.observations.join("\n"); + + expect(result.status).toBe("passed"); + expect(report).toContain("1000 IDs total median=1000.00 ms; range=1000.00–1000.00 ms; per ID=1.0000 ms"); + expect(report).toContain("1000 IDs total median=2000.00 ms; range=2000.00–2000.00 ms; per ID=2.0000 ms"); + expect(report).toContain("Obfuscated document IDs, legacy: 1000 IDs total median=3000.00 ms"); + expect(report).toContain("Obfuscated document IDs, independent: 1000 IDs total median=4000.00 ms"); + expect(report).toContain("ID key derivation at save time: median=42.00 ms"); + expect(report).toContain("JavaScript heap: unavailable on this device."); + expect(report).not.toContain("private-"); + expect(f.counts()).toEqual({ + derivations: 3, + chunkCounts: [9288, 9288], + documentCounts: [3096, 3096], + chunkSizes: [256, 4096, 32768], + }); + }); + + it("labels the highest sampled heap separately from total app RAM and allows a lower final sample", async () => { + const f = fixture(); + let reads = 0; + const clock = { + now: f.now, + get memory() { + return { usedJSHeapSize: (reads++ === 0 ? 2 : reads === 2 ? 5 : 1) * 1048576 }; + }, + }; + const result = await runReviewHarnessIdBenchmark(f.operations, clock, f.yieldControl); + + expect(result.observations).toContain( + "JavaScript heap: before=2.00 MiB; highest sampled=5.00 MiB; after=1.00 MiB." + ); + expect(result.observations.join("\n")).toContain("neither total app RAM nor a true peak"); + }); + + it.each([Number.NaN, Number.POSITIVE_INFINITY, -1, "throws"])( + "keeps timings usable when the heap API returns %s", + async (value) => { + const f = fixture(); + const result = await runReviewHarnessIdBenchmark( + f.operations, + { + now: f.now, + get memory() { + if (value === "throws") throw new Error("Heap API unavailable"); + return { usedJSHeapSize: value as number }; + }, + }, + f.yieldControl + ); + + expect(result.status).toBe("passed"); + expect(result.observations).toContain("JavaScript heap: unavailable on this device."); + expect(result.observations.join("\n")).not.toMatch(/NaN|Infinity|private-/u); + } + ); +}); diff --git a/src/features/ReviewHarness/reviewHarnessIdBenchmarkRuntime.ts b/src/features/ReviewHarness/reviewHarnessIdBenchmarkRuntime.ts new file mode 100644 index 00000000..f2ef7641 --- /dev/null +++ b/src/features/ReviewHarness/reviewHarnessIdBenchmarkRuntime.ts @@ -0,0 +1,35 @@ +import { DEFAULT_SETTINGS, deriveIdKey } from "@vrtmrz/livesync-commonlib/settings"; +import { path2id_base } from "@vrtmrz/livesync-commonlib/compat/string_and_binary/path"; +import type { FilePath } from "@vrtmrz/livesync-commonlib/compat/common/types"; +import { HashManager } from "@vrtmrz/livesync-commonlib/hashing"; +import type { IdBenchmarkOperations } from "./reviewHarnessIdBenchmark"; + +const FIXTURE_PASSPHRASE = "Self-hosted LiveSync ID benchmark passphrase"; +const FIXTURE_SOURCE = "Self-hosted LiveSync ID benchmark source"; +const FIXTURE_KEY = "ab".repeat(32); + +export async function createIdBenchmarkOperations(): Promise { + const managers: HashManager[] = []; + for (const independent of [false, true]) { + const settings = Object.freeze({ + ...DEFAULT_SETTINGS, + encrypt: true, + passphrase: FIXTURE_PASSPHRASE, + hashAlg: "xxhash64" as const, + idDerivationVersion: independent ? (1 as const) : (0 as const), + idDerivationKey: independent ? FIXTURE_KEY : "", + }); + // HashManager only reads currentSettings; this fixture has no storage or live service access. + const settingService = { currentSettings: () => settings } as HashManager["options"]["settingService"]; + const manager = new HashManager({ settingService }); + if (!(await manager.initialise())) throw new Error("The benchmark hash manager could not initialise."); + managers.push(manager); + } + return { + deriveKey: () => deriveIdKey(FIXTURE_SOURCE), + chunkId: (piece, independent) => managers[independent ? 1 : 0].computeHash(piece), + // Fixture paths are already normalised; use the same ID calculation as PathService. + documentId: (path, independent) => + path2id_base(path as FilePath, FIXTURE_PASSPHRASE, false, independent ? FIXTURE_KEY : undefined), + }; +} diff --git a/src/features/ReviewHarness/reviewHarnessIdBenchmarkRuntime.unit.spec.ts b/src/features/ReviewHarness/reviewHarnessIdBenchmarkRuntime.unit.spec.ts new file mode 100644 index 00000000..91a3c423 --- /dev/null +++ b/src/features/ReviewHarness/reviewHarnessIdBenchmarkRuntime.unit.spec.ts @@ -0,0 +1,38 @@ +import { describe, expect, it, vi } from "vitest"; +import { DEFAULT_SETTINGS } from "@vrtmrz/livesync-commonlib/settings"; +import { createIdBenchmarkOperations } from "./reviewHarnessIdBenchmarkRuntime"; + +describe("Review Harness benchmark implementation", () => { + it("uses the packaged legacy and independent algorithms with isolated fixed settings", async () => { + const originalDefaults = structuredClone(DEFAULT_SETTINGS); + const fetch = vi.spyOn(globalThis, "fetch").mockRejectedValue(new Error("Network access is forbidden")); + try { + const operations = await createIdBenchmarkOperations(); + const chunk = "r".repeat(256); + const legacy = await operations.chunkId(chunk, false); + const independent = await operations.chunkId(chunk, true); + + expect(legacy).toMatch(/^\+[0-9a-z]{1,13}$/u); + expect(independent).toMatch(/^\+[0-9a-f]{64}$/u); + expect(independent).toBe("+9223e53d99e80c29effee9e95e38ed168d13c14f717054f9e996a1cd0a597000"); + expect(await operations.chunkId(chunk, false)).toBe(legacy); + expect(await operations.chunkId(chunk, true)).toBe(independent); + expect(await operations.chunkId("s".repeat(256), true)).not.toBe(independent); + + const legacyPath = await operations.documentId("benchmark/path-1.md", false); + const independentPath = await operations.documentId("benchmark/path-1.md", true); + expect(legacyPath).toMatch(/^f:[0-9a-f]{64}$/u); + expect(independentPath).toMatch(/^f:[0-9a-f]{64}$/u); + expect(legacyPath).not.toBe(independentPath); + expect(await operations.documentId("benchmark/path-1.md", true)).toBe(independentPath); + + const second = await createIdBenchmarkOperations(); + expect(await second.chunkId(chunk, true)).toBe(independent); + expect(await operations.deriveKey()).toMatch(/^[0-9a-f]{64}$/u); + expect(fetch).not.toHaveBeenCalled(); + expect(DEFAULT_SETTINGS).toEqual(originalDefaults); + } finally { + fetch.mockRestore(); + } + }); +}); diff --git a/src/integrations/cloudflare/settings.ts b/src/integrations/cloudflare/settings.ts new file mode 100644 index 00000000..8d58c50c --- /dev/null +++ b/src/integrations/cloudflare/settings.ts @@ -0,0 +1,49 @@ +/** The provider identifier persisted in a P2P profile for Cloudflare TURN. */ +export const CLOUDFLARE_TURN_TYPE = "CF" as const; + +/** The lifetime requested from Cloudflare for each issued credential set. */ +export const CLOUDFLARE_TURN_CREDENTIAL_TTL_SECONDS = 86_400 as const; + +/** The Cloudflare TURN credential-generation endpoint. */ +export const CLOUDFLARE_TURN_CREDENTIAL_ENDPOINT = "https://rtc.live.cloudflare.com/v1/turn/keys" as const; + +/** A Cloudflare TURN configuration. */ +export interface CloudflareTurnConfiguration { + readonly turnKeyId: string; + readonly apiToken: string; +} + +// TURN Key IDs are inserted into one fixed URL path. Keep the accepted set +// deliberately narrower than URI escaping so a configuration cannot alter +// the request path or add a query string. +const TURN_KEY_ID_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._~-]{0,255}$/; + +// RFC 6750's b64token grammar, including optional trailing padding. This +// also excludes whitespace and control characters from the Authorization +// header without exposing the token in a validation message. +const BEARER_TOKEN_PATTERN = /^[A-Za-z0-9._~+/-]+={0,2}$/; +const MAX_BEARER_TOKEN_LENGTH = 4_096; + +/** + * Returns a safe validation message for a Cloudflare TURN configuration. + * The result never includes the supplied Key ID or API token. + */ +export function validateCloudflareTurnConfiguration(value: CloudflareTurnConfiguration): string | undefined { + const turnKeyId = value.turnKeyId; + if (typeof turnKeyId !== "string" || turnKeyId.length === 0) { + return "Enter a TURN Key ID."; + } + if (!TURN_KEY_ID_PATTERN.test(turnKeyId)) { + return "TURN Key ID contains unsupported characters."; + } + + const apiToken = value.apiToken; + if (typeof apiToken !== "string" || apiToken.length === 0) { + return "Enter a TURN Key API Token."; + } + if (apiToken.length > MAX_BEARER_TOKEN_LENGTH || !BEARER_TOKEN_PATTERN.test(apiToken)) { + return "TURN Key API Token must use Bearer token syntax."; + } + + return undefined; +} diff --git a/src/integrations/cloudflare/turnCredentials.ts b/src/integrations/cloudflare/turnCredentials.ts new file mode 100644 index 00000000..924b64b6 --- /dev/null +++ b/src/integrations/cloudflare/turnCredentials.ts @@ -0,0 +1,364 @@ +import { + CLOUDFLARE_TURN_CREDENTIAL_ENDPOINT, + CLOUDFLARE_TURN_CREDENTIAL_TTL_SECONDS, + type CloudflareTurnConfiguration, + validateCloudflareTurnConfiguration, +} from "./settings"; +import { compatGlobal, type CompatTimeoutHandle } from "@vrtmrz/livesync-commonlib/compat/common/coreEnvFunctions"; + +/** Fetch-compatible function supplied by the host composition. */ +export type CloudflareTurnFetch = (input: string | Request, init?: RequestInit) => Promise; + +export interface CloudflareTurnDependencies { + readonly fetch: CloudflareTurnFetch; + readonly now?: () => number; + readonly requestDeadlineMs?: number; +} + +export const CLOUDFLARE_TURN_REQUEST_DEADLINE_MS = 15_000 as const; +export const CLOUDFLARE_TURN_MAX_RESPONSE_BYTES = 32 * 1024; +export const CLOUDFLARE_TURN_MAX_ICE_SERVER_ENTRIES = 16 as const; +export const CLOUDFLARE_TURN_MAX_ICE_SERVER_URLS = 32 as const; +export const CLOUDFLARE_TURN_MIN_REMAINING_LIFETIME_MS = 30_000 as const; + +type TurnFailureCode = "configuration" | "authentication" | "unavailable" | "invalid-response"; + +const FAILURE_MESSAGES: Record = { + configuration: "The Cloudflare TURN configuration is invalid.", + authentication: "The Cloudflare TURN credential request was not authorised.", + unavailable: "The Cloudflare TURN service is unavailable.", + "invalid-response": "The Cloudflare TURN service returned an invalid response.", +}; + +function credentialFailure(code: TurnFailureCode, retryable: boolean): Error { + return Object.assign(new Error(FAILURE_MESSAGES[code]), { code, retryable }); +} + +function isRecord(value: unknown): value is Record { + return typeof value === "object" && value !== null && !Array.isArray(value); +} + +function abortError(): Error { + try { + return new DOMException("The operation was aborted.", "AbortError"); + } catch { + const error = new Error("The operation was aborted."); + error.name = "AbortError"; + return error; + } +} + +function throwIfAborted(signal: AbortSignal): void { + if (signal.aborted) { + throw abortError(); + } +} + +function isControlCharacter(value: string): boolean { + return Array.from(value).some((character) => { + const code = character.charCodeAt(0); + return code <= 0x1f || code === 0x7f; + }); +} + +function isPort(value: string): boolean { + if (!/^\d{1,5}$/.test(value)) return false; + const port = Number(value); + return port >= 1 && port <= 65_535; +} + +function isHost(value: string): boolean { + return value.length > 0 && /^[A-Za-z0-9._-]+$/.test(value); +} + +/** + * Validates the URL forms accepted by WebRTC's ICE server configuration. + * TURN URLs may carry only the standard transport query parameter; userinfo, + * paths, fragments, and arbitrary query values are not accepted. + */ +export function isSupportedIceServerUrl(value: string): boolean { + if (value.length === 0 || value.length > 2_048 || isControlCharacter(value)) return false; + const schemeMatch = /^(stun|stuns|turn|turns):(.+)$/i.exec(value); + if (!schemeMatch) return false; + + const remainder = schemeMatch[2]; + const queryIndex = remainder.indexOf("?"); + const authority = queryIndex >= 0 ? remainder.slice(0, queryIndex) : remainder; + const query = queryIndex >= 0 ? remainder.slice(queryIndex + 1) : ""; + if (authority.length === 0 || authority.includes("/") || authority.includes("#") || authority.includes("@")) { + return false; + } + if (authority.includes("%")) return false; + + if (authority.startsWith("[")) { + const closingBracket = authority.indexOf("]"); + if (closingBracket < 0) return false; + const host = authority.slice(1, closingBracket); + if (!/^[0-9A-Fa-f:.]+$/.test(host) || !host.includes(":")) return false; + const suffix = authority.slice(closingBracket + 1); + if (suffix !== "" && (!suffix.startsWith(":") || !isPort(suffix.slice(1)))) return false; + } else { + const colonIndex = authority.lastIndexOf(":"); + const host = colonIndex >= 0 ? authority.slice(0, colonIndex) : authority; + if (!isHost(host) || (colonIndex >= 0 && !isPort(authority.slice(colonIndex + 1)))) return false; + // IPv6 literals must use brackets so a colon cannot be interpreted as + // an ambiguous port separator. + if (colonIndex >= 0 && host.includes(":")) return false; + } + + if (query.length === 0) return true; + const queryParts = query.split("&"); + return queryParts.length === 1 && /^transport=(udp|tcp)$/i.test(queryParts[0]); +} + +function isTurnUrl(value: string): boolean { + return /^(turn|turns):/i.test(value); +} + +function isCredential(value: unknown): value is string { + return typeof value === "string" && value.length > 0 && value.length <= 4_096 && !isControlCharacter(value); +} + +function normaliseIceServers(value: unknown): readonly RTCIceServer[] { + if (!isRecord(value) || !Array.isArray(value.iceServers)) { + throw credentialFailure("invalid-response", false); + } + if (value.iceServers.length === 0 || value.iceServers.length > CLOUDFLARE_TURN_MAX_ICE_SERVER_ENTRIES) { + throw credentialFailure("invalid-response", false); + } + + const servers: RTCIceServer[] = []; + let urlCount = 0; + let hasTurnServer = false; + + for (const candidate of value.iceServers) { + if (!isRecord(candidate)) throw credentialFailure("invalid-response", false); + const rawUrls = candidate.urls; + const urls = + typeof rawUrls === "string" + ? [rawUrls] + : Array.isArray(rawUrls) && rawUrls.every((url): url is string => typeof url === "string") + ? [...rawUrls] + : undefined; + if (!urls || urls.length === 0) throw credentialFailure("invalid-response", false); + + urlCount += urls.length; + if (urlCount > CLOUDFLARE_TURN_MAX_ICE_SERVER_URLS || urls.some((url) => !isSupportedIceServerUrl(url))) { + throw credentialFailure("invalid-response", false); + } + + const turnEntry = urls.some(isTurnUrl); + hasTurnServer ||= turnEntry; + const normalised: RTCIceServer = { urls }; + if (turnEntry) { + if (!isCredential(candidate.username) || !isCredential(candidate.credential)) { + throw credentialFailure("invalid-response", false); + } + normalised.username = candidate.username; + normalised.credential = candidate.credential; + } + servers.push(normalised); + } + + if (!hasTurnServer) throw credentialFailure("invalid-response", false); + return Object.freeze(servers); +} + +class BoundedResponseError extends Error { + constructor(readonly kind: "too-large" | "invalid-length" | "read-failed") { + super(kind); + } +} + +async function readResponseBody(response: Response): Promise { + const contentLength = response.headers.get("content-length"); + if (contentLength !== null) { + const declaredLength = Number(contentLength); + if (!Number.isFinite(declaredLength) || declaredLength < 0) { + throw new BoundedResponseError("invalid-length"); + } + if (declaredLength > CLOUDFLARE_TURN_MAX_RESPONSE_BYTES) { + throw new BoundedResponseError("too-large"); + } + } + + if (!response.body) { + try { + const text = await response.text(); + if (new TextEncoder().encode(text).byteLength > CLOUDFLARE_TURN_MAX_RESPONSE_BYTES) { + throw new BoundedResponseError("too-large"); + } + return text; + } catch (error) { + if (error instanceof BoundedResponseError) throw error; + throw new BoundedResponseError("read-failed"); + } + } + + const reader = response.body.getReader(); + const chunks: Uint8Array[] = []; + let totalBytes = 0; + try { + while (true) { + const result = await reader.read(); + if (result.done) break; + totalBytes += result.value.byteLength; + if (totalBytes > CLOUDFLARE_TURN_MAX_RESPONSE_BYTES) { + try { + await reader.cancel(); + } catch { + // The response is already invalid because it exceeded the + // bound; cancellation failure must not change the safe + // classification or expose a host-specific error. + } + throw new BoundedResponseError("too-large"); + } + chunks.push(result.value); + } + } catch (error) { + if (error instanceof BoundedResponseError) throw error; + throw new BoundedResponseError("read-failed"); + } finally { + reader.releaseLock(); + } + + const bytes = new Uint8Array(totalBytes); + let offset = 0; + for (const chunk of chunks) { + bytes.set(chunk, offset); + offset += chunk.byteLength; + } + return new TextDecoder().decode(bytes); +} + +function classifyHttpFailure(status: number): Error { + if (status === 401 || status === 403) { + return credentialFailure("authentication", false); + } + if (status === 408 || status === 429 || status >= 500) { + return credentialFailure("unavailable", true); + } + return credentialFailure("unavailable", false); +} + +function parseResponseBody(body: string): readonly RTCIceServer[] { + let value: unknown; + try { + value = JSON.parse(body) as unknown; + } catch { + throw credentialFailure("invalid-response", false); + } + return normaliseIceServers(value); +} + +/** Acquire one temporary ICE configuration for a new room connection. */ +export async function acquireCloudflareTurnCredentials( + configuration: CloudflareTurnConfiguration, + dependencies: CloudflareTurnDependencies, + signal: AbortSignal +): Promise<{ iceServers: readonly RTCIceServer[]; expiresAt: number }> { + if (validateCloudflareTurnConfiguration(configuration)) throw credentialFailure("configuration", false); + const now = dependencies.now ?? Date.now; + const requestDeadlineMs = dependencies.requestDeadlineMs ?? CLOUDFLARE_TURN_REQUEST_DEADLINE_MS; + throwIfAborted(signal); + const requestStartedAt = now(); + if (!Number.isFinite(requestStartedAt)) { + throw credentialFailure("unavailable", true); + } + + const requestController = new AbortController(); + let cancelledByCaller = false; + let rejectCaller: ((reason?: unknown) => void) | undefined; + const callerAbort = new Promise((_resolve, reject) => { + rejectCaller = reject; + }); + let timedOut = false; + const onAbort = () => { + cancelledByCaller = true; + requestController.abort(); + rejectCaller?.(abortError()); + }; + signal.addEventListener("abort", onAbort, { once: true }); + if (signal.aborted) { + signal.removeEventListener("abort", onAbort); + requestController.abort(); + throw abortError(); + } + let timeoutId: CompatTimeoutHandle | undefined; + const deadline = new Promise((_resolve, reject) => { + timeoutId = compatGlobal.setTimeout(() => { + timedOut = true; + requestController.abort(); + reject(credentialFailure("unavailable", true)); + }, requestDeadlineMs); + }); + + const cleanup = () => { + if (timeoutId !== undefined) compatGlobal.clearTimeout(timeoutId); + signal.removeEventListener("abort", onAbort); + }; + + const endpoint = `${CLOUDFLARE_TURN_CREDENTIAL_ENDPOINT}/${configuration.turnKeyId}/credentials/generate-ice-servers`; + let response: Response; + try { + response = await Promise.race([ + dependencies.fetch(endpoint, { + method: "POST", + headers: { + Authorization: `Bearer ${configuration.apiToken}`, + "Content-Type": "application/json", + }, + body: JSON.stringify({ ttl: CLOUDFLARE_TURN_CREDENTIAL_TTL_SECONDS }), + signal: requestController.signal, + redirect: "error", + credentials: "omit", + cache: "no-store", + }), + callerAbort, + deadline, + ]); + } catch { + cleanup(); + if (cancelledByCaller || signal.aborted) throw abortError(); + if (timedOut) throw credentialFailure("unavailable", true); + throw credentialFailure("unavailable", true); + } + + if (cancelledByCaller || signal.aborted) { + cleanup(); + throw abortError(); + } + if (timedOut || requestController.signal.aborted) { + cleanup(); + throw credentialFailure("unavailable", true); + } + if (response.status !== 201) { + cleanup(); + throw classifyHttpFailure(response.status); + } + + let body: string; + try { + body = await Promise.race([readResponseBody(response), callerAbort, deadline]); + } catch (error) { + cleanup(); + if (cancelledByCaller || signal.aborted) throw abortError(); + if (timedOut) throw credentialFailure("unavailable", true); + if (error instanceof BoundedResponseError && error.kind === "read-failed") { + throw credentialFailure("unavailable", true); + } + throw credentialFailure("invalid-response", false); + } + + try { + throwIfAborted(signal); + const iceServers = parseResponseBody(body); + const expiresAt = requestStartedAt + CLOUDFLARE_TURN_CREDENTIAL_TTL_SECONDS * 1_000; + if (!Number.isFinite(expiresAt) || expiresAt <= now() + CLOUDFLARE_TURN_MIN_REMAINING_LIFETIME_MS) { + throw credentialFailure("invalid-response", false); + } + return { iceServers, expiresAt }; + } finally { + cleanup(); + } +} diff --git a/src/integrations/cloudflare/turnCredentials.unit.spec.ts b/src/integrations/cloudflare/turnCredentials.unit.spec.ts new file mode 100644 index 00000000..365e2a5e --- /dev/null +++ b/src/integrations/cloudflare/turnCredentials.unit.spec.ts @@ -0,0 +1,179 @@ +import { afterEach, describe, expect, it, vi } from "vitest"; +import { + CLOUDFLARE_TURN_MAX_RESPONSE_BYTES, + CLOUDFLARE_TURN_REQUEST_DEADLINE_MS, + acquireCloudflareTurnCredentials, +} from "./turnCredentials"; +import { + CLOUDFLARE_TURN_CREDENTIAL_ENDPOINT, + CLOUDFLARE_TURN_CREDENTIAL_TTL_SECONDS, + validateCloudflareTurnConfiguration, +} from "./settings"; + +const configuration = { + turnKeyId: "key-123", + apiToken: "token_abc-123", +} as const; + +function response(body: unknown, status = 201): Response { + return new Response(JSON.stringify(body), { + status, + headers: { "content-type": "application/json" }, + }); +} + +function validBody() { + return { + iceServers: [ + { + urls: ["turn:relay.example.test:3478?transport=udp", "turns:relay.example.test:5349"], + username: "turn-user", + credential: "turn-password", + }, + { urls: "stun:stun.example.test:3478" }, + ], + }; +} + +afterEach(() => { + vi.useRealTimers(); +}); + +describe("Cloudflare TURN credentials", () => { + it("requests the fixed endpoint with the bearer token and TTL", async () => { + const now = 1_000_000; + let requestUrl: string | Request | undefined; + let requestInit: RequestInit | undefined; + const fetch = vi.fn(async (input: string | Request, init?: RequestInit) => { + requestUrl = input; + requestInit = init; + return response(validBody()); + }); + const dependencies = { fetch, now: () => now }; + + const result = await acquireCloudflareTurnCredentials( + configuration, + dependencies, + new AbortController().signal + ); + + expect(requestUrl).toBe(`${CLOUDFLARE_TURN_CREDENTIAL_ENDPOINT}/key-123/credentials/generate-ice-servers`); + expect(requestInit).toMatchObject({ + method: "POST", + redirect: "error", + credentials: "omit", + cache: "no-store", + body: JSON.stringify({ ttl: CLOUDFLARE_TURN_CREDENTIAL_TTL_SECONDS }), + }); + expect(new Headers(requestInit?.headers).get("authorization")).toBe("Bearer token_abc-123"); + expect(new Headers(requestInit?.headers).get("content-type")).toBe("application/json"); + expect(requestInit?.signal).toBeInstanceOf(AbortSignal); + expect(result.iceServers).toHaveLength(2); + expect(result.expiresAt).toBe(now + CLOUDFLARE_TURN_CREDENTIAL_TTL_SECONDS * 1_000); + }); + + it("rejects malformed, oversized, and STUN-only responses without exposing secrets", async () => { + const cases: Array<{ body: unknown; expectedCode: string }> = [ + { body: { iceServers: [] }, expectedCode: "invalid-response" }, + { body: { iceServers: [{ urls: "turn:relay.example.test:3478" }] }, expectedCode: "invalid-response" }, + { body: { iceServers: [{ urls: "stun:stun.example.test:3478" }] }, expectedCode: "invalid-response" }, + ]; + for (const testCase of cases) { + const dependencies = { + fetch: vi.fn(async () => response(testCase.body)), + now: () => 1_000_000, + }; + const error = await acquireCloudflareTurnCredentials( + configuration, + dependencies, + new AbortController().signal + ).catch((reason: unknown) => reason); + expect(error).toMatchObject({ code: testCase.expectedCode }); + expect(String(error)).not.toContain(configuration.apiToken); + expect(String(error)).not.toContain(configuration.turnKeyId); + } + + const oversized = "x".repeat(CLOUDFLARE_TURN_MAX_RESPONSE_BYTES + 1); + const dependencies = { + fetch: vi.fn(async () => new Response(oversized, { status: 201 })), + now: () => 1_000_000, + }; + const error = await acquireCloudflareTurnCredentials( + configuration, + dependencies, + new AbortController().signal + ).catch((reason: unknown) => reason); + expect(error).toMatchObject({ code: "invalid-response" }); + }); + + it("classifies authentication and transient provider failures", async () => { + const authDependencies = { + fetch: vi.fn(async () => response({}, 401)), + }; + await expect( + acquireCloudflareTurnCredentials(configuration, authDependencies, new AbortController().signal) + ).rejects.toMatchObject({ + code: "authentication", + retryable: false, + }); + + const transientDependencies = { + fetch: vi.fn(async () => response({}, 503)), + }; + await expect( + acquireCloudflareTurnCredentials(configuration, transientDependencies, new AbortController().signal) + ).rejects.toMatchObject({ + code: "unavailable", + retryable: true, + }); + }); + + it("propagates caller cancellation and turns a deadline into an unavailable failure", async () => { + const controller = new AbortController(); + const fetch = vi.fn((_input: string | Request, init?: RequestInit) => { + return new Promise((_resolve, reject) => { + init?.signal?.addEventListener("abort", () => reject(new DOMException("aborted", "AbortError")), { + once: true, + }); + }); + }); + const dependencies = { fetch }; + const cancelled = acquireCloudflareTurnCredentials(configuration, dependencies, controller.signal); + controller.abort(); + await expect(cancelled).rejects.toMatchObject({ name: "AbortError" }); + + vi.useFakeTimers(); + const timedDependencies = { fetch }; + const timed = acquireCloudflareTurnCredentials(configuration, timedDependencies, new AbortController().signal); + const assertion = expect(timed).rejects.toMatchObject({ code: "unavailable", retryable: true }); + await vi.advanceTimersByTimeAsync(CLOUDFLARE_TURN_REQUEST_DEADLINE_MS); + await assertion; + }); + + it("rejects an issuance which has no usable remaining lifetime", async () => { + let now = 1_000_000; + const dependencies = { + fetch: vi.fn(async () => { + now += CLOUDFLARE_TURN_CREDENTIAL_TTL_SECONDS * 1_000; + return response(validBody()); + }), + now: () => now, + }; + await expect( + acquireCloudflareTurnCredentials(configuration, dependencies, new AbortController().signal) + ).rejects.toMatchObject({ + code: "invalid-response", + }); + }); +}); + +describe("Cloudflare TURN input validation", () => { + it("rejects unsafe key IDs and malformed bearer credentials", () => { + expect( + validateCloudflareTurnConfiguration({ turnKeyId: "key/id", apiToken: configuration.apiToken }) + ).toContain("unsupported characters"); + expect(validateCloudflareTurnConfiguration({ ...configuration, apiToken: "token with spaces" })).toContain( + "Bearer token syntax" + ); + }); +}); diff --git a/src/integrations/turnSettings.ts b/src/integrations/turnSettings.ts new file mode 100644 index 00000000..38348253 --- /dev/null +++ b/src/integrations/turnSettings.ts @@ -0,0 +1,14 @@ +import type { P2PConnectionInfo } from "@vrtmrz/livesync-commonlib/compat/common/types"; +import { CLOUDFLARE_TURN_TYPE, validateCloudflareTurnConfiguration } from "./cloudflare/settings"; + +/** Validate provider inputs without requesting credentials. */ +export function validateManagedTurnSettings(settings: Partial): string | undefined { + if (settings.P2P_managedType === undefined || settings.P2P_managedType === "") return undefined; + if (settings.P2P_managedType !== CLOUDFLARE_TURN_TYPE) { + return "The selected TURN configuration is not supported."; + } + return validateCloudflareTurnConfiguration({ + turnKeyId: settings.P2P_managedId ?? "", + apiToken: settings.P2P_managedToken ?? "", + }); +} diff --git a/src/main.ts b/src/main.ts index 549c7941..7ea2f730 100644 --- a/src/main.ts +++ b/src/main.ts @@ -1,3 +1,4 @@ +import { useP2PSettingsPreparation } from "@/serviceFeatures/useP2PSettingsPreparation"; import { getLanguage, Notice, Plugin, type App, type PluginManifest } from "./deps"; import { setGetLanguage } from "@vrtmrz/livesync-commonlib/compat/common/coreEnvFunctions"; setGetLanguage(getLanguage); @@ -182,7 +183,8 @@ export default class ObsidianLiveSyncPlugin extends Plugin { const replicator = useP2PReplicatorFeature( core, (_compatibilityReplicator, p2p) => createInteractiveP2PReplication(p2p), - createOpenRebuildUI(this.app) + createOpenRebuildUI(this.app), + { prepareP2PSettings: useP2PSettingsPreparation(core.services.API.webCompatFetch.bind(core.services.API)) } ); setupManager.registerP2PSetupConnectionProbe(replicator.connectionProbe); useP2PReplicatorCommands(core, replicator); diff --git a/src/modules/coreFeatures/ModuleResolveMismatchedTweaks.ts b/src/modules/coreFeatures/ModuleResolveMismatchedTweaks.ts index 2fe44d06..c5889223 100644 --- a/src/modules/coreFeatures/ModuleResolveMismatchedTweaks.ts +++ b/src/modules/coreFeatures/ModuleResolveMismatchedTweaks.ts @@ -99,6 +99,17 @@ function resolutionSettingsSignature(settings: ObsidianLiveSyncSettings): string } export class ModuleResolvingMismatchedTweaks extends AbstractModule { + private requiresIdConfigurationReview(assessment: TweakAssessment): boolean { + if (!assessment.entries.some(({ key, relation }) => key === "idDerivationVersion" && relation === "different")) { + return false; + } + Logger( + "The document ID configurations differ. Import the correct Setup URI, or configure the matching ID key, before synchronising.", + LOG_LEVEL_NOTICE + ); + return true; + } + private _selectNewerTweakSide(current: TweakValues, preferred: Partial): "REMOTE" | "CURRENT" { Logger(`Modified: ${current.tweakModified} (current) vs ${preferred.tweakModified} (preferred)`); const currentModified = current.tweakModified; @@ -196,6 +207,7 @@ export class ModuleResolvingMismatchedTweaks extends AbstractModule { assessment = assessTweakCompatibility(this.settings, preferred) ): Promise<[TweakValues | boolean, boolean]> { if (assessment.alignment === "matched") return [false, false]; + if (this.requiresIdConfigurationReview(assessment)) return [false, false]; const acceptedSettings = settingsAfterAdoption(assessment, "adoptPreferred"); const autoAcceptSide = await this._shouldAutoAcceptCompatibleLossy(assessment); if (autoAcceptSide === "REMOTE") return [acceptedSettings, false]; @@ -363,6 +375,7 @@ export class ModuleResolvingMismatchedTweaks extends AbstractModule { const trialSignature = JSON.stringify(trialSetting); const currentSignature = resolutionSettingsSignature(this.settings); const assessment = assessTweakCompatibility(trialSetting, preferred); + if (this.requiresIdConfigurationReview(assessment)) return { result: false, requireFetch: false }; if (assessment.alignment === "matched") { this._log("The settings in the remote database are the same as the local database.", LOG_LEVEL_NOTICE); return { result: false, requireFetch: false }; diff --git a/src/modules/coreFeatures/ModuleResolveMismatchedTweaks.unit.spec.ts b/src/modules/coreFeatures/ModuleResolveMismatchedTweaks.unit.spec.ts index 2b641f8a..d0a02bb8 100644 --- a/src/modules/coreFeatures/ModuleResolveMismatchedTweaks.unit.spec.ts +++ b/src/modules/coreFeatures/ModuleResolveMismatchedTweaks.unit.spec.ts @@ -7,7 +7,7 @@ import { type TweakValues, } from "@vrtmrz/livesync-commonlib/compat/common/types"; import { extractObject } from "octagonal-wheels/object"; -import { assessTweakCompatibility } from "@vrtmrz/livesync-commonlib/settings"; +import { assessTweakCompatibility, configuredIdKey } from "@vrtmrz/livesync-commonlib/settings"; import { ModuleResolvingMismatchedTweaks } from "./ModuleResolveMismatchedTweaks"; import { setLang } from "@/common/translation"; import { @@ -74,6 +74,68 @@ function createModule(settingsOverride: Partial = {}) { } describe("ModuleResolvingMismatchedTweaks", () => { + it.each([0, 1] as const)( + "keeps ID configuration %s when automatically aligning Chunk settings", + async (idDerivationVersion) => { + const idDerivationKey = idDerivationVersion === 1 ? "ab".repeat(32) : ""; + const { module, core, askSelectStringDialogue } = createModule({ + encrypt: true, + usePathObfuscation: false, + idDerivationVersion, + idDerivationKey, + autoAcceptCompatibleTweak: true, + hashAlg: "xxhash64", + tweakModified: 1, + }); + const preferred: TweakValues = { + ...extractObject(TweakValuesTemplate, core.settings), + idDerivationVersion: idDerivationVersion === 1 ? 0 : 1, + hashAlg: "xxhash32", + tweakModified: 2, + }; + core._services.tweakValue = { + checkAndAskResolvingMismatched: module._checkAndAskResolvingMismatchedTweaks.bind(module), + }; + core._services.setting.saveSettingData.mockImplementation(async () => { + configuredIdKey(core.settings); + }); + + await expect(module._askResolvingMismatchedTweaks(preferred, async () => true)).resolves.toBe("CHECKAGAIN"); + + expect(core.settings).toMatchObject({ idDerivationVersion, idDerivationKey, hashAlg: "xxhash32" }); + expect(askSelectStringDialogue).not.toHaveBeenCalled(); + } + ); + + it.each(["active", "trial"] as const)( + "withholds ordinary tweak adoption for different document ID modes (%s)", + async (route) => { + const { module, core, askSelectStringDialogue } = createModule({ + encrypt: true, + usePathObfuscation: true, + idDerivationVersion: 0, + idDerivationKey: "", + }); + const preferred: TweakValues = { + ...extractObject(TweakValuesTemplate, core.settings), + idDerivationVersion: 1, + }; + + if (route === "active") { + await expect(module._checkAndAskResolvingMismatchedTweaks(preferred)).resolves.toEqual([false, false]); + } else { + await expect(module._askUseRemoteConfiguration(core.settings, preferred)).resolves.toEqual({ + result: false, + requireFetch: false, + }); + } + + expect(askSelectStringDialogue).not.toHaveBeenCalled(); + expect(core._services.setting.saveSettingData).not.toHaveBeenCalled(); + expect(core.settings).toMatchObject({ idDerivationVersion: 0, idDerivationKey: "" }); + } + ); + it("compatibility: offers ordinary application for a missing legacy filename-case setting", async () => { const { module, askSelectStringDialogue } = createModule({ autoAcceptCompatibleTweak: false, diff --git a/src/modules/features/ModuleLog.ts b/src/modules/features/ModuleLog.ts index 14bfd36c..99170ec1 100644 --- a/src/modules/features/ModuleLog.ts +++ b/src/modules/features/ModuleLog.ts @@ -10,7 +10,7 @@ import { import { scheduleTask } from "octagonal-wheels/concurrency/task"; import { fireAndForget, isDirty, throttle } from "@vrtmrz/livesync-commonlib/compat/common/utils"; import { - collectingChunks, + chunkFetchCounts, pluginScanningCount, hiddenFilesEventCount, hiddenFilesProcessingCount, @@ -36,7 +36,11 @@ import { formatRemoteActivityStatusLabel, getTrackedRequestCount, } from "./RemoteActivityStatus.ts"; -import { createMinimumVisibleActivityCount, createPaddedCounterLabel } from "./StatusBarDisplay.ts"; +import { + createChunkFetchCounterLabel, + createMinimumVisibleActivityCount, + createPaddedCounterLabel, +} from "./StatusBarDisplay.ts"; import type { LiveSyncCore } from "@/main.ts"; import { LiveSyncError } from "@vrtmrz/livesync-commonlib/compat/common/LSError"; import { isValidPath } from "@/common/utils.ts"; @@ -140,7 +144,7 @@ export class ModuleLog extends AbstractObsidianModule { const labelStorageCount = registerDisplay( createPaddedCounterLabel(this.services.replication.storageApplyingCount, `πŸ’Ύ`) ); - const labelChunkCount = registerDisplay(createPaddedCounterLabel(collectingChunks, `🧩`)); + const labelChunkCount = registerDisplay(createChunkFetchCounterLabel(chunkFetchCounts)); const labelPluginScanCount = registerDisplay(createPaddedCounterLabel(pluginScanningCount, `πŸ”Œ`)); const labelConflictProcessCount = registerDisplay( createPaddedCounterLabel(this.services.conflict.conflictProcessQueueCount, `πŸ”©`) diff --git a/src/modules/features/ModuleObsidianSettingAsMarkdown.ts b/src/modules/features/ModuleObsidianSettingAsMarkdown.ts index cbbdab3e..bbbced4c 100644 --- a/src/modules/features/ModuleObsidianSettingAsMarkdown.ts +++ b/src/modules/features/ModuleObsidianSettingAsMarkdown.ts @@ -1,3 +1,8 @@ +import { + hasManagedTurnSettings, + omitManagedTurnProfilesFromMarkdown, + preserveManagedTurnProfilesOnMarkdownImport, +} from "@/common/turnSettingsPrivacy"; // import { PouchDB } from "../../lib/src/pouchdb/pouchdb-browser"; import { isObjectDifferent } from "octagonal-wheels/object"; import { EVENT_SETTING_SAVED, eventHub } from "@/common/events"; @@ -129,11 +134,14 @@ export class ModuleObsidianSettingsAsMarkdown extends AbstractModule { let settingToApply = { ...DEFAULT_SETTINGS } as ObsidianLiveSyncSettings; settingToApply = { ...settingToApply, ...newSetting }; + preserveManagedTurnProfilesOnMarkdownImport(newSetting, this.settings, settingToApply); if (!settingToApply?.writeCredentialsForSettingSync) { //New setting does not contains credentials. settingToApply.couchDB_USER = this.settings.couchDB_USER; settingToApply.couchDB_PASSWORD = this.settings.couchDB_PASSWORD; settingToApply.passphrase = this.settings.passphrase; + settingToApply.idDerivationVersion = this.settings.idDerivationVersion; + settingToApply.idDerivationKey = this.settings.idDerivationKey; } const oldSetting = this.generateSettingForMarkdown( this.settings, @@ -197,22 +205,31 @@ export class ModuleObsidianSettingsAsMarkdown extends AbstractModule { const saveData = { ...(settings ? settings : this.settings) } as Partial; delete saveData.encryptedCouchDBConnection; delete saveData.encryptedPassphrase; + delete saveData.encryptedIdDerivationKey; delete saveData.additionalSuffixOfDatabaseName; if (!saveData.writeCredentialsForSettingSync && !keepCredential) { delete saveData.couchDB_USER; delete saveData.couchDB_PASSWORD; delete saveData.passphrase; + delete saveData.idDerivationKey; delete saveData.jwtKey; delete saveData.jwtKid; delete saveData.jwtSub; delete saveData.couchDB_CustomHeaders; delete saveData.bucketCustomHeaders; } + omitManagedTurnProfilesFromMarkdown(saveData); return saveData; } async saveSettingToMarkdown(filename: string) { const saveData = this.generateSettingForMarkdown(); + if (hasManagedTurnSettings(this.settings)) { + this._log( + "Share TURN provider credentials through an encrypted Setup URI. Connection profiles are omitted from Markdown settings.", + LOG_LEVEL_INFO + ); + } const file = await this.core.storageAccess.isExists(filename); if (!file) { diff --git a/src/modules/features/SettingDialogue/ObsidianLiveSyncSettingTab.ts b/src/modules/features/SettingDialogue/ObsidianLiveSyncSettingTab.ts index 032caf76..5bf463d1 100644 --- a/src/modules/features/SettingDialogue/ObsidianLiveSyncSettingTab.ts +++ b/src/modules/features/SettingDialogue/ObsidianLiveSyncSettingTab.ts @@ -569,6 +569,7 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab { } } + // Internal Metadata encryption affects future Metadata writes and is not a rebuild requirement. isNeedRebuildLocal() { return this.isSomeDirty([ "useIndexedDBAdapter", diff --git a/src/modules/features/SettingDialogue/PaneRemoteConfig.ts b/src/modules/features/SettingDialogue/PaneRemoteConfig.ts index 260ec3c0..61310e4d 100644 --- a/src/modules/features/SettingDialogue/PaneRemoteConfig.ts +++ b/src/modules/features/SettingDialogue/PaneRemoteConfig.ts @@ -47,6 +47,12 @@ function getSettingsFromEditingSettings(editingSettings: AllSettings): ObsidianL } return workObj; } + +function syncIdDerivationSettings(target: Partial, source: ObsidianLiveSyncSettings): void { + target.idDerivationVersion = source.idDerivationVersion; + target.idDerivationKey = source.idDerivationKey; +} + function createRemoteConfigurationId(): string { return `remote-${Date.now().toString(36)}-${Math.random().toString(36).slice(2, 8)}`; } @@ -116,7 +122,28 @@ export function paneRemoteConfig( .onClick(async () => { const setupManager = this.core.getModule(SetupManager); const originalSettings = getSettingsFromEditingSettings(this.editingSettings); - await setupManager.onlyE2EEConfiguration(UserMode.Update, originalSettings); + const originalIdDerivationVersion = this.core.settings.idDerivationVersion; + const originalIdDerivationKey = this.core.settings.idDerivationKey; + const applied = await setupManager.onlyE2EEConfiguration(UserMode.Update, originalSettings); + if (applied) { + this.editingSettings.encryptInternalMetadata = + this.core.settings.encryptInternalMetadata; + if (this.initialSettings) { + this.initialSettings.encryptInternalMetadata = + this.core.settings.encryptInternalMetadata; + } + this.requestUpdate(); + } + if ( + this.core.settings.idDerivationVersion !== originalIdDerivationVersion || + this.core.settings.idDerivationKey !== originalIdDerivationKey + ) { + syncIdDerivationSettings(this.editingSettings, this.core.settings); + if (this.initialSettings) { + syncIdDerivationSettings(this.initialSettings, this.core.settings); + } + this.requestUpdate(); + } updateE2EESummary(); }) .setButtonText("Configure") @@ -155,9 +182,11 @@ export function paneRemoteConfig( const currentConfigs = cloneRemoteConfigurations(this.core.settings.remoteConfigurations); this.editingSettings.remoteConfigurations = currentConfigs; this.editingSettings.activeConfigurationId = this.core.settings.activeConfigurationId; + syncIdDerivationSettings(this.editingSettings, this.core.settings); if (this.initialSettings) { this.initialSettings.remoteConfigurations = cloneRemoteConfigurations(currentConfigs); this.initialSettings.activeConfigurationId = this.core.settings.activeConfigurationId; + syncIdDerivationSettings(this.initialSettings, this.core.settings); } }; const persistRemoteConfigurations = async (synchroniseActiveRemote: boolean = false) => { @@ -243,7 +272,10 @@ export function paneRemoteConfig( ...DEFAULT_SETTINGS, encrypt: this.editingSettings.encrypt, usePathObfuscation: this.editingSettings.usePathObfuscation, + encryptInternalMetadata: this.editingSettings.encryptInternalMetadata, passphrase: this.editingSettings.passphrase, + idDerivationVersion: this.editingSettings.idDerivationVersion, + idDerivationKey: this.editingSettings.idDerivationKey, configPassphraseStore: this.editingSettings.configPassphraseStore, }); const addRemoteConfiguration = async () => { diff --git a/src/modules/features/SettingDialogue/PaneRemoteConfig.unit.spec.ts b/src/modules/features/SettingDialogue/PaneRemoteConfig.unit.spec.ts index d61a998a..4d83048f 100644 --- a/src/modules/features/SettingDialogue/PaneRemoteConfig.unit.spec.ts +++ b/src/modules/features/SettingDialogue/PaneRemoteConfig.unit.spec.ts @@ -2,6 +2,7 @@ import { afterEach, describe, expect, it, vi } from "vitest"; const runtime = vi.hoisted(() => ({ buttonClasses: [] as string[], + clickHandlers: [] as Array<() => Promise | void>, panels: [] as Array<{ destroy: ReturnType }>, settingClasses: [] as string[], })); @@ -51,7 +52,8 @@ vi.mock("./LiveSyncSetting.ts", () => ({ setDestructive() { return this; }, - onClick() { + onClick(callback: () => Promise | void) { + runtime.clickHandlers.push(callback); return this; }, setButtonText() { @@ -97,6 +99,7 @@ vi.mock("@vrtmrz/livesync-commonlib/compat/common/ConnectionString", () => ({ }, })); vi.mock("@/modules/features/SetupWizard/dialogs/SetupRemote.svelte", () => ({ default: {} })); +vi.mock("@/modules/features/SetupWizard/dialogs/SetupRemoteE2EE.svelte", () => ({ default: {} })); vi.mock("@/modules/features/SetupWizard/dialogs/SetupRemoteCouchDB.svelte", () => ({ default: {} })); vi.mock("@/modules/features/SetupWizard/dialogs/SetupRemoteBucket.svelte", () => ({ default: {} })); vi.mock("@/modules/features/SetupWizard/dialogs/SetupRemoteP2P.svelte", () => ({ default: {} })); @@ -114,6 +117,7 @@ function createPanelElement(): HTMLElement { afterEach(() => { runtime.buttonClasses.length = 0; + runtime.clickHandlers.length = 0; runtime.panels.length = 0; runtime.settingClasses.length = 0; vi.clearAllMocks(); @@ -148,4 +152,93 @@ describe("paneRemoteConfig", () => { expect(runtime.panels[0].destroy).toHaveBeenCalledOnce(); }); + + it("applies an internal Metadata preference change without scheduling setup initialisation", async () => { + const originalSettings = { + encrypt: true, + passphrase: "passphrase", + E2EEAlgorithm: "v2", + usePathObfuscation: true, + encryptInternalMetadata: false, + remoteConfigurations: {}, + }; + const setupManager = { + onlyE2EEConfiguration: vi.fn(async () => { + host.core.settings.encryptInternalMetadata = true; + return true; + }), + }; + const host = { + editingSettings: { ...originalSettings }, + initialSettings: { ...originalSettings }, + core: { + settings: { ...originalSettings }, + getModule: vi.fn(() => setupManager), + }, + lifetimeComponent: { register: vi.fn() }, + requestUpdate: vi.fn(), + }; + const addPanel = vi.fn((_parent: HTMLElement, heading: string) => ({ + then(callback: (paneEl: HTMLElement) => void) { + if (heading === "E2EE Configuration") { + callback(createPanelElement()); + } + }, + })); + + paneRemoteConfig.call(host as never, {} as HTMLElement, { addPanel } as never); + await runtime.clickHandlers[0](); + + expect(setupManager.onlyE2EEConfiguration).toHaveBeenCalledOnce(); + expect(host.editingSettings.encryptInternalMetadata).toBe(true); + expect(host.initialSettings.encryptInternalMetadata).toBe(true); + expect(host.requestUpdate).toHaveBeenCalledOnce(); + }); + + it("copies applied ID derivation settings into both dialogue buffers", async () => { + const nextIdKey = "ab".repeat(32); + const originalSettings = { + encrypt: true, + passphrase: "passphrase", + E2EEAlgorithm: "v2", + usePathObfuscation: true, + encryptInternalMetadata: false, + idDerivationVersion: 0, + idDerivationKey: "", + remoteConfigurations: {}, + }; + const setupManager = { + onlyE2EEConfiguration: vi.fn(() => { + host.core.settings.idDerivationVersion = 1; + host.core.settings.idDerivationKey = nextIdKey; + return Promise.resolve(false); + }), + }; + const host = { + editingSettings: { ...originalSettings }, + initialSettings: { ...originalSettings }, + core: { + settings: { ...originalSettings }, + getModule: vi.fn(() => setupManager), + }, + lifetimeComponent: { register: vi.fn() }, + requestUpdate: vi.fn(), + }; + const addPanel = vi.fn((_parent: HTMLElement, heading: string) => ({ + then(callback: (paneEl: HTMLElement) => void) { + if (heading === "E2EE Configuration") { + callback(createPanelElement()); + } + }, + })); + + paneRemoteConfig.call(host as never, {} as HTMLElement, { addPanel } as never); + await runtime.clickHandlers[0](); + + expect(host.editingSettings.idDerivationVersion).toBe(1); + expect(host.editingSettings.idDerivationKey).toBe(nextIdKey); + expect(host.initialSettings.idDerivationVersion).toBe(1); + expect(host.initialSettings.idDerivationKey).toBe(nextIdKey); + expect(host.requestUpdate).toHaveBeenCalledOnce(); + }); }); diff --git a/src/modules/features/SettingDialogue/settingUtils.ts b/src/modules/features/SettingDialogue/settingUtils.ts index 8fc43895..debcab56 100644 --- a/src/modules/features/SettingDialogue/settingUtils.ts +++ b/src/modules/features/SettingDialogue/settingUtils.ts @@ -68,6 +68,7 @@ export function getE2EEConfigSummary(setting: ObsidianLiveSyncSettings, showAdva export function getSummaryFromPartialSettings(setting: Partial, showAdvanced = false) { const outputSummary: Record = {}; for (const key of Object.keys(setting) as (keyof ObsidianLiveSyncSettings)[]) { + if (key === "idDerivationKey" || key === "encryptedIdDerivationKey") continue; const config = getConfig(key as AllSettingItemKey); if (!config) continue; if (config.isAdvanced && !showAdvanced) continue; diff --git a/src/modules/features/SetupManager.ts b/src/modules/features/SetupManager.ts index acc5e87a..d4adff20 100644 --- a/src/modules/features/SetupManager.ts +++ b/src/modules/features/SetupManager.ts @@ -1,6 +1,5 @@ import { type BucketSyncSetting, - type EncryptionSettings, type ObsidianLiveSyncSettings, type P2PSyncSetting, LOG_LEVEL_NOTICE, @@ -36,6 +35,7 @@ import type { SetupRemoteCouchDBResultType, SetupRemoteCouchDBInitialData, SetupRemoteE2EEResultType, + SetupRemoteE2EEInitialData, SetupRemoteP2PInitialData, SetupRemoteP2PResultType, SetupRemoteResultType, @@ -58,6 +58,20 @@ function copySettingsForRemoteProfileUpdate(settings: ObsidianLiveSyncSettings): }; } +function normaliseImportedIdDerivationSettings(settings: ObsidianLiveSyncSettings): ObsidianLiveSyncSettings { + // Setup URIs are complete imports even when their encoder omitted default-valued fields. + // Fill each missing half so a receiving device cannot supply the unrelated saved key. + return { + ...settings, + idDerivationVersion: Object.prototype.hasOwnProperty.call(settings, "idDerivationVersion") + ? settings.idDerivationVersion + : 0, + idDerivationKey: Object.prototype.hasOwnProperty.call(settings, "idDerivationKey") + ? settings.idDerivationKey + : "", + }; +} + /** * User modes for onboarding and setup */ @@ -219,7 +233,7 @@ export class SetupManager extends AbstractModule { return false; } this._log("Setup URI dialog closed.", LOG_LEVEL_VERBOSE); - return await this.onConfirmApplySettingsFromWizard(newSetting, userMode); + return await this.onConfirmApplySettingsFromWizard(normaliseImportedIdDerivationSettings(newSetting), userMode); } /** @@ -328,14 +342,44 @@ export class SetupManager extends AbstractModule { * @returns */ async onlyE2EEConfiguration(userMode: UserMode, currentSetting: ObsidianLiveSyncSettings): Promise { - const e2eeConf = await this.dialogManager.openWithExplicitCancel( + const e2eeConf = await this.dialogManager.openWithExplicitCancel< + SetupRemoteE2EEResultType, + SetupRemoteE2EEInitialData + >( SetupRemoteE2EE, - currentSetting + { settings: currentSetting, newVault: userMode === UserMode.NewUser } ); if (e2eeConf === "cancelled") { this._log("E2EE configuration cancelled.", LOG_LEVEL_NOTICE); return false; } + const onlyInternalMetadataPreferenceChanged = + currentSetting.encryptInternalMetadata !== e2eeConf.encryptInternalMetadata && + currentSetting.encrypt === e2eeConf.encrypt && + currentSetting.passphrase === e2eeConf.passphrase && + currentSetting.E2EEAlgorithm === e2eeConf.E2EEAlgorithm && + currentSetting.usePathObfuscation === e2eeConf.usePathObfuscation && + currentSetting.idDerivationVersion === e2eeConf.idDerivationVersion && + currentSetting.idDerivationKey === e2eeConf.idDerivationKey; + if (userMode === UserMode.Update && onlyInternalMetadataPreferenceChanged) { + if (e2eeConf.encryptInternalMetadata && currentSetting.remoteType === REMOTE_COUCHDB) { + const proceed = "Enable without rebuilding β€” update every other device first"; + const choice = await this.core.confirm.askSelectStringDialogue( + "A manual remote Rebuild is strongly recommended to protect existing file properties. " + + "Before continuing without rebuilding, update every other synchronising device to a version " + + "which supports this option, including devices currently running LiveSync. " + + "Existing properties remain unchanged until they are rewritten or rebuilt.", + [proceed, "Cancel"], + { title: "Encrypt internal file Properties", defaultAction: "Cancel" } + ); + if (choice !== proceed) return false; + } + await this.services.setting.applyPartial( + { encryptInternalMetadata: e2eeConf.encryptInternalMetadata }, + true + ); + return true; + } const newSetting = { ...currentSetting, ...e2eeConf, @@ -350,9 +394,12 @@ export class SetupManager extends AbstractModule { * @returns */ async onConfigureManually(originalSetting: ObsidianLiveSyncSettings, userMode: UserMode): Promise { - const e2eeConf = await this.dialogManager.openWithExplicitCancel( + const e2eeConf = await this.dialogManager.openWithExplicitCancel< + SetupRemoteE2EEResultType, + SetupRemoteE2EEInitialData + >( SetupRemoteE2EE, - originalSetting + { settings: originalSetting, newVault: userMode === UserMode.NewUser } ); if (e2eeConf === "cancelled") { this._log("Manual configuration cancelled.", LOG_LEVEL_NOTICE); @@ -496,7 +543,13 @@ export class SetupManager extends AbstractModule { * @returns Promise that resolves to true if settings applied successfully, false otherwise */ async decodeQR(qr: string) { - const newSettings = decodeSettingsFromQRCodeData(qr); + let newSettings: ObsidianLiveSyncSettings; + try { + newSettings = normaliseImportedIdDerivationSettings(decodeSettingsFromQRCodeData(qr)); + } catch { + this._log("The QR configuration could not be decoded or contains unsupported settings.", LOG_LEVEL_NOTICE); + return false; + } return await this.onConfirmApplySettingsFromWizard(newSettings, UserMode.Unknown); } diff --git a/src/modules/features/SetupManager.unit.spec.ts b/src/modules/features/SetupManager.unit.spec.ts index 41d9ecaf..c8dfe4ec 100644 --- a/src/modules/features/SetupManager.unit.spec.ts +++ b/src/modules/features/SetupManager.unit.spec.ts @@ -193,6 +193,58 @@ describe("SetupManager", () => { expect(setting.currentSettings().activeConfigurationId).toBe("legacy-couchdb"); }); + it("compatibility: treats omitted ID derivation fields in a Setup URI as legacy defaults", async () => { + const { manager, setting, dialogManager } = createSetupManager(); + const savedKey = "12".repeat(32); + setting.settings = { + ...createLegacyRemoteSetting(), + isConfigured: true, + idDerivationVersion: 1, + idDerivationKey: savedKey, + }; + const imported = { + ...createLegacyRemoteSetting(), + isConfigured: true, + } as Partial; + delete imported.idDerivationVersion; + delete imported.idDerivationKey; + vi.spyOn(setting, "adjustSettings").mockImplementation((settings) => Promise.resolve(settings)); + dialogManager.openWithExplicitCancel.mockResolvedValueOnce(imported).mockResolvedValueOnce("cancelled"); + + await manager.onUseSetupURI(UserMode.Unknown, "mock-config://legacy-settings"); + + const mergedSettings = vi.mocked(setting.adjustSettings).mock.calls[0][0]; + expect(mergedSettings.idDerivationVersion).toBe(0); + expect(mergedSettings.idDerivationKey).toBe(""); + expect(setting.currentSettings().idDerivationKey).toBe(savedKey); + }); + + it("does not inherit the missing half of a partially present Setup URI ID configuration", async () => { + const { manager, setting, dialogManager } = createSetupManager(); + const savedKey = "34".repeat(32); + setting.settings = { + ...createLegacyRemoteSetting(), + isConfigured: true, + idDerivationVersion: 1, + idDerivationKey: savedKey, + }; + const imported = { + ...createLegacyRemoteSetting(), + isConfigured: true, + idDerivationVersion: 1, + } as Partial; + delete imported.idDerivationKey; + vi.spyOn(setting, "adjustSettings").mockImplementation((settings) => Promise.resolve(settings)); + dialogManager.openWithExplicitCancel.mockResolvedValueOnce(imported).mockResolvedValueOnce("cancelled"); + + await manager.onUseSetupURI(UserMode.Unknown, "mock-config://partial-settings"); + + const mergedSettings = vi.mocked(setting.adjustSettings).mock.calls[0][0]; + expect(mergedSettings.idDerivationVersion).toBe(1); + expect(mergedSettings.idDerivationKey).toBe(""); + expect(setting.currentSettings().idDerivationKey).toBe(savedKey); + }); + it("compatibility: normalises imported flat remote settings from QR data before applying", async () => { const { manager, setting, dialogManager } = createSetupManager(); vi.mocked(decodeSettingsFromQRCodeData).mockReturnValue(createLegacyRemoteSetting()); @@ -208,6 +260,79 @@ describe("SetupManager", () => { expect(setting.currentSettings().activeConfigurationId).toBe("legacy-couchdb"); }); + it("compatibility: applies legacy defaults when QR data omits ID derivation fields", async () => { + const { manager, setting, dialogManager } = createSetupManager(); + const savedKey = "56".repeat(32); + setting.settings = { + ...createLegacyRemoteSetting(), + isConfigured: true, + idDerivationVersion: 1, + idDerivationKey: savedKey, + }; + const imported = { ...createLegacyRemoteSetting(), isConfigured: true } as Partial; + delete imported.idDerivationVersion; + delete imported.idDerivationKey; + vi.mocked(decodeSettingsFromQRCodeData).mockReturnValue(imported as ObsidianLiveSyncSettings); + vi.spyOn(setting, "adjustSettings").mockImplementation((settings) => Promise.resolve(settings)); + dialogManager.openWithExplicitCancel.mockResolvedValueOnce("cancelled"); + + await manager.decodeQR("qr-data"); + + const mergedSettings = vi.mocked(setting.adjustSettings).mock.calls[0][0]; + expect(mergedSettings.idDerivationVersion).toBe(0); + expect(mergedSettings.idDerivationKey).toBe(""); + expect(setting.currentSettings().idDerivationKey).toBe(savedKey); + }); + + it("rejects invalid QR settings before applying them", async () => { + const { manager, setting } = createSetupManager(); + vi.mocked(decodeSettingsFromQRCodeData).mockImplementationOnce(() => { + throw new Error("Invalid ID derivation key"); + }); + const applyExternalSettings = vi.spyOn(setting, "applyExternalSettings"); + + await expect(manager.decodeQR("invalid-qr")).resolves.toBe(false); + expect(applyExternalSettings).not.toHaveBeenCalled(); + }); + + it("requires the normal Fetch choice when ID derivation changes with the Metadata preference", async () => { + const { manager, setting, dialogManager, core } = createSetupManager(); + const currentSettings: ObsidianLiveSyncSettings = { + ...createLegacyRemoteSetting(), + isConfigured: true, + encrypt: true, + passphrase: "e2ee-passphrase", + usePathObfuscation: true, + encryptInternalMetadata: false, + idDerivationVersion: 0, + idDerivationKey: "", + }; + const nextIdKey = "78".repeat(32); + setting.settings = currentSettings; + const applyPartial = vi.spyOn(setting, "applyPartial"); + core.confirm = { + askSelectStringDialogue: vi.fn(() => + Promise.resolve("Enable without rebuilding β€” update every other device first") + ), + }; + dialogManager.openWithExplicitCancel + .mockResolvedValueOnce({ + ...currentSettings, + encryptInternalMetadata: true, + idDerivationVersion: 1, + idDerivationKey: nextIdKey, + }) + .mockResolvedValueOnce("existing-user") + .mockResolvedValueOnce("apply"); + + await manager.onlyE2EEConfiguration(UserMode.Update, currentSettings); + + expect(applyPartial).not.toHaveBeenCalled(); + expect(core.rebuilder.scheduleFetch).toHaveBeenCalledWith(expect.any(Function)); + expect(setting.currentSettings().idDerivationVersion).toBe(1); + expect(setting.currentSettings().idDerivationKey).toBe(nextIdKey); + }); + it("reserves Rebuild before saving a new-user configuration", async () => { const { manager, setting, dialogManager, core } = createSetupManager(); setting.settings = { ...setting.currentSettings(), isConfigured: false }; @@ -659,3 +784,23 @@ describe("SetupManager", () => { expect(setting.currentSettings().P2P_ActiveRemoteConfigurationId).toBe("existing"); }); }); + +describe("internal Metadata configuration", () => { + it.each([true, false])( + "applies the preference only after accepting the no-Rebuild warning (%s)", + async (accept) => { + const { manager, setting, dialogManager, core } = createSetupManager(); + const current = { ...setting.settings, encryptInternalMetadata: false, remoteType: REMOTE_COUCHDB }; + dialogManager.openWithExplicitCancel.mockResolvedValue({ ...current, encryptInternalMetadata: true }); + const ask = vi.fn(async (_message: string, choices: string[]) => (accept ? choices[0] : "Cancel")); + core.confirm = { askSelectStringDialogue: ask }; + const apply = vi.spyOn(setting, "applyPartial").mockResolvedValue(undefined); + await expect(manager.onlyE2EEConfiguration(UserMode.Update, current)).resolves.toBe(accept); + expect(ask.mock.calls[0][1][0]).toContain("update every other device first"); + expect(ask.mock.calls[0][0]).toContain("currently running LiveSync"); + expect(apply).toHaveBeenCalledTimes(accept ? 1 : 0); + expect(core.rebuilder.scheduleRebuild).not.toHaveBeenCalled(); + expect(core.rebuilder.scheduleFetch).not.toHaveBeenCalled(); + } + ); +}); diff --git a/src/modules/features/SetupWizard/dialogs/SetupRemoteE2EE.svelte b/src/modules/features/SetupWizard/dialogs/SetupRemoteE2EE.svelte index 7340f31d..0a06a719 100644 --- a/src/modules/features/SetupWizard/dialogs/SetupRemoteE2EE.svelte +++ b/src/modules/features/SetupWizard/dialogs/SetupRemoteE2EE.svelte @@ -13,38 +13,144 @@ E2EEAlgorithms, type EncryptionSettings, } from "@vrtmrz/livesync-commonlib/compat/common/types"; + import { + deriveIdKey, + deriveOrImportIdKey, + formatIdRecoveryCode, + ID_DERIVATION_VERSION, + ID_RECOVERY_CODE_PREFIX, + } from "@vrtmrz/livesync-commonlib/settings"; import { onMount } from "svelte"; import type { GuestDialogProps } from "@/modules/services/LiveSyncUI/svelteDialog"; import { copyTo, pickEncryptionSettings } from "@vrtmrz/livesync-commonlib/compat/common/utils"; - import { TYPE_CANCELLED, type SetupRemoteE2EEResultType } from "./setupDialogTypes"; + import { + TYPE_CANCELLED, + type SetupRemoteE2EEInitialData, + type SetupRemoteE2EEResultType, + } from "./setupDialogTypes"; import { $msg as translateMessage } from "@/common/translation"; - type Props = GuestDialogProps; + type Props = GuestDialogProps; + type IdConfigurationChoice = "keep" | "random" | "custom"; + type IdCustomChoice = "passphrase" | "source" | "recovery"; const { setResult, getInitialData }: Props = $props(); let default_encryption: EncryptionSettings = { encrypt: true, passphrase: "", E2EEAlgorithm: DEFAULT_SETTINGS.E2EEAlgorithm, usePathObfuscation: true, - } as EncryptionSettings; + encryptInternalMetadata: true, + idDerivationVersion: 0, + idDerivationKey: "", + }; let encryptionSettings = $state({ ...default_encryption }); + let newVault = $state(false); + let idConfigurationChoice = $state("keep"); + let idCustomChoice = $state("source"); + let idDerivationSource = $state(""); + let idDerivationError = $state(""); + let recoveryCodeVisible = $state(false); + let recoveryCodeCopied = $state(false); + + const idDerivationConfigured = $derived( + encryptionSettings.idDerivationVersion === ID_DERIVATION_VERSION && + typeof encryptionSettings.idDerivationKey === "string" && + encryptionSettings.idDerivationKey.length > 0 + ); + const recoveryCode = $derived.by(() => + idDerivationConfigured ? formatIdRecoveryCode(encryptionSettings.idDerivationKey) : "" + ); onMount(() => { if (getInitialData) { const initialData = getInitialData(); if (initialData) { - copyTo(initialData, encryptionSettings); + copyTo(initialData.settings, encryptionSettings); + newVault = initialData.newVault; } } + idConfigurationChoice = !idDerivationConfigured && newVault ? "random" : "keep"; }); let e2eeValid = $derived.by(() => { if (!encryptionSettings.encrypt) return true; return encryptionSettings.passphrase.trim().length >= 1; }); + let canEncryptInternalMetadata = $derived( + encryptionSettings.encrypt && + encryptionSettings.E2EEAlgorithm === E2EEAlgorithms.V2 && + encryptionSettings.usePathObfuscation + ); - function commit() { - setResult(pickEncryptionSettings(encryptionSettings)); + function resetIdDerivationSource() { + idDerivationSource = ""; + idDerivationError = ""; + } + + function toggleEncryption(enabled: boolean) { + encryptionSettings.encrypt = enabled; + if (!enabled) resetIdDerivationSource(); + } + + function selectIdConfiguration() { + recoveryCodeVisible = false; + recoveryCodeCopied = false; + resetIdDerivationSource(); + } + + function selectIdCustomSource() { + resetIdDerivationSource(); + } + + async function copyRecoveryCode() { + try { + await navigator.clipboard.writeText(recoveryCode); + recoveryCodeCopied = true; + } catch { + idDerivationError = translateMessage("The recovery code could not be copied. Select and copy the visible code instead."); + } + } + + async function commit() { + idDerivationError = ""; + const result = pickEncryptionSettings(encryptionSettings); + + if (encryptionSettings.encrypt && idConfigurationChoice !== "keep") { + let source = idDerivationSource; + if (idConfigurationChoice === "random") { + const bytes = crypto.getRandomValues(new Uint8Array(32)); + source = Array.from(bytes, (byte) => byte.toString(16).padStart(2, "0")).join(""); + } else if (idCustomChoice === "passphrase") { + source = encryptionSettings.passphrase; + } + if (source.length === 0) { + if (!idDerivationConfigured) { + idDerivationError = translateMessage("An ID source is required to enable this option."); + return; + } + } else { + try { + result.idDerivationKey = + idConfigurationChoice === "custom" && idCustomChoice !== "passphrase" + ? await importOrDeriveEnteredIdKey(source, idCustomChoice) + : await deriveIdKey(source); + result.idDerivationVersion = ID_DERIVATION_VERSION; + } catch { + idDerivationError = translateMessage("The ID source or recovery code is invalid. Check it and try again."); + return; + } + } + } + + idDerivationSource = ""; + setResult(result); + } + + async function importOrDeriveEnteredIdKey(source: string, choice: IdCustomChoice): Promise { + if (choice === "recovery" && !source.trim().startsWith(ID_RECOVERY_CODE_PREFIX)) { + throw new Error("An ID recovery code is required."); + } + return await deriveOrImportIdKey(source); } @@ -52,7 +158,11 @@ {translateMessage("Please configure your end-to-end encryption settings.")} - + toggleEncryption(event.currentTarget.checked)} + /> {translateMessage( @@ -87,6 +197,182 @@ {/if} +
+ {translateMessage("ID generation")} + + + +
+ {#if encryptionSettings.encrypt && idConfigurationChoice === "keep" && !idDerivationConfigured} + + {translateMessage("Changing the E2EE passphrase changes IDs generated by the legacy configuration.")} + + {/if} + {#if (encryptionSettings.encrypt && idConfigurationChoice !== "keep") || idDerivationConfigured} + {#if encryptionSettings.encrypt} + + {translateMessage( + "This uses a saved key for new Chunk IDs and obfuscated Metadata document IDs, so changing the E2EE passphrase does not derive a new key automatically." + )} + + {/if} + {#if idDerivationConfigured} + + {translateMessage("The saved ID key is configured. Its source cannot be shown again.")} + + + {#if recoveryCodeVisible} + + + + + {#if recoveryCodeCopied} + {translateMessage("Recovery code copied.")} + {/if} + {/if} + {/if} + {#if encryptionSettings.encrypt} + {#if idConfigurationChoice === "custom"} +
+ {translateMessage("How to set the ID key")} + + + +
+ {#if idCustomChoice === "source" || idCustomChoice === "recovery"} + + + + {/if} + {/if} + {#if idDerivationConfigured && idConfigurationChoice !== "keep"} + + {translateMessage("The displayed recovery code belongs to the current key. Reopen this dialogue after saving to copy the replacement key.")} + + {/if} + {#if idConfigurationChoice === "custom" && idCustomChoice === "source"} + + {translateMessage("Choose a long, unpredictable source. It is used once and cannot be shown again after saving. A recovery code can be displayed on this device later. This input also accepts a tagged recovery code.")} + + {:else if idConfigurationChoice === "custom" && idCustomChoice === "recovery"} + + {translateMessage("Paste a tagged recovery code from an existing device to restore the same ID key.")} + + {:else if idConfigurationChoice === "random"} + + {translateMessage("For recovery after losing every device, save the recovery code after setup or choose an ID source you can reproduce.")} + + {:else if idConfigurationChoice === "custom" && idCustomChoice === "passphrase"} + + {translateMessage( + "The ID key is derived from the current E2EE passphrase and saved separately. Changing that passphrase later does not change the saved ID key. To reduce the risk of guessing that passphrase from known IDs, use a separate, unpredictable ID source instead." + )} + + {/if} + {#if idDerivationConfigured && idConfigurationChoice === "custom" && idCustomChoice !== "passphrase"} + {translateMessage("Leave this input empty to keep the saved ID key.")} + {/if} + {/if} + {idDerivationError} + {/if} + + + + + + This option encrypts file properties used by Hidden File Sync and Customisation Sync. +
+ It applies only to CouchDB and requires End-to-End Encryption, the V2 algorithm, and Property Encryption + (Obfuscate Properties). The remote type is selected later in this setup wizard. +
+ It protects properties written after the option is enabled; existing properties are not rewritten. A manual remote + Rebuild is strongly recommended to protect existing properties. Update every other synchronising device to a compatible + version before enabling this option, including devices currently running LiveSync. + + - - - - - - - + {error} diff --git a/src/modules/features/SetupWizard/dialogs/UseSetupURI.svelte b/src/modules/features/SetupWizard/dialogs/UseSetupURI.svelte index 5300558b..dcd29126 100644 --- a/src/modules/features/SetupWizard/dialogs/UseSetupURI.svelte +++ b/src/modules/features/SetupWizard/dialogs/UseSetupURI.svelte @@ -1,6 +1,5 @@