diff --git a/devs.md b/devs.md index 56a4dbf1..713eb6cc 100644 --- a/devs.md +++ b/devs.md @@ -235,6 +235,18 @@ Commonlib owns the typed English fallback for messages requested by its services ### Logging & Debugging +#### ID generation measurements on a device + +Enable **Enable Developers' Debug Tools.**, restart Obsidian, and run **Open review harness** from the command palette. Choose **Run** beside **ID generation performance**, keep Obsidian in the foreground, and use **Copy Markdown report** to retain the results. The **Automatic** action does not run this measurement; **Full review** includes it. + +The measurement uses fixed in-memory inputs and keys, with no Vault, database, settings, or remote writes. It compares legacy `xxhash64` and independent Chunk IDs for 256-byte, 4-KiB, and 32-KiB inputs, and compares obfuscated document IDs. Each result reports the median and range of three 1,000-ID samples and the median time per ID. Key derivation at save time is measured separately. Warm-up and pauses between batches are excluded from the timings. These measurements do not represent a full Rebuild. + +Where `performance.memory` is available, the report includes approximate JavaScript heap samples before, during, and after measurement. These may include other Obsidian activity and garbage collection; they are neither total process RAM nor an exact peak. Unsupported devices explicitly report that heap measurements are unavailable. + +The developer-only adapter in `src/features/ReviewHarness/reviewHarnessIdBenchmarkRuntime.ts` imports `HashManager` from Commonlib's public `/hashing` entry. Compilation, packed-package checks, and runtime tests cover this boundary. The algorithms remain owned by Commonlib. + +#### Logs + - Use `this._log(msg, LOG_LEVEL_INFO)` in modules (automatically prefixes with module name) - Log levels: `LOG_LEVEL_DEBUG`, `LOG_LEVEL_VERBOSE`, `LOG_LEVEL_INFO`, `LOG_LEVEL_NOTICE`, `LOG_LEVEL_URGENT` - LOG_LEVEL_NOTICE and above are reported to the user via Obsidian notices diff --git a/docs/design_docs/configurable_id_derivation.md b/docs/design_docs/configurable_id_derivation.md new file mode 100644 index 00000000..8a88ce5f --- /dev/null +++ b/docs/design_docs/configurable_id_derivation.md @@ -0,0 +1,515 @@ +--- +date: 2026-09-29 +commonlib-version: "0.1.33" +self-hosted-livesync-version: "1.0.32" +status: unreleased +--- + +# Configurable ID derivation + +## Purpose and baseline + +Introduce an optional, saved secret for deterministic Chunk IDs and obfuscated +Metadata document IDs. This allows an E2EE passphrase to change without also +changing those IDs, and allows their derivation to use an independent secret. +Identical inputs must produce identical IDs on participating devices so that +Chunks can be reused and edits to the same path share one document identity. + +The baseline is [PR #1222](https://github.com/vrtmrz/obsidian-livesync/pull/1222), +including its passphrase-persistence correction at commit +`126d6eadb858a79a08ad7f600061e54fc8d31196`. Commonlib `0.1.33`, published with +the `next` tag, provides the construction described here. It replaces the +independent Chunk algorithm from the `0.1.32` prerelease without a +compatibility branch or a new settings version. Legacy ID generation remains +unchanged. LiveSync pins the published `0.1.33` package and its registry +integrity in the lockfile. +Its [Internal Metadata encryption design](https://github.com/vrtmrz/obsidian-livesync/blob/126d6eadb858a79a08ad7f600061e54fc8d31196/docs/design_docs/internal_metadata_encryption.md) +remains the basis for Properties encryption and CouchDB feature admission. +This document records an unreleased LiveSync feature. It does not select a +plug-in release version. + +## Feasibility + +The change is feasible within the existing architecture. Commonlib already +centralises Chunk hashing, path-to-ID conversion, settings persistence, and +Setup URI encoding. Chunk reads follow stored IDs, so changing the generator +does not require a new Chunk reader or content representation. + +The work spans Commonlib and its consumers. The principal constraints are +agreement on document IDs, complete propagation of the saved secret, and cache +behaviour after a setting change. The construction and transport-specific +agreement checks are described below. No database migration framework is +required. + +## Scope + +| Value or operation | Proposed behaviour | +| -------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Encrypted Chunk IDs | Use the saved ID secret in the new mode. Keep legacy generation when the option is absent. | +| Obfuscated Metadata document IDs | Use the same saved secret with a separate derivation purpose. Preserve existing path normalisation and namespace handling, including ordinary files, `i:`, `ix:`, and supported legacy `ps:` entries. | +| Unobfuscated document IDs | Retain the current path-based identity. | +| Content and Properties encryption | Continue using the E2EE passphrase and existing encryption format. | +| CouchDB/PouchDB `_rev` | Retain existing revision generation and replication behaviour. | +| Internal content digests and transport bookkeeping | Retain existing behaviour unless they directly construct one of the IDs above. | + +Document IDs are already assigned in the local database. Properties encryption +protects the path and other fields during transfer, while preserving `_id`. +Consequently, the saved ID secret must reach local path conversion as well as +remote-facing code. Path Obfuscation continues to control whether this +conversion is used; this proposal does not enable it automatically. + +Journal keys which incorporate document IDs inherit the resulting IDs. They do +not need another secret. Content digests inside encrypted Customisation Sync +content also do not need a separate setting. + +Automatic migration or enablement of existing or already migrated users, +Setup URI expiry or revocation, QR format security changes, revision redesign, +remote-only E2EE passphrase rotation, and a new remote configuration management +protocol are outside this change. + +## Input and saved state + +New Vault setup selects independent ID derivation and a random source by +default when E2EE is enabled. An existing Vault with no saved key retains +legacy mode until the user selects a new key explicitly. The setup dialogue +shows three radio choices: + +1. Keep current configuration, selected by default for an existing Vault. It + retains the saved key when present and otherwise retains legacy ID + generation. A small description below this choice shows which configuration + is currently saved. In legacy mode, changing the E2EE passphrase still + changes IDs. +2. Generate a random ID key, selected by default for a new Vault. +3. Set an ID key. Three nested radio choices derive it once from the current + E2EE passphrase, accept a source string, or import a tagged recovery code. + Only the latter two show the text input. + +The action is not persisted. An ordinary source is converted to a key when the +settings are applied, and then discarded. A tagged `sls-id-v1:` recovery code +imports its exact 256-bit key without deriving it again, including when pasted +into the source-string input. The recovery-code choice accepts only tagged +codes. A malformed tagged code is rejected. If either input is empty, a saved +key is kept; without a saved key, the dialogue requests input. Changing the +E2EE passphrase later preserves the saved ID key. Cancelling or failing to +save preserves the previous settings. + +The source itself cannot be recovered from its key. A user can explicitly +display and copy the saved key as a tagged recovery code on the local device; +it is hidden when the dialogue opens. The dialogue warns that anyone who needs +recovery after losing every device should save that code or choose a source +they can reproduce. +Turning E2EE off retains the saved value but suspends its use for ID generation. +The setup dialogue disables new ID-key configuration while E2EE is off. Turning +E2EE on again reactivates the same value. Existing E2EE re-encryption and +Rebuild requirements still apply when the passphrase changes. + +The source-input warning concerns only that input. The E2EE passphrase retains +its separate, existing storage behaviour. The recovery code contains the actual +saved ID key and must be handled as a secret. + +Settings need to represent legacy mode or a supported version plus a derived +secret. Final field names belong in Commonlib. A declared new version with a +missing, malformed, or unavailable secret is an error; it must not silently +fall back to legacy generation. Loading or exporting an already derived value +must not derive it again. + +## Deterministic derivation + +The required contract is: + +```text +source string --versioned derivation at save--> saved ID secret +saved ID secret + Chunk content ------------> Chunk ID +saved ID secret + canonical path -----------> obfuscated document ID +E2EE passphrase ----------------------------> content and Properties encryption +``` + +Derivation is offline and deterministic across supported runtimes. Its version +fixes the text encoding, treatment of Unicode and whitespace, salt, parameters, +and saved representation. It must not depend on server state, the E2EE Security +Seed, a device identifier, time, or device-specific iteration calibration. +Repeated saving of the same source under the same version produces the same +value. Reusing that source in another Vault consequently also reuses the value. + +Version 1 uses PBKDF2-HMAC-SHA-256 with 310,000 iterations, the UTF-8 bytes of +the source after NFC normalisation, the fixed salt +`self-hosted-livesync:id-source:v1`, and a 256-bit output encoded as 64 lowercase +hexadecimal characters. Whitespace is preserved. The existing +`idDerivationVersion: 1` setting, saved-settings fields, and recovery-code format +remain unchanged; this implementation change does not add an ID format version +or migration path. + +Obfuscated document IDs continue to use the saved 256-bit value directly as the +key for full HMAC-SHA-256. Their message remains UTF-8 encoding of +`self-hosted-livesync:id-v1:document`, a NUL byte, and the canonical path. +Agreement proofs likewise retain their existing full-HMAC messages. Neither +path uses the new Chunk-specific cache. + +For encrypted Chunk IDs, first compute xxHash64 over the UTF-8 bytes of the +exact Chunk text with seed 0. Encode its result as a fixed 16-character +lowercase hexadecimal prehash. Derive a Chunk-specific subkey from the saved +32-byte value, then HMAC the domain-separated prehash: + +```text +Kchunk = HMAC-SHA-256( + saved 32-byte key, + UTF8('self-hosted-livesync:id-v1:chunk-key:xxhash64') +) +prehash = fixed16lowerhex(xxHash64(UTF8(exact Chunk text), seed 0)) +Chunk ID = full64lowerhex(HMAC-SHA-256( + Kchunk, + UTF8('self-hosted-livesync:id-v1:chunk:xxhash64' + NUL + prehash) +)) +``` + +The resulting Chunk ID is the full 64-character lowercase hexadecimal HMAC +output. Existing namespace prefixes remain outside the digest. Keyed Chunk IDs +use this fixed prehash regardless of `hashAlg`; legacy mode and its existing +hash selection remain unchanged. + +The Chunk-specific HMAC subkey and imported key, along with the WASM xxHash64 +generator, are cached per `HashManager` and active saved key. Concurrent +preparation is shared. Replacing the manager or key, turning E2EE off, or +returning to legacy mode clears the cache; failed preparation can be retried. +This cache adds no persistent state. The current E2EE setting also selects the +legacy encrypted or plain Chunk route when a manager remains alive while E2EE is +turned off. + +### Security properties and limits + +A derived value remains a secret capable of generating IDs. Hashing does not +increase the entropy of its source. A password KDF adds guessing cost; it does +not make a weak source strong. HKDF alone does not provide that password +stretching. See [RFC 8018](https://www.rfc-editor.org/rfc/rfc8018.html#section-8) +and [RFC 5869](https://www.rfc-editor.org/rfc/rfc5869.html#section-4). + +The random default separates ID generation from the E2EE passphrase. Deriving +both secrets from the same source retains a relationship with the original +passphrase, even after that passphrase changes. An independent source with +sufficient entropy provides the intended separation. HMAC with purpose +separation is the construction for using that secret; see +[RFC 2104](https://www.rfc-editor.org/rfc/rfc2104.html). +The fixed derivation salt means that reusing a source across Vaults reuses the +ID key; use separate sources when independent Vault identities are required. + +CouchDB authentication and database access control remain the first access +boundary. This design also considers exposure through database credentials, +server administration, or backups. It does not promise to conceal equality, +document counts, revision history, or ciphertext lengths from database readers. + +For Chunk IDs, xxHash64 is a public, non-cryptographic prehash. Distinct Chunk +texts which produce the same 64-bit prehash produce the same ID under the same +saved key. The final 256-bit HMAC does not restore distinctions lost at that +stage, so collision resistance for Chunk IDs is bounded by xxHash64 rather than +by the HMAC output width. This limit is an accepted trade-off for bounding the +content processed by HMAC. + +The independent ID key preserves existing file contents, Chunk representation, +and `_rev` behaviour. It does not change the privacy properties of those +formats. Payload and virtual file padding remain outside this change. + +## Sharing, import, and storage + +Include the saved derived value and its version in Setup URIs, protected by the +existing, separate Setup URI passphrase. Import the saved value directly. +Additional devices therefore need neither the original source nor another +derivation step. Manual setup can reproduce it by entering the same source and +version, or by importing the tagged recovery code. After an E2EE passphrase +change, the current passphrase cannot be assumed to reproduce the old ID secret. + +The standalone Setup URI generator uses a fresh random 256-bit ID key by +default, prints its tagged recovery code, and accepts that code for repeatable +generation for the same Vault. `id_mode=legacy` selects the old ID behaviour. +Running it again without the code produces a different key, so the generated +URI must not be treated as an update for an existing remote. + +QR sharing includes the same fields through the existing QR representation and +warnings. Its current payload is not encrypted like a Setup URI. The agreed +scope accepts that existing sharing model and user responsibility for keeping +QR material private; it adds no QR storage or expiry mechanism. + +| Boundary | Required handling | +| ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Existing settings and old complete Setup URI/QR/P2P imports | Missing fields select legacy behaviour. Do not inherit an unrelated value already present on the receiving device. | +| Ordinary partial setting updates | Preserve the current secret and version when neither is supplied. | +| New-format imports | Validate version and value together before applying or starting database work. | +| Local persistence | Integrate the secret explicitly with sensitive-configuration encryption and loading. Adding an arbitrary field does not currently provide this protection. | +| Reports, logs, and Markdown settings | Redact the secret in reports and logs; treat it as a credential in the existing Markdown export/import policy. An export which omits credentials must omit this secret. | +| Remote profiles, CLI, WebApp, WebPeer, and direct writers | Carry the effective value and version through every supported configuration path. A selected new mode must never degrade silently to legacy mode. | + +Setup URI JSON encoding can carry ordinary new settings, but QR encoding uses +an explicit key-index table. Append stable QR entries without reordering old +ones. Complete imports and partial edits must have distinct missing-value +semantics even where the current implementation merges settings objects. +In particular, `SetupManager` currently merges decoded URI settings over the +receiving device's settings. Complete imports must normalise the new fields +before that merge to prevent accidental inheritance. + +## Compatibility and changes to existing data + +| Difference or change | Consequence | +| ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | +| Different Chunk derivation only | Existing content remains readable through Metadata `children`; new writes can duplicate Chunks and reduce reuse. | +| Different obfuscated document ID derivation | The same path can become separate documents. Treat this as an incompatible configuration requiring resolution. | +| New E2EE passphrase, unchanged saved ID secret | IDs remain stable for unchanged content, paths, and other ID settings. Re-encryption still requires the existing E2EE workflow. | +| Enabling, replacing, or disabling the option with Path Obfuscation active | Document identity changes. Use the established authoritative Rebuild and secondary-device Fetch workflow. | +| Existing installation with no new option | Preserve its exact legacy behaviour; do not derive or copy a value during upgrade. | + +Copying the legacy passphrase into the new derivation does not preserve legacy +IDs, because the derivation itself changes. This proposal therefore makes no +automatic or seamless migration promise. Before an explicit transition, update +and stop the participating devices, select the authoritative data, and use the +existing [Rebuild and Fetch procedures](../recovery.md). Share the resulting +configuration before other devices rejoin. One-entry +[Metadata ID repair](metadata_document_id_validation_and_repair.md) does not +perform this transition. + +Commonlib's Chunk cache includes content-to-ID lookup before hashing. Changing +the hash function alone can keep producing old IDs, and reading old Chunks can +populate that lookup again. The implementation must distinguish read reuse +from the ID selected for a new write, including after manager replacement and +restart. Readers continue accepting referenced legacy Chunks. + +## Agreement checks and older clients + +Keep checks focused on preventing incompatible document identities. Reuse the +existing configuration review and replication admission paths. A mismatch +must not be treated as an automatically alignable Chunk setting when document +IDs depend on it. Show a mismatch or unsupported version without exposing the +saved secret. + +The advertised ID version is used for comparison only. Ordinary Tweak alignment +preserves each device's ID version and key together, including when only Chunk +IDs differ. A document ID mode mismatch requires explicit configuration through +a Setup URI or the matching key, rather than adopting a version without its key. + +For CouchDB, extend the supported feature set in the remote feature contract +introduced by PR #1222, then declare the new requirement before writing data +under it. That mechanism rejects unsupported features at admission and provides +a best-effort stop when an unsupported requirement arrives later. It checks +format support, not equality of saved secrets, and does not make a live +migration atomic. Journal can extend its existing milestone compatibility path; +P2P needs its separate admission handling. The CouchDB feature contract alone +cannot protect those transports. + +The implementation checks up to two remote documents in each ordinary and +internal obfuscated-ID namespace before CouchDB replication or direct writes. +This includes a legacy-mode caller connecting to a remote which uses keyed IDs. +For each available sample it recomputes the ID from the decrypted path; any +mismatch rejects the connection. An empty database, or one with no usable +sample, is reported as unverified and may proceed because there is no observed +document identity to conflict with. A sampled match is evidence, not a proof +that every document has the same identity; an unsampled mixture remains a +limitation of this bounded check. + +When E2EE and Path Obfuscation are both active, Journal stores an +E2EE-encrypted, domain-separated proof in its existing milestone. It is +encrypted before the milestone is uploaded and compared on later connections. +An established milestone without this proof requires a Rebuild before the new +document IDs can be used. Journal advertises a new compatibility range for +keyed document IDs so older clients reject it. P2P compares a +purpose-separated HMAC over a fresh challenge during peer admission; the proof +is not stored. When Path Obfuscation is off, different keys affect only Chunk +IDs, so Journal keeps its legacy compatibility range and P2P does not require +key agreement. Neither transport publishes the key or a plaintext verifier in +Tweak values. CouchDB and direct writers use the document sample check above +rather than a stored verifier. The sample check uses the host's path service +with the attempted settings snapshot so stored non-canonical paths are treated +the same way as ID generation. +The existing `_rev` behaviour for ordinary content remains unchanged. + +## Implementation responsibilities + +The following are the confirmed integration points in the reviewed baseline. +Commonlib paths refer to its package implementation, not a source mirror in +this repository. + +| Owner and entry points | Work | +| -------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Commonlib `HashManagerCore`, concrete hash managers, `PathService`, and `path2id_base` | Select legacy or new derivation consistently; preserve path and namespace semantics. | +| Commonlib `EntryManagerImpls`, `LayeredChunkManager`, and `LiveSyncManagers` | Keep referenced Chunks readable and invalidate or partition generation-dependent caches. | +| Commonlib settings definitions/lifecycle, `SettingService`, `pickEncryptionSettings`, and `API/processSetting` | Own version validation, persistence, copying, imports, Setup URI encoding, and QR slots. | +| Commonlib compatibility assessment and replication implementations | Classify identity differences, protect any comparison data, and enforce supported formats at each transport boundary. | +| Commonlib `API/DirectFileManipulatorV2` | Carry the option through its explicit settings and path-obfuscation configuration. | +| LiveSync `SetupRemoteE2EE.svelte`, `PaneRemoteConfig.ts`, and `SetupManager.ts` | Implement the three configuration actions and three nested ID-key inputs, configured state, local recovery-code reveal, and existing Apply/Rebuild/Fetch choices. | +| LiveSync `replicatorConfigurationIdentity.ts`, `reportTool.ts`, and `ModuleObsidianSettingAsMarkdown.ts` | Replace connections when effective settings change, redact the secret, and apply credential-sharing rules. | +| CLI, browser applications, and setup tools | Use the same Commonlib contract in manual setup and imports; generate new Setup URIs with a reusable random ID key by default. | + +Implement Commonlib changes in its own repository, validate its packed artefact, +and validate LiveSync against that exact dependency before adopting a released +version. Translations remain outside this implementation scope. + +## Validation + +The fixed-vector and cache tests first failed against unchanged Commonlib +`0.1.32`, then passed after the implementation change. Commonlib's 2,036 Unit +tests, type check, package boundary, and isolated packed-package checks pass. +Three Integration tests against real CouchDB and Object Storage verify direct +access, Journal agreement, and rejection before control-document changes. +The same-manager E2EE-off regression was reproduced and fixed. + +LiveSync's 1,049 Unit tests, type and lint checks, production build, and iOS 15 +bundle syntax check pass after installing the exact published `0.1.33` +package. Its tarball matches the validated publication candidate, and all 559 +installed package files match the registry artefact. The resulting bundle is +identical to the one checked before publication. + +Real Obsidian two-Vault checks with the accepted Chunk construction cover +matching-key synchronisation, incompatible document key rejection, and +differing Chunk keys with visible paths. The Review Harness also passes with +the published package, verifying actual ID calculations and report copying +without changing live settings. Its adapter imports `HashManager` through +Commonlib's focused `/hashing` entry, whose package checks cover public types, +Node execution, and browser bundling. + +### Current Chunk calculation performance + +The actual Commonlib `HashManager` implementations were compared in Obsidian +1.12.7 on ARM64 Linux. Six samples rotate all three variants through each +execution position twice. The table reports median total ID calculation time; +1,000 means the total for 1,000 IDs, not the time per ID. Inputs are synthetic. + +| Input | IDs per sample | Legacy xxHash64 | Previous independent HMAC | Updated independent ID | +| --------------------------------- | -------------: | --------------: | ------------------------: | ---------------------: | +| 256-byte text | 1,000 | 4.50 ms | 41.40 ms | 24.45 ms | +| 4 KiB text | 1,000 | 10.00 ms | 96.25 ms | 29.70 ms | +| 32 KiB text | 1,000 | 48.40 ms | 494.40 ms | 68.95 ms | +| 10 MiB binary, default splitting | 103 | 19.60 ms | 192.50 ms | 21.45 ms | +| 50 MiB binary, default splitting | 512 | 99.85 ms | 972.75 ms | 106.65 ms | +| 10 MiB binary, Self-hosted preset | 5 | 24.10 ms | 216.90 ms | 18.50 ms | +| 50 MiB binary, Self-hosted preset | 35 | 118.40 ms | 1,067.00 ms | 91.75 ms | + +The first updated ID, including Chunk-key preparation, took 1.3 ms in this +run. Repeated measurements exclude preparation, warm-up, and pauses. Binary +cases use the actual splitter and Base64 representation; all decoded bytes +and repeated IDs were checked. Splitting, Base64 conversion, database work, +payload encryption, and transfer are outside the measured interval. These +results establish lower ID calculation cost on this host, not a complete +Rebuild speedup or native mobile performance. + +### Native-device ID measurements + +User-supplied Review Harness reports compare the previous and updated builds +on Android 13 and iOS 18.7. Each value is the median total time for 1,000 +independent Chunk IDs, using three samples in each run. + +| Input | Android, previous | Android, updated | iOS, previous | iOS, updated | +| ------------- | ----------------: | ---------------: | ------------: | -----------: | +| 256-byte text | 53.6 ms | 37.8 ms | 21 ms | 20 ms | +| 4 KiB text | 70.9 ms | 43.4 ms | 22 ms | 22 ms | +| 32 KiB text | 151.1 ms | 66.5 ms | 36 ms | 39 ms | + +Android's 32-KiB result takes about 56% less time. The corresponding iOS +result increases by 3 ms per 1,000 IDs; separate runs with three samples do +not establish the cause of that difference. Across these sizes, the updated +independent calculation adds approximately 18–25 ms per 1,000 IDs over each +device's legacy xxHash64 calculation. Save-time key derivation has medians of +46.6 ms on Android and 51 ms on iOS. + +These reports measure synthetic ID calculations, excluding database work, +payload encryption, and transfer. Android's heap samples remain constant, +and iOS does not expose them, so the reports do not establish memory usage or +improvement. The updated build has not been measured on Windows. + +The checks below are historical reference evidence for the predecessor +independent-ID implementation, which used full-content HMAC-SHA-256 for Chunk +IDs. They do not validate the current xxHash64-prehash construction. + +### Historical predecessor checks + +Earlier consumer validation used a local Commonlib `0.1.32` candidate. Clean +installations with npm 10 and npm 11, type checking, lint, Svelte checks, the +production build, and the iOS 15 bundle compatibility check passed. LiveSync +had 1,039 passing Unit tests, six passing Integration tests against real +CouchDB, and seven passing Setup URI utility tests with the frozen Deno +lockfile. + +Predecessor Commonlib candidate checks covered deterministic vectors, Unicode +normalisation, legacy behaviour, encrypted settings persistence, imports, +cache transitions, and transport admission. Its Integration tests against real +Object Storage accept a matching Journal key, reject a different document ID +key before changing the milestone, and allow different Chunk keys when paths +remain visible. A direct-access Integration test against real CouchDB reads +with the same key and rejects a different key before changing the version +document. These library tests complemented the consumer checks for that +predecessor; they were not additional LiveSync Unit tests. + +Real Obsidian 1.12.7 on ARM64 Linux verifies the following consumer boundaries: + +| Boundary | Verified behaviour | +| ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| CouchDB synchronisation | Two Vaults exchange notes in both directions with matching document and Chunk IDs. Different Chunk keys also work with Path Obfuscation off. | +| CouchDB rejection | Ordinary replication rejects different or legacy document ID keys before downloading files or changing remote documents and checkpoints. | +| Visible onboarding | Both a separate source and the random default persist an encrypted key, transfer it through a Setup URI, complete Fast Fetch, and synchronise in both directions. The `%`-prefixed E2EE passphrase survives restart. | +| Input and recovery | The radio controls and disabled styling are exercised. An empty first source keeps the dialogue open with an error; empty input keeps an existing key. A recovery code restores the same key. | +| Journal upload | The uploaded documents and Chunks have keyed IDs, and the first Object Storage milestone contains the encrypted agreement proof. | +| Credential-free Markdown | Neither the saved ID key nor its encrypted representation appears in exported settings Markdown. | + +A CLI P2P E2E run with a local relay imports encrypted Setup URIs, transfers a +note with matching keys, and rejects a peer with a different document ID key. +A separate check loads the published `0.1.31` DirectFileManipulator in another +process: it reads a legacy remote, and rejects an independent-ID remote with +or without Path Obfuscation, leaving remote documents and checkpoints +unchanged. This checks the previous library API, rather than an older +Obsidian installation. Bounded sampling tests cover empty and mixed document +collections; they do not establish that every document in a remote is +compatible. + +### Performance reference for the predecessor + +The measurements below are historical results for a local predecessor +Commonlib `0.1.32` candidate which used full-content HMAC-SHA-256 for +independent Chunk IDs. They are reference evidence only, not performance +results for the accepted xxHash64-prehash construction. Both modes enable E2EE +and Path Obfuscation; the legacy baseline uses `xxhash64`. Three trials per +mode alternate their order. These are synthetic corpora and serial local +writes, excluding Vault enumeration, remote payload encryption, and transfer; +they are not timings of the complete Rebuild action. + +Saving an ordinary ID source takes 52–57 ms in Node.js 24, with a median of +56 ms. This PBKDF2 operation happens once when saving the source. Per-Chunk +and per-document IDs use the saved key and do not repeat PBKDF2. + +The actual Obsidian renderer gives these median times for 1,000 serial calls +to the Commonlib hash manager or Path Service, after warm-up: + +| Input | Legacy | Independent ID | +| ------------------------ | ------: | -------------: | +| Distinct 256-byte Chunks | 5.4 ms | 43.3 ms | +| Distinct 4 KiB Chunks | 15.0 ms | 104.8 ms | +| Distinct 32 KiB Chunks | 52.8 ms | 581.7 ms | +| Distinct document paths | 36.5 ms | 50.0 ms | + +These direct calls include no Chunk-content cache hits. The predecessor hash +had a measurable cost, especially when there were many small Chunks. The +local-write experiments exercise splitting, ID generation, Chunk reuse, and +PouchDB writes: + +| Workload and adapter | Legacy median (range) | Independent median (range) | +| ------------------------------------------------------------- | --------------------: | -------------------------: | +| 5,000 text/binary files, 100 MiB, Node PouchDB memory adapter | 102.6 s (87.4–109.3) | 86.6 s (75.4–93.1) | +| 1,000 text files, 19.5 MiB, actual Obsidian local database | 78.7 s (52.0–82.8) | 63.2 s (62.8–63.8) | + +The measurements do not show a large overall slowdown for these workloads, +but the variation does not support a general speedup claim. Both experiments +checked document counts, Chunk-reference counts, and sample content readback. +The Node experiment also checked that every referenced Chunk was present. The +100 MiB corpus includes 250 duplicate files and produces 213,788 Chunk +references to 199,468 distinct Chunks in both modes, preserving reuse. + +For that corpus, stored document JSON grows from 133,157,581 to 154,342,743 +UTF-8 bytes, an increase of 15.9%. The text-only Obsidian corpus produces many +small Chunks and grows from 27,893,656 to 33,594,994 bytes, or 20.4%, including +32 warm-up documents. Longer Chunk IDs occur in both Chunk documents and +Metadata references. Ordinary obfuscated document IDs remain 66 characters. +These totals measure serialised document JSON; physical database and index +growth depend on the adapter and have not been measured. + +### Remaining validation + +Larger binary workloads on mobile, a representative user's Vault, physical +storage growth, and the complete Rebuild wall time remain unmeasured. The +native-device reports above verify the synthetic ID calculation scenario; +desktop E2E and mobile viewport checks do not establish other mobile +operating-system behaviour. URI revocation, QR redesign, and automatic +migration remain outside this change. diff --git a/docs/glossary.md b/docs/glossary.md index 8af12013..9bc5ed31 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -92,6 +92,12 @@ recovery guidance, or diagnostics intended for users. reports, and advanced edge-case settings. - **Hidden File Sync:** The feature which synchronises files in hidden directories, such as `.obsidian`. +- **ID key:** A saved secret used to generate encrypted Chunk IDs and + obfuscated Metadata document IDs when independent ID derivation is enabled. + It is separate from the current E2EE passphrase. +- **ID recovery code:** A versioned text form of the saved ID key which can be + shown on the current device and imported without deriving a different key. + Treat it as a secret. - **JWT Authentication:** An experimental CouchDB authentication option which uses a JSON Web Token instead of standard credentials. It requires a private key or secret, algorithm, expiry duration, subject, and key ID. diff --git a/docs/settings.md b/docs/settings.md index 449fa5ab..44b21a4b 100644 --- a/docs/settings.md +++ b/docs/settings.md @@ -239,6 +239,20 @@ Setting key: passphrase Encrypting passphrase. If you change the passphrase, you need to rebuild databases (You will be informed). +#### Independent ID derivation + +Setting keys: `idDerivationVersion`, `idDerivationKey` + +This setting saves a separate key for encrypted Chunk IDs and obfuscated Metadata document IDs. New Vault setup selects **Generate a random ID key** by default when E2EE is enabled. Existing Vaults select **Keep current configuration** by default. The radio choices show the available configurations together. A small description under **Keep current configuration** identifies the saved configuration: an existing ID key, or legacy IDs linked to the E2EE passphrase. That choice retains either one; on a new Vault, choosing it explicitly uses legacy IDs. If the configuration is legacy, changing the E2EE passphrase also changes IDs. + +To set a key yourself, choose **Set an ID key**. Three further radio choices then appear: **Derive from current E2EE passphrase**, **Enter an ID source**, and **Import an ID recovery code**. The last two choices show a text input. The source input also recognises a tagged recovery code. An empty input keeps an existing key; a first key requires input. An ordinary source is converted to a key when you apply the settings and cannot be shown again. A recovery code imports the saved key directly. + +Use **Show current recovery code** to display and copy the saved key on this device. The code starts with `sls-id-v1:` and can be pasted into the manual input on another device without deriving a different key. A Setup URI carries the same saved key under its separate passphrase. If you need to restore the configuration after losing every device, save the recovery code or choose a source you can reproduce before relying on the random default. Keep the code private. + +Using the E2EE passphrase as the source keeps IDs stable after later passphrase changes, but it does not separate the original passphrase from guesses based on known IDs. Use a long, unpredictable, separate source when that separation matters. Hashing a weak source does not make it strong. + +Changing the E2EE passphrase later does not change the saved ID key, although the existing re-encryption and Rebuild procedure still applies to the encrypted data. While E2EE is off, the saved ID key is retained but is not used; existing legacy ID generation applies until E2EE is enabled again. Devices with different ID keys can synchronise when Path Obfuscation is off, although identical content may produce duplicate Chunks. Enabling, replacing, or disabling the ID key can change document IDs when Path Obfuscation is active. Update participating devices, Rebuild from the authoritative Vault, and Fetch on other devices before resuming ordinary synchronisation. A QR code includes the saved key under the existing QR sharing rules, so keep the QR code private. + #### Path Obfuscation Setting key: usePathObfuscation @@ -1033,6 +1047,8 @@ Setting key: hashAlg `xxhash64` is the supported current value. Older algorithms remain selectable only as an edge-case compatibility path for existing databases. Changing the algorithm can reduce chunk reuse between devices and requires the normal tweak review. +When independent ID derivation is enabled, encrypted Chunk IDs use its versioned HMAC construction instead of `hashAlg`. The selected `hashAlg` continues to apply to legacy IDs. + ### 6. Edge case addressing (Behaviour) #### Fetch database with previous behaviour diff --git a/docs/setup_object_storage.md b/docs/setup_object_storage.md index 0244a766..4c72b716 100644 --- a/docs/setup_object_storage.md +++ b/docs/setup_object_storage.md @@ -31,7 +31,7 @@ deno run --minimum-dependency-age=0 --allow-env https://raw.githubusercontent.co For providers which require them, set `force_path_style`, `use_custom_request_handler`, or `bucket_custom_headers` as described in the [setup utility reference](../utils/readme.md#object-storage). -Store the generated Setup URI and Setup URI passphrase separately. The URI is encrypted, but it contains the Object Storage credentials. +Store the generated Setup URI and Setup URI passphrase separately. The URI is encrypted, but it contains the Object Storage credentials. The generator also prints an ID recovery code; reuse it through `id_recovery_code` if you regenerate a URI for the same Vault. A new run without it creates a different ID key. The [setup utility reference](../utils/readme.md#setup-uri-generation) describes the `id_mode=legacy` option for existing Vaults. ## Set up the first device diff --git a/docs/setup_own_server.md b/docs/setup_own_server.md index 0eac36b2..1177b820 100644 --- a/docs/setup_own_server.md +++ b/docs/setup_own_server.md @@ -190,6 +190,8 @@ deno run --minimum-dependency-age=0 --allow-env https://raw.githubusercontent.co > > If `uri_passphrase` is omitted, the generator creates a cryptographically random value and prints it once. +The generator also prints an ID recovery code for its random ID key. Save that code if you may need to regenerate a Setup URI for the same Vault. Pass it back as `id_recovery_code`; otherwise a later run creates a different key. Set `id_mode=legacy` only when connecting to a Vault which uses the previous ID behaviour. See the [setup utility reference](../utils/readme.md#setup-uri-generation). + The generator consumes the exact registry-pinned Commonlib release used by the provisioning utility. It creates a configured CouchDB remote profile, applies the current defaults for a new Vault, and encodes them with Commonlib's Setup URI contract. You will then get the following output: @@ -198,6 +200,8 @@ You will then get the following output: Generated couchdb Setup URI. Your passphrase for the Setup URI is: H7vX...a-random-32-character-value This passphrase is never shown again, so store it safely. +ID recovery code: sls-id-v1:<64 lowercase hexadecimal characters> +Use id_recovery_code with this value and reuse the same remote settings when generating another Setup URI for the same Vault. obsidian://setuplivesync?settings=%5B%22tm2DpsOE74nJAryprZO2M93wF%2Fvg.......4b26ed33230729%22%5D ``` diff --git a/docs/setup_p2p.md b/docs/setup_p2p.md index fb13a9c1..82b74ba6 100644 --- a/docs/setup_p2p.md +++ b/docs/setup_p2p.md @@ -135,4 +135,4 @@ export uri_passphrase= deno run --minimum-dependency-age=0 --allow-env https://raw.githubusercontent.com/vrtmrz/obsidian-livesync/main/utils/setup/generate_setup_uri.ts ``` -The generated Setup URI contains the encrypted room, relay, and Vault settings. It deliberately omits the device-specific name. Store the URI and its passphrase separately. After importing it on the first device, continue from the initialisation step above, then generate a fresh Setup URI for an additional device from that working device. +The generated Setup URI contains the encrypted room, relay, and Vault settings. It deliberately omits the device-specific name. Store the URI and its passphrase separately. The generator prints an ID recovery code; pass it as `id_recovery_code` if you regenerate a URI for the same Vault. A run without it creates a different ID key. Also reuse the original `p2p_room_id` and `p2p_passphrase`, since omitted values are generated afresh. The [setup utility reference](../utils/readme.md#setup-uri-generation) describes the `id_mode=legacy` option for existing Vaults. After importing the URI on the first device, continue from the initialisation step above, then generate a fresh Setup URI for an additional device from that working device. diff --git a/package-lock.json b/package-lock.json index 6b81bcf3..5939f6ed 100644 --- a/package-lock.json +++ b/package-lock.json @@ -23,7 +23,7 @@ "@smithy/types": "^4.14.3", "@smithy/util-retry": "^4.4.5", "@vrtmrz/browser-ui-kit": "0.1.0", - "@vrtmrz/livesync-commonlib": "0.1.31", + "@vrtmrz/livesync-commonlib": "0.1.33", "@vrtmrz/obsidian-plugin-kit": "0.1.4", "@vrtmrz/ui-interactions": "0.1.2", "diff-match-patch": "^1.0.5", @@ -4567,9 +4567,9 @@ } }, "node_modules/@vrtmrz/livesync-commonlib": { - "version": "0.1.31", - "resolved": "https://registry.npmjs.org/@vrtmrz/livesync-commonlib/-/livesync-commonlib-0.1.31.tgz", - "integrity": "sha512-ZFspZQmVlsCD41us7OdVc6Fl7T8ckCMiz/BKjif7pE7FO6/vJ4hhEGfcYRSYY+2iyuWyUklJcMgW7nEsO4p2UA==", + "version": "0.1.33", + "resolved": "https://registry.npmjs.org/@vrtmrz/livesync-commonlib/-/livesync-commonlib-0.1.33.tgz", + "integrity": "sha512-jziEUYCrGty3btR8pj/nkQrfLIGsQBbMug9opN8vCwAZmAznS0Zuh/l1TJpoKl8nqijZT1AJ4w5jkgU9pEyz4Q==", "license": "MIT", "dependencies": { "@aws-sdk/client-s3": "^3.808.0", diff --git a/package.json b/package.json index 6162c67d..92005b6e 100644 --- a/package.json +++ b/package.json @@ -185,7 +185,7 @@ "@smithy/types": "^4.14.3", "@smithy/util-retry": "^4.4.5", "@vrtmrz/browser-ui-kit": "0.1.0", - "@vrtmrz/livesync-commonlib": "0.1.31", + "@vrtmrz/livesync-commonlib": "0.1.33", "@vrtmrz/obsidian-plugin-kit": "0.1.4", "@vrtmrz/ui-interactions": "0.1.2", "diff-match-patch": "^1.0.5", diff --git a/src/apps/cli/testdeno/helpers/docker.ts b/src/apps/cli/testdeno/helpers/docker.ts index 159d6e60..822f0211 100644 --- a/src/apps/cli/testdeno/helpers/docker.ts +++ b/src/apps/cli/testdeno/helpers/docker.ts @@ -628,7 +628,7 @@ export async function startP2pRelay(): Promise { //TODO: port mapping should be configurable. "4000:7777", "--tmpfs", - "/app/strfry-db:rw,size=256m", + "/app/strfry-db:rw,size=256m,mode=1777", "--entrypoint", "sh", P2P_RELAY_IMAGE, diff --git a/src/apps/cli/testdeno/helpers/settings.ts b/src/apps/cli/testdeno/helpers/settings.ts index 524622c0..bb999b13 100644 --- a/src/apps/cli/testdeno/helpers/settings.ts +++ b/src/apps/cli/testdeno/helpers/settings.ts @@ -13,7 +13,11 @@ export async function initSettingsFile(settingsFile: string): Promise { * Generate a full setup URI from a settings file via the Commonlib package API. * Mirrors the bash flow in test-setup-put-cat-linux.sh. */ -export async function generateSetupUriFromSettings(settingsFile: string, setupPassphrase: string): Promise { +export async function generateSetupUriFromSettings( + settingsFile: string, + setupPassphrase: string, + preserveRemoteSettings = false +): Promise { const script = [ "import { fs } from '@vrtmrz/livesync-commonlib/node';", "import { encodeSettingsToSetupURI } from '@vrtmrz/livesync-commonlib/compat/API/processSetting';", @@ -21,13 +25,17 @@ export async function generateSetupUriFromSettings(settingsFile: string, setupPa " const settingsPath = process.env.SETTINGS_FILE;", " const passphrase = process.env.SETUP_PASSPHRASE;", " const settings = JSON.parse(fs.readFileSync(settingsPath, 'utf-8'));", - " settings.couchDB_DBNAME = 'setup-put-cat-db';", - " settings.couchDB_URI = 'http://127.0.0.1:5999';", - " settings.couchDB_USER = 'dummy';", - " settings.couchDB_PASSWORD = 'dummy';", - " settings.liveSync = false;", - " settings.syncOnStart = false;", - " settings.syncOnSave = false;", + ...(preserveRemoteSettings + ? [] + : [ + " settings.couchDB_DBNAME = 'setup-put-cat-db';", + " settings.couchDB_URI = 'http://127.0.0.1:5999';", + " settings.couchDB_USER = 'dummy';", + " settings.couchDB_PASSWORD = 'dummy';", + " settings.liveSync = false;", + " settings.syncOnStart = false;", + " settings.syncOnSave = false;", + ]), " const uri = await encodeSettingsToSetupURI(settings, passphrase);", " process.stdout.write(uri.trim());", "})();", diff --git a/src/apps/cli/testdeno/test-p2p-sync.ts b/src/apps/cli/testdeno/test-p2p-sync.ts index e05c6efe..32f3d1ad 100644 --- a/src/apps/cli/testdeno/test-p2p-sync.ts +++ b/src/apps/cli/testdeno/test-p2p-sync.ts @@ -1,6 +1,11 @@ import { assert } from "@std/assert"; import { TempDir } from "./helpers/temp.ts"; -import { initSettingsFile, applyP2pSettings, applyP2pTestTweaks } from "./helpers/settings.ts"; +import { + initSettingsFile, + applyP2pSettings, + applyP2pTestTweaks, + generateSetupUriFromSettings, +} from "./helpers/settings.ts"; import { startCliInBackground } from "./helpers/backgroundCli.ts"; import { discoverPeer, @@ -9,10 +14,10 @@ import { maybeStartCoturn, stopCoturnIfStarted, } from "./helpers/p2p.ts"; -import { runCli } from "./helpers/cli.ts"; +import { runCli, runCliOrFail, runCliWithInputOrFail, sanitiseCatStdout } from "./helpers/cli.ts"; import { getOptimalLoopbackIp } from "./helpers/net.ts"; -Deno.test("p2p-sync: discovers peer and completes sync", async () => { +Deno.test("p2p-sync: transfers with the same ID key and rejects a different document ID key", async () => { const loopbackIp = await getOptimalLoopbackIp(); const loopbackHost = loopbackIp === "::1" ? "[::1]" : loopbackIp; @@ -32,14 +37,18 @@ Deno.test("p2p-sync: discovers peer and completes sync", async () => { const hostSettings = workDir.join("settings-host.json"); const clientVault = workDir.join("vault-sync"); const clientSettings = workDir.join("settings-sync.json"); + const rejectedVault = workDir.join("vault-rejected"); + const rejectedSettings = workDir.join("settings-rejected.json"); await Deno.mkdir(hostVault, { recursive: true }); await Deno.mkdir(clientVault, { recursive: true }); + await Deno.mkdir(rejectedVault, { recursive: true }); const relayStarted = await maybeStartLocalRelay(relay); const coturnStarted = await maybeStartCoturn(turnServers); try { await initSettingsFile(hostSettings); await initSettingsFile(clientSettings); + await initSettingsFile(rejectedSettings); await applyP2pSettings( hostSettings, roomId, @@ -58,8 +67,52 @@ Deno.test("p2p-sync: discovers peer and completes sync", async () => { "~.*", turnServers ); + await applyP2pSettings( + rejectedSettings, + roomId, + passphrase, + "self-hosted-livesync-cli-tests", + relay, + "~.*", + turnServers + ); await applyP2pTestTweaks(hostSettings, hostPeerName, passphrase); await applyP2pTestTweaks(clientSettings, clientPeerName, passphrase); + await applyP2pTestTweaks(rejectedSettings, "p2p-rejected-" + nonce, passphrase); + for (const [vault, path, key, label] of [ + [hostVault, hostSettings, "ab".repeat(32), "host"], + [clientVault, clientSettings, "ab".repeat(32), "client"], + [rejectedVault, rejectedSettings, "cd".repeat(32), "rejected"], + ]) { + const settings = JSON.parse(await Deno.readTextFile(path)); + settings.idDerivationVersion = 1; + settings.idDerivationKey = key; + const sourcePath = workDir.join("setup-source-" + label + ".json"); + await Deno.writeTextFile(sourcePath, JSON.stringify(settings)); + const setupPassphrase = "independent-id-setup-passphrase"; + const setupUri = await generateSetupUriFromSettings(sourcePath, setupPassphrase, true); + await runCliWithInputOrFail(setupPassphrase + "\n", vault, "--settings", path, "setup", setupUri); + const persisted = JSON.parse(await Deno.readTextFile(path)); + assert(persisted.idDerivationVersion === 1, "The Setup URI lost the ID derivation version."); + assert(persisted.idDerivationKey === "", "The CLI stored the ID key in plain text."); + assert( + typeof persisted.encryptedIdDerivationKey === "string" && persisted.encryptedIdDerivationKey.length > 0, + "The CLI did not encrypt the saved ID key." + ); + assert(persisted.P2P_Enabled === true, "The Setup URI disabled P2P."); + assert(persisted.P2P_roomID === roomId, "The Setup URI changed the P2P room."); + assert(persisted.P2P_relays === relay, "The Setup URI changed the P2P relay."); + assert(persisted.remoteType === "ONLY_P2P", "The Setup URI changed the remote type."); + } + const notePath = "p2p/independent-id-note.md"; + await runCliWithInputOrFail( + "A note transferred with the saved ID key.\n", + clientVault, + "--settings", + clientSettings, + "put", + notePath + ); const host = startCliInBackground(hostVault, "--settings", hostSettings, "p2p-host"); try { @@ -82,9 +135,32 @@ Deno.test("p2p-sync: discovers peer and completes sync", async () => { syncResult.code === 0, `p2p-sync failed\nstdout: ${syncResult.stdout}\nstderr: ${syncResult.stderr}` ); + const rejectedPeer = await discoverPeer(rejectedVault, rejectedSettings, peersTimeout, hostPeerName); + const rejectedSync = await runCli( + rejectedVault, + "--settings", + rejectedSettings, + "p2p-sync", + rejectedPeer.id, + String(syncTimeout) + ); + assert( + rejectedSync.code !== 0, + `P2P accepted a different key for obfuscated document IDs.\nstdout: ${rejectedSync.stdout}\nstderr: ${rejectedSync.stderr}` + ); + assert( + rejectedSync.combined.includes("Tweak values are not matched"), + `P2P failed before checking peer settings.\nstdout: ${rejectedSync.stdout}\nstderr: ${rejectedSync.stderr}` + ); } finally { await host.stop(); } + const received = sanitiseCatStdout( + await runCliOrFail(hostVault, "--settings", hostSettings, "cat", notePath) + ).trimEnd(); + assert(received === "A note transferred with the saved ID key.", "The host did not receive the keyed note."); + const rejectedRead = await runCli(rejectedVault, "--settings", rejectedSettings, "cat", notePath); + assert(rejectedRead.code !== 0, "The rejected device received the keyed note."); } finally { await stopLocalRelayIfStarted(relayStarted); await stopCoturnIfStarted(coturnStarted); diff --git a/src/common/messages/LiveSyncProvisionalMessages.ts b/src/common/messages/LiveSyncProvisionalMessages.ts index 59d7d6cc..4825633f 100644 --- a/src/common/messages/LiveSyncProvisionalMessages.ts +++ b/src/common/messages/LiveSyncProvisionalMessages.ts @@ -204,6 +204,50 @@ export const liveSyncProvisionalEnglishMessages = { "Repair failed before the source was removed. Run inspection again before retrying.", "Connection settings": "Connection settings", "Saved connections": "Saved connections", + "ID generation": "ID generation", + "Keep current configuration": "Keep current configuration", + "Set an ID key": "Set an ID key", + "Current configuration: a saved ID key is used.": "Current configuration: a saved ID key is used.", + "Current configuration: the saved ID key is retained while E2EE is off.": + "Current configuration: the saved ID key is retained while E2EE is off.", + "Current configuration: no ID key is saved. With E2EE enabled, keeping it uses legacy IDs tied to the E2EE passphrase.": + "Current configuration: no ID key is saved. With E2EE enabled, keeping it uses legacy IDs tied to the E2EE passphrase.", + "Changing the E2EE passphrase changes IDs generated by the legacy configuration.": + "Changing the E2EE passphrase changes IDs generated by the legacy configuration.", + "This uses a saved key for new Chunk IDs and obfuscated Metadata document IDs, so changing the E2EE passphrase does not derive a new key automatically.": + "This uses a saved key for new Chunk IDs and obfuscated Metadata document IDs, so changing the E2EE passphrase does not derive a new key automatically.", + Configured: "Configured", + "The saved ID key is configured. Its source cannot be shown again.": + "The saved ID key is configured. Its source cannot be shown again.", + "Leave this input empty to keep the saved ID key.": "Leave this input empty to keep the saved ID key.", + "Generate a random ID key": "Generate a random ID key", + "How to set the ID key": "How to set the ID key", + "Derive from current E2EE passphrase": "Derive from current E2EE passphrase", + "Enter an ID source": "Enter an ID source", + "Import an ID recovery code": "Import an ID recovery code", + "ID source": "ID source", + "ID recovery code": "ID recovery code", + "Enter an ID recovery code": "Enter an ID recovery code", + "Choose a long, unpredictable source. It is used once and cannot be shown again after saving. A recovery code can be displayed on this device later. This input also accepts a tagged recovery code.": + "Choose a long, unpredictable source. It is used once and cannot be shown again after saving. A recovery code can be displayed on this device later. This input also accepts a tagged recovery code.", + "Paste a tagged recovery code from an existing device to restore the same ID key.": + "Paste a tagged recovery code from an existing device to restore the same ID key.", + "For recovery after losing every device, save the recovery code after setup or choose an ID source you can reproduce.": + "For recovery after losing every device, save the recovery code after setup or choose an ID source you can reproduce.", + "Show current recovery code": "Show current recovery code", + "Hide current recovery code": "Hide current recovery code", + "Current ID recovery code": "Current ID recovery code", + "Copy recovery code": "Copy recovery code", + "Recovery code copied.": "Recovery code copied.", + "The displayed recovery code belongs to the current key. Reopen this dialogue after saving to copy the replacement key.": + "The displayed recovery code belongs to the current key. Reopen this dialogue after saving to copy the replacement key.", + "The recovery code could not be copied. Select and copy the visible code instead.": + "The recovery code could not be copied. Select and copy the visible code instead.", + "The ID key is derived from the current E2EE passphrase and saved separately. Changing that passphrase later does not change the saved ID key. To reduce the risk of guessing that passphrase from known IDs, use a separate, unpredictable ID source instead.": + "The ID key is derived from the current E2EE passphrase and saved separately. Changing that passphrase later does not change the saved ID key. To reduce the risk of guessing that passphrase from known IDs, use a separate, unpredictable ID source instead.", + "An ID source is required to enable this option.": "An ID source is required to enable this option.", + "The ID source or recovery code is invalid. Check it and try again.": + "The ID source or recovery code is invalid. Check it and try again.", } as const; export type LiveSyncProvisionalMessageKey = keyof typeof liveSyncProvisionalEnglishMessages; diff --git a/src/common/replicatorConfigurationIdentity.ts b/src/common/replicatorConfigurationIdentity.ts index af72f6d6..03f10306 100644 --- a/src/common/replicatorConfigurationIdentity.ts +++ b/src/common/replicatorConfigurationIdentity.ts @@ -43,7 +43,10 @@ function projectHeaders(value: string): readonly (readonly [name: string, value: } function projectRemoteSecurity(settings: RemoteDBSettings) { - return settings.encrypt + return [ + settings.idDerivationVersion, + settings.idDerivationKey, + settings.encrypt ? ([ "encrypted", settings.passphrase, @@ -51,7 +54,8 @@ function projectRemoteSecurity(settings: RemoteDBSettings) { settings.E2EEAlgorithm, settings.permitEmptyPassphrase, ] as const) - : (["plain"] as const); + : (["plain"] as const), + ] as const; } /** diff --git a/src/common/replicatorConfigurationIdentity.unit.spec.ts b/src/common/replicatorConfigurationIdentity.unit.spec.ts index a5ef0f9b..f5a540b4 100644 --- a/src/common/replicatorConfigurationIdentity.unit.spec.ts +++ b/src/common/replicatorConfigurationIdentity.unit.spec.ts @@ -31,6 +31,17 @@ describe("active Replicator configuration identity", () => { }); } + it("replaces a connection when the independent ID key changes", () => { + const first = configuredSettings({ idDerivationVersion: 1, idDerivationKey: "a".repeat(64) }); + const second = { ...first, idDerivationKey: "b".repeat(64) }; + expect(getCouchDBReplicatorConfigurationIdentity(second)).not.toBe( + getCouchDBReplicatorConfigurationIdentity(first) + ); + expect(getObjectStorageReplicatorConfigurationIdentity(second)).not.toBe( + getObjectStorageReplicatorConfigurationIdentity(first) + ); + }); + it.each([ ["couchDB_URI", "https://other.example.test/base"], ["couchDB_DBNAME", "other-vault"], diff --git a/src/common/reportTool.ts b/src/common/reportTool.ts index ab174bb8..5c10cbf6 100644 --- a/src/common/reportTool.ts +++ b/src/common/reportTool.ts @@ -80,6 +80,8 @@ export async function generateReport(settings: ObsidianLiveSyncSettings, core: L pluginConfig.couchDB_USER = REDACTED; pluginConfig.passphrase = REDACTED; pluginConfig.encryptedPassphrase = REDACTED; + pluginConfig.idDerivationKey = REDACTED; + pluginConfig.encryptedIdDerivationKey = REDACTED; pluginConfig.encryptedCouchDBConnection = REDACTED; pluginConfig.accessKey = REDACTED; pluginConfig.secretKey = REDACTED; diff --git a/src/common/reportTool.unit.spec.ts b/src/common/reportTool.unit.spec.ts index 0fe7aa23..8436b313 100644 --- a/src/common/reportTool.unit.spec.ts +++ b/src/common/reportTool.unit.spec.ts @@ -10,6 +10,22 @@ vi.mock("@vrtmrz/livesync-commonlib/compat/common/coreEnvFunctions", () => ({ })); describe("TURN credentials in diagnostic reports", () => { + it("redacts the derived ID key and its encrypted local wrapper", async () => { + const key = "f3205cc41d24116d8c2484993c9d9a2e667373af338ba02f2ee71199adb82f2e"; + const wrapper = "encrypted-id-key-test-wrapper"; + const settings = { + ...DEFAULT_SETTINGS, + idDerivationVersion: 1 as const, + idDerivationKey: key, + encryptedIdDerivationKey: wrapper, + }; + const core = { services: { vault: { isStorageInsensitive: () => false } } } as unknown as LiveSyncBaseCore; + const report = await generateReport(settings, core); + const text = JSON.stringify(report); + expect(text).not.toContain(key); + expect(text).not.toContain(wrapper); + }); + it("redacts provider tokens in all profiles and runtime credentials", async () => { const token = "private+token/with=symbols"; const provider = { P2P_managedType: "CF", P2P_managedId: "private-key", P2P_managedToken: token }; diff --git a/src/features/ReviewHarness/reviewHarnessContract.ts b/src/features/ReviewHarness/reviewHarnessContract.ts index 29a76a8c..3925857d 100644 --- a/src/features/ReviewHarness/reviewHarnessContract.ts +++ b/src/features/ReviewHarness/reviewHarnessContract.ts @@ -1,9 +1,6 @@ import type { ObsidianLiveSyncSettings, SettingsMigrationState } from "@vrtmrz/livesync-commonlib/settings"; import type { CompatibilityPause } from "@/common/databaseCompatibility.ts"; -import type { - ReviewHarnessScenarioResult, - ReviewHarnessScenarioStatus, -} from "./reviewHarnessTypes"; +import type { ReviewHarnessScenarioResult, ReviewHarnessScenarioStatus } from "./reviewHarnessTypes"; export type { ReviewHarnessScenarioResult, ReviewHarnessScenarioStatus } from "./reviewHarnessTypes"; @@ -32,6 +29,14 @@ export const REVIEW_HARNESS_SCENARIOS = [ mode: "automatic", access: "dedicated-vault-fixtures", }, + { + id: "id-generation-performance", + title: "ID generation performance", + description: + "Measures legacy and independent IDs with fixed in-memory inputs. Reports time per 1,000 IDs and per ID, key derivation time, and JavaScript heap samples where available. Keep Obsidian in the foreground.", + mode: "automatic", + access: "read-only", + }, ] as const; export const REVIEW_HARNESS_SCENARIO_IDS = REVIEW_HARNESS_SCENARIOS.map(({ id }) => id); @@ -114,7 +119,9 @@ const NEW_VAULT_RECOMMENDATION_KEYS = [ "E2EEAlgorithm", ] as const; -type LifecycleSettingKey = (typeof PRESERVED_SYNC_SETTING_KEYS)[number] | (typeof NEW_VAULT_RECOMMENDATION_KEYS)[number]; +type LifecycleSettingKey = + | (typeof PRESERVED_SYNC_SETTING_KEYS)[number] + | (typeof NEW_VAULT_RECOMMENDATION_KEYS)[number]; type SettingsForLifecycleInspection = Partial>; export function inspectSettingsLifecycle(input: { @@ -130,9 +137,7 @@ export function inspectSettingsLifecycle(input: { }; } - const invalidSyncSettings = PRESERVED_SYNC_SETTING_KEYS.filter( - (key) => typeof input.settings[key] !== "boolean" - ); + const invalidSyncSettings = PRESERVED_SYNC_SETTING_KEYS.filter((key) => typeof input.settings[key] !== "boolean"); if (invalidSyncSettings.length > 0) { return { status: "failed", @@ -205,6 +210,7 @@ export interface ReviewHarnessReportScenario { readonly mode: ReviewHarnessScenarioMode; readonly status: ReviewHarnessScenarioStatus; readonly detail: string; + readonly observations?: readonly string[]; } export interface ReviewHarnessReportInput { @@ -248,13 +254,15 @@ export function formatReviewHarnessReport(input: ReviewHarnessReportInput): stri ); const scenarios = table( ["Scenario", "Mode", "Status", "Detail"], - input.scenarios.map(({ id, title, mode, status, detail }) => [ - `${title} (${id})`, - mode, - status, - detail, - ]) + input.scenarios.map(({ id, title, mode, status, detail }) => [`${title} (${id})`, mode, status, detail]) ); + const observations = input.scenarios + .filter((scenario) => scenario.observations?.length) + .map( + ({ title, observations }) => + `### ${title}\n\n${observations!.map((value) => `- ${tableCell(value)}`).join("\n")}` + ) + .join("\n\n"); return `## Self-hosted LiveSync Review Harness report Generated at \`${tableCell(input.generatedAt)}\`. @@ -267,6 +275,8 @@ ${environment} ${scenarios} +${observations} +
Event transcript diff --git a/src/features/ReviewHarness/reviewHarnessContract.unit.spec.ts b/src/features/ReviewHarness/reviewHarnessContract.unit.spec.ts index a24b53e1..bc33c775 100644 --- a/src/features/ReviewHarness/reviewHarnessContract.unit.spec.ts +++ b/src/features/ReviewHarness/reviewHarnessContract.unit.spec.ts @@ -75,6 +75,7 @@ describe("Review Harness contract", () => { "settings-lifecycle", "compatibility-review", "vault-round-trip", + "id-generation-performance", ]); }); diff --git a/src/features/ReviewHarness/reviewHarnessController.ts b/src/features/ReviewHarness/reviewHarnessController.ts index 7bfc755c..12c0e370 100644 --- a/src/features/ReviewHarness/reviewHarnessController.ts +++ b/src/features/ReviewHarness/reviewHarnessController.ts @@ -21,6 +21,7 @@ export interface ReviewHarnessRuntime { getCompatibilityPause(): CompatibilityPause | undefined; openCompatibilityReview(): Promise; runVaultRoundTrip(): Promise; + runIdBenchmark(): Promise; readContinuation(): string | null; writeContinuation(value: string): void; deleteContinuation(): void; @@ -159,6 +160,8 @@ export class ReviewHarnessController { }); } else if (id === "vault-round-trip") { result = await this.runtime.runVaultRoundTrip(); + } else if (id === "id-generation-performance") { + result = await this.runtime.runIdBenchmark(); } else { const inspection = this.inspectCompatibilityReview(); result = @@ -206,10 +209,7 @@ export class ReviewHarnessController { detail: "The device-local compatibility review remains pending.", observations: inspection.observations, }; - this.record( - "compatibility-review-updated", - this.results["compatibility-review"].status - ); + this.record("compatibility-review-updated", this.results["compatibility-review"].status); } catch (error) { this.setUnexpectedFailure("compatibility-review", error); } finally { @@ -259,6 +259,7 @@ export class ReviewHarnessController { mode, status: this.results[id].status, detail: this.results[id].detail, + observations: this.results[id].observations, })), transcript: this.transcript, }); diff --git a/src/features/ReviewHarness/reviewHarnessController.unit.spec.ts b/src/features/ReviewHarness/reviewHarnessController.unit.spec.ts index 04b07bee..dee43186 100644 --- a/src/features/ReviewHarness/reviewHarnessController.unit.spec.ts +++ b/src/features/ReviewHarness/reviewHarnessController.unit.spec.ts @@ -80,6 +80,11 @@ function createRuntime(): ReviewHarnessRuntime & { detail: "The owned fixture tree was exercised and removed.", observations: [], })), + runIdBenchmark: vi.fn(async () => ({ + status: "passed" as const, + detail: "ID generation measurements completed.", + observations: ["Chunk 256 B: 1000 IDs total=43.00 ms; per ID=0.0430 ms"], + })), readContinuation() { return this.continuation; }, @@ -150,6 +155,60 @@ describe("ReviewHarnessController", () => { expect(runtime.reportError).toHaveBeenCalledOnce(); }); + it("runs ID measurements on request and includes their units in the copied report", async () => { + const runtime = createRuntime(); + const controller = new ReviewHarnessController(runtime); + + await controller.runAutomaticScenarios(); + expect(runtime.runIdBenchmark).not.toHaveBeenCalled(); + + await controller.runScenario("id-generation-performance"); + await controller.copyReport(); + + expect(runtime.runIdBenchmark).toHaveBeenCalledOnce(); + expect(controller.snapshot().results["id-generation-performance"].status).toBe("passed"); + expect(vi.mocked(runtime.copyText).mock.calls[0][0]).toContain("1000 IDs total=43.00 ms; per ID=0.0430 ms"); + expect(runtime.runVaultRoundTrip).not.toHaveBeenCalled(); + expect(runtime.events).toEqual([]); + expect(runtime.continuation).toBeNull(); + }); + + it("excludes an unexpected measurement error from the copied report", async () => { + const runtime = createRuntime(); + runtime.runIdBenchmark = vi.fn().mockRejectedValue(new Error("private measurement error")); + const controller = new ReviewHarnessController(runtime); + + await controller.runScenario("id-generation-performance"); + + expect(controller.snapshot().results["id-generation-performance"].status).toBe("failed"); + expect(controller.createReport()).not.toContain("private measurement error"); + expect(runtime.reportError).toHaveBeenCalledOnce(); + }); + + it("does not overlap an ID measurement with another scenario", async () => { + const runtime = createRuntime(); + let finish!: () => void; + const pending = new Promise((resolve) => { + finish = resolve; + }); + runtime.runIdBenchmark = vi.fn(async () => { + await pending; + return { status: "passed" as const, detail: "Measured", observations: [] }; + }); + const controller = new ReviewHarnessController(runtime); + + const running = controller.runScenario("id-generation-performance"); + await controller.runScenario("id-generation-performance"); + await controller.runScenario("vault-round-trip"); + + expect(runtime.runIdBenchmark).toHaveBeenCalledOnce(); + expect(runtime.runVaultRoundTrip).not.toHaveBeenCalled(); + expect(controller.snapshot().running).toBe(true); + finish(); + await running; + expect(controller.snapshot().running).toBe(false); + }); + it("deletes a one-shot continuation before exposing the resumed guided step", () => { const runtime = createRuntime(); runtime.continuation = JSON.stringify({ @@ -167,9 +226,7 @@ describe("ReviewHarnessController", () => { expect(controller.snapshot().results["compatibility-review"]).toMatchObject({ status: "waiting-for-user", }); - expect(controller.snapshot().resumedRequestId).toBe( - "compatibility-review-2026-07-18T11:59:00.000Z" - ); + expect(controller.snapshot().resumedRequestId).toBe("compatibility-review-2026-07-18T11:59:00.000Z"); }); it("does not copy rejected continuation values into the report", () => { diff --git a/src/features/ReviewHarness/reviewHarnessIdBenchmark.ts b/src/features/ReviewHarness/reviewHarnessIdBenchmark.ts new file mode 100644 index 00000000..f9212724 --- /dev/null +++ b/src/features/ReviewHarness/reviewHarnessIdBenchmark.ts @@ -0,0 +1,109 @@ +import type { ReviewHarnessScenarioResult } from "./reviewHarnessTypes"; + +export interface IdBenchmarkOperations { + deriveKey(): Promise; + chunkId(piece: string, independent: boolean): Promise; + documentId(path: string, independent: boolean): Promise; +} + +type BenchmarkPerformance = Pick & { + readonly memory?: { readonly usedJSHeapSize: number }; +}; + +const ID_COUNT = 1000; +const SAMPLES = 3; +const BATCH_SIZE = 100; +const WARMUP_COUNT = 32; + +function readHeap(clock: BenchmarkPerformance): number | undefined { + try { + const bytes = clock.memory?.usedJSHeapSize; + return typeof bytes === "number" && Number.isFinite(bytes) && bytes >= 0 ? bytes : undefined; + } catch { + return undefined; + } +} + +function summary(samples: readonly number[]): string { + const sorted = [...samples].sort((a, b) => a - b); + return `median=${sorted[1].toFixed(2)} ms; range=${sorted[0].toFixed(2)}–${sorted[2].toFixed(2)} ms`; +} + +export async function runReviewHarnessIdBenchmark( + operations: IdBenchmarkOperations, + clock: BenchmarkPerformance = performance, + yieldControl: () => Promise = () => new Promise((resolve) => window.setTimeout(resolve, 0)) +): Promise { + const before = readHeap(clock); + let highest = before; + const sampleHeap = () => { + const value = readHeap(clock); + if (value !== undefined) highest = Math.max(highest ?? value, value); + return value; + }; + const observations = [ + "Fixed synthetic inputs; 3 samples, alternating legacy/independent order; 32 warm-up IDs per sample. Legacy Chunk algorithm: xxhash64.", + "Compute timings include input construction and awaited ID generation. Initialisation, warm-up, and pauses between batches are excluded. This does not measure a Rebuild or remote transfer.", + ]; + const derivationSamples: number[] = []; + for (let sample = 0; sample < SAMPLES; sample++) { + await yieldControl(); + const started = clock.now(); + await operations.deriveKey(); + derivationSamples.push(clock.now() - started); + sampleHeap(); + } + observations.push(`ID key derivation at save time: ${summary(derivationSamples)} per derivation.`); + + const cases = [ + ...[256, 4096, 32768].map((bytes) => { + const prefix = "r".repeat(bytes - 8); + return { + label: `Chunk IDs, ${bytes} B`, + run: (i: number, independent: boolean) => + operations.chunkId(prefix + i.toString(36).padStart(8, "0"), independent), + }; + }), + { + label: "Obfuscated document IDs", + run: (i: number, independent: boolean) => operations.documentId(`benchmark/path-${i}.md`, independent), + }, + ]; + for (const scenario of cases) { + const samples: [number[], number[]] = [[], []]; + for (let sample = 0; sample < SAMPLES; sample++) { + for (const independent of sample % 2 === 0 ? [false, true] : [true, false]) { + for (let i = 0; i < WARMUP_COUNT; i++) await scenario.run(i, independent); + let elapsed = 0; + for (let batch = 0; batch < ID_COUNT; batch += BATCH_SIZE) { + await yieldControl(); + const started = clock.now(); + for (let i = batch; i < batch + BATCH_SIZE; i++) await scenario.run(i, independent); + elapsed += clock.now() - started; + sampleHeap(); + } + samples[independent ? 1 : 0].push(elapsed); + } + } + for (const [index, values] of samples.entries()) { + const median = [...values].sort((a, b) => a - b)[1]; + observations.push( + `${scenario.label}, ${index === 0 ? "legacy" : "independent"}: ${ID_COUNT} IDs total ${summary(values)}; per ID=${(median / ID_COUNT).toFixed(4)} ms.` + ); + } + } + const after = sampleHeap(); + if (highest === undefined) { + observations.push("JavaScript heap: unavailable on this device."); + } else { + const mib = (bytes: number | undefined) => + bytes === undefined ? "unavailable" : `${(bytes / 1048576).toFixed(2)} MiB`; + observations.push( + `JavaScript heap: before=${mib(before)}; highest sampled=${mib(highest)}; after=${mib(after)}.` + ); + } + observations.push( + "Heap samples are approximate, may include other Obsidian work, and are affected by garbage collection. They are neither total app RAM nor a true peak." + ); + return { status: "passed", detail: "ID generation measurements completed.", observations }; +} diff --git a/src/features/ReviewHarness/reviewHarnessIdBenchmark.unit.spec.ts b/src/features/ReviewHarness/reviewHarnessIdBenchmark.unit.spec.ts new file mode 100644 index 00000000..2e68837a --- /dev/null +++ b/src/features/ReviewHarness/reviewHarnessIdBenchmark.unit.spec.ts @@ -0,0 +1,99 @@ +import { describe, expect, it } from "vitest"; +import { runReviewHarnessIdBenchmark, type IdBenchmarkOperations } from "./reviewHarnessIdBenchmark"; + +function fixture() { + let elapsed = 0; + let derivations = 0; + const chunkCounts = [0, 0]; + const documentCounts = [0, 0]; + const chunkSizes = new Set(); + const operations: IdBenchmarkOperations = { + deriveKey: () => { + derivations++; + elapsed += 42; + return Promise.resolve("private-derived-key"); + }, + chunkId: (piece, independent) => { + chunkCounts[independent ? 1 : 0]++; + chunkSizes.add(piece.length); + elapsed += independent ? 2 : 1; + return Promise.resolve("private-chunk-id"); + }, + documentId: (_path, independent) => { + documentCounts[independent ? 1 : 0]++; + elapsed += independent ? 4 : 3; + return Promise.resolve("private-document-id"); + }, + }; + return { + operations, + now: () => elapsed, + yieldControl: () => { + elapsed += 100; + return Promise.resolve(); + }, + counts: () => ({ derivations, chunkCounts, documentCounts, chunkSizes: [...chunkSizes] }), + }; +} + +describe("Review Harness ID measurements", () => { + it("reports totals and per-ID timings separately, excluding warm-up and cooperative pauses", async () => { + const f = fixture(); + const result = await runReviewHarnessIdBenchmark(f.operations, { now: f.now }, f.yieldControl); + const report = result.observations.join("\n"); + + expect(result.status).toBe("passed"); + expect(report).toContain("1000 IDs total median=1000.00 ms; range=1000.00–1000.00 ms; per ID=1.0000 ms"); + expect(report).toContain("1000 IDs total median=2000.00 ms; range=2000.00–2000.00 ms; per ID=2.0000 ms"); + expect(report).toContain("Obfuscated document IDs, legacy: 1000 IDs total median=3000.00 ms"); + expect(report).toContain("Obfuscated document IDs, independent: 1000 IDs total median=4000.00 ms"); + expect(report).toContain("ID key derivation at save time: median=42.00 ms"); + expect(report).toContain("JavaScript heap: unavailable on this device."); + expect(report).not.toContain("private-"); + expect(f.counts()).toEqual({ + derivations: 3, + chunkCounts: [9288, 9288], + documentCounts: [3096, 3096], + chunkSizes: [256, 4096, 32768], + }); + }); + + it("labels the highest sampled heap separately from total app RAM and allows a lower final sample", async () => { + const f = fixture(); + let reads = 0; + const clock = { + now: f.now, + get memory() { + return { usedJSHeapSize: (reads++ === 0 ? 2 : reads === 2 ? 5 : 1) * 1048576 }; + }, + }; + const result = await runReviewHarnessIdBenchmark(f.operations, clock, f.yieldControl); + + expect(result.observations).toContain( + "JavaScript heap: before=2.00 MiB; highest sampled=5.00 MiB; after=1.00 MiB." + ); + expect(result.observations.join("\n")).toContain("neither total app RAM nor a true peak"); + }); + + it.each([Number.NaN, Number.POSITIVE_INFINITY, -1, "throws"])( + "keeps timings usable when the heap API returns %s", + async (value) => { + const f = fixture(); + const result = await runReviewHarnessIdBenchmark( + f.operations, + { + now: f.now, + get memory() { + if (value === "throws") throw new Error("Heap API unavailable"); + return { usedJSHeapSize: value as number }; + }, + }, + f.yieldControl + ); + + expect(result.status).toBe("passed"); + expect(result.observations).toContain("JavaScript heap: unavailable on this device."); + expect(result.observations.join("\n")).not.toMatch(/NaN|Infinity|private-/u); + } + ); +}); diff --git a/src/features/ReviewHarness/reviewHarnessIdBenchmarkRuntime.ts b/src/features/ReviewHarness/reviewHarnessIdBenchmarkRuntime.ts new file mode 100644 index 00000000..f2ef7641 --- /dev/null +++ b/src/features/ReviewHarness/reviewHarnessIdBenchmarkRuntime.ts @@ -0,0 +1,35 @@ +import { DEFAULT_SETTINGS, deriveIdKey } from "@vrtmrz/livesync-commonlib/settings"; +import { path2id_base } from "@vrtmrz/livesync-commonlib/compat/string_and_binary/path"; +import type { FilePath } from "@vrtmrz/livesync-commonlib/compat/common/types"; +import { HashManager } from "@vrtmrz/livesync-commonlib/hashing"; +import type { IdBenchmarkOperations } from "./reviewHarnessIdBenchmark"; + +const FIXTURE_PASSPHRASE = "Self-hosted LiveSync ID benchmark passphrase"; +const FIXTURE_SOURCE = "Self-hosted LiveSync ID benchmark source"; +const FIXTURE_KEY = "ab".repeat(32); + +export async function createIdBenchmarkOperations(): Promise { + const managers: HashManager[] = []; + for (const independent of [false, true]) { + const settings = Object.freeze({ + ...DEFAULT_SETTINGS, + encrypt: true, + passphrase: FIXTURE_PASSPHRASE, + hashAlg: "xxhash64" as const, + idDerivationVersion: independent ? (1 as const) : (0 as const), + idDerivationKey: independent ? FIXTURE_KEY : "", + }); + // HashManager only reads currentSettings; this fixture has no storage or live service access. + const settingService = { currentSettings: () => settings } as HashManager["options"]["settingService"]; + const manager = new HashManager({ settingService }); + if (!(await manager.initialise())) throw new Error("The benchmark hash manager could not initialise."); + managers.push(manager); + } + return { + deriveKey: () => deriveIdKey(FIXTURE_SOURCE), + chunkId: (piece, independent) => managers[independent ? 1 : 0].computeHash(piece), + // Fixture paths are already normalised; use the same ID calculation as PathService. + documentId: (path, independent) => + path2id_base(path as FilePath, FIXTURE_PASSPHRASE, false, independent ? FIXTURE_KEY : undefined), + }; +} diff --git a/src/features/ReviewHarness/reviewHarnessIdBenchmarkRuntime.unit.spec.ts b/src/features/ReviewHarness/reviewHarnessIdBenchmarkRuntime.unit.spec.ts new file mode 100644 index 00000000..91a3c423 --- /dev/null +++ b/src/features/ReviewHarness/reviewHarnessIdBenchmarkRuntime.unit.spec.ts @@ -0,0 +1,38 @@ +import { describe, expect, it, vi } from "vitest"; +import { DEFAULT_SETTINGS } from "@vrtmrz/livesync-commonlib/settings"; +import { createIdBenchmarkOperations } from "./reviewHarnessIdBenchmarkRuntime"; + +describe("Review Harness benchmark implementation", () => { + it("uses the packaged legacy and independent algorithms with isolated fixed settings", async () => { + const originalDefaults = structuredClone(DEFAULT_SETTINGS); + const fetch = vi.spyOn(globalThis, "fetch").mockRejectedValue(new Error("Network access is forbidden")); + try { + const operations = await createIdBenchmarkOperations(); + const chunk = "r".repeat(256); + const legacy = await operations.chunkId(chunk, false); + const independent = await operations.chunkId(chunk, true); + + expect(legacy).toMatch(/^\+[0-9a-z]{1,13}$/u); + expect(independent).toMatch(/^\+[0-9a-f]{64}$/u); + expect(independent).toBe("+9223e53d99e80c29effee9e95e38ed168d13c14f717054f9e996a1cd0a597000"); + expect(await operations.chunkId(chunk, false)).toBe(legacy); + expect(await operations.chunkId(chunk, true)).toBe(independent); + expect(await operations.chunkId("s".repeat(256), true)).not.toBe(independent); + + const legacyPath = await operations.documentId("benchmark/path-1.md", false); + const independentPath = await operations.documentId("benchmark/path-1.md", true); + expect(legacyPath).toMatch(/^f:[0-9a-f]{64}$/u); + expect(independentPath).toMatch(/^f:[0-9a-f]{64}$/u); + expect(legacyPath).not.toBe(independentPath); + expect(await operations.documentId("benchmark/path-1.md", true)).toBe(independentPath); + + const second = await createIdBenchmarkOperations(); + expect(await second.chunkId(chunk, true)).toBe(independent); + expect(await operations.deriveKey()).toMatch(/^[0-9a-f]{64}$/u); + expect(fetch).not.toHaveBeenCalled(); + expect(DEFAULT_SETTINGS).toEqual(originalDefaults); + } finally { + fetch.mockRestore(); + } + }); +}); diff --git a/src/modules/coreFeatures/ModuleResolveMismatchedTweaks.ts b/src/modules/coreFeatures/ModuleResolveMismatchedTweaks.ts index 2fe44d06..c5889223 100644 --- a/src/modules/coreFeatures/ModuleResolveMismatchedTweaks.ts +++ b/src/modules/coreFeatures/ModuleResolveMismatchedTweaks.ts @@ -99,6 +99,17 @@ function resolutionSettingsSignature(settings: ObsidianLiveSyncSettings): string } export class ModuleResolvingMismatchedTweaks extends AbstractModule { + private requiresIdConfigurationReview(assessment: TweakAssessment): boolean { + if (!assessment.entries.some(({ key, relation }) => key === "idDerivationVersion" && relation === "different")) { + return false; + } + Logger( + "The document ID configurations differ. Import the correct Setup URI, or configure the matching ID key, before synchronising.", + LOG_LEVEL_NOTICE + ); + return true; + } + private _selectNewerTweakSide(current: TweakValues, preferred: Partial): "REMOTE" | "CURRENT" { Logger(`Modified: ${current.tweakModified} (current) vs ${preferred.tweakModified} (preferred)`); const currentModified = current.tweakModified; @@ -196,6 +207,7 @@ export class ModuleResolvingMismatchedTweaks extends AbstractModule { assessment = assessTweakCompatibility(this.settings, preferred) ): Promise<[TweakValues | boolean, boolean]> { if (assessment.alignment === "matched") return [false, false]; + if (this.requiresIdConfigurationReview(assessment)) return [false, false]; const acceptedSettings = settingsAfterAdoption(assessment, "adoptPreferred"); const autoAcceptSide = await this._shouldAutoAcceptCompatibleLossy(assessment); if (autoAcceptSide === "REMOTE") return [acceptedSettings, false]; @@ -363,6 +375,7 @@ export class ModuleResolvingMismatchedTweaks extends AbstractModule { const trialSignature = JSON.stringify(trialSetting); const currentSignature = resolutionSettingsSignature(this.settings); const assessment = assessTweakCompatibility(trialSetting, preferred); + if (this.requiresIdConfigurationReview(assessment)) return { result: false, requireFetch: false }; if (assessment.alignment === "matched") { this._log("The settings in the remote database are the same as the local database.", LOG_LEVEL_NOTICE); return { result: false, requireFetch: false }; diff --git a/src/modules/coreFeatures/ModuleResolveMismatchedTweaks.unit.spec.ts b/src/modules/coreFeatures/ModuleResolveMismatchedTweaks.unit.spec.ts index 2b641f8a..d0a02bb8 100644 --- a/src/modules/coreFeatures/ModuleResolveMismatchedTweaks.unit.spec.ts +++ b/src/modules/coreFeatures/ModuleResolveMismatchedTweaks.unit.spec.ts @@ -7,7 +7,7 @@ import { type TweakValues, } from "@vrtmrz/livesync-commonlib/compat/common/types"; import { extractObject } from "octagonal-wheels/object"; -import { assessTweakCompatibility } from "@vrtmrz/livesync-commonlib/settings"; +import { assessTweakCompatibility, configuredIdKey } from "@vrtmrz/livesync-commonlib/settings"; import { ModuleResolvingMismatchedTweaks } from "./ModuleResolveMismatchedTweaks"; import { setLang } from "@/common/translation"; import { @@ -74,6 +74,68 @@ function createModule(settingsOverride: Partial = {}) { } describe("ModuleResolvingMismatchedTweaks", () => { + it.each([0, 1] as const)( + "keeps ID configuration %s when automatically aligning Chunk settings", + async (idDerivationVersion) => { + const idDerivationKey = idDerivationVersion === 1 ? "ab".repeat(32) : ""; + const { module, core, askSelectStringDialogue } = createModule({ + encrypt: true, + usePathObfuscation: false, + idDerivationVersion, + idDerivationKey, + autoAcceptCompatibleTweak: true, + hashAlg: "xxhash64", + tweakModified: 1, + }); + const preferred: TweakValues = { + ...extractObject(TweakValuesTemplate, core.settings), + idDerivationVersion: idDerivationVersion === 1 ? 0 : 1, + hashAlg: "xxhash32", + tweakModified: 2, + }; + core._services.tweakValue = { + checkAndAskResolvingMismatched: module._checkAndAskResolvingMismatchedTweaks.bind(module), + }; + core._services.setting.saveSettingData.mockImplementation(async () => { + configuredIdKey(core.settings); + }); + + await expect(module._askResolvingMismatchedTweaks(preferred, async () => true)).resolves.toBe("CHECKAGAIN"); + + expect(core.settings).toMatchObject({ idDerivationVersion, idDerivationKey, hashAlg: "xxhash32" }); + expect(askSelectStringDialogue).not.toHaveBeenCalled(); + } + ); + + it.each(["active", "trial"] as const)( + "withholds ordinary tweak adoption for different document ID modes (%s)", + async (route) => { + const { module, core, askSelectStringDialogue } = createModule({ + encrypt: true, + usePathObfuscation: true, + idDerivationVersion: 0, + idDerivationKey: "", + }); + const preferred: TweakValues = { + ...extractObject(TweakValuesTemplate, core.settings), + idDerivationVersion: 1, + }; + + if (route === "active") { + await expect(module._checkAndAskResolvingMismatchedTweaks(preferred)).resolves.toEqual([false, false]); + } else { + await expect(module._askUseRemoteConfiguration(core.settings, preferred)).resolves.toEqual({ + result: false, + requireFetch: false, + }); + } + + expect(askSelectStringDialogue).not.toHaveBeenCalled(); + expect(core._services.setting.saveSettingData).not.toHaveBeenCalled(); + expect(core.settings).toMatchObject({ idDerivationVersion: 0, idDerivationKey: "" }); + } + ); + it("compatibility: offers ordinary application for a missing legacy filename-case setting", async () => { const { module, askSelectStringDialogue } = createModule({ autoAcceptCompatibleTweak: false, diff --git a/src/modules/features/ModuleObsidianSettingAsMarkdown.ts b/src/modules/features/ModuleObsidianSettingAsMarkdown.ts index b44e140c..bbbced4c 100644 --- a/src/modules/features/ModuleObsidianSettingAsMarkdown.ts +++ b/src/modules/features/ModuleObsidianSettingAsMarkdown.ts @@ -140,6 +140,8 @@ export class ModuleObsidianSettingsAsMarkdown extends AbstractModule { settingToApply.couchDB_USER = this.settings.couchDB_USER; settingToApply.couchDB_PASSWORD = this.settings.couchDB_PASSWORD; settingToApply.passphrase = this.settings.passphrase; + settingToApply.idDerivationVersion = this.settings.idDerivationVersion; + settingToApply.idDerivationKey = this.settings.idDerivationKey; } const oldSetting = this.generateSettingForMarkdown( this.settings, @@ -203,11 +205,13 @@ export class ModuleObsidianSettingsAsMarkdown extends AbstractModule { const saveData = { ...(settings ? settings : this.settings) } as Partial; delete saveData.encryptedCouchDBConnection; delete saveData.encryptedPassphrase; + delete saveData.encryptedIdDerivationKey; delete saveData.additionalSuffixOfDatabaseName; if (!saveData.writeCredentialsForSettingSync && !keepCredential) { delete saveData.couchDB_USER; delete saveData.couchDB_PASSWORD; delete saveData.passphrase; + delete saveData.idDerivationKey; delete saveData.jwtKey; delete saveData.jwtKid; delete saveData.jwtSub; diff --git a/src/modules/features/SettingDialogue/PaneRemoteConfig.ts b/src/modules/features/SettingDialogue/PaneRemoteConfig.ts index e193c986..61310e4d 100644 --- a/src/modules/features/SettingDialogue/PaneRemoteConfig.ts +++ b/src/modules/features/SettingDialogue/PaneRemoteConfig.ts @@ -47,6 +47,12 @@ function getSettingsFromEditingSettings(editingSettings: AllSettings): ObsidianL } return workObj; } + +function syncIdDerivationSettings(target: Partial, source: ObsidianLiveSyncSettings): void { + target.idDerivationVersion = source.idDerivationVersion; + target.idDerivationKey = source.idDerivationKey; +} + function createRemoteConfigurationId(): string { return `remote-${Date.now().toString(36)}-${Math.random().toString(36).slice(2, 8)}`; } @@ -116,6 +122,8 @@ export function paneRemoteConfig( .onClick(async () => { const setupManager = this.core.getModule(SetupManager); const originalSettings = getSettingsFromEditingSettings(this.editingSettings); + const originalIdDerivationVersion = this.core.settings.idDerivationVersion; + const originalIdDerivationKey = this.core.settings.idDerivationKey; const applied = await setupManager.onlyE2EEConfiguration(UserMode.Update, originalSettings); if (applied) { this.editingSettings.encryptInternalMetadata = @@ -126,6 +134,16 @@ export function paneRemoteConfig( } this.requestUpdate(); } + if ( + this.core.settings.idDerivationVersion !== originalIdDerivationVersion || + this.core.settings.idDerivationKey !== originalIdDerivationKey + ) { + syncIdDerivationSettings(this.editingSettings, this.core.settings); + if (this.initialSettings) { + syncIdDerivationSettings(this.initialSettings, this.core.settings); + } + this.requestUpdate(); + } updateE2EESummary(); }) .setButtonText("Configure") @@ -164,9 +182,11 @@ export function paneRemoteConfig( const currentConfigs = cloneRemoteConfigurations(this.core.settings.remoteConfigurations); this.editingSettings.remoteConfigurations = currentConfigs; this.editingSettings.activeConfigurationId = this.core.settings.activeConfigurationId; + syncIdDerivationSettings(this.editingSettings, this.core.settings); if (this.initialSettings) { this.initialSettings.remoteConfigurations = cloneRemoteConfigurations(currentConfigs); this.initialSettings.activeConfigurationId = this.core.settings.activeConfigurationId; + syncIdDerivationSettings(this.initialSettings, this.core.settings); } }; const persistRemoteConfigurations = async (synchroniseActiveRemote: boolean = false) => { @@ -254,6 +274,8 @@ export function paneRemoteConfig( usePathObfuscation: this.editingSettings.usePathObfuscation, encryptInternalMetadata: this.editingSettings.encryptInternalMetadata, passphrase: this.editingSettings.passphrase, + idDerivationVersion: this.editingSettings.idDerivationVersion, + idDerivationKey: this.editingSettings.idDerivationKey, configPassphraseStore: this.editingSettings.configPassphraseStore, }); const addRemoteConfiguration = async () => { diff --git a/src/modules/features/SettingDialogue/PaneRemoteConfig.unit.spec.ts b/src/modules/features/SettingDialogue/PaneRemoteConfig.unit.spec.ts index efa79c83..4d83048f 100644 --- a/src/modules/features/SettingDialogue/PaneRemoteConfig.unit.spec.ts +++ b/src/modules/features/SettingDialogue/PaneRemoteConfig.unit.spec.ts @@ -194,4 +194,51 @@ describe("paneRemoteConfig", () => { expect(host.initialSettings.encryptInternalMetadata).toBe(true); expect(host.requestUpdate).toHaveBeenCalledOnce(); }); + + it("copies applied ID derivation settings into both dialogue buffers", async () => { + const nextIdKey = "ab".repeat(32); + const originalSettings = { + encrypt: true, + passphrase: "passphrase", + E2EEAlgorithm: "v2", + usePathObfuscation: true, + encryptInternalMetadata: false, + idDerivationVersion: 0, + idDerivationKey: "", + remoteConfigurations: {}, + }; + const setupManager = { + onlyE2EEConfiguration: vi.fn(() => { + host.core.settings.idDerivationVersion = 1; + host.core.settings.idDerivationKey = nextIdKey; + return Promise.resolve(false); + }), + }; + const host = { + editingSettings: { ...originalSettings }, + initialSettings: { ...originalSettings }, + core: { + settings: { ...originalSettings }, + getModule: vi.fn(() => setupManager), + }, + lifetimeComponent: { register: vi.fn() }, + requestUpdate: vi.fn(), + }; + const addPanel = vi.fn((_parent: HTMLElement, heading: string) => ({ + then(callback: (paneEl: HTMLElement) => void) { + if (heading === "E2EE Configuration") { + callback(createPanelElement()); + } + }, + })); + + paneRemoteConfig.call(host as never, {} as HTMLElement, { addPanel } as never); + await runtime.clickHandlers[0](); + + expect(host.editingSettings.idDerivationVersion).toBe(1); + expect(host.editingSettings.idDerivationKey).toBe(nextIdKey); + expect(host.initialSettings.idDerivationVersion).toBe(1); + expect(host.initialSettings.idDerivationKey).toBe(nextIdKey); + expect(host.requestUpdate).toHaveBeenCalledOnce(); + }); }); diff --git a/src/modules/features/SettingDialogue/settingUtils.ts b/src/modules/features/SettingDialogue/settingUtils.ts index 8fc43895..debcab56 100644 --- a/src/modules/features/SettingDialogue/settingUtils.ts +++ b/src/modules/features/SettingDialogue/settingUtils.ts @@ -68,6 +68,7 @@ export function getE2EEConfigSummary(setting: ObsidianLiveSyncSettings, showAdva export function getSummaryFromPartialSettings(setting: Partial, showAdvanced = false) { const outputSummary: Record = {}; for (const key of Object.keys(setting) as (keyof ObsidianLiveSyncSettings)[]) { + if (key === "idDerivationKey" || key === "encryptedIdDerivationKey") continue; const config = getConfig(key as AllSettingItemKey); if (!config) continue; if (config.isAdvanced && !showAdvanced) continue; diff --git a/src/modules/features/SetupManager.ts b/src/modules/features/SetupManager.ts index 3dfc3802..d4adff20 100644 --- a/src/modules/features/SetupManager.ts +++ b/src/modules/features/SetupManager.ts @@ -1,6 +1,5 @@ import { type BucketSyncSetting, - type EncryptionSettings, type ObsidianLiveSyncSettings, type P2PSyncSetting, LOG_LEVEL_NOTICE, @@ -36,6 +35,7 @@ import type { SetupRemoteCouchDBResultType, SetupRemoteCouchDBInitialData, SetupRemoteE2EEResultType, + SetupRemoteE2EEInitialData, SetupRemoteP2PInitialData, SetupRemoteP2PResultType, SetupRemoteResultType, @@ -58,6 +58,20 @@ function copySettingsForRemoteProfileUpdate(settings: ObsidianLiveSyncSettings): }; } +function normaliseImportedIdDerivationSettings(settings: ObsidianLiveSyncSettings): ObsidianLiveSyncSettings { + // Setup URIs are complete imports even when their encoder omitted default-valued fields. + // Fill each missing half so a receiving device cannot supply the unrelated saved key. + return { + ...settings, + idDerivationVersion: Object.prototype.hasOwnProperty.call(settings, "idDerivationVersion") + ? settings.idDerivationVersion + : 0, + idDerivationKey: Object.prototype.hasOwnProperty.call(settings, "idDerivationKey") + ? settings.idDerivationKey + : "", + }; +} + /** * User modes for onboarding and setup */ @@ -219,7 +233,7 @@ export class SetupManager extends AbstractModule { return false; } this._log("Setup URI dialog closed.", LOG_LEVEL_VERBOSE); - return await this.onConfirmApplySettingsFromWizard(newSetting, userMode); + return await this.onConfirmApplySettingsFromWizard(normaliseImportedIdDerivationSettings(newSetting), userMode); } /** @@ -328,9 +342,12 @@ export class SetupManager extends AbstractModule { * @returns */ async onlyE2EEConfiguration(userMode: UserMode, currentSetting: ObsidianLiveSyncSettings): Promise { - const e2eeConf = await this.dialogManager.openWithExplicitCancel( + const e2eeConf = await this.dialogManager.openWithExplicitCancel< + SetupRemoteE2EEResultType, + SetupRemoteE2EEInitialData + >( SetupRemoteE2EE, - currentSetting + { settings: currentSetting, newVault: userMode === UserMode.NewUser } ); if (e2eeConf === "cancelled") { this._log("E2EE configuration cancelled.", LOG_LEVEL_NOTICE); @@ -341,7 +358,9 @@ export class SetupManager extends AbstractModule { currentSetting.encrypt === e2eeConf.encrypt && currentSetting.passphrase === e2eeConf.passphrase && currentSetting.E2EEAlgorithm === e2eeConf.E2EEAlgorithm && - currentSetting.usePathObfuscation === e2eeConf.usePathObfuscation; + currentSetting.usePathObfuscation === e2eeConf.usePathObfuscation && + currentSetting.idDerivationVersion === e2eeConf.idDerivationVersion && + currentSetting.idDerivationKey === e2eeConf.idDerivationKey; if (userMode === UserMode.Update && onlyInternalMetadataPreferenceChanged) { if (e2eeConf.encryptInternalMetadata && currentSetting.remoteType === REMOTE_COUCHDB) { const proceed = "Enable without rebuilding — update every other device first"; @@ -375,9 +394,12 @@ export class SetupManager extends AbstractModule { * @returns */ async onConfigureManually(originalSetting: ObsidianLiveSyncSettings, userMode: UserMode): Promise { - const e2eeConf = await this.dialogManager.openWithExplicitCancel( + const e2eeConf = await this.dialogManager.openWithExplicitCancel< + SetupRemoteE2EEResultType, + SetupRemoteE2EEInitialData + >( SetupRemoteE2EE, - originalSetting + { settings: originalSetting, newVault: userMode === UserMode.NewUser } ); if (e2eeConf === "cancelled") { this._log("Manual configuration cancelled.", LOG_LEVEL_NOTICE); @@ -521,7 +543,13 @@ export class SetupManager extends AbstractModule { * @returns Promise that resolves to true if settings applied successfully, false otherwise */ async decodeQR(qr: string) { - const newSettings = decodeSettingsFromQRCodeData(qr); + let newSettings: ObsidianLiveSyncSettings; + try { + newSettings = normaliseImportedIdDerivationSettings(decodeSettingsFromQRCodeData(qr)); + } catch { + this._log("The QR configuration could not be decoded or contains unsupported settings.", LOG_LEVEL_NOTICE); + return false; + } return await this.onConfirmApplySettingsFromWizard(newSettings, UserMode.Unknown); } diff --git a/src/modules/features/SetupManager.unit.spec.ts b/src/modules/features/SetupManager.unit.spec.ts index c55c0354..c8dfe4ec 100644 --- a/src/modules/features/SetupManager.unit.spec.ts +++ b/src/modules/features/SetupManager.unit.spec.ts @@ -193,6 +193,58 @@ describe("SetupManager", () => { expect(setting.currentSettings().activeConfigurationId).toBe("legacy-couchdb"); }); + it("compatibility: treats omitted ID derivation fields in a Setup URI as legacy defaults", async () => { + const { manager, setting, dialogManager } = createSetupManager(); + const savedKey = "12".repeat(32); + setting.settings = { + ...createLegacyRemoteSetting(), + isConfigured: true, + idDerivationVersion: 1, + idDerivationKey: savedKey, + }; + const imported = { + ...createLegacyRemoteSetting(), + isConfigured: true, + } as Partial; + delete imported.idDerivationVersion; + delete imported.idDerivationKey; + vi.spyOn(setting, "adjustSettings").mockImplementation((settings) => Promise.resolve(settings)); + dialogManager.openWithExplicitCancel.mockResolvedValueOnce(imported).mockResolvedValueOnce("cancelled"); + + await manager.onUseSetupURI(UserMode.Unknown, "mock-config://legacy-settings"); + + const mergedSettings = vi.mocked(setting.adjustSettings).mock.calls[0][0]; + expect(mergedSettings.idDerivationVersion).toBe(0); + expect(mergedSettings.idDerivationKey).toBe(""); + expect(setting.currentSettings().idDerivationKey).toBe(savedKey); + }); + + it("does not inherit the missing half of a partially present Setup URI ID configuration", async () => { + const { manager, setting, dialogManager } = createSetupManager(); + const savedKey = "34".repeat(32); + setting.settings = { + ...createLegacyRemoteSetting(), + isConfigured: true, + idDerivationVersion: 1, + idDerivationKey: savedKey, + }; + const imported = { + ...createLegacyRemoteSetting(), + isConfigured: true, + idDerivationVersion: 1, + } as Partial; + delete imported.idDerivationKey; + vi.spyOn(setting, "adjustSettings").mockImplementation((settings) => Promise.resolve(settings)); + dialogManager.openWithExplicitCancel.mockResolvedValueOnce(imported).mockResolvedValueOnce("cancelled"); + + await manager.onUseSetupURI(UserMode.Unknown, "mock-config://partial-settings"); + + const mergedSettings = vi.mocked(setting.adjustSettings).mock.calls[0][0]; + expect(mergedSettings.idDerivationVersion).toBe(1); + expect(mergedSettings.idDerivationKey).toBe(""); + expect(setting.currentSettings().idDerivationKey).toBe(savedKey); + }); + it("compatibility: normalises imported flat remote settings from QR data before applying", async () => { const { manager, setting, dialogManager } = createSetupManager(); vi.mocked(decodeSettingsFromQRCodeData).mockReturnValue(createLegacyRemoteSetting()); @@ -208,6 +260,79 @@ describe("SetupManager", () => { expect(setting.currentSettings().activeConfigurationId).toBe("legacy-couchdb"); }); + it("compatibility: applies legacy defaults when QR data omits ID derivation fields", async () => { + const { manager, setting, dialogManager } = createSetupManager(); + const savedKey = "56".repeat(32); + setting.settings = { + ...createLegacyRemoteSetting(), + isConfigured: true, + idDerivationVersion: 1, + idDerivationKey: savedKey, + }; + const imported = { ...createLegacyRemoteSetting(), isConfigured: true } as Partial; + delete imported.idDerivationVersion; + delete imported.idDerivationKey; + vi.mocked(decodeSettingsFromQRCodeData).mockReturnValue(imported as ObsidianLiveSyncSettings); + vi.spyOn(setting, "adjustSettings").mockImplementation((settings) => Promise.resolve(settings)); + dialogManager.openWithExplicitCancel.mockResolvedValueOnce("cancelled"); + + await manager.decodeQR("qr-data"); + + const mergedSettings = vi.mocked(setting.adjustSettings).mock.calls[0][0]; + expect(mergedSettings.idDerivationVersion).toBe(0); + expect(mergedSettings.idDerivationKey).toBe(""); + expect(setting.currentSettings().idDerivationKey).toBe(savedKey); + }); + + it("rejects invalid QR settings before applying them", async () => { + const { manager, setting } = createSetupManager(); + vi.mocked(decodeSettingsFromQRCodeData).mockImplementationOnce(() => { + throw new Error("Invalid ID derivation key"); + }); + const applyExternalSettings = vi.spyOn(setting, "applyExternalSettings"); + + await expect(manager.decodeQR("invalid-qr")).resolves.toBe(false); + expect(applyExternalSettings).not.toHaveBeenCalled(); + }); + + it("requires the normal Fetch choice when ID derivation changes with the Metadata preference", async () => { + const { manager, setting, dialogManager, core } = createSetupManager(); + const currentSettings: ObsidianLiveSyncSettings = { + ...createLegacyRemoteSetting(), + isConfigured: true, + encrypt: true, + passphrase: "e2ee-passphrase", + usePathObfuscation: true, + encryptInternalMetadata: false, + idDerivationVersion: 0, + idDerivationKey: "", + }; + const nextIdKey = "78".repeat(32); + setting.settings = currentSettings; + const applyPartial = vi.spyOn(setting, "applyPartial"); + core.confirm = { + askSelectStringDialogue: vi.fn(() => + Promise.resolve("Enable without rebuilding — update every other device first") + ), + }; + dialogManager.openWithExplicitCancel + .mockResolvedValueOnce({ + ...currentSettings, + encryptInternalMetadata: true, + idDerivationVersion: 1, + idDerivationKey: nextIdKey, + }) + .mockResolvedValueOnce("existing-user") + .mockResolvedValueOnce("apply"); + + await manager.onlyE2EEConfiguration(UserMode.Update, currentSettings); + + expect(applyPartial).not.toHaveBeenCalled(); + expect(core.rebuilder.scheduleFetch).toHaveBeenCalledWith(expect.any(Function)); + expect(setting.currentSettings().idDerivationVersion).toBe(1); + expect(setting.currentSettings().idDerivationKey).toBe(nextIdKey); + }); + it("reserves Rebuild before saving a new-user configuration", async () => { const { manager, setting, dialogManager, core } = createSetupManager(); setting.settings = { ...setting.currentSettings(), isConfigured: false }; diff --git a/src/modules/features/SetupWizard/dialogs/SetupRemoteE2EE.svelte b/src/modules/features/SetupWizard/dialogs/SetupRemoteE2EE.svelte index dff27936..0a06a719 100644 --- a/src/modules/features/SetupWizard/dialogs/SetupRemoteE2EE.svelte +++ b/src/modules/features/SetupWizard/dialogs/SetupRemoteE2EE.svelte @@ -13,13 +13,26 @@ E2EEAlgorithms, type EncryptionSettings, } from "@vrtmrz/livesync-commonlib/compat/common/types"; + import { + deriveIdKey, + deriveOrImportIdKey, + formatIdRecoveryCode, + ID_DERIVATION_VERSION, + ID_RECOVERY_CODE_PREFIX, + } from "@vrtmrz/livesync-commonlib/settings"; import { onMount } from "svelte"; import type { GuestDialogProps } from "@/modules/services/LiveSyncUI/svelteDialog"; import { copyTo, pickEncryptionSettings } from "@vrtmrz/livesync-commonlib/compat/common/utils"; - import { TYPE_CANCELLED, type SetupRemoteE2EEResultType } from "./setupDialogTypes"; + import { + TYPE_CANCELLED, + type SetupRemoteE2EEInitialData, + type SetupRemoteE2EEResultType, + } from "./setupDialogTypes"; import { $msg as translateMessage } from "@/common/translation"; - type Props = GuestDialogProps; + type Props = GuestDialogProps; + type IdConfigurationChoice = "keep" | "random" | "custom"; + type IdCustomChoice = "passphrase" | "source" | "recovery"; const { setResult, getInitialData }: Props = $props(); let default_encryption: EncryptionSettings = { encrypt: true, @@ -27,17 +40,37 @@ E2EEAlgorithm: DEFAULT_SETTINGS.E2EEAlgorithm, usePathObfuscation: true, encryptInternalMetadata: true, + idDerivationVersion: 0, + idDerivationKey: "", }; let encryptionSettings = $state({ ...default_encryption }); + let newVault = $state(false); + let idConfigurationChoice = $state("keep"); + let idCustomChoice = $state("source"); + let idDerivationSource = $state(""); + let idDerivationError = $state(""); + let recoveryCodeVisible = $state(false); + let recoveryCodeCopied = $state(false); + + const idDerivationConfigured = $derived( + encryptionSettings.idDerivationVersion === ID_DERIVATION_VERSION && + typeof encryptionSettings.idDerivationKey === "string" && + encryptionSettings.idDerivationKey.length > 0 + ); + const recoveryCode = $derived.by(() => + idDerivationConfigured ? formatIdRecoveryCode(encryptionSettings.idDerivationKey) : "" + ); onMount(() => { if (getInitialData) { const initialData = getInitialData(); if (initialData) { - copyTo(initialData, encryptionSettings); + copyTo(initialData.settings, encryptionSettings); + newVault = initialData.newVault; } } + idConfigurationChoice = !idDerivationConfigured && newVault ? "random" : "keep"; }); let e2eeValid = $derived.by(() => { if (!encryptionSettings.encrypt) return true; @@ -49,8 +82,75 @@ encryptionSettings.usePathObfuscation ); - function commit() { - setResult(pickEncryptionSettings(encryptionSettings)); + function resetIdDerivationSource() { + idDerivationSource = ""; + idDerivationError = ""; + } + + function toggleEncryption(enabled: boolean) { + encryptionSettings.encrypt = enabled; + if (!enabled) resetIdDerivationSource(); + } + + function selectIdConfiguration() { + recoveryCodeVisible = false; + recoveryCodeCopied = false; + resetIdDerivationSource(); + } + + function selectIdCustomSource() { + resetIdDerivationSource(); + } + + async function copyRecoveryCode() { + try { + await navigator.clipboard.writeText(recoveryCode); + recoveryCodeCopied = true; + } catch { + idDerivationError = translateMessage("The recovery code could not be copied. Select and copy the visible code instead."); + } + } + + async function commit() { + idDerivationError = ""; + const result = pickEncryptionSettings(encryptionSettings); + + if (encryptionSettings.encrypt && idConfigurationChoice !== "keep") { + let source = idDerivationSource; + if (idConfigurationChoice === "random") { + const bytes = crypto.getRandomValues(new Uint8Array(32)); + source = Array.from(bytes, (byte) => byte.toString(16).padStart(2, "0")).join(""); + } else if (idCustomChoice === "passphrase") { + source = encryptionSettings.passphrase; + } + if (source.length === 0) { + if (!idDerivationConfigured) { + idDerivationError = translateMessage("An ID source is required to enable this option."); + return; + } + } else { + try { + result.idDerivationKey = + idConfigurationChoice === "custom" && idCustomChoice !== "passphrase" + ? await importOrDeriveEnteredIdKey(source, idCustomChoice) + : await deriveIdKey(source); + result.idDerivationVersion = ID_DERIVATION_VERSION; + } catch { + idDerivationError = translateMessage("The ID source or recovery code is invalid. Check it and try again."); + return; + } + } + } + + idDerivationSource = ""; + setResult(result); + } + + async function importOrDeriveEnteredIdKey(source: string, choice: IdCustomChoice): Promise { + if (choice === "recovery" && !source.trim().startsWith(ID_RECOVERY_CODE_PREFIX)) { + throw new Error("An ID recovery code is required."); + } + return await deriveOrImportIdKey(source); } @@ -58,7 +158,11 @@ {translateMessage("Please configure your end-to-end encryption settings.")} - + toggleEncryption(event.currentTarget.checked)} + /> {translateMessage( @@ -93,6 +197,164 @@ {/if} +
+ {translateMessage("ID generation")} + + + +
+ {#if encryptionSettings.encrypt && idConfigurationChoice === "keep" && !idDerivationConfigured} + + {translateMessage("Changing the E2EE passphrase changes IDs generated by the legacy configuration.")} + + {/if} + {#if (encryptionSettings.encrypt && idConfigurationChoice !== "keep") || idDerivationConfigured} + {#if encryptionSettings.encrypt} + + {translateMessage( + "This uses a saved key for new Chunk IDs and obfuscated Metadata document IDs, so changing the E2EE passphrase does not derive a new key automatically." + )} + + {/if} + {#if idDerivationConfigured} + + {translateMessage("The saved ID key is configured. Its source cannot be shown again.")} + + + {#if recoveryCodeVisible} + + + + + {#if recoveryCodeCopied} + {translateMessage("Recovery code copied.")} + {/if} + {/if} + {/if} + {#if encryptionSettings.encrypt} + {#if idConfigurationChoice === "custom"} +
+ {translateMessage("How to set the ID key")} + + + +
+ {#if idCustomChoice === "source" || idCustomChoice === "recovery"} + + + + {/if} + {/if} + {#if idDerivationConfigured && idConfigurationChoice !== "keep"} + + {translateMessage("The displayed recovery code belongs to the current key. Reopen this dialogue after saving to copy the replacement key.")} + + {/if} + {#if idConfigurationChoice === "custom" && idCustomChoice === "source"} + + {translateMessage("Choose a long, unpredictable source. It is used once and cannot be shown again after saving. A recovery code can be displayed on this device later. This input also accepts a tagged recovery code.")} + + {:else if idConfigurationChoice === "custom" && idCustomChoice === "recovery"} + + {translateMessage("Paste a tagged recovery code from an existing device to restore the same ID key.")} + + {:else if idConfigurationChoice === "random"} + + {translateMessage("For recovery after losing every device, save the recovery code after setup or choose an ID source you can reproduce.")} + + {:else if idConfigurationChoice === "custom" && idCustomChoice === "passphrase"} + + {translateMessage( + "The ID key is derived from the current E2EE passphrase and saved separately. Changing that passphrase later does not change the saved ID key. To reduce the risk of guessing that passphrase from known IDs, use a separate, unpredictable ID source instead." + )} + + {/if} + {#if idDerivationConfigured && idConfigurationChoice === "custom" && idCustomChoice !== "passphrase"} + {translateMessage("Leave this input empty to keep the saved ID key.")} + {/if} + {/if} + {idDerivationError} + {/if} + diff --git a/src/modules/features/SetupWizard/dialogs/setupDialogTypes.ts b/src/modules/features/SetupWizard/dialogs/setupDialogTypes.ts index 2c9ef58e..816fc8c9 100644 --- a/src/modules/features/SetupWizard/dialogs/setupDialogTypes.ts +++ b/src/modules/features/SetupWizard/dialogs/setupDialogTypes.ts @@ -110,6 +110,10 @@ export type SetupRemoteResultType = typeof TYPE_COUCHDB | typeof TYPE_BUCKET | t export type UseSetupURIResultType = typeof TYPE_CANCELLED | ObsidianLiveSyncSettings; export type SetupRemoteE2EEResultType = typeof TYPE_CANCELLED | EncryptionSettings; +export type SetupRemoteE2EEInitialData = { + settings: EncryptionSettings; + newVault: boolean; +}; export type SetupRemoteBucketResultType = typeof TYPE_CANCELLED | BucketSyncSetting; diff --git a/src/serviceFeatures/useReviewHarness.ts b/src/serviceFeatures/useReviewHarness.ts index 2f9c9791..cb56eb93 100644 --- a/src/serviceFeatures/useReviewHarness.ts +++ b/src/serviceFeatures/useReviewHarness.ts @@ -15,6 +15,8 @@ import { runReviewHarnessVaultRoundTrip, } from "@/features/ReviewHarness/reviewHarnessVaultFixture"; import type { CompatibilityReviewController } from "./compatibilityReview"; +import { runReviewHarnessIdBenchmark } from "@/features/ReviewHarness/reviewHarnessIdBenchmark"; +import { createIdBenchmarkOperations } from "@/features/ReviewHarness/reviewHarnessIdBenchmarkRuntime"; async function runVaultRoundTrip(plugin: ObsidianLiveSyncPlugin): Promise { const vault = plugin.app.vault; @@ -58,6 +60,8 @@ export function useReviewHarness( getCompatibilityPause: () => compatibilityReview.pendingPause, openCompatibilityReview: () => compatibilityReview.openReview(), runVaultRoundTrip: () => runVaultRoundTrip(plugin), + runIdBenchmark: async () => + runReviewHarnessIdBenchmark(await createIdBenchmarkOperations(), activeWindow.performance), readContinuation: () => services.setting.getSmallConfig(REVIEW_HARNESS_STATE_KEY), writeContinuation: (value) => services.setting.setSmallConfig(REVIEW_HARNESS_STATE_KEY, value), deleteContinuation: () => services.setting.deleteSmallConfig(REVIEW_HARNESS_STATE_KEY), diff --git a/test/e2e-obsidian/README.md b/test/e2e-obsidian/README.md index 39128c1b..e5aea396 100644 --- a/test/e2e-obsidian/README.md +++ b/test/e2e-obsidian/README.md @@ -129,6 +129,8 @@ The mobile pass uses Obsidian's `app.emulateMobile(true)`, a 390 by 844 CSS-pixe `test:e2e:obsidian:review-harness` exercises only the boundaries owned by the opt-in maintainer Harness. It retains a real compatibility pause, uses the fixed Harness restart action to persist a device-local continuation and reload Obsidian, and requires the Harness to delete that state before reopening. It also runs the bounded settings-lifecycle observation, confirms the dedicated Vault fixture root is removed, captures the copied privacy-bounded Markdown report, and checks the Harness layout and touch targets in mobile test mode. Compatibility explanation and persistence details remain owned by `settings-ui`, real P2P transfer remains owned by the dedicated P2P suites, and general Vault reflection remains owned by `vault-reflection`; the Harness test does not duplicate those workflows. +The Harness also measures ID generation with fixed in-memory data on desktop and mobile displays, checks that live settings remain unchanged, and verifies that the copied report includes both per-1,000-ID and per-ID timings, key derivation, and JavaScript heap availability. Mobile test mode verifies the UI and execution path; measure native device performance by running the same Harness on that device. + `test:e2e:obsidian:p2p-pane` starts one configured CouchDB-only session with no P2P profile and separate configured P2P sessions for desktop and mobile. It proves that the command remains registered while the retired command, automatic pane, and ribbon entry without a P2P configuration are absent. For the configured P2P profiles, it verifies that the desktop ribbon is available, the current status command reaches the pane without it opening at start-up, checks its connection control and horizontal layout, and captures unobstructed desktop and mobile screenshots. The mobile session uses a fresh Vault, profile, and Obsidian process, enters `app.emulateMobile(true)` through `lifecycle.beforePluginStart`, and requires the P2P view to belong to the right drawer rather than inheriting desktop workspace state. It deliberately uses no relay or peer: replacement of the active replicator is covered by focused unit tests, the Deno and Compose CLI P2P lifecycle suite covers the headless transport, and `p2p-setup-uri-workflow` owns the visible transfer path between two real Obsidian sessions. `test:e2e:obsidian:local-suite` builds the plug-in and, unless `LIVESYNC_CLI_COMMAND` selects an external CLI, the local LiveSync CLI. It then runs discovery, smoke, the onboarding invitation, Svelte dialogue mounting, revision repair, settings UI, the Review Harness, the P2P status pane, Vault reflection, CouchDB upload and manual setup, CLI-to-Obsidian synchronisation, Object Storage upload and Setup URI round-trip, P2P Setup URI round-trip, startup scan, provisioned CouchDB Setup URI, two-vault synchronisation, Hidden File Sync, Customisation Sync, and setting Markdown export in sequence. Start the local CouchDB, RustFS, and P2P relay fixtures before running it, or use `test:e2e:obsidian:local-suite:services` to let the wrapper stop leftover fixtures, start fresh fixtures, and stop them again after the run. @@ -139,10 +141,16 @@ The same workflow checks the two remote-activity status boundaries. It first hol `test:e2e:obsidian:couchdb-manual-setup-workflow` follows the visible onboarding path for the first device when no Setup URI is available. It enters end-to-end encryption and CouchDB details, runs the read-only `Check server requirements` step, requires the prepared fixture to pass without applying a server fix, and lets the onboarding connection test create the named database. After Rebuild completes on the first device, it creates an ordinary note, asks that working device to generate a Setup URI for a second device, completes Fetch there, and verifies a bidirectional note round-trip. The workflow captures each decision point and the expanded server-check result; password controls remain visually masked. It uses an E2EE passphrase beginning with `%`, confirms that the saved settings do not contain it in plain text, and checks that Obsidian restores it after restarting with the first Vault. +The ordinary workflow now checks that all three ID-configuration radio choices are visible, disabled and dimmed while E2EE is off, and fully visible when it is enabled. It also checks that the random key is selected by default for a new Vault, **Keep current configuration** shows its legacy explanation, and the saved key is encrypted locally and transferred by Setup URI. A screenshot of the disabled group is saved as `guide-couchdb-manual-id-generation-disabled.png`. Set `E2E_OBSIDIAN_INDEPENDENT_IDS=true` for the same visible workflow with an explicitly entered, randomly generated source. That variant checks all three nested radio choices, requires a source when no key is saved, retains the saved key when a custom source is empty, rejects an ordinary string in the recovery-code input, restores the same key from a tagged code, verifies that the source is absent from local settings, and checks that both devices compute the same obfuscated document IDs after Setup URI import and Fast Fetch. + If this status workflow fails while Obsidian is running, it writes a full-page screenshot and a JSON snapshot of the status text and counters under `/tmp/obsidian-livesync-e2e`. The dialogue-mount workflow leaves desktop and mobile screenshots for both representative Svelte routes, the Hidden File Sync workflow captures the successfully displayed JSON Resolve dialogue before selecting an option, and the Security Seed reconnect workflow captures each significant application state. The suite therefore records representative evidence without capturing every interaction. Set `E2E_OBSIDIAN_DIAGNOSTICS_DIR` to use another directory. The two-Vault workflow performs the missing-marker review once for each isolated Vault. Later process launches reuse the same profile-backed acknowledgement, rather than seeding a replacement or repeatedly applying a decision for the first device. The Hidden File Sync scenario is narrower: it starts from an explicitly acknowledged marker because it tests consumer-owned hidden-file behaviour, JSON resolution, target filtering, and grouped mobile Notices rather than duplicating the compatibility workflow. After `app.emulateMobile(true)`, its fixture operations use the active DevTools renderer because Obsidian can remove desktop-only CLI commands in mobile mode. +The two-Vault workflow also covers independent ID derivation with two real Obsidian sessions: a note travels in each direction, both devices retain the same obfuscated document IDs, and identical content reuses the same Chunk IDs. Fresh devices with a different ID key or legacy ID configuration must be rejected by ordinary CouchDB replication before any remote document or checkpoint changes. Set `E2E_OBSIDIAN_ONLY_INDEPENDENT_IDS=true` to run that case without the other two-Vault scenarios. + +Set `E2E_OBSIDIAN_ONLY_DIFFERENT_CHUNK_ID_KEYS=true` to run the focused case where two devices use different saved ID keys with Path Obfuscation off. It verifies that each device can read the other's note, visible document IDs agree, and writing the same content produces different Chunk IDs. + `test:e2e:obsidian:cli-to-obsidian-sync` is the cross-runtime compatibility check for the official LiveSync CLI and the real Obsidian plug-in. Build the plug-in first, and build the local CLI too when no external CLI command is selected. The script uses E2EE, Path Obfuscation, and the current preferred chunk settings to create and synchronise a note through the CLI, starts real Obsidian with an isolated Vault and profile, synchronises the same CouchDB database, and verifies that the plug-in materialises identical note content. This covers the boundary that CLI-only and plug-in-only round trips do not exercise. The isolated Obsidian session starts with its CouchDB settings and device-local compatibility acknowledgement already in place. This keeps the scenario focused on cross-runtime data compatibility; unconfigured start-up and visible CouchDB onboarding are covered by their dedicated workflows. @@ -166,6 +174,9 @@ LIVESYNC_CLI_COMMAND="docker run --rm --network host --user $(id -u):$(id -g) -- `test:e2e:obsidian:minio-upload` reuses the Object Storage variables from `.test.env` or the process environment. It expects a reachable S3-compatible service and starts with isolated Object Storage settings and the device-local compatibility acknowledgement already in place, keeping the scenario focused on upload rather than unconfigured start-up or setup. It confirms those settings through `obsidian-cli eval`, creates a note in real Obsidian, runs one-shot Journal Sync, and verifies through the AWS SDK that objects were written under a unique bucket prefix. Adapter tests separately observe an in-progress SDK command, while this real-runtime workflow verifies the resulting request counters advance and rebalance. +Set `E2E_OBSIDIAN_INDEPENDENT_IDS=true` to run the same upload with E2EE, Path Obfuscation, and a separately derived ID key. The scenario verifies the local document and Chunk ID shapes before the Journal transfer. +Set `E2E_OBSIDIAN_CUSTOM_HTTP_HANDLER=true` when the local Object Storage fixture does not allow browser requests from Obsidian's renderer. + `test:e2e:obsidian:object-storage-setup-uri-workflow` uses the public Commonlib-backed tool to generate the initial Setup URI for a unique Object Storage prefix, completes visible initialisation on the first device, and then asks that working real Obsidian device to create a new Setup URI through the registered command. A second real Obsidian device imports only the device-generated URI. The workflow verifies the A-to-B note through explicit replication, then verifies that the B-to-A note arrives through `syncOnStart` after restarting the first device, without requesting manual replication. It captures the documented onboarding choices, and removes the Object Storage prefix only after both sessions have stopped. `test:e2e:obsidian:p2p-setup-uri-workflow` runs two concurrent isolated real Obsidian sessions against the local Compose Nostr relay fixture. The first device imports a generated initial Setup URI and completes its signalling test with zero peers, creates a Setup URI for the second device through the registered command, and remains online while the second device imports it. The second device must select the expected online source before Fetch can rebuild its local database. The workflow accepts each connection request visibly on the receiving device, verifies the initial A-to-B fetch, checks that the menu for the three persistent per-peer actions remains within the viewport, reconnects both P2P sessions in join order, and verifies the B-to-A return journey. Every started session remains tracked until teardown completes. @@ -204,7 +215,7 @@ This proves in real Obsidian the plug-in behaviour shared by supported platforms `test:e2e:obsidian:internal-metadata-migration` enables internal Metadata encryption through the settings UI without Rebuild. It checks unchanged plaintext and rewritten encrypted Hidden File Sync and Customisation Sync Metadata in CouchDB, stable document IDs, mismatch rejection on a second device, and file restoration after aligning settings. It then turns the preference OFF, runs Fast Fetch, and compares content loaded from both Metadata representations and their Chunks while retaining the remote feature declaration. These focused tests use the local CouchDB fixture and are outside `test:e2e:obsidian:local-suite`. -`test:e2e:obsidian:setting-markdown-export` enables setting Markdown export, waits for the generated Markdown file in the vault, and verifies that credentials are omitted when `writeCredentialsForSettingSync=false`. +`test:e2e:obsidian:setting-markdown-export` enables setting Markdown export, waits for the generated Markdown file in the vault, and verifies that credentials are omitted when `writeCredentialsForSettingSync=false`, including both the plaintext ID key and its encrypted local representation. `test:e2e:obsidian:upgrade-from-stable` is the release-acceptance upgrade workflow. It installs the exact published 0.25.83 artefacts into an isolated Vault, verifies their pinned SHA-256 values, and then replaces only the plug-in artefacts with the current target while retaining the same Vault and isolated Obsidian profile. The first run downloads the old release into the ignored `_testdata/releases` cache; every later run verifies the cached bytes before use. diff --git a/test/e2e-obsidian/scripts/couchdb-manual-setup-workflow.ts b/test/e2e-obsidian/scripts/couchdb-manual-setup-workflow.ts index 4e1945b4..b4468ffc 100644 --- a/test/e2e-obsidian/scripts/couchdb-manual-setup-workflow.ts +++ b/test/e2e-obsidian/scripts/couchdb-manual-setup-workflow.ts @@ -2,6 +2,7 @@ import { randomBytes } from "node:crypto"; import { readFile } from "node:fs/promises"; import { join } from "node:path"; import { DEVICE_ID_PREFERRED, MILESTONE_DOCID } from "@vrtmrz/livesync-commonlib/compat/common/types"; +import { deriveIdKey, formatIdRecoveryCode } from "@vrtmrz/livesync-commonlib/settings"; import { evalObsidianJson } from "../runner/cli.ts"; import { assertCouchDbReachable, @@ -99,7 +100,12 @@ async function captureFailure(session: ObsidianLiveSyncSession, label: string): } } -async function enterManualCouchDBSettings(port: number, couchDb: CouchDbConfig, dbName: string): Promise { +async function enterManualCouchDBSettings( + port: number, + couchDb: CouchDbConfig, + dbName: string, + independentIdSource?: string +): Promise { const screenshots: string[] = []; await withObsidianPage(port, async (page) => { const invitation = page.locator(".notice").filter({ hasText: "Welcome to Self-hosted LiveSync" }); @@ -134,12 +140,41 @@ async function enterManualCouchDBSettings(port: number, couchDb: CouchDbConfig, 0, "The Obfuscate Properties row was present before end-to-end encryption was enabled." ); + const disabledIdChoices = encryption.locator("fieldset.sls-id-choices").first(); + assertEqual( + await disabledIdChoices.evaluate((element) => element.hasAttribute("disabled")), + true, + "The ID configuration was enabled while E2EE was off." + ); + for (const value of ["keep", "random", "custom"]) { + assertEqual( + await disabledIdChoices.locator(`input[value="${value}"]`).isDisabled(), + true, + `The ${value} ID configuration was enabled while E2EE was off.` + ); + } + const disabledIdScreenshot = join( + process.env.E2E_OBSIDIAN_DIAGNOSTICS_DIR ?? "/tmp/obsidian-livesync-e2e", + "guide-couchdb-manual-id-generation-disabled.png" + ); + await disabledIdChoices.screenshot({ path: disabledIdScreenshot }); + screenshots.push(disabledIdScreenshot); + assertEqual( + await disabledIdChoices.evaluate((element) => Number(getComputedStyle(element).opacity) < 1), + true, + "The disabled ID configuration did not look disabled in the default theme." + ); await encryption .locator("label.row") .filter({ hasText: "End-to-End Encryption" }) .locator('input[type="checkbox"]') .first() .check({ timeout: uiTimeoutMs }); + assertEqual( + await disabledIdChoices.evaluate((element) => Number(getComputedStyle(element).opacity)), + 1, + "The ID configuration remained dimmed after E2EE was enabled." + ); const passphraseInput = encryption.locator('input[name="e2ee-passphrase"]'); await passphraseInput.waitFor({ state: "visible", timeout: uiTimeoutMs }); await encryption @@ -149,7 +184,85 @@ async function enterManualCouchDBSettings(port: number, couchDb: CouchDbConfig, .first() .check({ timeout: uiTimeoutMs }); await passphraseInput.fill(e2eePassphrase); - const passwordToggle = encryption.locator("button.sls-password-toggle"); + const idChoices = encryption.locator('input[type="radio"][name="id-derivation-choice"]'); + assertEqual(await idChoices.count(), 3, "The three ID configurations were not all shown."); + for (const value of ["keep", "random", "custom"]) { + assertEqual( + await encryption.locator(`input[name="id-derivation-choice"][value="${value}"]`).isVisible(), + true, + `The ${value} ID configuration was not visible.` + ); + } + const keepChoice = encryption.locator('input[name="id-derivation-choice"][value="keep"]'); + const randomChoice = encryption.locator('input[name="id-derivation-choice"][value="random"]'); + const customChoice = encryption.locator('input[name="id-derivation-choice"][value="custom"]'); + assertEqual(await randomChoice.isChecked(), true, "The default ID configuration was not random."); + assertEqual( + await encryption.getByText("Keep current configuration", { exact: true }).count(), + 1, + "The current-configuration choice was not labelled consistently." + ); + assertEqual( + await encryption.getByText("Current configuration: no ID key is saved.", { exact: false }).count(), + 1, + "The current legacy configuration was not explained." + ); + await keepChoice.check({ timeout: uiTimeoutMs }); + assertEqual( + await encryption.getByText("Changing the E2EE passphrase changes IDs", { exact: false }).count(), + 1, + "Keeping legacy IDs did not explain the effect of changing the E2EE passphrase." + ); + await randomChoice.check({ timeout: uiTimeoutMs }); + if (independentIdSource) { + await customChoice.check({ timeout: uiTimeoutMs }); + const customChoices = encryption.locator('input[type="radio"][name="id-custom-choice"]'); + assertEqual(await customChoices.count(), 3, "The three custom ID inputs were not all shown."); + for (const value of ["passphrase", "source", "recovery"]) { + assertEqual( + await encryption.locator(`input[name="id-custom-choice"][value="${value}"]`).isVisible(), + true, + `The ${value} custom ID input was not visible.` + ); + } + const sourceChoice = encryption.locator('input[name="id-custom-choice"][value="source"]'); + assertEqual(await sourceChoice.isChecked(), true, "The custom ID input was not selected by default."); + await encryption + .locator('input[name="id-custom-choice"][value="passphrase"]') + .check({ timeout: uiTimeoutMs }); + assertEqual( + await encryption.locator('input[name="id-derivation-source"]').count(), + 0, + "The E2EE passphrase choice exposed a second source input." + ); + await encryption + .locator('input[name="id-custom-choice"][value="recovery"]') + .check({ timeout: uiTimeoutMs }); + assertEqual( + await encryption.locator('input[name="id-derivation-source"]').getAttribute("placeholder"), + "Enter an ID recovery code", + "The recovery-code choice did not request a recovery code." + ); + const recoveryChoice = encryption.locator('input[name="id-custom-choice"][value="recovery"]'); + await sourceChoice.check({ timeout: uiTimeoutMs }); + const sourceInput = encryption.locator('input[name="id-derivation-source"]'); + await sourceInput.fill(""); + await encryption.getByRole("button", { name: "Proceed", exact: true }).click({ timeout: uiTimeoutMs }); + assertEqual( + await encryption.getByText("An ID source is required to enable this option.", { exact: false }).count(), + 1, + "A first-time independent ID configuration did not require a source." + ); + assertEqual( + await encryption.isVisible(), + true, + "The E2EE dialogue closed after a first-time ID source was omitted." + ); + await recoveryChoice.check({ timeout: uiTimeoutMs }); + await sourceChoice.check({ timeout: uiTimeoutMs }); + await sourceInput.fill(independentIdSource); + } + const passwordToggle = passphraseInput.locator("..").locator("button.sls-password-toggle"); await passwordToggle.click({ timeout: uiTimeoutMs }); assertEqual( await passphraseInput.getAttribute("type"), @@ -167,16 +280,13 @@ async function enterManualCouchDBSettings(port: number, couchDb: CouchDbConfig, "password", "Toggling visibility again did not re-mask the passphrase." ); - assertEqual( - await passphraseInput.inputValue(), - e2eePassphrase, - "Re-masking the passphrase changed its value." - ); + assertEqual(await passphraseInput.inputValue(), e2eePassphrase, "Re-masking the passphrase changed its value."); }); screenshots.push(await captureGuideDialogue(port, "guide-couchdb-manual-encryption.png", "End-to-End Encryption")); await withObsidianPage(port, async (page) => { const encryption = modalByTitle(page, "End-to-End Encryption"); await encryption.getByRole("button", { name: "Proceed", exact: true }).click({ timeout: uiTimeoutMs }); + await encryption.waitFor({ state: "hidden", timeout: uiTimeoutMs }); }); screenshots.push( @@ -288,13 +398,18 @@ async function waitForRemoteEntry(context: RunnerContext, entry: { id: string; c }); } -async function assertPersistedE2EE(vault: TemporaryVault): Promise { - const persisted = JSON.parse( - await readFile(join(vault.path, ".obsidian", "plugins", "obsidian-livesync", "data.json"), "utf8") - ) as { +async function assertPersistedE2EE(vault: TemporaryVault, independentIdSource?: string): Promise { + const rawSettings = await readFile( + join(vault.path, ".obsidian", "plugins", "obsidian-livesync", "data.json"), + "utf8" + ); + const persisted = JSON.parse(rawSettings) as { encrypt?: unknown; encryptedPassphrase?: unknown; passphrase?: unknown; + idDerivationVersion?: unknown; + idDerivationKey?: unknown; + encryptedIdDerivationKey?: unknown; }; assertEqual(persisted.encrypt, true, "Manual CouchDB setup did not persist E2EE as enabled."); assertEqual(persisted.passphrase, "", "Manual CouchDB setup persisted the E2EE passphrase in plain text."); @@ -304,6 +419,14 @@ async function assertPersistedE2EE(vault: TemporaryVault): Promise { if (JSON.stringify(persisted).includes(e2eePassphrase)) { throw new Error("Manual CouchDB setup persisted the E2EE passphrase in plain text."); } + assertEqual(persisted.idDerivationVersion, 1, "The independent ID mode was not persisted."); + assertEqual(persisted.idDerivationKey, "", "The derived ID key was stored in plain text."); + if (typeof persisted.encryptedIdDerivationKey !== "string" || !persisted.encryptedIdDerivationKey) { + throw new Error("The derived ID key was not encrypted in local settings."); + } + if (independentIdSource) { + if (rawSettings.includes(independentIdSource)) throw new Error("The ID source was stored in local settings."); + } } async function assertRestoredE2EEPassphrase(session: ObsidianLiveSyncSession, cliBinary: string): Promise { @@ -320,6 +443,126 @@ async function assertRestoredE2EEPassphrase(session: ObsidianLiveSyncSession, cl assertEqual(restored, true, "The E2EE passphrase was not restored after Obsidian restarted."); } +async function assertCurrentIdDerivationKey( + session: ObsidianLiveSyncSession, + cliBinary: string, + expected: string, + context: string +): Promise { + const settings = await evalObsidianJson<{ idDerivationVersion: number; idDerivationKey: string }>( + cliBinary, + [ + "(()=>{", + "const settings=app.plugins.plugins['obsidian-livesync'].core.services.setting.currentSettings();", + "return JSON.stringify({idDerivationVersion:settings.idDerivationVersion,idDerivationKey:settings.idDerivationKey});", + "})()", + ].join(""), + session.cliEnv + ); + assertEqual(settings.idDerivationVersion, 1, `${context}: the independent ID version was not retained.`); + assertEqual(settings.idDerivationKey, expected, `${context}: the saved ID key changed.`); +} + +async function assertRecoveryCodeCanBeRevealed( + session: ObsidianLiveSyncSession, + cliBinary: string, + source: string +): Promise { + const port = session.remoteDebuggingPort; + const expected = formatIdRecoveryCode(await deriveIdKey(source)); + await withObsidianPage(port, async (page) => { + const settingsNavigator = await openLiveSyncSettings(page, uiTimeoutMs); + const remotePage = await settingsNavigator.openPage("Remote Configuration"); + await remotePage + .locator(".setting-item") + .filter({ hasText: "Configure E2EE" }) + .getByRole("button", { name: "Configure", exact: true }) + .click({ timeout: uiTimeoutMs }); + const encryption = modalByTitle(page, "End-to-End Encryption"); + await encryption.waitFor({ state: "visible", timeout: uiTimeoutMs }); + assertEqual( + await encryption.locator('input[name="id-derivation-choice"][value="keep"]').isChecked(), + true, + "An existing ID key was not selected for reuse." + ); + assertEqual( + await encryption.getByText("Current configuration: a saved ID key is used.").count(), + 1, + "The saved ID key was not explained." + ); + await encryption.getByRole("button", { name: "Show current recovery code" }).click({ timeout: uiTimeoutMs }); + assertEqual( + await encryption.getByRole("textbox", { name: "Current ID recovery code" }).inputValue(), + expected, + "The displayed recovery code did not contain the saved ID key." + ); + await encryption.locator('input[name="id-derivation-choice"][value="custom"]').check({ timeout: uiTimeoutMs }); + await encryption.locator('input[name="id-custom-choice"][value="recovery"]').check({ timeout: uiTimeoutMs }); + const recoveryInput = encryption.locator('input[name="id-derivation-source"]'); + await recoveryInput.fill("not-a-recovery-code"); + await encryption.getByRole("button", { name: "Proceed" }).click({ timeout: uiTimeoutMs }); + assertEqual( + await encryption.getByText("The ID source or recovery code is invalid.", { exact: false }).count(), + 1, + "The recovery-code choice accepted an ordinary source string." + ); + await recoveryInput.fill(expected); + await encryption.getByRole("button", { name: "Proceed" }).click({ timeout: uiTimeoutMs }); + await encryption.waitFor({ state: "hidden", timeout: uiTimeoutMs }); + assertEqual( + await modalByTitle(page, "Mostly Complete: Decision Required").count(), + 0, + "Recovering the existing ID key opened a new setup decision." + ); + }); + const expectedIdKey = await deriveIdKey(source); + await assertCurrentIdDerivationKey(session, cliBinary, expectedIdKey, "Recovering the saved ID key"); + + await withObsidianPage(port, async (page) => { + const settingsNavigator = await openLiveSyncSettings(page, uiTimeoutMs); + const remotePage = await settingsNavigator.openPage("Remote Configuration"); + await remotePage + .locator(".setting-item") + .filter({ hasText: "Configure E2EE" }) + .getByRole("button", { name: "Configure", exact: true }) + .click({ timeout: uiTimeoutMs }); + const encryption = modalByTitle(page, "End-to-End Encryption"); + await encryption.waitFor({ state: "visible", timeout: uiTimeoutMs }); + await encryption.locator('input[name="id-derivation-choice"][value="custom"]').check({ timeout: uiTimeoutMs }); + await encryption.locator('input[name="id-custom-choice"][value="source"]').check({ timeout: uiTimeoutMs }); + const sourceInput = encryption.locator('input[name="id-derivation-source"]'); + await sourceInput.fill(""); + assertEqual(await sourceInput.inputValue(), "", "The independent ID source input was not empty."); + assertEqual( + await encryption.getByText("Leave this input empty to keep the saved ID key.", { exact: true }).count(), + 1, + "The configured ID source did not explain that an empty input keeps the saved ID key." + ); + await encryption.getByRole("button", { name: "Proceed", exact: true }).click({ timeout: uiTimeoutMs }); + await encryption.waitFor({ state: "hidden", timeout: uiTimeoutMs }); + assertEqual( + await modalByTitle(page, "Mostly Complete: Decision Required").count(), + 0, + "Keeping the saved ID key after an empty source opened a new setup decision." + ); + }); + await assertCurrentIdDerivationKey(session, cliBinary, expectedIdKey, "Saving an empty custom ID source"); +} + +async function readDocumentId(cliBinary: string, environment: NodeJS.ProcessEnv, path: string): Promise { + return await evalObsidianJson( + cliBinary, + [ + "(async()=>{", + `const path=${JSON.stringify(path)};`, + "const core=app.plugins.plugins['obsidian-livesync'].core;", + "return JSON.stringify(await core.services.path.path2id(path));", + "})()", + ].join(""), + environment + ); +} + async function setRemotePreferredE2EEDisabled(context: RunnerContext): Promise { const milestone = await fetchCouchDbDocument(context.couchDb, context.dbName, MILESTONE_DOCID); const tweakValues = milestone.tweak_values; @@ -430,6 +673,10 @@ async function main(): Promise { }; const screenshots: string[] = []; let secondDeviceArtifact: SetupArtifact | undefined; + const independentIdSource = + process.env.E2E_OBSIDIAN_INDEPENDENT_IDS === "true" ? randomBytes(32).toString("base64url") : undefined; + let firstEntryId: string | undefined; + let returnEntryId: string | undefined; try { await assertCouchDbReachable(couchDb); @@ -440,7 +687,9 @@ async function main(): Promise { let session = await startUnconfiguredSession(context, vaultA); try { - screenshots.push(...(await enterManualCouchDBSettings(session.remoteDebuggingPort, couchDb, dbName))); + screenshots.push( + ...(await enterManualCouchDBSettings(session.remoteDebuggingPort, couchDb, dbName, independentIdSource)) + ); screenshots.push(await captureAndStartInitialisation(session.remoteDebuggingPort, "new", captures)); screenshots.push(await confirmRebuild(session.remoteDebuggingPort, captures)); screenshots.push(await continueWithoutRemoteSettings(session.remoteDebuggingPort, captures)); @@ -453,10 +702,14 @@ async function main(): Promise { 1, "Manual CouchDB setup did not persist exactly one remote profile." ); - await assertPersistedE2EE(vaultA); + await assertPersistedE2EE(vaultA, independentIdSource); + if (independentIdSource) { + await assertRecoveryCodeCanBeRevealed(session, context.cliBinary, independentIdSource); + } await writeNoteViaObsidian(context.cliBinary, session.cliEnv, notePath, noteContent); const entry = await waitForLocalDatabaseEntry(context.cliBinary, session.cliEnv, notePath); + firstEntryId = entry.id; await pushLocalChanges(context.cliBinary, session.cliEnv); await waitForRemoteEntry(context, entry); } catch (error) { @@ -479,9 +732,12 @@ async function main(): Promise { ); await finishInitialisation(session.remoteDebuggingPort, context.cliBinary, session.cliEnv); await resumeCompatibilityReviewIfShown(session.remoteDebuggingPort); - await assertPersistedE2EE(vaultA); + await assertPersistedE2EE(vaultA, independentIdSource); const rebuiltEntry = await waitForLocalDatabaseEntry(context.cliBinary, session.cliEnv, notePath); + if (independentIdSource) { + assertEqual(rebuiltEntry.id, firstEntryId, "Rebuild changed the configured document ID."); + } await waitForRemoteEntry(context, rebuiltEntry); await assertRemoteEntryEncrypted(context, rebuiltEntry, notePath, noteContent); await assertRemotePreferredE2EE(context, true); @@ -512,11 +768,20 @@ async function main(): Promise { screenshots.push(...(await confirmFastFetch(session.remoteDebuggingPort, captures))); await finishInitialisation(session.remoteDebuggingPort, context.cliBinary, session.cliEnv); await resumeCompatibilityReviewIfShown(session.remoteDebuggingPort); + await assertPersistedE2EE(vaultB, independentIdSource); await pushLocalChanges(context.cliBinary, session.cliEnv); await waitForVaultFile(vaultB, notePath, noteContent); + if (independentIdSource) { + assertEqual( + await readDocumentId(context.cliBinary, session.cliEnv, notePath), + firstEntryId, + "The Setup URI did not restore the document ID key on the second device." + ); + } await writeNoteViaObsidian(context.cliBinary, session.cliEnv, returnNotePath, returnNoteContent); const returnEntry = await waitForLocalDatabaseEntry(context.cliBinary, session.cliEnv, returnNotePath); + returnEntryId = returnEntry.id; await pushLocalChanges(context.cliBinary, session.cliEnv); await waitForRemoteEntry(context, returnEntry); } catch (error) { @@ -531,6 +796,13 @@ async function main(): Promise { await resumeCompatibilityReviewIfShown(session.remoteDebuggingPort); await pushLocalChanges(context.cliBinary, session.cliEnv); await waitForVaultFile(vaultA, returnNotePath, returnNoteContent); + if (independentIdSource) { + assertEqual( + await readDocumentId(context.cliBinary, session.cliEnv, returnNotePath), + returnEntryId, + "The first device did not retain the shared document ID key." + ); + } } catch (error) { await captureFailure(session, "return-journey"); throw error; diff --git a/test/e2e-obsidian/scripts/minio-upload.ts b/test/e2e-obsidian/scripts/minio-upload.ts index b0899f12..7de062e8 100644 --- a/test/e2e-obsidian/scripts/minio-upload.ts +++ b/test/e2e-obsidian/scripts/minio-upload.ts @@ -15,6 +15,8 @@ * Separate successes would not prove that those observations belonged to the * same upload. */ +import { randomBytes } from "node:crypto"; +import { deriveIdKey } from "@vrtmrz/livesync-commonlib/settings"; import { evalObsidianJson } from "../runner/cli.ts"; import { discoverObsidianCli, requireObsidianBinary } from "../runner/environment.ts"; import { @@ -33,6 +35,7 @@ import { listObjectStorageObjects, loadObjectStorageConfig, makeUniqueBucketPrefix, + readObjectStorageJson, } from "../runner/objectStorage.ts"; import { startObsidianLiveSyncSession, type ObsidianLiveSyncSession } from "../runner/session.ts"; import { createTemporaryVault } from "../runner/vault.ts"; @@ -41,6 +44,8 @@ import { REMOTE_ACTIVITY_EXPECTED_STATE, waitForRemoteActivityState } from "../r process.env.E2E_OBSIDIAN_CLI_TIMEOUT_MS ??= "30000"; const notePath = "E2E/minio-upload.md"; +const useIndependentIds = process.env.E2E_OBSIDIAN_INDEPENDENT_IDS === "true"; +const useCustomRequestHandler = process.env.E2E_OBSIDIAN_CUSTOM_HTTP_HANDLER === "true"; const noteContent = [ "# Object Storage upload from real Obsidian", "", @@ -127,10 +132,23 @@ async function main(): Promise { }); await waitForLiveSyncCoreReady(cli.binary, session.cliEnv); - const configured = await configureObjectStorage(cli.binary, session.cliEnv, { - ...objectStorage, - bucketPrefix, - }); + const configured = await configureObjectStorage( + cli.binary, + session.cliEnv, + { ...objectStorage, bucketPrefix }, + { + ...(useIndependentIds + ? { + encrypt: true, + usePathObfuscation: true, + passphrase: randomBytes(32).toString("base64url"), + idDerivationVersion: 1, + idDerivationKey: await deriveIdKey(randomBytes(32).toString("base64url")), + } + : {}), + ...(useCustomRequestHandler ? { useCustomRequestHandler: true } : {}), + } + ); await waitForLiveSyncCoreReady(cli.binary, session.cliEnv); assertEqual(configured.isConfigured, true, "Self-hosted LiveSync was not marked as configured."); assertEqual(configured.remoteType, "MINIO", "Remote type was not Object Storage."); @@ -145,6 +163,16 @@ async function main(): Promise { REMOTE_ACTIVITY_EXPECTED_STATE.idle ); const localEntry = await createNoteAndWaitForLocalDb(cli.binary, session.cliEnv); + if (useIndependentIds) { + if ( + !/^f:[0-9a-f]{64}$/u.test(localEntry.id) || + localEntry.children.some((child) => !/^h:\+[0-9a-f]{64}$/u.test(child)) + ) { + throw new Error( + `The real Obsidian Journal upload did not use independent document and Chunk IDs (document length ${localEntry.id.length}, Chunk lengths ${localEntry.children.map((child) => child.length).join(",")}).` + ); + } + } await pushLocalChanges(cli.binary, session.cliEnv); const activityAfterUpload = await waitForRemoteActivityState( session.remoteDebuggingPort, @@ -160,6 +188,15 @@ async function main(): Promise { ); const keys = await waitForObjectStorageObjects(bucketPrefix); + if (useIndependentIds) { + const milestone = await readObjectStorageJson<{ encrypted_id_derivation_proof?: string }>( + objectStorage, + `${bucketPrefix}_00000000-milestone.json` + ); + if (!milestone.encrypted_id_derivation_proof) { + throw new Error("The Journal milestone did not retain an encrypted ID agreement proof."); + } + } console.log( `Uploaded ${localEntry.path} through Journal Sync to ${objectStorage.bucket}/${bucketPrefix} (${keys.length} object(s)); tracked requests: ${activityAfterUpload.requestCount - activityBeforeUpload.requestCount}` diff --git a/test/e2e-obsidian/scripts/review-harness.ts b/test/e2e-obsidian/scripts/review-harness.ts index 383074c7..853b1233 100644 --- a/test/e2e-obsidian/scripts/review-harness.ts +++ b/test/e2e-obsidian/scripts/review-harness.ts @@ -6,6 +6,7 @@ import { assertNoHorizontalOverflow, } from "@vrtmrz/obsidian-test-session"; import { CURRENT_SETTING_VERSION } from "@vrtmrz/livesync-commonlib/compat/common/models/setting.const"; +import { DoctorRegulation } from "@vrtmrz/livesync-commonlib/compat/common/configForDoc"; import { REVIEW_HARNESS_STATE_KEY } from "../../../src/features/ReviewHarness/reviewHarnessController.ts"; import { REVIEW_HARNESS_FIXTURE_ROOT } from "../../../src/features/ReviewHarness/reviewHarnessVaultFixture.ts"; import { evalObsidianJson } from "../runner/cli.ts"; @@ -166,7 +167,8 @@ async function captureReadinessFailure( async function openHarness(): Promise { const opened = await withObsidianPage(obsidianRemoteDebuggingPort(), async (page) => { return await page.evaluate( - (commandId) => (globalThis as ReviewHarnessTestGlobal).app?.commands?.executeCommandById(commandId) === true, + (commandId) => + (globalThis as ReviewHarnessTestGlobal).app?.commands?.executeCommandById(commandId) === true, "obsidian-livesync:open-review-harness" ); }); @@ -205,11 +207,54 @@ async function runAutomaticScenarios(): Promise { }); } +async function runIdBenchmark(): Promise { + await withObsidianPage(obsidianRemoteDebuggingPort(), async (page) => { + const snapshotSettings = () => + page.evaluate(() => { + const plugin = (globalThis as ReviewHarnessTestGlobal).app?.plugins?.plugins["obsidian-livesync"] as { + core: { services: { setting: { currentSettings(): unknown } } }; + }; + return JSON.stringify(plugin.core.services.setting.currentSettings()); + }); + const before = await snapshotSettings(); + const harness = page.locator('[data-testid="review-harness"]'); + await harness + .locator('[data-testid="review-harness-run-id-generation-performance"]') + .click({ timeout: uiTimeoutMs }); + const result = harness.locator('[data-testid="review-harness-result-id-generation-performance"]'); + await result.getByText("Passed:", { exact: false }).waitFor({ state: "visible", timeout: uiTimeoutMs * 4 }); + const observations = await result.locator("li").allTextContents(); + for (const label of [ + "Chunk IDs, 256 B", + "Chunk IDs, 4096 B", + "Chunk IDs, 32768 B", + "Obfuscated document IDs", + ]) { + for (const mode of ["legacy", "independent"]) { + if ( + !observations.some( + (line) => + line.startsWith(`${label}, ${mode}: 1000 IDs total median=`) && line.includes("; per ID=") + ) + ) { + throw new Error(`Missing benchmark timing and units: ${label}, ${mode}`); + } + } + } + if ( + !observations.some((line) => line.startsWith("ID key derivation at save time:")) || + !observations.some((line) => line.startsWith("JavaScript heap:")) + ) { + throw new Error("The benchmark did not report derivation and heap observations."); + } + if ((await snapshotSettings()) !== before) throw new Error("The benchmark changed the live settings."); + await assertNoHorizontalOverflow(page, harness, { label: "ID benchmark results" }); + }); +} + async function runVaultFixture(): Promise { await withObsidianPage(obsidianRemoteDebuggingPort(), async (page) => { - await page - .locator('[data-testid="review-harness-run-vault-round-trip"]') - .click({ timeout: uiTimeoutMs }); + await page.locator('[data-testid="review-harness-run-vault-round-trip"]').click({ timeout: uiTimeoutMs }); const confirmation = page.locator(".modal-container").filter({ has: page.getByText("Review Harness: Vault fixture access", { exact: true }), }); @@ -272,27 +317,22 @@ async function restartAndResumeHarness(): Promise { }); await keepCompatibilityPaused(); await waitForHarness(); - return await captureObsidianDialogue( - obsidianRemoteDebuggingPort(), - "review-harness-resumed.png", - async (page) => { - const harness = page.locator('[data-testid="review-harness"]'); - await harness - .locator('[data-testid="review-harness-resumed"]') - .waitFor({ state: "visible", timeout: uiTimeoutMs }); - const continuationRemoved = await page.evaluate((stateKey) => { - const plugin = (globalThis as ReviewHarnessTestGlobal).app?.plugins?.plugins["obsidian-livesync"]; - if (typeof plugin !== "object" || plugin === null || !("core" in plugin)) { - throw new Error("Self-hosted LiveSync is unavailable after restart."); - } - const core = (plugin as { core: { services: { setting: { getSmallConfig(key: string): string } } } }) - .core; - return core.services.setting.getSmallConfig(stateKey) === ""; - }, REVIEW_HARNESS_STATE_KEY); - if (!continuationRemoved) throw new Error("The one-shot continuation was not removed before use."); - await assertNoHorizontalOverflow(page, harness, { label: "resumed Review Harness" }); - } - ); + return await captureObsidianDialogue(obsidianRemoteDebuggingPort(), "review-harness-resumed.png", async (page) => { + const harness = page.locator('[data-testid="review-harness"]'); + await harness + .locator('[data-testid="review-harness-resumed"]') + .waitFor({ state: "visible", timeout: uiTimeoutMs }); + const continuationRemoved = await page.evaluate((stateKey) => { + const plugin = (globalThis as ReviewHarnessTestGlobal).app?.plugins?.plugins["obsidian-livesync"]; + if (typeof plugin !== "object" || plugin === null || !("core" in plugin)) { + throw new Error("Self-hosted LiveSync is unavailable after restart."); + } + const core = (plugin as { core: { services: { setting: { getSmallConfig(key: string): string } } } }).core; + return core.services.setting.getSmallConfig(stateKey) === ""; + }, REVIEW_HARNESS_STATE_KEY); + if (!continuationRemoved) throw new Error("The one-shot continuation was not removed before use."); + await assertNoHorizontalOverflow(page, harness, { label: "resumed Review Harness" }); + }); } async function completeResumedCompatibilityStep(): Promise { @@ -331,9 +371,7 @@ async function copyAndReadReport(): Promise { undefined, { timeout: uiTimeoutMs } ); - return await page.evaluate( - () => (globalThis as ReviewHarnessTestGlobal).reviewHarnessCopiedReport ?? "" - ); + return await page.evaluate(() => (globalThis as ReviewHarnessTestGlobal).reviewHarnessCopiedReport ?? ""); }); } @@ -347,35 +385,36 @@ async function verifyMobileHarness(): Promise { if (typeof plugin !== "object" || plugin === null || !("core" in plugin)) { throw new Error("Self-hosted LiveSync is unavailable in mobile test mode."); } - const core = (plugin as { - core: { services: { API: { showWindow(type: string): Promise } } }; - }).core; + const core = ( + plugin as { + core: { services: { API: { showWindow(type: string): Promise } } }; + } + ).core; await core.services.API.showWindow(viewType); }, "self-hosted-livesync-review-harness"); }); - return await captureObsidianDialogue( - obsidianRemoteDebuggingPort(), - "review-harness-mobile.png", - async (page) => { - const harness = page.locator('[data-testid="review-harness"]'); - await harness.waitFor({ state: "visible", timeout: uiTimeoutMs }); - await assertNoHorizontalOverflow(page, harness, { label: "mobile Review Harness" }); - const heading = harness.getByRole("heading", { name: "Self-hosted LiveSync review harness" }); - await assertLocatorWithinSafeArea(page, heading, { - label: "mobile Review Harness heading", - safeAreaInsets: iPhoneSafeArea, + await runIdBenchmark(); + return await captureObsidianDialogue(obsidianRemoteDebuggingPort(), "review-harness-mobile.png", async (page) => { + const harness = page.locator('[data-testid="review-harness"]'); + await harness.waitFor({ state: "visible", timeout: uiTimeoutMs }); + await harness.getByRole("heading", { name: "Self-hosted LiveSync review harness" }).scrollIntoViewIfNeeded(); + await assertNoHorizontalOverflow(page, harness, { label: "mobile Review Harness" }); + const heading = harness.getByRole("heading", { name: "Self-hosted LiveSync review harness" }); + await assertLocatorWithinSafeArea(page, heading, { + label: "mobile Review Harness heading", + safeAreaInsets: iPhoneSafeArea, + }); + for (const testId of [ + "review-harness-run-automatic", + "review-harness-run-full", + "review-harness-copy-report", + "review-harness-run-id-generation-performance", + ]) { + await assertLocatorHasMinimumTouchTarget(page, harness.locator(`[data-testid="${testId}"]`), { + label: testId, }); - for (const testId of [ - "review-harness-run-automatic", - "review-harness-run-full", - "review-harness-copy-report", - ]) { - await assertLocatorHasMinimumTouchTarget(page, harness.locator(`[data-testid="${testId}"]`), { - label: testId, - }); - } } - ); + }); } async function main(): Promise { @@ -391,7 +430,8 @@ async function main(): Promise { vault, startupGraceMs: Number(process.env.E2E_OBSIDIAN_STARTUP_GRACE_MS ?? 1000), pluginData: { - doctorProcessedVersion: "1.0.0", + // Config Doctor is covered by settings-ui; this fixture exercises the Harness. + doctorProcessedVersion: DoctorRegulation.version, settingVersion: CURRENT_SETTING_VERSION, isConfigured: true, additionalSuffixOfDatabaseName: "", @@ -440,15 +480,34 @@ async function main(): Promise { const vaultConfirmationScreenshot = await runVaultFixture(); const resumedScreenshot = await restartAndResumeHarness(); await completeResumedCompatibilityStep(); + await runIdBenchmark(); const report = await copyAndReadReport(); if (!report.includes("## Self-hosted LiveSync Review Harness report")) { throw new Error("The copied Review Harness report was not Markdown evidence."); } - for (const forbidden of [vault.name, REVIEW_HARNESS_FIXTURE_ROOT]) { - if (report.includes(forbidden)) throw new Error(`The Review Harness report exposed local state: ${forbidden}`); + for (const expected of [ + "1000 IDs total median=", + "; per ID=", + "ID key derivation at save time:", + "JavaScript heap:", + ]) { + if (!report.includes(expected)) throw new Error(`Missing copied benchmark observation: ${expected}`); + } + for (const forbidden of [ + vault.name, + REVIEW_HARNESS_FIXTURE_ROOT, + "ab".repeat(32), + "Self-hosted LiveSync ID benchmark passphrase", + "Self-hosted LiveSync ID benchmark source", + ]) { + if (report.includes(forbidden)) + throw new Error(`The Review Harness report exposed local state: ${forbidden}`); } const mobileScreenshot = await verifyMobileHarness(); + const outputDirectory = process.env.E2E_OBSIDIAN_DIAGNOSTICS_DIR ?? "/tmp/obsidian-livesync-e2e"; + await mkdir(outputDirectory, { recursive: true }); + await writeFile(join(outputDirectory, "review-harness-report.md"), report, "utf8"); console.log( `Review Harness passed one-shot, fixture, report, and mobile checks. Screenshots: ${[ initialScreenshot, diff --git a/test/e2e-obsidian/scripts/setting-markdown-export.ts b/test/e2e-obsidian/scripts/setting-markdown-export.ts index 44a5825e..d4a24291 100644 --- a/test/e2e-obsidian/scripts/setting-markdown-export.ts +++ b/test/e2e-obsidian/scripts/setting-markdown-export.ts @@ -1,5 +1,6 @@ import { readFile } from "node:fs/promises"; import { join } from "node:path"; +import { deriveIdKey } from "@vrtmrz/livesync-commonlib/settings"; import { evalObsidianJson } from "../runner/cli.ts"; import { discoverObsidianCli, requireObsidianBinary } from "../runner/environment.ts"; import { assertEqual } from "../runner/liveSyncWorkflow.ts"; @@ -34,7 +35,11 @@ async function waitForFileContaining( throw new Error(`Timed out waiting for setting Markdown: ${fullPath}\nLast error: ${String(lastError)}`); } -async function configureSettingMarkdown(cliBinary: string, env: NodeJS.ProcessEnv): Promise { +async function configureSettingMarkdown( + cliBinary: string, + env: NodeJS.ProcessEnv, + idDerivationKey: string +): Promise { await evalObsidianJson( cliBinary, [ @@ -46,6 +51,8 @@ async function configureSettingMarkdown(cliBinary: string, env: NodeJS.ProcessEn "couchDB_USER:'e2e-user',", "couchDB_PASSWORD:'e2e-password',", "passphrase:'e2e-passphrase',", + "idDerivationVersion:1,", + `idDerivationKey:${JSON.stringify(idDerivationKey)},`, "showVerboseLog:true,", "},true);", "await core.services.setting.saveSettingData();", @@ -64,6 +71,7 @@ async function main(): Promise { } const vault = await createTemporaryVault(); + const idDerivationKey = await deriveIdKey("setting-markdown-export-independent-id-key-fixture"); let session: ObsidianLiveSyncSession | undefined; try { console.log(`Using Obsidian executable: ${binary}`); @@ -77,19 +85,39 @@ async function main(): Promise { }); // The export is available while an unconfigured Vault remains outside // application readiness; the session helper has already loaded the plug-in. - await configureSettingMarkdown(cli.binary, session.cliEnv); + await configureSettingMarkdown(cli.binary, session.cliEnv, idDerivationKey); const content = await waitForFileContaining(vault.path, settingPath, [ (value) => value.includes("````yaml:livesync-setting"), (value) => value.includes(`settingSyncFile: ${settingPath}`), (value) => value.includes("showVerboseLog: true"), ]); + const persisted = JSON.parse( + await readFile(join(vault.path, ".obsidian", "plugins", "obsidian-livesync", "data.json"), "utf-8") + ) as { + idDerivationVersion?: unknown; + idDerivationKey?: unknown; + encryptedIdDerivationKey?: unknown; + }; + assertEqual(persisted.idDerivationVersion, 1, "The independent ID key fixture was not persisted."); + assertEqual(persisted.idDerivationKey, "", "The independent ID key was stored in plain text locally."); + const encryptedIdDerivationKey = persisted.encryptedIdDerivationKey; + if (typeof encryptedIdDerivationKey !== "string" || encryptedIdDerivationKey.length === 0) { + throw new Error("The independent ID key fixture was not saved in encrypted local settings."); + } + assertEqual( content.includes("couchDB_PASSWORD: e2e-password"), false, "Credential leaked into setting Markdown." ); assertEqual(content.includes("passphrase: e2e-passphrase"), false, "Passphrase leaked into setting Markdown."); + assertEqual(content.includes(idDerivationKey), false, "Plaintext ID key leaked into setting Markdown."); + assertEqual( + content.includes(encryptedIdDerivationKey), + false, + "Encrypted ID key leaked into setting Markdown." + ); console.log(`Generated setting Markdown without credentials: ${settingPath}`); } finally { diff --git a/test/e2e-obsidian/scripts/two-vault-sync.ts b/test/e2e-obsidian/scripts/two-vault-sync.ts index dbb97913..e006f1a7 100644 --- a/test/e2e-obsidian/scripts/two-vault-sync.ts +++ b/test/e2e-obsidian/scripts/two-vault-sync.ts @@ -1,11 +1,16 @@ import { mkdir, readFile, rename as renameFilesystemPath, rm, writeFile } from "node:fs/promises"; import { dirname, join } from "node:path"; +import { SALT_OF_PASSPHRASE } from "@vrtmrz/livesync-commonlib/compat/common/types"; +import { encryptString } from "@vrtmrz/livesync-commonlib/compat/encryption/stringEncryption"; +import { deriveIdKey } from "@vrtmrz/livesync-commonlib/settings"; +import { CENTRAL_COMPATIBILITY_REJECTION_REASONS } from "@vrtmrz/livesync-commonlib/replication"; import { evalObsidianJson } from "../runner/cli.ts"; import { assertCouchDbReachable, createCouchDbDatabase, deleteCouchDbDatabase, fetchAllCouchDbDocs, + fetchCouchDbLocalDocs, loadCouchDbConfig, makeUniqueDatabaseName, waitForCouchDbDocs, @@ -19,6 +24,7 @@ import { assertE2eCompatibilityReviewPending, configureCouchDb, createE2eCouchDbPluginData, + createE2eObsidianDeviceLocalState, prepareRemote, pushLocalChanges, resumeCompatibilityReview, @@ -442,7 +448,8 @@ async function renameNoteViaObsidian(cliBinary: string, env: NodeJS.ProcessEnv, async function startConfiguredSession( context: RunnerContext, vault: TemporaryVault, - overrides: Record = {} + overrides: Record = {}, + persistedOverrides: Record = overrides ): Promise { const couchDbSettings = { uri: context.couchDb.uri, @@ -456,7 +463,7 @@ async function startConfiguredSession( cliBinary: context.cliBinary, vault, startupGraceMs: Number(process.env.E2E_OBSIDIAN_STARTUP_GRACE_MS ?? 1000), - pluginData: createE2eCouchDbPluginData(couchDbSettings, overrides), + pluginData: createE2eCouchDbPluginData(couchDbSettings, persistedOverrides), }); context.activeSessions.add(session); try { @@ -970,6 +977,193 @@ async function runEncryptedRoundTrip( console.log("Two-vault encrypted note synchronisation round-tripped."); } +async function runIndependentIdRoundTrip( + context: RunnerContext, + vaultA: TemporaryVault, + vaultB: TemporaryVault +): Promise { + const source = "real-obsidian-e2e-independent-id-source"; + const key = await deriveIdKey(source); + const content = "# Shared content with an independent ID key.\n"; + const pathA = "E2E/independent-ids/from-a.md"; + const pathB = "E2E/independent-ids/from-b.md"; + const overrides = { + encrypt: true, + passphrase: "real-obsidian-e2e-independent-passphrase", + usePathObfuscation: true, + E2EEAlgorithm: "v2", + idDerivationVersion: 1, + idDerivationKey: key, + }; + const persistedOverrides = { + ...overrides, + idDerivationKey: "", + encryptedIdDerivationKey: await encryptString(key, `*${SALT_OF_PASSPHRASE}`), + }; + + let session = await startConfiguredSession(context, vaultA, overrides, persistedOverrides); + await writeNoteViaObsidian(context.cliBinary, session.cliEnv, pathA, content); + const entryA = await uploadNote(context, session, pathA); + if (entryA.children.length === 0) throw new Error("Independent ID mode produced no Chunks."); + await stopTrackedSession(context, session); + + session = await startConfiguredSession(context, vaultB, overrides, persistedOverrides); + await syncAndApply(context, session); + await waitForPathContent(vaultB.path, pathA, (received) => received === content); + const receivedA = await waitForLocalDatabaseEntry(context.cliBinary, session.cliEnv, pathA); + assertEqual(receivedA.id, entryA.id, "The second device did not preserve the obfuscated document ID."); + + await writeNoteViaObsidian(context.cliBinary, session.cliEnv, pathB, content); + const entryB = await uploadNote(context, session, pathB); + assertEqual( + JSON.stringify(entryB.children), + JSON.stringify(entryA.children), + "The second device did not reuse the same content-derived Chunk IDs." + ); + await stopTrackedSession(context, session); + + session = await startConfiguredSession(context, vaultA, overrides, persistedOverrides); + await syncAndApply(context, session); + await waitForPathContent(vaultA.path, pathB, (received) => received === content); + const receivedB = await waitForLocalDatabaseEntry(context.cliBinary, session.cliEnv, pathB); + assertEqual(receivedB.id, entryB.id, "The first device did not preserve the return document ID."); + await stopTrackedSession(context, session); + console.log("Two real Obsidian devices shared independent document and Chunk IDs in both directions."); + + for (const [label, candidateKey] of [ + ["different key", "cd".repeat(32)], + ["legacy IDs", ""], + ] as const) { + const remoteBefore = await fetchAllCouchDbDocs(context.couchDb, context.dbName); + const checkpointsBefore = await fetchCouchDbLocalDocs(context.couchDb, context.dbName); + const rejectedVault = await createTemporaryVault(); + let rejectedSession: ObsidianLiveSyncSession | undefined; + try { + rejectedSession = await startObsidianLiveSyncSession({ + binary: context.binary, + cliBinary: context.cliBinary, + vault: rejectedVault, + localStorageEntries: createE2eObsidianDeviceLocalState(rejectedVault.name), + pluginData: createE2eCouchDbPluginData( + { ...context.couchDb, dbName: context.dbName }, + { + ...overrides, + idDerivationVersion: candidateKey ? 1 : 0, + idDerivationKey: "", + encryptedIdDerivationKey: candidateKey + ? await encryptString(candidateKey, `*${SALT_OF_PASSPHRASE}`) + : "", + } + ), + }); + context.activeSessions.add(rejectedSession); + await waitForLiveSyncCoreReady(context.cliBinary, rejectedSession.cliEnv); + const unsentPath = "E2E/independent-ids/rejected.md"; + await writeNoteViaObsidian(context.cliBinary, rejectedSession.cliEnv, unsentPath, content); + await waitForLocalDatabaseEntry(context.cliBinary, rejectedSession.cliEnv, unsentPath); + const attempt = await evalObsidianJson<{ admitted: boolean; reason: string; replicated: boolean }>( + context.cliBinary, + [ + "(async()=>{", + "const core=app.plugins.plugins['obsidian-livesync'].core;", + "const replicator=core.services.replicator.getActiveReplicator();", + "const settings=core.services.setting.currentSettings();", + "let reason='';", + "const connection=await replicator.checkReplicationConnectivity(settings,false,false,false,false,undefined,(decision)=>{reason=decision.reason??'';});", + "if(connection) await connection.close();", + "const replicated=await core.services.replication.replicate(true);", + "return JSON.stringify({admitted:!!connection,reason,replicated:!!replicated});", + "})()", + ].join(""), + rejectedSession.cliEnv + ); + assertEqual(attempt.admitted, false, `CouchDB admitted ${label} for obfuscated document IDs.`); + assertEqual( + attempt.reason, + CENTRAL_COMPATIBILITY_REJECTION_REASONS.ID_DERIVATION_MISMATCH, + `CouchDB rejected ${label} for an unrelated reason.` + ); + assertEqual(attempt.replicated, false, `Ordinary replication accepted ${label}.`); + assertEqual( + await pathExists(rejectedVault.path, pathA), + false, + "A rejected device received a remote note." + ); + await stopTrackedSession(context, rejectedSession); + rejectedSession = undefined; + assertEqual( + JSON.stringify(await fetchAllCouchDbDocs(context.couchDb, context.dbName)), + JSON.stringify(remoteBefore), + `A rejected ${label} connection changed remote documents.` + ); + assertEqual( + JSON.stringify(await fetchCouchDbLocalDocs(context.couchDb, context.dbName)), + JSON.stringify(checkpointsBefore), + `A rejected ${label} connection changed remote checkpoints.` + ); + } finally { + if (rejectedSession) await stopTrackedSession(context, rejectedSession); + await rejectedVault.dispose(); + } + } + console.log("Ordinary CouchDB replication rejected different and legacy document ID keys without remote writes."); +} + +async function runDifferentChunkIdKeysRoundTrip( + context: RunnerContext, + vaultA: TemporaryVault, + vaultB: TemporaryVault +): Promise { + const keyA = await deriveIdKey("real-obsidian-e2e-chunk-source-a"); + const keyB = await deriveIdKey("real-obsidian-e2e-chunk-source-b"); + const content = "# Shared content with different Chunk ID keys.\n"; + const pathA = "E2E/chunk-id-keys/from-a.md"; + const pathB = "E2E/chunk-id-keys/from-b.md"; + const commonSettings = { + encrypt: true, + passphrase: "real-obsidian-e2e-chunk-passphrase", + usePathObfuscation: false, + E2EEAlgorithm: "v2", + idDerivationVersion: 1, + }; + const settingsFor = (key: string) => ({ ...commonSettings, idDerivationKey: key }); + const persistedSettingsFor = async (key: string) => ({ + ...settingsFor(key), + idDerivationKey: "", + encryptedIdDerivationKey: await encryptString(key, `*${SALT_OF_PASSPHRASE}`), + }); + const persistedA = await persistedSettingsFor(keyA); + const persistedB = await persistedSettingsFor(keyB); + + let session = await startConfiguredSession(context, vaultA, settingsFor(keyA), persistedA); + await writeNoteViaObsidian(context.cliBinary, session.cliEnv, pathA, content); + const entryA = await uploadNote(context, session, pathA); + if (entryA.children.length === 0) throw new Error("The first device produced no Chunks."); + await stopTrackedSession(context, session); + + session = await startConfiguredSession(context, vaultB, settingsFor(keyB), persistedB); + await syncAndApply(context, session); + await waitForPathContent(vaultB.path, pathA, (received) => received === content); + const receivedA = await waitForLocalDatabaseEntry(context.cliBinary, session.cliEnv, pathA); + assertEqual(receivedA.id, entryA.id, "The second device changed the visible document ID."); + + await writeNoteViaObsidian(context.cliBinary, session.cliEnv, pathB, content); + const entryB = await uploadNote(context, session, pathB); + if (entryB.children.length === 0) throw new Error("The second device produced no Chunks."); + if (JSON.stringify(entryB.children) === JSON.stringify(entryA.children)) { + throw new Error("Different ID keys unexpectedly generated the same Chunk IDs."); + } + await stopTrackedSession(context, session); + + session = await startConfiguredSession(context, vaultA, settingsFor(keyA), persistedA); + await syncAndApply(context, session); + await waitForPathContent(vaultA.path, pathB, (received) => received === content); + const receivedB = await waitForLocalDatabaseEntry(context.cliBinary, session.cliEnv, pathB); + assertEqual(receivedB.id, entryB.id, "The first device changed the return document ID."); + await stopTrackedSession(context, session); + console.log("Two real Obsidian devices exchanged notes with different Chunk ID keys and visible document paths."); +} + async function runMarkdownAutoMerge( context: RunnerContext, vaultA: TemporaryVault, @@ -1094,10 +1288,9 @@ async function runConflictTimeStorageOperations( showMergeDialogOnlyOnActive: true, handleFilenameCaseSensitive: false, }; - const baseContent = Object.fromEntries(paths.map((path) => [path, `# Conflict operation\n\nBase for ${path}.\n`])) as Record< - (typeof paths)[number], - string - >; + const baseContent = Object.fromEntries( + paths.map((path) => [path, `# Conflict operation\n\nBase for ${path}.\n`]) + ) as Record<(typeof paths)[number], string>; const leftContent = Object.fromEntries( paths.map((path) => [path, `${baseContent[path]}\nEdit made on Vault A.\n`]) ) as Record<(typeof paths)[number], string>; @@ -1138,7 +1331,9 @@ async function runConflictTimeStorageOperations( const initialBranchRevisions = new Map>(); for (const path of paths) { const state = await waitForFileConflict(context.cliBinary, session.cliEnv, path); - const displayedBranch = state.branches.find((branch) => branch.content === rightContent[path] && !branch.deleted); + const displayedBranch = state.branches.find( + (branch) => branch.content === rightContent[path] && !branch.deleted + ); if (!displayedBranch) { throw new Error(`Could not identify the branch displayed by Vault B: ${path}; ${JSON.stringify(state)}`); } @@ -1179,12 +1374,7 @@ async function runConflictTimeStorageOperations( "A conflict-time deletion did not extend the displayed revision." ); - await renameNoteViaObsidian( - context.cliBinary, - session.cliEnv, - conflictCaseFromPath, - conflictCaseToPath - ); + await renameNoteViaObsidian(context.cliBinary, session.cliEnv, conflictCaseFromPath, conflictCaseToPath); const caseRenamedBranch = await waitForConflictBranch( context.cliBinary, session.cliEnv, @@ -1225,12 +1415,7 @@ async function runConflictTimeStorageOperations( "A conflict-time case-only rename did not record the new displayed revision." ); - await renameNoteViaObsidian( - context.cliBinary, - session.cliEnv, - conflictRenameFromPath, - conflictRenameToPath - ); + await renameNoteViaObsidian(context.cliBinary, session.cliEnv, conflictRenameFromPath, conflictRenameToPath); const renamedTarget = await waitForLocalDatabaseEntry(context.cliBinary, session.cliEnv, conflictRenameToPath); const renamedSourceDeletion = await waitForConflictBranch( context.cliBinary, @@ -1376,10 +1561,13 @@ async function main(): Promise { const couchDb = await loadCouchDbConfig(); const dbName = makeUniqueDatabaseName(couchDb.dbPrefix, "two-vault-sync"); const encryptedDbName = makeUniqueDatabaseName(couchDb.dbPrefix, "two-vault-sync-e2ee"); + const independentDbName = makeUniqueDatabaseName(couchDb.dbPrefix, "two-vault-sync-independent-ids"); const vaultA = await createTemporaryVault(); const vaultB = await createTemporaryVault(); const encryptedVaultA = await createTemporaryVault(); const encryptedVaultB = await createTemporaryVault(); + const independentVaultA = await createTemporaryVault(); + const independentVaultB = await createTemporaryVault(); const context: RunnerContext = { binary, cliBinary: cli.binary, @@ -1396,11 +1584,20 @@ async function main(): Promise { reviewedVaults: new Set(), activeSessions: new Set(), }; + const independentContext: RunnerContext = { + binary, + cliBinary: cli.binary, + couchDb, + dbName: independentDbName, + reviewedVaults: new Set(), + activeSessions: new Set(), + }; try { await assertCouchDbReachable(couchDb); await createCouchDbDatabase(couchDb, dbName); await createCouchDbDatabase(couchDb, encryptedDbName); + await createCouchDbDatabase(couchDb, independentDbName); console.log(`Using Obsidian executable: ${binary}`); console.log(`Temporary vault A: ${vaultA.path}`); @@ -1409,11 +1606,13 @@ async function main(): Promise { console.log(`Temporary encrypted CouchDB database: ${encryptedDbName}`); const onlyParentCaseDeletion = process.env.E2E_OBSIDIAN_ONLY_PARENT_CASE_DELETION === "true"; + const onlyIndependentIds = process.env.E2E_OBSIDIAN_ONLY_INDEPENDENT_IDS === "true"; + const onlyDifferentChunkIdKeys = process.env.E2E_OBSIDIAN_ONLY_DIFFERENT_CHUNK_ID_KEYS === "true"; if (onlyParentCaseDeletion) { await runParentCaseDeletionProtection(context, vaultA, vaultB); } const onlyConflictOperations = process.env.E2E_OBSIDIAN_ONLY_CONFLICT_OPERATIONS === "true"; - if (!onlyParentCaseDeletion && !onlyConflictOperations) { + if (!onlyParentCaseDeletion && !onlyConflictOperations && !onlyIndependentIds && !onlyDifferentChunkIdKeys) { await runCreateUpdateDelete(context, vaultA, vaultB); await runRename(context, vaultA, vaultB); await runCaseOnlyRename(context, vaultA, vaultB); @@ -1427,17 +1626,27 @@ async function main(): Promise { ) { await runConflictTimeStorageOperations(context, vaultA, vaultB); } - if (!onlyParentCaseDeletion && !onlyConflictOperations) { + if (!onlyParentCaseDeletion && !onlyConflictOperations && !onlyIndependentIds && !onlyDifferentChunkIdKeys) { await runTargetMismatch(context, vaultA, vaultB); await runEncryptedRoundTrip(encryptedContext, encryptedVaultA, encryptedVaultB); } + if (!onlyParentCaseDeletion && !onlyConflictOperations) { + if (onlyDifferentChunkIdKeys) { + await runDifferentChunkIdKeysRoundTrip(independentContext, independentVaultA, independentVaultB); + } else { + await runIndependentIdRoundTrip(independentContext, independentVaultA, independentVaultB); + } + } } finally { await stopTrackedSessions(context); await stopTrackedSessions(encryptedContext); + await stopTrackedSessions(independentContext); await vaultA.dispose(); await vaultB.dispose(); await encryptedVaultA.dispose(); await encryptedVaultB.dispose(); + await independentVaultA.dispose(); + await independentVaultB.dispose(); if (process.env.E2E_OBSIDIAN_KEEP_COUCHDB !== "true") { await deleteCouchDbDatabase(couchDb, dbName).catch((error: unknown) => { console.warn(error instanceof Error ? error.message : error); @@ -1445,6 +1654,9 @@ async function main(): Promise { await deleteCouchDbDatabase(couchDb, encryptedDbName).catch((error: unknown) => { console.warn(error instanceof Error ? error.message : error); }); + await deleteCouchDbDatabase(couchDb, independentDbName).catch((error: unknown) => { + console.warn(error instanceof Error ? error.message : error); + }); } } } diff --git a/updates.md b/updates.md index aec7077d..f11a6b45 100644 --- a/updates.md +++ b/updates.md @@ -16,6 +16,9 @@ Earlier releases remain available in the 1.0 release history, the 1.0 preview hi #### New Feature +- An optional saved ID key can generate encrypted Chunk IDs and obfuscated Metadata document IDs independently of the current E2EE passphrase. + - New Vaults use a random key by default; existing Vaults keep their current ID configuration by default. You can also derive a key from the current E2EE passphrase, enter a separate source, or import a recovery code. The source is not retained; the saved key can be revealed locally as a recovery code. + - The saved key stays in place when the E2EE passphrase changes or E2EE is turned off. Share it with another device through a protected Setup URI. Changing document IDs on an existing remote requires the usual Rebuild and Fetch procedure. - We can now keep the file properties used by Hidden File Sync and Customisation Sync private in CouchDB. - **Encrypt internal file Properties** extends E2EE V2 and Property Encryption to their paths, times, sizes, and Chunk references. - Existing configurations keep this preference disabled. New Vaults enable it for use when the required encryption settings are active. @@ -23,6 +26,9 @@ Earlier releases remain available in the 1.0 release history, the 1.0 preview hi - We can now see which unsupported feature prevents a client from synchronising with CouchDB. - Clients check the features required by the remote before transferring data or resetting the local database for Fast Fetch. Receiving an unsupported requirement also stops active replication. +- We can now compare ID generation performance on a desktop or mobile device through **Open review harness**, available with the developers' debug tools enabled. + - The copied report includes legacy and independent ID timings and, where available, approximate JavaScript heap samples. The measurement uses fixed test data and keeps our Vault and settings unchanged. + #### Fixed - We can now keep using an E2EE passphrase beginning with `%` after restarting Obsidian. (#1221) diff --git a/utils/couchdb/livesync-commonlib.ts b/utils/couchdb/livesync-commonlib.ts index b1a3a30a..b17ec92a 100644 --- a/utils/couchdb/livesync-commonlib.ts +++ b/utils/couchdb/livesync-commonlib.ts @@ -1,5 +1,5 @@ // Keep CouchDB database-version negotiation isolated from Setup URI generation. // The exact release must match utils/livesync-commonlib-version.ts; the setup // tool suite checks every static specifier before release. -export { checkRemoteVersion } from "npm:@vrtmrz/livesync-commonlib@0.1.0-rc.4/compat/pouchdb/negotiation"; -export { PouchDB } from "npm:@vrtmrz/livesync-commonlib@0.1.0-rc.4/compat/pouchdb/pouchdb-browser"; +export { checkRemoteVersion } from "npm:@vrtmrz/livesync-commonlib@0.1.32/compat/pouchdb/negotiation"; +export { PouchDB } from "npm:@vrtmrz/livesync-commonlib@0.1.32/compat/pouchdb/pouchdb-browser"; diff --git a/utils/flyio/deno.lock b/utils/flyio/deno.lock index 7ca3f1f5..6b9f40b0 100644 --- a/utils/flyio/deno.lock +++ b/utils/flyio/deno.lock @@ -1,7 +1,7 @@ { "version": "5", "specifiers": { - "npm:@vrtmrz/livesync-commonlib@0.1.0-rc.4": "0.1.0-rc.4" + "npm:@vrtmrz/livesync-commonlib@0.1.32": "0.1.32" }, "npm": { "@aws-sdk/checksums@3.1000.18": { @@ -284,8 +284,8 @@ "@trystero-p2p/core" ] }, - "@vrtmrz/livesync-commonlib@0.1.0-rc.4": { - "integrity": "sha512-u4FdbjnYg7lAf38z7eUv4eq4vxEdrl4rFMxiDZiJ7T701awKiflkXGIJQnaHdLaMa3zkRBU15qqEo7GtextmlA==", + "@vrtmrz/livesync-commonlib@0.1.32": { + "integrity": "sha512-gzjhd7bg+DKHhd/24WGxBM7557wsw3HJkc0YjKll1n8/8vuhqPnlLuZmM33fj+4bv9nGFUnDlnoVowpQrTns6w==", "dependencies": [ "@aws-sdk/client-s3", "@smithy/fetch-http-handler", @@ -509,8 +509,8 @@ "whatwg-url" ] }, - "octagonal-wheels@0.1.51": { - "integrity": "sha512-KTlfqKPjobHJg/t3A539srnFf+VHr1aXkHSmsNDDpiI5UFC7FamZ95dWpJfGE2EI/HULR5hveQDgkazmz8SAcg==", + "octagonal-wheels@0.1.54": { + "integrity": "sha512-Je3ancYhjKX7UY2K19T/qTjG8C9nK8YVrACr5naIf78mN4bbjQkYyWmlj+ooifV/moWVsQrp4fEWz/7mv6It3A==", "dependencies": [ "idb" ] diff --git a/utils/flyio/generate_setupuri.test.ts b/utils/flyio/generate_setupuri.test.ts index ec36f369..7c35f03e 100644 --- a/utils/flyio/generate_setupuri.test.ts +++ b/utils/flyio/generate_setupuri.test.ts @@ -35,6 +35,13 @@ Deno.test("generates a current self-hosted Setup URI through the published Commo const decoded = await decodeSettingsFromSetupURI(setupURI, "setup-secret"); assert(decoded, "Commonlib could not decode the generated Setup URI"); const effectiveSettings = { ...DEFAULT_SETTINGS, ...decoded }; + const recoveryCode = stdout.match(/sls-id-v1:[0-9a-f]{64}/u)?.[0]; + assert(recoveryCode, "the generator did not print an ID recovery code"); + assert( + (effectiveSettings as typeof effectiveSettings & { idDerivationKey?: string }).idDerivationKey === + recoveryCode.slice("sls-id-v1:".length), + "the CouchDB Setup URI did not contain the generated ID key", + ); assert( effectiveSettings.isConfigured, "the CouchDB Setup URI left the imported device unconfigured", diff --git a/utils/livesync-commonlib-version.ts b/utils/livesync-commonlib-version.ts index 945f0039..47913bdd 100644 --- a/utils/livesync-commonlib-version.ts +++ b/utils/livesync-commonlib-version.ts @@ -2,4 +2,4 @@ // Commonlib registry release. Static npm specifiers cannot interpolate this // value, so livesync-commonlib-version.test.ts verifies the domain-specific // facades against it. -export const LIVESYNC_COMMONLIB_VERSION = "0.1.0-rc.4"; +export const LIVESYNC_COMMONLIB_VERSION = "0.1.32"; diff --git a/utils/readme.md b/utils/readme.md index df4c3c64..894e6baf 100644 --- a/utils/readme.md +++ b/utils/readme.md @@ -31,6 +31,8 @@ Authentication and other non-retryable HTTP failures stop immediately. Network a The existing `flyio/generate_setupuri.ts` path remains a CouchDB-only compatibility wrapper for the Fly.io deployment script. +The generator creates a fresh random ID key by default and includes it in the encrypted Setup URI. It prints a tagged `sls-id-v1:` recovery code. Set `id_recovery_code` to that code when generating another URI for the same Vault; a new run without it creates a different key. This restores only the ID key: reuse the original connection details too. For P2P, provide the original `p2p_room_id` and `p2p_passphrase` because omitted values are generated afresh. Set `id_mode=legacy` to generate a URI with the previous ID behaviour. `id_mode=legacy` and `id_recovery_code` cannot be combined. Keep the recovery code private and retain it if every device might be lost. + ### CouchDB ```sh diff --git a/utils/setup/generate_setup_uri.test.ts b/utils/setup/generate_setup_uri.test.ts index 725725bf..2dea3dbf 100644 --- a/utils/setup/generate_setup_uri.test.ts +++ b/utils/setup/generate_setup_uri.test.ts @@ -26,6 +26,20 @@ Deno.test("generates an Object Storage Setup URI with a selected S3 profile", as ); assert(decoded, "Commonlib could not decode the Object Storage Setup URI"); const effective = { ...DEFAULT_SETTINGS, ...decoded }; + const recoveryCode = generated.idRecoveryCode; + assert( + typeof recoveryCode === "string" && recoveryCode.startsWith("sls-id-v1:"), + "the generator did not return an ID recovery code", + ); + assert( + (effective as typeof effective & { idDerivationVersion?: number }).idDerivationVersion === 1, + "the Setup URI did not enable independent IDs", + ); + assert( + (effective as typeof effective & { idDerivationKey?: string }).idDerivationKey === + recoveryCode.slice("sls-id-v1:".length), + "the Setup URI did not contain the generated ID key", + ); assert( effective.isConfigured, "the Setup URI left the imported device unconfigured", @@ -74,6 +88,11 @@ Deno.test("generates a random-room P2P Setup URI without copying a device identi ); assert(decoded, "Commonlib could not decode the P2P Setup URI"); const effective = { ...DEFAULT_SETTINGS, ...decoded }; + assert( + (effective as typeof effective & { idDerivationKey?: string }).idDerivationKey === + generated.idRecoveryCode?.slice("sls-id-v1:".length), + "the P2P Setup URI did not contain the generated ID key", + ); assert( /^\d{3}-\d{3}-\d{3}-[a-z0-9]{3}$/.test(effective.P2P_roomID), "Commonlib did not generate the expected random room ID", @@ -121,3 +140,46 @@ Deno.test("generates a random-room P2P Setup URI without copying a device identi "the selected profile was not a P2P connection URI", ); }); + +Deno.test("reuses the ID key from a recovery code and permits explicit legacy IDs", async () => { + const environment = { + remote_type: "p2p", + passphrase: "vault-secret", + uri_passphrase: "setup-secret", + }; + const first = await generateSetupURI(environment); + const second = await generateSetupURI({ ...environment, id_recovery_code: first.idRecoveryCode }); + const independentlyGenerated = await generateSetupURI(environment); + assert(second.idRecoveryCode === first.idRecoveryCode, "the recovery code changed on repeat generation"); + assert(independentlyGenerated.idRecoveryCode !== first.idRecoveryCode, "the default ID key was reused"); + const repeatedSettings = await decodeSettingsFromSetupURI(second.setupURI, second.setupPassphrase); + assert(repeatedSettings, "the repeated Setup URI could not be decoded"); + assert( + (repeatedSettings as typeof repeatedSettings & { idDerivationKey?: string }).idDerivationKey === + first.idRecoveryCode?.slice("sls-id-v1:".length), + "the recovery code did not restore the original ID key", + ); + + const legacy = await generateSetupURI({ ...environment, id_mode: "legacy" }); + const decoded = await decodeSettingsFromSetupURI(legacy.setupURI, legacy.setupPassphrase); + assert(decoded, "the legacy Setup URI could not be decoded"); + assert(legacy.idRecoveryCode === undefined, "legacy mode returned an ID recovery code"); + assert( + (decoded as typeof decoded & { idDerivationVersion?: number }).idDerivationVersion !== 1, + "legacy mode enabled independent IDs", + ); + let rejected = false; + try { + await generateSetupURI({ ...environment, id_recovery_code: "sls-id-v1:wrong" }); + } catch { + rejected = true; + } + assert(rejected, "an invalid recovery code was accepted"); + rejected = false; + try { + await generateSetupURI({ ...environment, id_mode: "legacy", id_recovery_code: first.idRecoveryCode }); + } catch { + rejected = true; + } + assert(rejected, "legacy mode silently ignored a recovery code"); +}); diff --git a/utils/setup/generate_setup_uri.ts b/utils/setup/generate_setup_uri.ts index 07c4f1cd..d090d289 100644 --- a/utils/setup/generate_setup_uri.ts +++ b/utils/setup/generate_setup_uri.ts @@ -19,6 +19,36 @@ export interface GeneratedSetupURI { remoteType: SetupRemoteType; setupURI: string; setupPassphrase: string; + idRecoveryCode?: string; +} + +const ID_RECOVERY_CODE_PREFIX = "sls-id-v1:"; +const ID_RECOVERY_CODE_PATTERN = /^sls-id-v1:([0-9a-f]{64})$/u; + +function generateRandomIdKey(): string { + const bytes = crypto.getRandomValues(new Uint8Array(32)); + return Array.from(bytes, (byte) => byte.toString(16).padStart(2, "0")).join(""); +} + +function configureIdDerivation( + settings: ObsidianLiveSyncSettings, + environment: SetupGeneratorEnvironment, +): string | undefined { + const mode = environment.id_mode?.trim().toLowerCase() || "random"; + if (mode !== "random" && mode !== "legacy") { + throw new Error("id_mode must be random or legacy"); + } + const suppliedCode = environment.id_recovery_code?.trim(); + if (mode === "legacy") { + if (suppliedCode) throw new Error("id_recovery_code cannot be used with id_mode=legacy"); + return undefined; + } + const key = suppliedCode + ? ID_RECOVERY_CODE_PATTERN.exec(suppliedCode)?.[1] + : generateRandomIdKey(); + if (!key) throw new Error("id_recovery_code must be a valid sls-id-v1 recovery code"); + Object.assign(settings, { idDerivationVersion: 1, idDerivationKey: key }); + return `${ID_RECOVERY_CODE_PREFIX}${key}`; } function requireValue( @@ -166,11 +196,12 @@ export async function generateSetupURI( const setupPassphrase = environment.uri_passphrase?.trim() || generateSecret(); const { remoteType, settings } = createSetupSettings(environment); + const idRecoveryCode = configureIdDerivation(settings, environment); const setupURI = await encodeSettingsToSetupURI(settings, setupPassphrase, [ "pluginSyncExtendedSetting", "doNotUseFixedRevisionForChunks", ], true); - return { remoteType, setupURI: setupURI.trim(), setupPassphrase }; + return { remoteType, setupURI: setupURI.trim(), setupPassphrase, idRecoveryCode }; } export async function runSetupURIGenerator( @@ -183,6 +214,10 @@ export async function runSetupURIGenerator( generated.setupPassphrase, ); console.log("This passphrase is never shown again, so store it safely."); + if (generated.idRecoveryCode) { + console.log("ID recovery code:", generated.idRecoveryCode); + console.log("Use id_recovery_code with this value and reuse the same remote settings when generating another Setup URI for the same Vault."); + } console.log(generated.setupURI); } diff --git a/utils/setup/livesync-commonlib.ts b/utils/setup/livesync-commonlib.ts index d8091d4d..2443266c 100644 --- a/utils/setup/livesync-commonlib.ts +++ b/utils/setup/livesync-commonlib.ts @@ -4,9 +4,9 @@ export { decodeSettingsFromSetupURI, encodeSettingsToSetupURI, -} from "npm:@vrtmrz/livesync-commonlib@0.1.0-rc.4/compat/API/processSetting"; -export { generateP2PRoomId } from "npm:@vrtmrz/livesync-commonlib@0.1.0-rc.4/compat/common/utils"; -export { upsertRemoteConfigurationInPlace } from "npm:@vrtmrz/livesync-commonlib@0.1.0-rc.4/remote-configurations"; +} from "npm:@vrtmrz/livesync-commonlib@0.1.32/compat/API/processSetting"; +export { generateP2PRoomId } from "npm:@vrtmrz/livesync-commonlib@0.1.32/compat/common/utils"; +export { upsertRemoteConfigurationInPlace } from "npm:@vrtmrz/livesync-commonlib@0.1.32/remote-configurations"; export { createNewVaultSettings, DEFAULT_SETTINGS, @@ -14,5 +14,5 @@ export { PREFERRED_BASE, PREFERRED_JOURNAL_SYNC, PREFERRED_SETTING_SELF_HOSTED, -} from "npm:@vrtmrz/livesync-commonlib@0.1.0-rc.4/settings"; -export type { ObsidianLiveSyncSettings } from "npm:@vrtmrz/livesync-commonlib@0.1.0-rc.4/settings"; +} from "npm:@vrtmrz/livesync-commonlib@0.1.32/settings"; +export type { ObsidianLiveSyncSettings } from "npm:@vrtmrz/livesync-commonlib@0.1.32/settings";