From 0ad71ccecbe404b01cc772cbd35592de580c3afa Mon Sep 17 00:00:00 2001 From: vorotamoroz Date: Fri, 31 Jul 2026 07:31:43 +0000 Subject: [PATCH] docs: specify adaptive journal sync design --- docs/design_docs/adaptive_journal_sync.md | 1117 +++++++++++++++++++-- 1 file changed, 1037 insertions(+), 80 deletions(-) diff --git a/docs/design_docs/adaptive_journal_sync.md b/docs/design_docs/adaptive_journal_sync.md index bd6f028b..c882ca84 100644 --- a/docs/design_docs/adaptive_journal_sync.md +++ b/docs/design_docs/adaptive_journal_sync.md @@ -13,8 +13,8 @@ Journal replication contract across different remote capabilities. The logical model would always treat Chunks as content-addressable, immutable values. The physical representation would adapt to the remote: -- a batch-capable database such as PostgreSQL exposed through PostgREST could store one encrypted Chunk per immutable - row and answer multi-key requests through one RPC; and +- a batch-capable database such as PostgreSQL exposed through PostgREST could store one independently framed Chunk + record per immutable row and answer multi-key requests through one RPC; and - an object-shaped remote such as S3-compatible Object Storage or WebDAV could aggregate Chunks into immutable packs, then resolve Chunk identifiers through an append-only, versioned catalogue. @@ -39,6 +39,24 @@ boundary. Adaptive Journal Sync proposes a domain-level repository boundary above physical object storage. +## Existing implementation seams + +The proposal can remain inside Journal Sync and reuse maintained boundaries rather than adding another top-level +replicator: + +- `JournalSyncCore` already reads exact PouchDB revisions, serialises Journal batches, advances a send checkpoint only + after publication, stores received Chunks before Metadata, and applies Metadata with its supplied revision history. +- `LayeredChunkManager` already provides read and write middleware around the local Chunk database. Its dormant + `HotPackLayer` is an implementation seam for a pack-backed Chunk source, not a requirement to revive the earlier + mutable hot-log design. +- `ChunkFetcher` already collects missing Chunk IDs into bounded groups, owns request claims and terminal conditions, + validates returned Chunks, persists them locally, and notifies waiters. Adaptive Journal Sync should implement the + Journal `fetchRemoteChunks` producer at that boundary rather than introduce another waiting mechanism. +- `putDBEntry` already creates Chunks before Metadata inside the Chunk manager transaction boundary. + +These seams establish where the new repository composes. They do not make CouchDB, P2P, or their connection settings +aware of Adaptive Journal Sync. + ## Relationship to existing designs This proposal builds on, rather than silently replaces, the following documents: @@ -87,6 +105,13 @@ index documents with immutable Level 0 segments, append-only catalogue deltas, a An immutable content value identified by a Chunk ID. If bytes change, the result is a new logical Chunk with a new ID. An existing logical Chunk is never updated in place. +### Remote Chunk key + +A repository-scoped, 32-byte address derived locally from the exact logical Chunk ID after the repository manifest has +been accepted. It is a query and object-routing identifier, not an encryption key. A client can reproduce it when +needed, so it does not persist a secret Chunk-ID-to-remote-key mapping. The same logical Chunk ID always derives the +same Remote Chunk key within one repository namespace. + ### Native CAS A remote representation in which one logical Chunk maps to one insert-only value addressable by an opaque Chunk key. @@ -94,8 +119,9 @@ PostgreSQL is the initial candidate because it can answer `hasMany`, `getMany`, ### Chunk pack -An immutable physical object containing one or more independently verifiable encrypted Chunk records. A pack is a -request-amortisation and compaction unit, not the logical identity of its Chunks. +An immutable physical object containing one or more independently verifiable Chunk records. Encrypted mode +authenticates and encrypts each record; unencrypted mode provides corruption detection without claiming authentication +against a hostile remote. A pack is a request-amortisation and compaction unit, not the logical identity of its Chunks. ### Catalogue @@ -112,6 +138,12 @@ An immutable group of Metadata revision events. It refers to logical Chunk IDs a The final publication record for a Metadata batch. A reader ignores an incomplete batch until its commit manifest is visible and valid. +### Writer stream + +One append-only sequence owned by a stable host ID and a writer epoch. The writer epoch is a random, collision-resistant +identifier persisted with the repository binding. It is not a wall-clock timestamp. A host creates a new writer epoch +after losing or deliberately replacing its local send state. + ## Required invariants 1. Metadata contains Chunk references, never raw file content. @@ -120,12 +152,16 @@ visible and valid. 4. Ordinary writers create immutable records. They do not edit an existing pack, catalogue snapshot, or catalogue delta. 5. A retry may create a duplicate physical Chunk or pack, but it must not create two logical meanings for one Chunk ID. -6. A receiver validates identity and authenticated encryption before accepting a Chunk. +6. A receiver validates identity and, in encrypted mode, authenticated encryption before accepting a Chunk. 7. A derived catalogue, cache, or PostgreSQL materialised index can be discarded and reconstructed. 8. Repacking does not require Metadata revision changes because Metadata refers to logical Chunk IDs rather than pack locations. 9. Remote reset, protocol migration, compaction, retirement, and deletion remain explicit protected operations. 10. The protocol must define a bounded completion or failure condition for each requested Chunk batch. +11. Each writer stream publishes a dense monotonic sequence and never publishes sequence `N + 1` before sequence `N` + has been verified as committed. +12. A receiver records one frontier per writer stream and never treats a global wall-clock or lexicographic maximum as + proof that all earlier work from every writer has been observed. ## Proposed repository boundary @@ -134,14 +170,14 @@ higher-level repository composition: ```ts interface JournalEventStore { - appendMetadataBatch(batch: EncryptedMetadataBatch): Promise; - listMetadataBatches(after: BatchCursor, limit: number): Promise; + appendMetadataBatch(batch: StoredMetadataBatch): Promise; + listMetadataBatches(after: BatchCursor, limit: number): Promise; } interface ChunkStore { capabilities(): ChunkStoreCapabilities; hasMany(ids: readonly RemoteChunkKey[]): Promise; - putMany(chunks: readonly EncryptedChunk[]): Promise; + putMany(chunks: readonly StoredChunkRecord[]): Promise; getMany(ids: readonly RemoteChunkKey[]): Promise; } @@ -154,6 +190,47 @@ interface AdaptiveJournalRepository { The interface is batched even when the physical remote lacks a native batch operation. The implementation owns request planning and must not make the caller loop over one network request per Chunk. +The current opaque `IJournalStorage` return values deliberately collapse several transport outcomes into `false` or an +empty list. Adaptive publication cannot use those values to decide that an object is absent or that an empty repository +is safe to initialise. The adaptive repository therefore needs typed outcomes at its correctness boundary: + +```ts +type RemoteRead = + | { status: "found"; value: T; identity?: string } + | { status: "missing" } + | { status: "failed"; failure: RemoteFailure }; + +type ImmutableCreate = + | { status: "created"; identity?: string } + | { status: "already-exists"; identity?: string } + | { status: "failed"; failure: RemoteFailure }; + +type ListingPage = + | { status: "page"; values: readonly T[]; next: string } + | { status: "complete"; values: readonly T[] } + | { status: "failed"; failure: RemoteFailure }; + +type RemoteFailure = { + category: + | "authentication" + | "permission" + | "rate-limited" + | "unavailable" + | "invalid-response" + | "unknown"; + retry: "never" | "later" | "verify-first"; +}; +``` + +Authentication failure, permission denial, timeout, malformed response, server failure, and an explicitly missing +object remain distinguishable. An adapter may implement these operations by HTTP, SQL RPC, an SDK, or another remote +mechanism. The repository contract defines the observable result, not the provider or wire method. + +`missing` and `already-exists` are expected outcomes rather than failures. A mutation which loses its response is +`unknown` with `verify-first`: the caller reads the exact immutable key before deciding whether to retry. A read-only +timeout may be retried later, while authentication, permission, and invalid-response failures require intervention +rather than an existence assumption. + Capability selection should describe semantics rather than provider names. Candidate capabilities include: - bounded native multi-key lookup; @@ -164,8 +241,116 @@ Capability selection should describe semantics rather than provider names. Candi - inexpensive ordered listing; and - server-side immutable CAS insertion. -The first implementation can use a fixed strategy per adapter. Dynamic thresholds may be added only after measurements -show that they improve the same contract. +The capability report does not benchmark the deployment or recommend a retrieval policy. The initial object-pack +implementation exposes a user preference of `whole-pack` or `range`. `range` is available only when the endpoint probe +has confirmed byte-range semantics; `whole-pack` is the portable default. An automatic policy may be considered later +without changing repository correctness. + +The recommended v1 interpretation treats `range` as a performance preference, not a correctness requirement. If a +previously capable endpoint stops honouring Range requests, normal synchronisation reports the capability change and +falls back to `whole-pack`. A separate strict diagnostic may fail instead, but loss of an optional optimisation should +not make readable data unavailable. + +### Repository initialisation + +An adaptive remote starts with one immutable repository manifest created through `putIfAbsent` semantics. The +recommended v1 manifest owns the Adaptive Security Seed rather than creating an empty Sync Parameters object and +updating its salt in a second operation: + +```json +{ + "chunkKeyMode": "hmac-sha256", + "cipherSuite": "aes-256-gcm", + "format": "adaptive-journal", + "formatVersion": 1, + "manifestAuth": "", + "passwordKdf": { + "iterations": 310000, + "name": "pbkdf2-hmac-sha256", + "outputBytes": 32 + }, + "recordKdf": "hkdf-sha256", + "repositoryId": "<32 random bytes, base64url>", + "requiredCapabilities": [ + "binary-fidelity", + "complete-listing", + "conditional-create", + "delete-visibility", + "read-after-write" + ], + "securitySeed": "<32 random bytes, base64url>" +} +``` + +The public manifest uses a restricted canonical JSON representation: UTF-8, lexicographically sorted object keys, +integers rather than floating-point values, and no insignificant whitespace. `manifestAuth` is calculated over the +canonical bytes with that field omitted. This makes the manifest independently reproducible without making ordinary +control records depend on JSON property order. + +`requiredCapabilities` declares protocol preconditions which the selected adapter must verify before the first write. +It is not a claim made by the server and is not accepted without an adapter check or, for WebDAV, the empirical safety +probe. Optional byte-range support is deliberately absent from this list. + +The v1 encrypted and unencrypted parameter tuples are fixed format constants rather than server-selected negotiation. +For encrypted mode this includes the KDF names, iteration count, output length, Chunk-key mode, and cipher suite shown +above. Unsupported or weaker values fail closed and require a later format version. The trusted local connection +profile also records whether encryption is expected; an encrypted profile never accepts a manifest which switches to +the unencrypted digest mode. + +Repository opening receives an explicit local intent of `create-new` or `attach-existing`. This is an operation +precondition, not a value negotiated with the remote. Initialisation proceeds as follows: + +1. Read the manifest with an outcome which distinguishes `missing` from failure. +2. Under `create-new`, treat an already present manifest as `repository-already-exists`; do not attach or overwrite + implicitly. +3. Under `attach-existing`, treat `missing` as a repository error and create nothing. +4. Under `create-new`, if the manifest is missing, verify the required adapter capabilities, generate `repositoryId` and + `securitySeed` locally, derive the secure repository keys when applicable, calculate `manifestAuth`, and create the + candidate's exact bytes conditionally. +5. If another concurrent creator wins, discard the complete local candidate, read the winner, derive secure keys when + applicable, and verify `manifestAuth`. +6. If create has an unknown outcome, read and compare the exact key before retrying. Never merge fields or overwrite the + manifest. +7. Pin the accepted `repositoryId` in the local repository binding. An existing binding must match it, even when a + substituted manifest has a valid HMAC under a reused passphrase. +8. Verify required adapter capabilities before the first writer registration or other Adaptive write in this client + session. +9. Initialise a device-local writer epoch only after the repository manifest has been accepted. + +The resulting state decisions are: + +| Local expectation | Remote result | Decision | +| --- | --- | --- | +| `create-new`, no binding | Explicitly missing | Conditionally create one candidate | +| `create-new`, no binding | Found | Stop with `repository-already-exists` | +| `attach-existing`, no binding | Found and authenticated as expected | Pin and open | +| `attach-existing`, no binding | Missing | Stop without writing | +| Existing pinned binding | Same authenticated repository ID | Open | +| Existing pinned binding | Missing or different repository ID | Stop for explicit reset or rebind | +| Any state | Authentication, permission, timeout, or invalid response | Stop without treating it as missing | + +Before the first conditional manifest create, the client durably stores the exact candidate bytes and digest as a small +`pendingInitialisation` value. After a restart it resolves that value before starting another create: explicit missing +retries the same candidate, an exact remote digest confirms success, and a different valid manifest means that another +creator won and the complete local candidate is discarded. Pinning the accepted repository ID and clearing +`pendingInitialisation` form one local atomic update. + +The first device therefore generates the random public inputs locally, but the conditionally created manifest decides +which inputs belong to the repository. Additional devices do not copy a device-generated identity key: they derive the +same secure role keys, or the same public identifiers, locally from the accepted manifest. A Setup URI exported from an +accepted binding should also carry the expected, non-secret `repositoryId`; `attach-existing` compares it before any +write. + +The existing opaque Journal Sync Parameters object remains the compatibility mechanism for `opaque-v1`. An +`adaptive-v1` repository uses the single manifest above and does not inherit the current empty-object-then-salt-update +race. A fresh adaptive repository, including an ordinary migration from the opaque layout, generates a new Security +Seed by default. Carrying an existing seed would require an explicit migration format and is not implicit compatibility +behaviour. + +The existing opaque adapters do not yet guarantee this initialisation race. S3-compatible storage needs an +`If-None-Match` condition, WebDAV needs empirically verified `If-None-Match: *` behaviour, and PostgREST needs an +insert-only unique operation rather than an upsert. Adaptive mode must not become writable until its adapter provides +that guarantee. ## Physical strategies @@ -176,26 +361,148 @@ PostgREST can expose a Vault-scoped, row-level-security-protected Chunk table: ```sql create table chunks ( vault_id text not null, - chunk_key text collate "C" not null, - encrypted_body bytea not null, - size_bytes bigint not null, - created_at timestamptz not null, - primary key (vault_id, chunk_key) + repository_id bytea not null check (octet_length(repository_id) = 32), + chunk_key bytea not null check (octet_length(chunk_key) = 32), + record_frame bytea not null, + frame_digest bytea not null check (octet_length(frame_digest) = 32), + size_bytes bigint generated always as (octet_length(record_frame)::bigint) stored, + created_at timestamptz not null default statement_timestamp(), + primary key (vault_id, repository_id, chunk_key) ); ``` +The corresponding writer and committed-Metadata rows can remain small and append-only: + +```sql +create table writer_streams ( + vault_id text not null, + repository_id bytea not null check (octet_length(repository_id) = 32), + writer_stream_id bytea not null check (octet_length(writer_stream_id) = 32), + descriptor_frame bytea not null, + descriptor_digest bytea not null check (octet_length(descriptor_digest) = 32), + created_at timestamptz not null default statement_timestamp(), + primary key (vault_id, repository_id, writer_stream_id) +); + +create table metadata_batches ( + vault_id text not null, + repository_id bytea not null check (octet_length(repository_id) = 32), + writer_stream_id bytea not null check (octet_length(writer_stream_id) = 32), + sequence bigint not null check (sequence > 0), + metadata_frame bytea not null, + metadata_digest bytea not null check (octet_length(metadata_digest) = 32), + created_at timestamptz not null default statement_timestamp(), + primary key (vault_id, repository_id, writer_stream_id, sequence), + unique (vault_id, repository_id, writer_stream_id, sequence, metadata_digest), + foreign key (vault_id, repository_id, writer_stream_id) + references writer_streams (vault_id, repository_id, writer_stream_id) +); + +create table journal_commits ( + vault_id text not null, + repository_id bytea not null check (octet_length(repository_id) = 32), + writer_stream_id bytea not null check (octet_length(writer_stream_id) = 32), + sequence bigint not null check (sequence > 0), + previous_commit_digest bytea null check ( + previous_commit_digest is null or octet_length(previous_commit_digest) = 32 + ), + required_chunk_keys_digest bytea not null check (octet_length(required_chunk_keys_digest) = 32), + metadata_digest bytea not null check (octet_length(metadata_digest) = 32), + commit_frame bytea not null, + commit_digest bytea not null check (octet_length(commit_digest) = 32), + created_at timestamptz not null default statement_timestamp(), + primary key (vault_id, repository_id, writer_stream_id, sequence), + foreign key (vault_id, repository_id, writer_stream_id) + references writer_streams (vault_id, repository_id, writer_stream_id), + foreign key (vault_id, repository_id, writer_stream_id, sequence, metadata_digest) + references metadata_batches ( + vault_id, + repository_id, + writer_stream_id, + sequence, + metadata_digest + ) +); +``` + +These definitions illustrate the semantic constraints rather than a ready migration: the packaged SQL must also install +row-level-security policies, immutable insert-only RPCs, grants, digest calculation, bounded queries, and indexes. The +RPC computes frame digests from supplied bytes, and a client recomputes them after reads. + +`vault_id` is obtained from the authenticated JWT claim by the database policy or security-invoker RPC. It is not +trusted from a field supplied by the binary request. + The public API would use bounded RPCs rather than one ordinary REST request per row: - `has_chunks(keys[])` returns a compact availability result; - `get_chunks(keys[])` returns a framed or streamed binary batch; - `put_chunks(keys[], bodies[])` inserts missing immutable rows and verifies existing identities; and -- `commit_metadata_batch(...)` publishes the Metadata batch only after its required Chunks are present. +- `put_metadata_batch(...)` inserts one immutable, not-yet-visible Metadata row; and +- `commit_metadata_batch(...)` publishes that row only after its required Chunks are present. The exact request limits must be byte-based as well as count-based. A request containing hundreds of small Chunks and a request containing hundreds of large Chunks do not have equivalent memory or proxy cost. PostgreSQL may maintain mutable derived indexes transactionally. The Chunk rows themselves remain immutable CAS values, and any separately maintained operational index must be rebuildable from authoritative rows and batch commits. +`created_at` is diagnostic server time only. It is never a replication cursor, a cross-writer ordering source, or +evidence that a Chunk is safe to collect. + +#### Recommended v1 PostgREST framing + +The initial PostgREST experiment should continue the existing binary RPC approach rather than encode every `bytea` value +inside JSON. A bounded `application/octet-stream` argument and response use: + +```text +BatchEnvelopeV1 := + magic[4] // "LSAB" + formatVersion:u8 + operation:u8 + flags:u16be + entryCount:u32be + totalLength:u64be + entries[entryCount] +``` + +Availability entries contain a 32-byte remote Chunk key. Put entries add a frame length and one complete +`RecordFrameV1`. Get responses add an explicit per-key `found` or `missing` status and include a frame only for `found`. +The request and response preserve input order so the client does not need a second key map. + +The recommended operation and status codes are: + +| Operation | Request entry | Response entry | +| --- | --- | --- | +| `0x01 HAS` | `remoteChunkKey[32]` | `status:u8`, where `0` is missing and `1` is present | +| `0x02 GET` | `remoteChunkKey[32]` | `status:u8`; found adds `frameDigest[32]`, `frameLength:u64be`, and `RecordFrameV1` | +| `0x03 PUT` | `remoteChunkKey[32]`, `frameDigest[32]`, `frameLength:u64be`, and `RecordFrameV1` | `status:u8`, where `0` is inserted, `1` is the exact existing frame, and `2` requires existing-frame validation | + +`flags & 0x0001` marks a response; every other v1 batch flag is zero. Authentication, authorisation, malformed input, +server unavailability, and transaction failure remain typed RPC failures rather than per-entry Chunk statuses. A PUT +digest is an early framing check only: the server recomputes SHA-256 over the supplied frame. + +The final Metadata publication uses a separate bounded `CommitEnvelopeV1`, because it has one writer identity and +sequence rather than homogeneous Chunk entries. It contains the repository ID, writer stream ID, sequence, nullable +previous commit digest, sorted unique required-key set and digest, Metadata logical key and digest, and exact commit +frame. Stage 0 fixtures freeze its field offsets before the SQL function is installed. + +Every RPC enforces both an entry-count limit and a total-byte limit before allocating or concatenating a response. v1 +does not claim unbounded streaming through PostgREST: the client pages large work into bounded binary RPCs. This avoids +JSON base64 or hexadecimal expansion while keeping PostgreSQL `bytea` materialisation explicit and measurable. +`totalLength` is the exact complete envelope length, including its fixed header, and the decoder rejects a mismatch. +Unsigned 64-bit lengths are parsed without conversion to a JavaScript `number` until the configured maximum has been +checked. + +The RPC computes the SHA-256 frame digest from the supplied bytes rather than trusting a client assertion, and clients +recompute it after every read. `put_chunks` inserts missing rows and treats an exact existing frame digest as an +idempotent retry. If the key exists with a different frame digest, the RPC returns a per-key conflict rather than +replacing the row. Different random record salts mean that two honest clients can produce different valid frames for +the same logical Chunk, so the client must fetch, decrypt when applicable, and validate the existing frame against the +requested remote Chunk key. It then accepts an equivalent existing Chunk or reports corruption; the server does not +infer plaintext equality from a frame digest. + +The transactional Metadata commit RPC verifies the required-key-set digest, checks all required Chunk and Metadata rows, +enforces the writer sequence and previous commit digest, and inserts only the final commit row. Unique constraints +provide idempotence for repository ID, writer stream, sequence, and remote Chunk key. ### Immutable pack CAS @@ -208,13 +515,187 @@ The pack strategy groups newly published Chunks into immutable objects: ```text packs/.bin indexes/.idx -catalogue/deltas/ +catalogue/deltas//.delta ``` -The pack index maps each opaque remote Chunk key to its ciphertext offset, ciphertext length, and integrity information. -S3-compatible storage can use byte-range reads when the pack format encrypts and authenticates records independently. -WebDAV can fetch the complete pack when reliable range reads are unavailable. Both paths expose the same batched -`getMany` result. +The pack index maps each opaque remote Chunk key to its complete frame offset, frame length, and integrity information. +The canonical retrieval path downloads a complete pack. A confirmed byte-range capability may instead retrieve selected +records when the user has chosen that policy. Both paths expose the same batched `getMany` result, and Range support is +not an Adaptive Journal safety requirement. + +Transfer and local materialisation are separate choices. A client may cache one downloaded pack, extract only the +currently requested Chunks, and retain the indexed pack for later requests. It need not immediately write every +unrequested record into the local Chunk database. Conversely, Fast Setup or another dense read can materialise the +complete pack in one pass. + +The pack must contain independently framed, compressed, and integrity-checked records. Encrypted mode encrypts and +authenticates each record independently. Whole-pack compression or whole-pack AEAD would make an isolated Range response +unusable. The verified index supplies the byte offsets needed for Range retrieval, while the complete-pack path +validates the same records. + +#### Recommended v1 record envelope + +Bulk data uses a small binary frame rather than JSON or a provider-specific row representation: + +```text +RecordFrameV1 := + magic[4] // "LSAR" + formatVersion:u8 // 1 + recordKind:u8 + flags:u16be + publicHeaderLength:u32be + payloadLength:u64be + publicHeader[publicHeaderLength] + payload[payloadLength] +``` + +The v1 record kinds and their provider-independent logical keys are: + +| Code | Record kind | Canonical logical key | Encryption role | +| --- | --- | --- | --- | +| `0x01` | Chunk | `remoteChunkKey[32]` | `chunk-record` | +| `0x02` | Pack index | `packIdDigest[32]` | `pack-index` | +| `0x03` | Metadata batch | `writerStreamId[32] || sequence:u64be` | `metadata-record` | +| `0x04` | Catalogue delta | `writerStreamId[32] || sequence:u64be` | `catalogue-record` | +| `0x05` | Catalogue snapshot | `snapshotId[32]` | `catalogue-record` | +| `0x06` | Writer descriptor | `writerStreamId[32]` | `writer-record` | +| `0x07` | Commit | `writerStreamId[32] || sequence:u64be` | `commit-record` | + +`snapshotId` is a random identifier generated before encoding; its exact frame digest is calculated afterwards. The +logical key is independent of the provider's physical object-name encoding. S3-compatible storage may use hierarchical +paths, WebDAV may use flat names, and PostgREST may use columns without changing record encryption or identity. + +`flags & 0x0001` means encrypted, and every other v1 flag bit is zero. The flag must agree with the encryption mode +pinned in the local connection profile and repository manifest. The public header is exactly one of: + +```text +EncryptedHeaderV1 := + codec:u8 + reserved[3] // zero + plaintextLength:u64be + recordSalt[32] + iv[12] + +UnencryptedHeaderV1 := + codec:u8 + reserved[3] // zero + plaintextLength:u64be + payloadDigest[32] +``` + +The corresponding `publicHeaderLength` values are 56 and 44 bytes. Codec `0x00` is no compression. Codec `0x01` is the +raw DEFLATE stream accepted by the maintained `fflate` `deflate` and `inflate` boundary; a writer uses it only when the +resulting payload is smaller. Valid encoders do not need to produce identical compressed bytes. Adding another codec +requires a later format version. + +An encrypted record's payload is AES-256-GCM ciphertext with its 16-byte tag appended by the cipher operation. An +unencrypted record's payload is compressed or uncompressed plaintext, and `payloadDigest` is SHA-256 over those exact +stored bytes. That digest detects accidental damage but does not authenticate data against a hostile remote. + +Numeric lengths are unsigned big-endian integers and every decoder applies explicit maximum values before allocating +memory. Reserved bytes and flag bits must be zero, unsupported record kinds, codecs, or modes fail closed, and the +declared public-header and payload lengths must equal the available bytes. The baseline v1 record, batch, and pack byte +ceilings must be frozen with Stage 0 fixtures before a writer is enabled; a local implementation may impose a lower +operational limit only if it reports the incompatibility instead of treating an oversized valid record as corruption. + +In encrypted mode, AEAD additional authenticated data is: + +```text +aad = canonicalBytes( + "livesync/adaptive-journal/record-aad/v1", + repositoryId[32], + logicalKeyLength:u16be, + logicalKey[logicalKeyLength], + fixedRecordPrefix[20], + publicHeader[publicHeaderLength] +) +``` + +The fixed record prefix already contains the magic, format version, record kind, flags, and both lengths. The logical +key is supplied by the PostgreSQL row, verified pack index, or decoded provider-independent control-record route rather +than repeated in a pack frame. The per-record encryption key is: + +```text +recordKey = HKDF( + roleKey, + salt = recordSalt, + info = canonicalBytes( + "livesync/adaptive-journal/record/v1", + recordKind, + logicalKey + ) +) +``` + +`canonicalBytes` uses fixed-width numeric fields and explicit byte lengths, rather than ambiguous string +concatenation. + +The existing `encryptBinary` helper cannot be reused unchanged because its HKDF `info` is empty and its AES-GCM call +does not accept additional authenticated data. The new format should reuse its reviewed PBKDF2, HKDF, IV, and AES-GCM +primitives behind a new domain-separated envelope API. + +#### Chunk payload + +`ChunkPayloadV1` contains the exact local Chunk ID, the Chunk-format version needed to validate it, and the canonical +Chunk bytes before transport compression. The receiver: + +1. verifies the frame and payload digests and, in encrypted mode, AEAD; +2. decompresses according to the record codec; +3. recomputes the local Chunk ID from the bytes and declared Chunk-format version; +4. compares it with the payload ID; +5. derives the remote Chunk key again and compares it with the requested or indexed key; and +6. only then writes the Chunk through the maintained Chunk manager. + +Compression is independent per record and may be `none` when compression does not reduce size. Old and new codec choices +can coexist because the codec is covered by record integrity protection and, in encrypted mode, AEAD. + +#### Pack and index + +`pack.bin` is a concatenation of complete `RecordFrameV1` values. It has no whole-pack compression or whole-pack +encryption. Its identifier is the base64url SHA-256 digest of its exact bytes, so an ordinary complete download verifies +the object before reading frames: + +```text +packId = base64url(SHA-256(packBytes)) +``` + +`indexes/.idx` is one index record, encrypted in encrypted mode and digest-protected plaintext in unencrypted +mode. Its payload contains entries sorted by the 32-byte binary remote Chunk key: + +```text +PackIndexPayloadV1 := + entryCount:u32be + entries[entryCount] + +PackIndexEntryV1 := + remoteChunkKey[32] + offset:u64be + frameLength:u64be + plaintextLength:u64be + frameDigest[32] +``` + +The index envelope binds `repositoryId`, `packId`, its entry count, and its exact payload digest. A Range read verifies +`frameDigest` before the record's mode-specific integrity and AEAD checks; a whole-pack read verifies `packId` and then +the same per-record checks. The index is downloaded and cached as one small object. PostgREST stores the same +`RecordFrameV1` value directly in a Chunk row, so both physical strategies share framing and validation rather than only +TypeScript method names. + +Keys are strictly increasing and unique. Every offset and length must identify one complete frame within the exact pack, +entries must not overlap, and the indexed frames must cover the pack without unindexed gaps. A decoder checks all +arithmetic for overflow before issuing a Range request or allocating a complete-pack buffer. + +#### Recommended local pack cache + +A complete-pack read stores the exact remote pack bytes under `(repositoryId, packId)` in a bounded, disposable LRU +cache. Normal incremental synchronisation materialises only the Chunks currently requested by `ChunkFetcher`; later +requests reuse the verified cached pack. Fast Setup may deliberately validate and materialise every record in the pack +to favour throughput. + +A Range read may cache verified individual frames without pretending that a sparse file is a complete pack. Persistent +cache entries retain exact remote bytes, not separately decoded Chunk plaintext. In encrypted mode those bytes remain +encrypted. Cache loss or eviction affects performance only and never changes the receive frontier or authoritative +catalogue. Pack targets must be expressed primarily in bytes. The initial experiment should compare small immutable Level 0 micro-packs with larger compacted Level 1 packs. A latency-sensitive synchronisation may publish a pack containing one @@ -259,9 +740,11 @@ loose objects only when measurements justify them. The remote object-store catalogue is logically versioned but not rewritten in full for every ordinary write: ```text -catalogue/snapshots/ -catalogue/deltas// -catalogue/commits/ +catalogue/snapshots/.snapshot +writers/.writer +catalogue/deltas//.delta +metadata//.batch +commits//.commit ``` A delta can register a new pack: @@ -269,9 +752,15 @@ A delta can register a new pack: ```json { "add": { - "packId": "pack-002", - "index": "indexes/pack-002.idx" - } + "indexDigest": "", + "indexKey": "indexes/pack-002.idx", + "packBytes": 123456, + "packId": "pack-002" + }, + "formatVersion": 1, + "repositoryId": "", + "sequence": "00000000000000000001", + "writerStreamId": "" } ``` @@ -279,9 +768,57 @@ The index contains the Chunk-to-location entries, so a catalogue delta stays sma independent immutable deltas; their additions merge as a set and do not compete to replace one shared catalogue object. The same logical Chunk may temporarily resolve to several valid packs. +A delta becomes authoritative only when a valid commit in the same writer sequence references its exact key and digest. +Listing a raw delta object is not evidence that its publication completed. A catalogue snapshot likewise records the +covered per-writer commit frontier and commit digests, rather than only the largest object name it observed. + +These are logical keys. A WebDAV adapter may encode them as flat object names when the configured server cannot create +or list nested collections reliably. + +The recommended WebDAV v1 mapping stays inside the one configured collection and uses only ASCII base64url identifiers, +20-digit sequences, and a reserved `a1~~...` prefix. For example: + +```text +a1~manifest.json +a1~writer~.writer +a1~pack~.bin +a1~index~.idx +a1~delta~~.delta +a1~metadata~~.batch +a1~commit~~.commit +a1~snapshot~.snapshot +a1~catalogue-latest.hint +``` + +Base64url fields have no padding and sequences have exactly 20 decimal digits. The mapping does not require nested +`MKCOL`. S3-compatible storage may retain hierarchical logical keys, while both mappings round-trip to the same +repository identifiers. + +### Control record encoding and visibility + +In encrypted mode, the recommended v1 privacy boundary leaves only the repository manifest, immutable object keys, +writer-stream routing, sequence numbers, object sizes, and access patterns visible. Pack indexes, catalogue deltas, +catalogue snapshots, Metadata batches, writer descriptors, and commit bodies use role-specific encrypted record +envelopes. Unencrypted mode uses the same envelopes with `cipherSuite: "none"` and makes those payloads visible. + +After decryption, catalogue, writer-descriptor, and commit payloads use the same restricted canonical JSON subset as the +repository manifest. IDs and digests are base64url strings, counters and byte sizes are integers, and no field uses a +floating-point value. Metadata batches retain the maintained Journal document encoding instead. The canonical +representation is intended for deterministic hashing and fixtures; AEAD authenticates the exact stored bytes in +encrypted mode. + +Every control payload repeats its repository ID and complete logical record identity. A decoder compares those fields +with the manifest, writer route, sequence, and frame kind before using the payload. This redundancy detects accidental +cross-key placement in unencrypted mode; it is not presented as authentication against a hostile remote. + +An optional plaintext `catalogue/latest` hint may contain only a format version, snapshot object key, and ciphertext +digest. Encrypted mode authenticates the hint with a catalogue-hint key. It is never authoritative: a missing, stale, or +invalid hint causes snapshot and delta discovery from immutable keys. + ### Local derived catalogue -A client loads the latest trusted snapshot it knows, applies later deltas, and builds a mutable local mapping: +A client loads the latest trusted snapshot it knows, follows each writer's later valid commits, applies their referenced +catalogue deltas, and builds a mutable local mapping: ```text Remote Chunk key -> [pack ID, offset, length, generation] @@ -306,37 +843,216 @@ The snapshot commit records its covered delta frontier, integrity hash, format v Readers apply deltas outside that frontier. Old snapshots and covered deltas remain available for a grace period before protected deletion. +### Writer discovery and frontiers + +A writer stream is keyed by `(hostId, writerEpoch)`. `hostId` is the stable device identity. `writerEpoch` is generated +randomly and persisted with that remote repository binding. It changes when a cloned host or a host which has lost its +send sequence begins writing, preventing two processes with stale local state from sharing one sequence. + +The tuple is the logical identity. Encrypted mode may encode the physical `writerStreamId` as an HMAC under a +domain-separated writer-name key so that the remote does not receive the raw stable host ID. Unencrypted mode may use a +repository-scoped digest. Readers use the immutable writer descriptor, encrypted in encrypted mode, to recover the +logical fields. + +```text +writerStreamId = HMAC-SHA-256( + roleKey("writer-name"), + canonical(hostId, writerEpoch) +) +``` + +Each stream uses a positive 63-bit sequence in the range `1..9223372036854775807`. This fits PostgreSQL `bigint`, +uses `u64be` with the high bit clear in canonical binary records, and uses zero-padded 20-digit decimal in object names. +Sequence assignment is dense: + +```text +commits/host-a/epoch-x/00000000000000000001.commit +commits/host-a/epoch-x/00000000000000000002.commit +commits/host-b/epoch-y/00000000000000000001.commit +``` + +A writer which could exhaust the range creates a new writer epoch before assigning another sequence; wrapping to zero +or a negative database value is invalid. + +A reader discovers immutable writer descriptors, then lists each writer stream after its own stored sequence. The +receive checkpoint is a map from writer stream ID to the highest contiguous, fully applied sequence. A reader does not +advance past a missing sequence, and it continues processing independent streams while one stream is incomplete. + +Writer discovery itself must not use one global `StartAfter` maximum, because a newly registered writer may sort before +that maximum. The descriptor set is expected to remain small enough for complete listing initially. A reconstructible +writer snapshot may optimise discovery later. + +There is no cross-writer timestamp order. PouchDB revision ancestry and conflict branches carry the causal information +needed for convergence. Adaptive Journal Sync preserves those exact revision histories; the repository only guarantees +publication order within one writer stream. + +### Writer state and crash recovery + +The device-local repository binding persists: + +```ts +type AdaptiveWriterState = { + repositoryId: string; + writerEpoch: string; + pendingWriterDescriptor?: { + key: string; + exactBytes: Uint8Array; + digest: string; + }; + lastCommittedSequence: string; + lastCommitDigest: string | null; + pendingCommit?: { + sequence: string; + key: string; + exactBytes: Uint8Array; + digest: string; + }; +}; +``` + +The epoch is generated locally, saved before publication, and registered through an immutable conditional writer +descriptor. The exact descriptor frame is persisted with `pendingWriterDescriptor` before that create. Success or an +unknown response is resolved by reading the descriptor key and comparing its exact digest; no commit is attempted until +registration has been confirmed and the pending value cleared atomically. A different digest at the same writer stream +ID is a collision. + +The writer does not reconstruct and continue an old epoch after losing this local state; it creates a new epoch and +leaves the old stream readable. + +Publication uses the following crash boundary: + +1. Allocate `lastCommittedSequence + 1` without publishing a later sequence. +2. Upload and verify any Chunk frames, pack, index, catalogue delta, and Metadata batch. A crash here may leave harmless + unreachable objects. +3. Build the final commit record, then durably store its exact key, bytes, and digest as `pendingCommit`. +4. Create the commit conditionally. +5. On success or an unknown response, read the commit key and require an exact digest match. +6. Atomically advance `lastCommittedSequence` and `lastCommitDigest`, then clear `pendingCommit`. +7. After restart, resolve `pendingCommit` before creating any new publication. + +Only the small final commit must be retained byte-for-byte across the uncertain mutation. Earlier immutable objects can +be verified and reused. Before rebuilding an object with a fixed logical key, such as a Metadata batch, catalogue delta, +or pack index, a restarted writer reads that key first. A valid existing record is adopted, a missing record may be +rebuilt, and a different logical value is a collision rather than permission to overwrite. A not-yet-referenced pack +candidate may instead be abandoned and replaced under a new content-derived pack ID. + +PostgREST uses the same idempotency key, sequence, and previous digest even when one transaction performs the final +server-side verification and commit. + ## Publication protocol +### Recommended Metadata and commit records + +`MetadataBatchV1` reuses the maintained Journal document encoding after excluding Chunk documents. It preserves each +PouchDB document ID, exact revision, revision ancestry, deletion marker, conflict branch, Metadata fields, and logical +Chunk references. The batch also records the sorted set of required remote Chunk keys. It is compressed as one bounded +batch, then placed in a metadata-record envelope. Encrypted mode encrypts it with AAD binding the repository ID, writer +stream, and sequence; unencrypted mode stores the same bounded payload without encryption. + +The final `CommitV1` control payload, encrypted in encrypted mode, contains: + +```json +{ + "catalogueDeltas": [ + { + "digest": "", + "key": "" + } + ], + "formatVersion": 1, + "metadata": { + "bytes": 1234, + "digest": "", + "key": "" + }, + "previousCommitDigest": "", + "repositoryId": "", + "requiredChunkKeysDigest": "", + "sequence": "00000000000000000001", + "writerStreamId": "" +} +``` + +The commit's own digest is SHA-256 over its exact stored bytes. `previousCommitDigest` forms a per-writer hash chain in +addition to the dense sequence, detecting an unexpected predecessor or a listing gap. An optional creation time may +exist inside the payload for diagnostics, but it has no ordering semantics. + +For PostgREST, the transaction receives the same sorted required-key set and verifies its digest before checking Chunk +and Metadata row existence and inserting the commit. For object stores, the client verifies the referenced pack, index, +catalogue, and Metadata objects before creating the commit. Readers ignore all uncommitted data objects. + +### Common publication state machine + +The final commit, rather than successful upload of any preceding object, changes reader-visible state: + +| Observed state | Reader and writer interpretation | +| --- | --- | +| Data objects exist, commit is missing | Incomplete or abandoned publication; readers ignore it | +| Commit exists and every referenced digest and dependency validates | Publication is eligible for receive | +| Commit is missing after an unknown create result | Writer retries the read; it does not allocate the next sequence | +| Commit key exists with the intended exact digest | Idempotent success | +| Commit key exists with a different digest | Writer-state collision or repository corruption; stop the stream | +| Commit is valid but a dependency is temporarily unavailable | Do not advance this stream; other writer streams may continue | +| Sequence `N + 1` is visible while `N` is missing | Do not skip the gap or infer deletion from listing order | + +An adapter may combine physical steps in one database transaction, but it must expose these same outcomes. In +particular, an HTTP success for a data upload is not a commit, and an HTTP timeout is not evidence that a commit failed. + ### Object-store publication 1. Build and validate a Chunk micro-pack locally. 2. Upload the immutable pack under a unique ID. 3. Upload its immutable index. 4. Publish an immutable catalogue delta which registers the pack. -5. Upload the encrypted Metadata batch. -6. Publish the batch commit manifest last. +5. Upload the Metadata batch record. +6. Publish the batch commit manifest at the next sequence in this writer stream. 7. Advance the local send checkpoint only after the commit manifest succeeds. A failure before step 6 may leave unreachable objects, but it does not publish Metadata with unavailable Chunk references. A retry can reuse verified immutable objects or publish replacements under new IDs. Later maintenance can remove unreachable incomplete publications. +The writer does not begin sequence `N + 1` while the result of sequence `N` is unknown. After a transport timeout, it +reads and verifies the intended immutable commit or retries the same key and bytes. A conditional conflict is acceptable +only when the existing object has the same authenticated identity. Otherwise the writer stops and reports repository +corruption or writer-state collision. + ### Native CAS publication -PostgreSQL can perform Chunk insertion, required-Chunk verification, Metadata batch insertion, and commit publication -inside a transaction. The external semantics remain equivalent to the object-store commit-manifest sequence. +The recommended PostgreSQL path keeps potentially large Chunk insertion outside the small final publication +transaction: + +1. Use bounded `has_chunks` and `put_chunks` RPCs to insert missing immutable Chunk rows. +2. Resolve every different-frame conflict by fetching and validating the existing logical Chunk. +3. Build and conditionally insert the immutable Metadata frame. On conflict or an unknown response, read it and validate + its logical identity and content before reuse. +4. Build the exact commit frame, then persist it as the local `pendingCommit`. +5. Call `commit_metadata_batch` with the sorted, unique required-key set, Metadata digest, and commit frame. +6. In one transaction, derive `vault_id` from the signed claim, verify the repository and writer stream, recompute the + required-key-set digest, require every Chunk and Metadata row, enforce the dense sequence and previous commit digest, + and insert the commit row. +7. On success or an unknown response, read the commit by `(repositoryId, writerStreamId, sequence)` and compare its exact + digest before advancing local state. + +The final transaction does not resend Chunk or Metadata bodies and does not need a native Chunk catalogue; its +`catalogueDeltas` list is empty. Bounded pre-publication transactions may leave harmless unreferenced rows after a +crash, just as object storage may leave unreferenced packs. This separation limits transaction memory and lock duration +while retaining atomic Metadata visibility. ### Receive -1. List committed Metadata batches after the local cursor. -2. Decrypt and validate Metadata events. -3. Collect all referenced Chunks missing from the local database. -4. Call `getMany` once per bounded byte and count window. -5. Let the Chunk Store group object-store requests by pack or issue a native batch RPC. -6. Validate and persist Chunks. -7. Apply Metadata revisions through the maintained PouchDB revision contract. -8. Advance the receive checkpoint only after the batch reaches its terminal state. +1. Discover writer streams and list commit records after each per-writer frontier. +2. Require the next dense sequence, validate the writer descriptor, commit frame, previous digest, and every referenced + exact-byte digest. +3. Load, decrypt when applicable, and validate the Metadata batch. +4. Derive the sorted required remote Chunk keys from its logical Chunk references and compare their digest with the + commit. +5. Collect the referenced Chunks missing from the local database. +6. Call `getMany` once per bounded byte and count window. +7. Let the Chunk Store group object-store requests by pack or issue a native batch RPC. +8. Validate and persist Chunks. +9. Apply Metadata revisions through the maintained PouchDB revision contract. +10. Advance the receive checkpoint only after the batch reaches its terminal state. A commit manifest may include non-authoritative location hints for newly published Chunks. Metadata continues to refer only to logical Chunk IDs so that repacking never changes file revisions. @@ -346,11 +1062,16 @@ only to logical Chunk IDs so that repacking never changes file revisions. Ordinary publication is multi-writer and append-only: - pack IDs, delta IDs, and batch IDs must be globally collision-resistant; +- one writer stream has exactly one publisher and one dense sequence; - concurrent publication of the same logical Chunk may create physical duplicates; - catalogue additions merge without a last-write-wins replacement; - failed uploads remain unreachable until maintenance removes them; and - retries are idempotent at the logical Chunk and Metadata revision boundaries. +The current opaque Journal filename cursor takes one global maximum after sorting timestamp-shaped keys. Adaptive +Journal Sync must not reuse that rule. A late writer, reset clock, or delayed upload can sort before the recorded +maximum. Per-writer frontiers make `StartAfter` safe only inside an already discovered stream. + Physical retirement and deletion require stronger coordination than addition. A compactor must use a bounded lease or fencing token appropriate to the remote. A stale compactor may upload a redundant replacement pack, but it must not publish a retirement or delete an object after losing its authority. @@ -358,30 +1079,163 @@ publish a retirement or delete an object after losing its authority. The protocol must test each failure boundary independently: pack upload, index upload, catalogue delta publication, Metadata upload, commit publication, checkpoint persistence, snapshot publication, retirement, and deletion. +## WebDAV server safety checker + +WebDAV implementations, gateways, and authentication layers vary even when they expose the same methods. Server +implementations should strive to preserve the required semantics, but configuration must rely on a non-destructive +empirical checker rather than a product-name allowlist or a standards-compliance assumption. + +The checker creates random probe keys in a reserved temporary namespace inside the configured collection. It touches no +existing Journal key, attempts to remove every probe in a `finally` path, and reports any residual key when cleanup +cannot be confirmed. It does not log credentials, custom headers, response bodies, or secret-bearing URLs. + +Each check reports `passed`, `failed`, or `inconclusive`, together with a bounded, sanitised reason: + +| Profile | Capability | Probe | +| --- | --- | --- | +| Opaque Journal safety | Collection access | Create or access the configured collection | +| Opaque Journal safety | Binary fidelity | `PUT` non-text bytes, then `GET` and compare exactly | +| Opaque Journal safety | Read-after-write | Observe the newly written bytes without stale-cache substitution | +| Opaque Journal safety | Complete depth-one listing | Confirm that `PROPFIND` includes the owned probe key | +| Opaque Journal safety | Delete visibility | Delete the probe, then confirm that its `GET` is explicitly missing | +| Adaptive Journal safety | Conditional create | Create with `If-None-Match: *`, reject a second different body, and retain the first body | +| Optional optimisation | Byte-range read | Request a non-trivial byte interval and require an exact `206` response | + +An HTTP `200` whole body in response to the Range probe means that byte-range optimisation is unsupported; it does not +make the server unsafe. A `206` response must contain the requested bytes and a consistent `Content-Range`. Timeout or +transport ambiguity is `inconclusive`, not evidence that an object is missing. + +The report answers only whether the endpoint exhibits each capability. It does not benchmark the endpoint, predict +which retrieval policy is faster, or recommend a user preference. Existing opaque Journal mode requires the opaque +safety profile. Adaptive mode additionally requires confirmed conditional create. A user may choose `whole-pack` +regardless of Range support, or choose `range` after the optional probe passes. + +The recommended schedule runs the checker during an explicit connection test and before the first Adaptive write in a +new repository-client session. A session may reuse the report until endpoint, collection, credentials, or custom headers +change. A semantic failure of a previously passed operation invalidates the cached report. It need not create probe +objects for every ordinary synchronisation. + ## Encryption and privacy The current whole-pack encryption can hide document IDs, Chunk IDs, and pack contents from the storage service. Native CAS and searchable pack indexes introduce different leakage, so they require an explicit threat-model review. -The remote Chunk key should be an opaque Vault-scoped derivation rather than a raw content hash. One candidate is: +### Recommended key lifecycle + +The per-Chunk `remoteChunkKey` is never randomly generated. It is deterministically derived locally on demand after the +immutable manifest has established the repository key schedule, and it is not stored as a secret mapping. + +The recommended derivation and creation boundary is: + +| Value | Creation or derivation time | Persistence | +| --- | --- | --- | +| `repositoryId`, Security Seed | Generated with a secure local random source only when a manifest read returns explicit `missing`; the winning conditional create selects the repository values | Immutable remote manifest and local repository binding | +| Master key and role keys | Derived locally in encrypted mode after every repository-client open and successful manifest authentication | Memory only | +| Writer epoch | Generated locally after manifest acceptance and before the first publication by that device binding | Device-local writer state and immutable remote writer descriptor | +| `remoteChunkKey` | Derived on demand when a referenced Chunk is checked, stored, or fetched | Normally not persisted; a disposable local cache is permitted | +| Record salt and IV | Generated locally each time a secure record frame is first encoded | Inside that immutable frame | +| Pack ID and frame digest | Derived after the exact immutable bytes have been finalised | Immutable object key, index, row, or commit reference | + +The order matters: an explicit remote failure never triggers local candidate generation, and no writer or Chunk +identity is committed before the manifest has been read back and accepted. A candidate generated by a losing +initialiser is discarded as one unit. + +For an encrypted repository: ```text -remoteChunkKey = HMAC(vaultChunkIdentityKey, localChunkID) +masterKey = PBKDF2-HMAC-SHA-256( + passphrase, + manifest.securitySeed, + iterations = 310000, + outputBytes = 32 +) + +roleKey(role) = HKDF-SHA-256( + masterKey, + salt = decodeBase64url(manifest.repositoryId), + info = UTF8("livesync/adaptive-journal/v1/" || role), + outputBytes = 32 +) + +remoteChunkKey = HMAC-SHA-256( + roleKey("chunk-identity"), + canonicalUTF8(localChunkID) +) ``` -This prevents correlation of the same content across independent Vaults while preserving equality and deduplication -inside one Vault. It still reveals within-Vault equality, object count, sizes, and access patterns to the remote. +`canonicalUTF8` includes a format-domain prefix and an explicit byte length before the exact local Chunk ID, avoiding +string-concatenation ambiguity. Binary remote Chunk keys remain 32 bytes internally and use base64url only in an object +key or JSON field. -For byte-range pack reads, each Chunk record must be encrypted and authenticated independently. Encrypting one complete -pack as a single AEAD value would require downloading the complete pack before authenticating or decrypting one range. -The pack header and index must be authenticated, and the design must define whether the object-store catalogue itself -is encrypted. +Record salt, IV, compression choice, and pack location are not inputs to this derivation. Two clients therefore derive +the same `remoteChunkKey` for the same local Chunk ID even when they independently produce different encrypted +`RecordFrameV1` bytes. The derived key changes only when the local Chunk ID or repository key namespace changes; the +latter is an explicit full migration. -PostgREST needs server-visible opaque keys to execute a multi-key query. It does not need plaintext file paths, local -Chunk hashes, or decrypted content. +Recommended role labels are: -Credential rotation must not change remote Chunk identity. Rotation of the Vault Chunk identity key is a data migration -which creates a new namespace and cannot be treated as an ordinary bearer-token or password change. +- `manifest-auth`; +- `chunk-identity`; +- `chunk-record`; +- `pack-index`; +- `metadata-record`; +- `catalogue-record`; +- `commit-record`; +- `writer-record`; +- `catalogue-hint`; and +- `writer-name`. + +The first client creates the manifest's random `repositoryId` and Security Seed locally, but no random +`vaultChunkIdentityKey` is persisted or uploaded. Every device derives the same role keys after reading and +authenticating the winning manifest. Derived keys are cached in memory and discarded with the repository client. + +Generating a separate random identity secret only in local settings is not recommended for v1. It would need a new +secure multi-device transfer, backup, rotation, and loss-recovery contract even when the repository content is not +encrypted. Domain-separated derivation from the existing shared passphrase avoids that second secret lifecycle. + +### Unencrypted repositories + +The current opaque Journal code also skips encryption when its passphrase is empty. Adaptive v1 must not treat an empty +passphrase as protocol negotiation: `encrypt: false` explicitly selects unencrypted mode, while `encrypt: true` requires +a non-empty passphrase and fails configuration before reading or creating an encrypted manifest. In unencrypted mode, +the implementation must not describe a derived key as secret. The recommended functional mode is: + +```text +remoteChunkKey = SHA-256( + "livesync/adaptive-journal/v1/public-chunk-identity" || + repositoryId || + canonicalUTF8(localChunkID) +) +``` + +The manifest records `chunkKeyMode: "repository-scoped-sha256"` and `cipherSuite: "none"`. +`manifestAuth` is then a public SHA-256 integrity digest rather than authentication against a hostile remote. This +prevents accidental cross-repository equality but does not prevent content guessing or deliberate correlation. That +weaker property is consistent with storing the corresponding Journal records without E2EE and must be presented +explicitly. + +Public-mode frame, pack, index, and commit digests detect accidental corruption and bind references by exact bytes. +They do not prevent a remote administrator from rewriting a public repository. Logical Chunk-ID recomputation and +remote-key derivation still prevent an unrelated payload from being accepted accidentally under a requested Chunk key. + +### Verification and rotation + +In encrypted mode, `manifestAuth` is HMAC-SHA-256 under `roleKey("manifest-auth")`. A wrong passphrase, changed Security +Seed, substituted repository ID, or modified format field fails before any repository write. As with any +password-verification value, it permits offline guesses against a captured manifest; the existing encrypted Journal +already exposes an equivalent password-verification opportunity through authenticated ciphertext. + +For byte-range pack reads, each Chunk record is encrypted and authenticated independently as defined above. Physical +pack location is deliberately absent from the logical Chunk AAD, allowing a validated frame to be copied during +repacking without changing Metadata or remote Chunk identity. + +PostgREST needs server-visible remote Chunk keys to execute a multi-key query. It does not need plaintext file paths, +local Chunk IDs, or decrypted content. Object-store catalogue and index bodies remain encrypted; the remote still sees +object names, sizes, equality within one repository, and access patterns. + +Bearer-token, WebDAV password, custom-header, and server-credential rotation do not change repository keys. Changing +the E2EE passphrase, Security Seed, repository ID, chunk-key mode, or role-label version creates a new repository +namespace and requires an explicit full migration. A remote reset creates a new repository ID and Security Seed. ## Garbage Collection and repacking @@ -412,10 +1266,33 @@ policy. PostgreSQL transactionality simplifies mutation; it does not prove that Adaptive Journal Sync requires an explicit protocol and capability version recorded in the remote milestone or its successor. A device which understands only opaque Journal packs must not partially consume Metadata-only batches. +Journal connection settings should separate three independent choices internally: + +```ts +type JournalConnectionProfile = { + replicator: "journal"; + provider: "s3" | "webdav" | "postgrest"; + protocol: "opaque-v1" | "adaptive-v1"; + expectedEncryption: "encrypted" | "unencrypted"; + expectedRepositoryId?: string; + packReadPolicy?: "whole-pack" | "range"; +}; +``` + +This does not require one `adaptive` boolean for every provider. Existing provider-specific URI schemes may remain +serialisation formats and carry one shared `journalFormat=adaptive-v1` value. Its absence means `opaque-v1`. The remote +repository manifest is the authority for the actual format, while the connection profile records the expected format. +The expected encryption mode and, after the first accepted binding, repository ID are pinned locally as well. They must +match before any write. + The existing remote layout and the adaptive layout should use disjoint prefixes or schemas during experimentation. An opt-in test Vault may rebuild into the adaptive representation, but an ordinary upgrade must not migrate or delete an existing remote implicitly. +Protocol mismatch is a repository-safety failure, not an ordinary tweak mismatch. A setting which relaxes tweak-value +checks must not bypass it. Provider credentials may rotate without changing the format, repository identity, or +checkpoint; changing provider, storage location, repository ID, or protocol creates a distinct binding. + The compatibility plan must define: - negotiation between old and adaptive clients; @@ -426,6 +1303,21 @@ The compatibility plan must define: - encryption-key and Chunk-identity-key migration; and - exact conditions which require Fetch, Rebuild, or a new remote profile. +### Initial provider capability matrix + +| Capability | PostgREST | S3-compatible storage | WebDAV | +| --- | --- | --- | --- | +| Typed explicit missing | RPC/HTTP status mapping | SDK/HTTP status mapping | HTTP status mapping | +| Immutable conditional create | Unique insert-only RPC | `If-None-Match` | Probe `If-None-Match: *` | +| Native multi-key Chunk lookup | Yes, bounded RPC | No | No | +| Atomic Chunk verification and Metadata commit | Yes, transaction | No, commit manifest | No, commit manifest | +| Complete object listing | Indexed keyset query | Prefix listing | Depth-one `PROPFIND` | +| Whole-pack read | Available | Available | Required | +| Byte-range optimisation | Not needed for row CAS | Usually available, verify | Optional, probe | + +This matrix chooses a likely implementation strategy but does not replace endpoint checks. The common adaptive +repository is defined by the semantic rows, not by identical adapter methods or identical physical layouts. + ## Performance model Adaptive selection must optimise total work rather than request count alone. Measurements need to include: @@ -443,44 +1335,60 @@ Adaptive selection must optimise total work rather than request count alone. Mea The benchmark matrix should vary Vault size, retained history, Chunk-size distribution, editing pattern, network round-trip time, proxy path, concurrent writers, and new-device versus incremental synchronisation. -Initial thresholds must be conservative and observable. Candidate inputs include pending byte count, pending Chunk -count, expected pack reuse, range-read capability, and an upper bound on one RPC or pack response. Provider names alone -must not become performance policy. +The first implementation does not calculate a recommendation from these measurements. A user selects whole-pack or +Range retrieval according to the endpoint capability report and their own deployment knowledge. Measurements may later +justify an optional automatic policy, but provider names alone must not become performance policy. ## Planned implementation stages ### Stage 0: Executable model -- Define the repository, Chunk Store, catalogue, and commit state machines independently of a transport. +- Define typed remote outcomes and the repository, Chunk Store, per-writer stream, catalogue, and commit state machines + independently of a transport. - Add property and fault-injection tests for publication ordering, retry, duplicate physical locations, and recovery. -- Record the threat model and remote-visible metadata. +- Freeze canonical manifest, key-schedule, frame, control-record, and digest fixtures across supported runtimes. +- Record the threat model and remote-visible metadata for encrypted and unencrypted modes. -### Stage 1: Native batch CAS experiment +### Stage 1: Manifest and endpoint safety -- Add disposable PostgreSQL/PostgREST RPCs for batched Chunk availability, insertion, retrieval, and Metadata commit. +- Add the single immutable repository manifest, Security Seed, role-key derivation, and read-back authentication. +- Implement conditional-create semantics in each adaptive adapter. +- Add the non-destructive WebDAV capability checker, with Range reported as optional. +- Fail closed on ambiguous initialisation and repository-format mismatch. + +### Stage 2: Native batch CAS experiment + +- Add disposable PostgreSQL/PostgREST bounded binary RPCs for batched Chunk availability, insertion, retrieval, and + Metadata commit. - Measure RPC framing, transaction size, memory, and proxy limits. - Verify Vault isolation and failure rollback through real service integration tests. -### Stage 2: Immutable object-pack experiment +### Stage 3: Chunk retrieval integration -- Define a versioned, independently encrypted Chunk-record pack format. +- Connect the adaptive repository to the existing Journal `fetchRemoteChunks` producer. +- Reuse `ChunkFetcher` batching, validation, persistence, waiting, and terminal-condition behaviour. +- Keep received PouchDB revision histories on the maintained Journal application path. + +### Stage 4: Immutable object-pack experiment + +- Implement the reviewed `RecordFrameV1`, content-digest pack ID, and mode-appropriate index format. - Implement catalogue deltas and a local derived catalogue. -- Compare S3-compatible range reads with WebDAV whole-pack fallback. +- Implement the whole-pack canonical path for S3-compatible storage and WebDAV. +- Add manually selected Range retrieval where the endpoint probe confirms support. - Validate bounded request behaviour without per-Chunk `HEAD` or `GET` loops. -### Stage 3: Snapshot and compaction +### Stage 5: Snapshot and compaction - Add catalogue snapshots, retirement records, fencing, grace conditions, and recovery. - Implement conservative repacking without Metadata rewrites. - Exercise concurrent writers and interrupted compactors. -### Stage 4: Consumer integration +### Stage 6: Consumer integration -- Extend Chunk retrieval with an explicit Adaptive Journal producer and terminal-condition contract. - Add multi-device Commonlib integration, CLI consumption, and focused real-Obsidian tests. - Define opt-in migration, rollback, diagnostics, and maintenance controls. -### Stage 5: Release decision +### Stage 7: Release decision - Compare the adaptive paths with the current opaque Journal baseline. - Review privacy, corruption recovery, storage amplification, and operational complexity. @@ -488,12 +1396,51 @@ must not become performance policy. ## Test obligations +Stage 0 must commit language-neutral fixture inputs and exact expected bytes: + +| Fixture group | Contract frozen by the fixture | +| --- | --- | +| Encrypted and unencrypted manifests | Canonical JSON, manifest check, fixed parameter tuple, and repository-ID pinning | +| Key schedule | Master and role keys, encrypted and unencrypted `remoteChunkKey`, and writer stream ID | +| Record frames | Every v1 record kind, both encryption modes, both codecs, header lengths, AAD, digest, and rejection cases | +| Native batch envelopes | HAS, GET, and PUT ordering, every per-entry status, length fields, and byte ceilings | +| Commit envelope | Required-key sorting and digest, previous-commit link, Metadata digest, and exact idempotency digest | +| Pack and index | Pack digest, sorted index bytes, offsets, complete reads, and Range reads | +| Writer recovery | Pending initialisation, descriptor registration, uncertain commit, sequence gap, and new-epoch recovery | + +Fixtures use fixed passphrases, seeds, IDs, salts, IVs, payloads, and expected failure categories. Randomised production +paths inject those values at the cryptographic boundary so the same encoder and decoder can be tested without weakening +runtime randomness. DEFLATE fixtures freeze accepted stored streams and expected plaintext, not one mandatory compressor +output; independently valid compressed frames may differ while deriving the same Remote Chunk key. + - Metadata is never committed before all newly required Chunks have a durable source. - One logical Chunk ID never resolves to different accepted plaintext bytes. - Batch retry, duplicate physical storage, and concurrent publication are idempotent. +- Missing, conflict, authentication failure, permission denial, timeout, and server failure remain distinguishable at + the adaptive repository boundary. +- Concurrent repository initialisation selects one immutable manifest without last-write-wins replacement. +- `attach-existing` never initialises a missing remote, and an exported expected repository ID must match. +- Restart resolves `pendingInitialisation` and `pendingWriterDescriptor` before creating another identity or commit. +- Manifest fixtures derive identical role keys across supported runtimes, and a wrong passphrase fails before writes. +- A previously pinned repository ID rejects a substituted manifest, including one authenticated by a reused passphrase. +- An encrypted local profile rejects an unencrypted or weakened manifest before writes, and encrypted and unencrypted + modes expose exactly their documented identity and privacy properties. +- Every binary frame rejects unsupported versions, excessive lengths, truncated fields, invalid AEAD, wrong logical + keys, invalid unencrypted payload digests, and mismatched local Chunk IDs before persistence. +- Writer streams recover an uncertain commit before publishing the next sequence. +- Restart resolves an exact persisted pending commit before assigning a later writer sequence. +- A delayed or newly discovered writer cannot be hidden by another writer's frontier. +- Receivers stop at a per-stream sequence gap without blocking complete independent streams. - Native CAS batch limits are enforced by count and bytes. +- PostgREST binary RPCs preserve input order, distinguish missing rows, force read-validation of a differently framed + existing Chunk, reject a different logical value, and roll back an incomplete transactional Metadata commit. +- Two independently encrypted frames for one logical Chunk derive the same remote Chunk key, and the losing insert + validates and accepts the winning plaintext. - Object-pack retrieval groups requests by pack and avoids per-Chunk network loops. - Range and whole-pack retrieval produce identical validated Chunk results. +- Complete-pack digest, index digest, frame digest, and record AEAD failures are independently detected. +- The WebDAV checker touches only its random probe keys, reports conditional-create and Range capabilities separately, + and reports incomplete cleanup. - Catalogue reconstruction from snapshots and deltas is deterministic. - A stale snapshot hint cannot hide later valid deltas. - Interrupted snapshotting, repacking, retirement, and deletion preserve readable data. @@ -502,19 +1449,29 @@ must not become performance policy. - Vault-scoped access and opaque remote Chunk identity prevent cross-Vault retrieval. - Mixed protocol versions fail closed before publishing incompatible Metadata. -## Open decisions +## Recommended v1 decisions -- Which Vault secret, derivation, and rotation contract should produce remote Chunk keys? -- Is the object-store catalogue encrypted, and which non-secret header remains listable? -- What exact pack framing supports independent authentication and efficient range reads? -- Should a small urgent publication always be a micro-pack, or may it use a loose immutable Chunk object? -- How are catalogue delta frontiers represented without relying on wall-clock ordering? -- Which conditional-write, lease, or fencing mechanisms form the portable minimum across S3-compatible storage and - WebDAV? -- What reachability and retention policy protects non-live revision history? -- What payload and response framing lets PostgREST stream large bounded batches safely? -- When does PostgreSQL `bytea` remain appropriate, and when should a PostgREST deployment use an external object store? -- Which benchmark results justify adaptive thresholds rather than one fixed strategy per remote? +The Planning baseline uses these choices unless implementation evidence requires a deliberate revision: -These decisions remain Planning work. No implementation stage should silently choose them through incidental adapter -behaviour. +- one conditionally created Adaptive manifest owns both repository identity and the Security Seed; +- encrypted repositories derive every role key locally from the existing shared passphrase, while unencrypted + repositories use an explicitly weaker repository-scoped digest rather than a new hidden local secret; +- in encrypted mode, remotely visible routing is limited to the repository manifest, immutable object names, writer + streams, sequences, sizes, and access patterns, while indexes, catalogue records, Metadata, and commits are encrypted; +- independently verifiable binary Chunk frames are shared by PostgREST rows and immutable object packs, with AEAD in + encrypted mode; +- pack IDs, index entries, commit links, and pending writer state use exact-byte digests; +- PostgREST v1 uses bounded binary `bytea` RPC envelopes rather than JSON-encoded Chunk bodies; +- a lost local writer state starts a new writer epoch rather than guessing how to continue the old stream; and +- Range is a manual performance preference with a safe whole-pack fallback. + +The following remain deferred design work and do not block the first read/write experiment: + +- whether a small urgent publication may use a loose immutable Chunk object instead of a micro-pack; +- which lease or fencing mechanisms protect maintenance across S3-compatible storage, WebDAV, and PostgREST; +- what reachability and retention policy protects non-live revision history; +- when PostgreSQL `bytea` remains operationally appropriate and when a deployment should use an external object store; + and +- which measurements would justify adding an optional automatic pack-read policy. + +No implementation stage should silently revise the reviewed choices through incidental adapter behaviour.