From d1733720e6ee6d853568f271abf0ab3b810f3d0d Mon Sep 17 00:00:00 2001 From: vorotamoroz Date: Fri, 31 Jul 2026 12:56:47 +0000 Subject: [PATCH] docs: split adaptive journal decisions by provider --- .../adr/2026_07_adaptive_journal_postgrest.md | 90 +++++++++++++++++ docs/adr/2026_07_adaptive_journal_protocol.md | 98 +++++++++++++++++++ docs/adr/2026_07_adaptive_journal_s3.md | 96 ++++++++++++++++++ docs/adr/2026_07_adaptive_journal_webdav.md | 96 ++++++++++++++++++ docs/design_docs/adaptive_journal_sync.md | 10 ++ 5 files changed, 390 insertions(+) create mode 100644 docs/adr/2026_07_adaptive_journal_postgrest.md create mode 100644 docs/adr/2026_07_adaptive_journal_protocol.md create mode 100644 docs/adr/2026_07_adaptive_journal_s3.md create mode 100644 docs/adr/2026_07_adaptive_journal_webdav.md diff --git a/docs/adr/2026_07_adaptive_journal_postgrest.md b/docs/adr/2026_07_adaptive_journal_postgrest.md new file mode 100644 index 00000000..ae5eed79 --- /dev/null +++ b/docs/adr/2026_07_adaptive_journal_postgrest.md @@ -0,0 +1,90 @@ +# Architectural Decision Record: Use Native Batch CAS for Adaptive PostgREST + +## Status + +Proposed as the final experimental provider, after the S3 and WebDAV delivery sequences have established the common +protocol, CLI, host, and end-to-end boundaries. + +## Context + +PostgREST can expose PostgreSQL uniqueness, row-level security, bounded set operations, and transactions. Treating it +as another opaque object store works, but leaves Metadata and Chunks combined and cannot use a multi-key Chunk query. +Using one HTTP request per logical Chunk would make latency dominate synchronisation. + +The common protocol decision is recorded in +[Adaptive Journal as an explicit protocol](2026_07_adaptive_journal_protocol.md). The tables, binary RPC framing, +transaction rules, and privacy model are specified in the +[Adaptive Journal Sync design](../design_docs/adaptive_journal_sync.md). + +## Decision + +Adaptive PostgREST separates Metadata publication from immutable Chunk storage physically as well as logically. + +- Vault- and repository-scoped Chunk rows use the Remote Chunk key as an insert-only unique address. +- Bounded binary RPCs implement `hasMany`, `getMany`, and `putMany` while preserving input order and per-entry status. +- Writer descriptors, Metadata batches, and commits remain small append-only records. +- A transactional commit verifies the sorted required-Chunk-key digest and the existence of every referenced Chunk + before making Metadata visible. +- Concurrent insertion of a different encrypted frame for the same logical Chunk reads, validates, and accepts the + winning plaintext only when it represents the same logical value. +- Row-level security scopes every operation to the configured Vault. Server-visible Remote Chunk keys reveal equality + within that repository but do not reveal plaintext Chunk IDs. + +PostgREST does not use object packs, catalogue deltas, or Range retrieval for native Chunk rows. The shared binary Chunk +record remains independently verifiable, but PostgreSQL supplies the batch index and uniqueness boundary. + +The SQL schema and RPC contract are versioned together with the Adaptive format. A missing or incompatible schema is +detected before publication and requires applying the reviewed schema or rebuilding the remote. The client does not +perform an implicit data-format migration. + +## Staged acceptance + +### Adapter and Commonlib integration + +Unit tests own binary-envelope limits, ordering, status decoding, insert conflicts, rollback, error classification, +Vault isolation, and malformed responses. Disposable PostgreSQL and PostgREST integration tests own the real SQL +schema, row-level security, RPC transactions, and two-client Metadata and Chunk synchronisation. + +### CLI end-to-end acceptance + +The built CLI applies an Adaptive PostgREST Setup URI to a second independent database and synchronises text and binary +Chunk-backed files through disposable PostgreSQL and PostgREST services. The scenario proves bounded native batching at +the headless product boundary without repeating the SQL failure matrix. + +### Host settings and UI + +The PostgREST dialogue persists the endpoint, Vault identifier, authentication configuration, Adaptive format, and +expected repository ID. Focused tests own connection-string and Setup URI preservation, validation, and format-mismatch +guidance; they do not execute SQL. + +### Real-host end-to-end acceptance + +One real-Obsidian workflow applies the reviewed schema and performs a representative Chunk-backed transfer. The +disposable Commonlib integration remains authoritative for RLS, transaction rollback, and the complete RPC matrix. + +## Alternatives rejected + +### Reuse immutable object packs in PostgreSQL + +This would preserve portability at the cost of native multi-key lookup and transaction guarantees. The repository +contract already permits a different physical representation. + +### Store raw file content with Metadata + +Metadata must remain small and must continue to refer to immutable logical Chunks. Combining content with Metadata +would break the maintained PouchDB model and duplicate unchanged content across revisions. + +### Encode Chunk bodies as JSON values + +Base64 and large JSON arrays add framing and memory overhead and make limits harder to enforce. Bounded binary `bytea` +RPC envelopes provide deterministic lengths and status ordering. + +## Consequences + +- PostgREST has the largest provider-specific implementation because it includes reviewed SQL, RLS, RPC framing, and + transaction behaviour. +- Placing it last lets the common protocol, CLI, and host boundaries stabilise before introducing that larger surface. +- Native batching can reduce request count substantially relative to object-per-Chunk storage, but actual speed still + depends on database, proxy, network, and workload measurements. +- PostgREST and object stores share logical records and correctness rules without pretending to share a physical + layout. diff --git a/docs/adr/2026_07_adaptive_journal_protocol.md b/docs/adr/2026_07_adaptive_journal_protocol.md new file mode 100644 index 00000000..4ec5b3f4 --- /dev/null +++ b/docs/adr/2026_07_adaptive_journal_protocol.md @@ -0,0 +1,98 @@ +# Architectural Decision Record: Introduce Adaptive Journal as an Explicit Protocol + +## Status + +Proposed for staged implementation. This decision does not change the current default Journal format or promise a +compatible in-place migration. + +## Context + +The existing Journal protocol publishes PouchDB Metadata and Chunk documents together in opaque immutable objects. +That representation is portable and remains the compatibility baseline, but it prevents a remote from answering a +bounded multi-key Chunk query or amortising object-store requests independently from Metadata publication. + +S3-compatible Object Storage, WebDAV, and PostgREST have materially different physical capabilities. Making each one +implement a separate replication algorithm would duplicate ordering, encryption, recovery, and Chunk-delivery rules. +Making every one use an identical physical layout would discard useful native batching and transactions. + +The detailed binary formats, state machines, privacy properties, and recovery rules are specified in the +[Adaptive Journal Sync design](../design_docs/adaptive_journal_sync.md). + +## Decision + +Adaptive Journal is a second, explicit Journal protocol version owned by Commonlib. It keeps one logical repository +contract while allowing provider-specific physical storage. + +- Metadata events and raw Chunk records remain separate throughout publication. +- Logical Chunks are immutable and content-addressable. A changed value has a new logical Chunk ID. +- A repository-scoped Remote Chunk key is derived locally from the exact logical Chunk ID after accepting the + repository manifest. +- Chunks become durable before a commit makes Metadata references visible. +- Ordinary publication creates immutable records. Catalogue snapshots, caches, and indexes are derived and + reconstructible. +- Each writer publishes a dense sequence identified by a stable host ID and a persisted random writer epoch. A reader + tracks one frontier per writer stream; it does not depend on a remote `startAfter` ordering contract. +- Remote operations return typed outcomes which distinguish absence, conflict, permanent rejection, retryable failure, + and an ambiguous mutation which must be verified before retrying. + +The adaptive repository exposes batched Chunk availability, publication, and retrieval even when its implementation +uses immutable packs internally. The caller does not issue one remote request per Chunk. + +### Repository identity and compatibility + +A new repository has one immutable, conditionally created manifest. The first device generates its candidate locally, +and the winning manifest fixes the repository ID, Security Seed, protocol parameters, and required capabilities. +Every client pins the accepted repository ID in local repository state. A Setup URI exported from an accepted binding +should carry that non-secret expected ID. + +`opaque-v1` and `adaptive-v1` are separate remote formats. Format selection is explicit in the remote configuration, +and a mismatch fails before publication. The implementation detects incompatible remote data, but it does not migrate +it. Changing format requires an explicit remote rebuild or a new remote namespace. + +Deprecated data formats do not enlarge the Adaptive protocol. Compatibility remains at the existing decoding +boundaries, and rebuilding the remote is the recovery path for an unsupported Adaptive layout. + +### Provider delivery sequence + +Each provider is delivered and reviewed through four boundaries, in order: + +1. **Adapter and Commonlib integration.** Implement semantic capabilities, typed failure handling, format detection, + unit tests, and disposable real-service integration tests. +2. **CLI end-to-end acceptance.** Use the built CLI and a real disposable service to apply a Setup URI, synchronise two + independent local databases, and restore text and binary Chunk content. This proves the headless product boundary + without Obsidian or Svelte. +3. **Host settings and UI.** Add only provider-specific controls, validation, profile persistence, Setup URI transport, + and focused host tests. +4. **Real-host end-to-end acceptance.** Exercise one representative setup and synchronisation path in a real Obsidian + instance. This test verifies host composition and does not repeat the adapter capability matrix or the complete CLI + conflict suite. + +The initial order is S3-compatible Object Storage, WebDAV, then PostgREST. A provider completes these boundaries before +the next provider is presented for integration review. The sequence keeps each review independently attributable and +keeps the real-host tests small. + +## Alternatives rejected + +### One cross-provider implementation change + +This hides which provider requires a shared-core change, makes failures difficult to attribute, and forces reviewers to +understand SQL RPCs, object packs, WebDAV behaviour, CLI composition, and Obsidian UI in one change. + +### Add the Host UI before headless acceptance + +This makes an application-level failure ambiguous between the protocol, adapter, CLI-independent host composition, and +presentation. The CLI provides the smaller executable boundary first. + +### Use one physical representation for every provider + +One-object-per-Chunk storage causes excessive object requests, while forcing packs into PostgreSQL discards bounded +multi-key RPCs and transactions. Common semantics do not require common physical storage. + +## Consequences + +- Commonlib owns protocol correctness and provider adapters; Self-hosted LiveSync owns CLI composition, Obsidian host + integration, and presentation. +- Provider reviews can stop at the first failed boundary without involving later UI or real-host tests. +- The final end-to-end suite remains an acceptance layer rather than a duplicate protocol test suite. +- Adding another provider requires the same semantic contract and staged evidence, not another replication algorithm. +- Remote rebuild remains an explicit operational requirement while Adaptive Journal is evolving. diff --git a/docs/adr/2026_07_adaptive_journal_s3.md b/docs/adr/2026_07_adaptive_journal_s3.md new file mode 100644 index 00000000..af431f3b --- /dev/null +++ b/docs/adr/2026_07_adaptive_journal_s3.md @@ -0,0 +1,96 @@ +# Architectural Decision Record: Use S3 as the Reference Adaptive Object Store + +## Status + +Proposed as an improvement to the maintained S3-compatible Journal path. Adaptive Journal remains explicit and +opt-in; the existing opaque format remains the compatibility default. + +## Context + +S3-compatible Object Storage already supplies the maintained Journal object model and is the smallest provider on +which to prove Adaptive immutable packs. It has a standard conditional-create request, paginated listing, binary +objects, and optional byte-range retrieval, but compatible endpoints can still differ in consistency, proxy behaviour, +and Range handling. + +The common protocol decision is recorded in +[Adaptive Journal as an explicit protocol](2026_07_adaptive_journal_protocol.md). The complete object-pack and catalogue +formats are specified in the [Adaptive Journal Sync design](../design_docs/adaptive_journal_sync.md). + +## Decision + +S3-compatible storage is the reference implementation of the Adaptive object-store strategy. + +- It stores the manifest, writer control records, Metadata batches, commits, Chunk packs, indexes, and catalogue records + as immutable objects under the configured bucket namespace. +- Manifest and every immutable publication use conditional create. A successful create response is sufficient; the + client does not add a confirmation request solely for caution. +- A lost or ambiguous mutation response returns `verify-first`. The caller reads the exact key before retrying. +- A conditional-request conflict which does not establish an existing immutable value remains retryable and is not + reported as a successful create. +- Listing follows every continuation token and proves complete prefix enumeration. Folder marker objects and stale + capability-probe objects are not repository data. +- Format inspection distinguishes empty, `opaque-v1`, `adaptive-v1`, and mixed repositories. Mixed or mismatched data + fails closed, and reset remains an explicit batched operation. + +The portable retrieval policy is `whole-pack`. A user may select `range` when the endpoint capability probe has +confirmed exact byte-range behaviour. Range responses must use `206`, carry a matching `Content-Range`, and return the +requested number of bytes. Loss of optional Range support falls back to whole-pack retrieval after reporting the +capability change; it does not make valid packs unreadable. + +Capability results which prove support or lack of support may be cached for the endpoint identity. A transient probe +failure is not cached as a permanent result. Probe objects use a reserved random prefix and are removed when possible; +incomplete cleanup is reported and ignored by format detection. + +## Staged acceptance + +### Adapter and Commonlib integration + +Focused tests cover conditional creation, ambiguous responses, binary fidelity, read-after-write visibility, paginated +listing, deletion visibility, Range validation, format detection, reset batching, and probe cleanup. A disposable +MinIO integration proves two-client Adaptive Metadata and Chunk synchronisation and both retrieval paths. + +### CLI end-to-end acceptance + +One real-MinIO scenario uses two independent CLI databases. Device A uses whole-pack retrieval. Device B receives an +Adaptive S3 Setup URI selecting Range retrieval. The scenario verifies that the URI preserves the format and policy, +that both devices can publish and receive more than one synchronisation round, and that text and binary content are +reconstructed from Chunks. + +This test owns the built CLI, settings persistence, Setup URI decoding, and headless composition. It does not repeat +every S3 error classification already owned by Commonlib. + +### Host settings and UI + +The Object Storage dialogue exposes `opaque-v1` and `adaptive-v1`, with whole-pack as the Adaptive default and Range as +an explicit preference. It preserves the expected repository ID and read policy through saved profiles and Setup URI +handling. Focused host tests own normalisation, validation, persistence, and warnings; they do not contact S3. + +### Real-host end-to-end acceptance + +One real-Obsidian workflow applies an Adaptive S3 configuration and proves a representative Chunk-backed file transfer +against disposable MinIO. It verifies the Obsidian composition boundary only. The CLI suite remains the broader +headless synchronisation acceptance test. + +## Alternatives rejected + +### Re-upload a mutable pack when one Chunk changes + +Packs are immutable publication units. A changed logical Chunk is added to a new pack, and a new catalogue delta makes +that location discoverable. Existing packs are replaced only by a separately protected compaction process. + +### Require Range for Adaptive S3 + +Whole-pack reads often provide better throughput and work on more endpoints. Range is a deployment preference, not a +correctness requirement. + +### Confirm every successful mutation with another request + +This doubles request count in the ordinary success path. Exact-key verification is reserved for ambiguous outcomes and +explicit diagnostics. + +## Consequences + +- S3 establishes the object-store contract before the more variable WebDAV implementation. +- The two retrieval policies are exercised without duplicating the complete CLI scenario. +- Existing opaque S3 repositories remain readable and never become Adaptive implicitly. +- Endpoint capability evidence, rather than the S3-compatible label alone, controls optional behaviour. diff --git a/docs/adr/2026_07_adaptive_journal_webdav.md b/docs/adr/2026_07_adaptive_journal_webdav.md new file mode 100644 index 00000000..b77ebc7a --- /dev/null +++ b/docs/adr/2026_07_adaptive_journal_webdav.md @@ -0,0 +1,96 @@ +# Architectural Decision Record: Gate Adaptive WebDAV with an Endpoint Safety Check + +## Status + +Proposed as an experimental provider after the S3 Adaptive path has completed adapter, CLI, host, and real-host +acceptance. + +## Context + +WebDAV exposes an object-shaped interface suitable for immutable packs, but method support and conditional semantics +vary across servers, gateways, reverse proxies, and authentication layers. A server can advertise WebDAV while failing +binary fidelity, replacing an object despite `If-None-Match: *`, returning incomplete listings, or ignoring Range. + +The common protocol decision is recorded in +[Adaptive Journal as an explicit protocol](2026_07_adaptive_journal_protocol.md). The pack format, flat object mapping, +and detailed safety-check sequence are specified in the +[Adaptive Journal Sync design](../design_docs/adaptive_journal_sync.md). + +## Decision + +Adaptive WebDAV uses the same immutable pack and catalogue semantics proven by S3, with a WebDAV-specific flat object +mapping inside the configured collection. Logical writer ordering comes from the host ID, writer epoch, and dense +sequence embedded in authenticated records. Correctness does not depend on a server-provided `startAfter` order. + +Before a new Adaptive repository becomes writable, a non-destructive endpoint safety checker exercises random reserved +probe keys and reports observed semantic capabilities: + +- binary write and exact read-back; +- read-after-write visibility; +- complete collection listing for the probe keys; +- `If-None-Match: *` preventing replacement; +- delete visibility; and +- exact byte-range behaviour, reported separately as optional. + +The checker is an implementation acceptance target for compatible servers, not a claim that every WebDAV server must +support Adaptive Journal. It touches no repository objects, attempts to remove every probe, reports incomplete cleanup, +and never interprets authentication, permission, timeout, malformed response, or server failure as absence. + +Conditional create, binary fidelity, complete listing, read-after-write visibility, and delete visibility are required. +Range remains optional. The user selects whole-pack or Range retrieval based on their endpoint and own latency and +throughput preference; the checker does not benchmark or recommend a policy. + +Successful mutations do not receive unconditional confirmation requests. An ambiguous response is classified as +`verify-first`, and the exact immutable key is read before retrying. The existing opaque WebDAV layout and Adaptive +layout are detected separately; a mismatch requires a remote rebuild or another namespace. + +## Staged acceptance + +### Adapter and Commonlib integration + +Unit tests own HTTP status classification, conditional-create verification, flat-name round trips, complete listing, +probe isolation, cleanup reporting, Range validation, and whole-pack fallback. A disposable WebDAV integration runs +the safety checker before exercising Adaptive Metadata and Chunk synchronisation. + +### CLI end-to-end acceptance + +The built CLI applies an Adaptive WebDAV Setup URI to a second independent database and synchronises text and binary +Chunk-backed files through a real disposable server. One client uses whole-pack retrieval; Range is added to this layer +only when the selected test server proves it. The CLI test does not repeat the complete endpoint capability matrix. + +### Host settings and UI + +The WebDAV dialogue exposes Adaptive mode only with clear capability-check results. It persists the expected repository +ID and the selected retrieval policy, defaults to whole-pack, and reports that Range is optional. Focused tests own +profile and Setup URI preservation without contacting a server. + +### Real-host end-to-end acceptance + +One real-Obsidian workflow uses the same known disposable WebDAV implementation, runs the safety gate, and proves a +representative Chunk-backed transfer. Servers outside that fixture are diagnosed by the checker rather than added to a +large real-host matrix. + +## Alternatives rejected + +### Trust advertised WebDAV methods + +Method advertisement does not prove the conditional, listing, visibility, and byte semantics required for immutable +publication. + +### Treat Range as mandatory + +Whole-pack retrieval is correct and often competitive for throughput. Requiring Range would exclude otherwise safe +servers for an optional optimisation. + +### Add server-specific compatibility branches + +The remote may be any implementation or proxy composition. Semantic checks produce a maintainable contract, while a +growing server-name table would remain incomplete and become stale. + +## Consequences + +- WebDAV remains experimental because suitability is endpoint-specific. +- The safety checker gives a concrete reason when a server cannot host Adaptive Journal. +- Common object-pack behaviour is inherited from the earlier S3 boundary, keeping WebDAV-specific tests focused on HTTP + semantics and name mapping. +- Optional Range support can improve request efficiency without becoming a data-availability requirement. diff --git a/docs/design_docs/adaptive_journal_sync.md b/docs/design_docs/adaptive_journal_sync.md index c882ca84..69c42af2 100644 --- a/docs/design_docs/adaptive_journal_sync.md +++ b/docs/design_docs/adaptive_journal_sync.md @@ -5,6 +5,16 @@ > instruction, or statement of current user-visible behaviour. The existing opaque Journal pack protocol remains the > implementation baseline while this proposal is evaluated. +The architectural decisions and staged acceptance boundaries are recorded separately: + +- [Adaptive Journal protocol](../adr/2026_07_adaptive_journal_protocol.md); +- [Adaptive Journal on S3-compatible Object Storage](../adr/2026_07_adaptive_journal_s3.md); +- [Adaptive Journal on WebDAV](../adr/2026_07_adaptive_journal_webdav.md); and +- [Adaptive Journal on PostgREST](../adr/2026_07_adaptive_journal_postgrest.md). + +This document remains the detailed protocol design. The ADRs decide how that design is divided, introduced, and +verified; they do not duplicate every binary layout or recovery state described here. + ## Summary Adaptive Journal Sync would separate lightweight Metadata events from Chunk delivery while preserving one common