obsidian-livesync/docs/adr/2026_07_adaptive_journal_protocol.md

Architectural Decision Record: Introduce Adaptive Journal as an Explicit Protocol

Status

Proposed for staged implementation. This decision does not change the current default Journal format or promise a compatible in-place migration.

Context

The existing Journal protocol publishes PouchDB Metadata and Chunk documents together in opaque immutable objects. That representation is portable and remains the compatibility baseline, but it prevents a remote from answering a bounded multi-key Chunk query or amortising object-store requests independently from Metadata publication.

S3-compatible Object Storage, WebDAV, and PostgREST have materially different physical capabilities. Making each one implement a separate replication algorithm would duplicate ordering, encryption, recovery, and Chunk-delivery rules. Making every one use an identical physical layout would discard useful native batching and transactions.

The detailed binary formats, state machines, privacy properties, and recovery rules are specified in the Adaptive Journal Sync design.

Decision

Adaptive Journal is a second, explicit Journal protocol version owned by Commonlib. It keeps one logical repository contract while allowing provider-specific physical storage.

• Metadata events and raw Chunk records remain separate throughout publication.
• Logical Chunks are immutable and content-addressable. A changed value has a new logical Chunk ID.
• A repository-scoped Remote Chunk key is derived locally from the exact logical Chunk ID after accepting the repository manifest.
• Chunks become durable before a commit makes Metadata references visible.
• Ordinary publication creates immutable records. Catalogue snapshots, caches, and indexes are derived and reconstructible.
• Each writer publishes a dense sequence identified by a stable host ID and a persisted random writer epoch. A reader tracks one frontier per writer stream; it does not depend on a remote startAfter ordering contract.
• Remote operations return typed outcomes which distinguish absence, conflict, permanent rejection, retryable failure, and an ambiguous mutation which must be verified before retrying.

The adaptive repository exposes batched Chunk availability, publication, and retrieval even when its implementation uses immutable packs internally. The caller does not issue one remote request per Chunk.

Repository identity and compatibility

A new repository has one immutable, conditionally created manifest. The first device generates its candidate locally, and the winning manifest fixes the repository ID, Security Seed, protocol parameters, and required capabilities. Every client pins the accepted repository ID in local repository state. A Setup URI exported from an accepted binding should carry that non-secret expected ID.

opaque-v1 and adaptive-v1 are separate remote formats. Format selection is explicit in the remote configuration, and a mismatch fails before publication. The implementation detects incompatible remote data, but it does not migrate it. Changing format requires an explicit remote rebuild or a new remote namespace.

Deprecated data formats do not enlarge the Adaptive protocol. Compatibility remains at the existing decoding boundaries, and rebuilding the remote is the recovery path for an unsupported Adaptive layout.

Provider delivery sequence

Each provider is delivered and reviewed through four boundaries, in order:

1. Adapter and Commonlib integration. Implement semantic capabilities, typed failure handling, format detection, unit tests, and disposable real-service integration tests.
2. CLI end-to-end acceptance. Use the built CLI and a real disposable service to apply a Setup URI, synchronise two independent local databases, and restore text and binary Chunk content. This proves the headless product boundary without Obsidian or Svelte.
3. Host settings and UI. Add only provider-specific controls, validation, profile persistence, Setup URI transport, and focused host tests.
4. Real-host end-to-end acceptance. Exercise one representative setup and synchronisation path in a real Obsidian instance. This test verifies host composition and does not repeat the adapter capability matrix or the complete CLI conflict suite.

The initial order is S3-compatible Object Storage, WebDAV, then PostgREST. A provider completes these boundaries before the next provider is presented for integration review. The sequence keeps each review independently attributable and keeps the real-host tests small.

Alternatives rejected

One cross-provider implementation change

This hides which provider requires a shared-core change, makes failures difficult to attribute, and forces reviewers to understand SQL RPCs, object packs, WebDAV behaviour, CLI composition, and Obsidian UI in one change.

Add the Host UI before headless acceptance

This makes an application-level failure ambiguous between the protocol, adapter, CLI-independent host composition, and presentation. The CLI provides the smaller executable boundary first.

Use one physical representation for every provider

One-object-per-Chunk storage causes excessive object requests, while forcing packs into PostgreSQL discards bounded multi-key RPCs and transactions. Common semantics do not require common physical storage.

Consequences

• Commonlib owns protocol correctness and provider adapters; Self-hosted LiveSync owns CLI composition, Obsidian host integration, and presentation.
• Provider reviews can stop at the first failed boundary without involving later UI or real-host tests.
• The final end-to-end suite remains an acceptance layer rather than a duplicate protocol test suite.
• Adding another provider requires the same semantic contract and staged evidence, not another replication algorithm.
• Remote rebuild remains an explicit operational requirement while Adaptive Journal is evolving.