obsidian-livesync/docs/adr/2026_07_adaptive_journal_s3.md

Architectural Decision Record: Use S3 as the Reference Adaptive Object Store

Status

Proposed as an improvement to the maintained S3-compatible Journal path. Adaptive Journal remains explicit and opt-in; the existing opaque format remains the compatibility default.

Context

S3-compatible Object Storage already supplies the maintained Journal object model and is the smallest provider on which to prove Adaptive immutable packs. It has a standard conditional-create request, paginated listing, binary objects, and optional byte-range retrieval, but compatible endpoints can still differ in consistency, proxy behaviour, and Range handling.

The common protocol decision is recorded in Adaptive Journal as an explicit protocol. The complete object-pack and catalogue formats are specified in the Adaptive Journal Sync design.

Decision

S3-compatible storage is the reference implementation of the Adaptive object-store strategy.

• It stores the manifest, writer control records, Metadata batches, commits, Chunk packs, indexes, and catalogue records as immutable objects under the configured bucket namespace.
• Manifest and every immutable publication use conditional create. A successful create response is sufficient; the client does not add a confirmation request solely for caution.
• A lost or ambiguous mutation response returns verify-first. The caller reads the exact key before retrying.
• A conditional-request conflict which does not establish an existing immutable value remains retryable and is not reported as a successful create.
• Listing follows every continuation token and proves complete prefix enumeration. Folder marker objects and stale capability-probe objects are not repository data.
• Format inspection distinguishes empty, opaque-v1, adaptive-v1, and mixed repositories. Mixed or mismatched data fails closed, and reset remains an explicit batched operation.

The portable retrieval policy is whole-pack. A user may select range when the endpoint capability probe has confirmed exact byte-range behaviour. Range responses must use 206, carry a matching Content-Range, and return the requested number of bytes. Loss of optional Range support falls back to whole-pack retrieval after reporting the capability change; it does not make valid packs unreadable.

Capability results which prove support or lack of support may be cached for the endpoint identity. A transient probe failure is not cached as a permanent result. Probe objects use a reserved random prefix and are removed when possible; incomplete cleanup is reported and ignored by format detection.

Staged acceptance

Adapter and Commonlib integration

Focused tests cover conditional creation, ambiguous responses, binary fidelity, read-after-write visibility, paginated listing, deletion visibility, Range validation, format detection, reset batching, and probe cleanup. A disposable MinIO integration proves two-client Adaptive Metadata and Chunk synchronisation and both retrieval paths.

CLI end-to-end acceptance

One real-MinIO scenario uses two independent CLI databases. Device A uses whole-pack retrieval. Device B receives an Adaptive S3 Setup URI selecting Range retrieval. The scenario verifies that the URI preserves the format and policy, that both devices can publish and receive more than one synchronisation round, and that text and binary content are reconstructed from Chunks.

This test owns the built CLI, settings persistence, Setup URI decoding, and headless composition. It does not repeat every S3 error classification already owned by Commonlib.

Host settings and UI

The Object Storage dialogue exposes opaque-v1 and adaptive-v1, with whole-pack as the Adaptive default and Range as an explicit preference. It preserves the expected repository ID and read policy through saved profiles and Setup URI handling. Focused host tests own normalisation, validation, persistence, and warnings; they do not contact S3.

Real-host end-to-end acceptance

One real-Obsidian workflow applies an Adaptive S3 configuration and proves a representative Chunk-backed file transfer against disposable MinIO. It verifies the Obsidian composition boundary only. The CLI suite remains the broader headless synchronisation acceptance test.

Alternatives rejected

Re-upload a mutable pack when one Chunk changes

Packs are immutable publication units. A changed logical Chunk is added to a new pack, and a new catalogue delta makes that location discoverable. Existing packs are replaced only by a separately protected compaction process.

Require Range for Adaptive S3

Whole-pack reads often provide better throughput and work on more endpoints. Range is a deployment preference, not a correctness requirement.

Confirm every successful mutation with another request

This doubles request count in the ordinary success path. Exact-key verification is reserved for ambiguous outcomes and explicit diagnostics.

Consequences

• S3 establishes the object-store contract before the more variable WebDAV implementation.
• The two retrieval policies are exercised without duplicating the complete CLI scenario.
• Existing opaque S3 repositories remain readable and never become Adaptive implicitly.
• Endpoint capability evidence, rather than the S3-compatible label alone, controls optional behaviour.