diff --git a/docs/adr/2026_07_adaptive_journal_webdav.md b/docs/adr/2026_07_adaptive_journal_webdav.md index b77ebc7a..835404f8 100644 --- a/docs/adr/2026_07_adaptive_journal_webdav.md +++ b/docs/adr/2026_07_adaptive_journal_webdav.md @@ -7,20 +7,24 @@ acceptance. ## Context -WebDAV exposes an object-shaped interface suitable for immutable packs, but method support and conditional semantics -vary across servers, gateways, reverse proxies, and authentication layers. A server can advertise WebDAV while failing -binary fidelity, replacing an object despite `If-None-Match: *`, returning incomplete listings, or ignoring Range. +WebDAV exposes an object-shaped interface suitable for immutable Commit Bundles and Packs, but method support and +conditional semantics vary across servers, gateways, reverse proxies, and authentication layers. A server can +advertise WebDAV while failing binary fidelity, replacing an object despite `If-None-Match: *`, returning incomplete +listings, or ignoring Range. The common protocol decision is recorded in -[Adaptive Journal as an explicit protocol](2026_07_adaptive_journal_protocol.md). The pack format, flat object mapping, -and detailed safety-check sequence are specified in the -[Adaptive Journal Sync design](../design_docs/adaptive_journal_sync.md). +[Adaptive Journal as an explicit protocol](2026_07_adaptive_journal_protocol.md). The Pack background, flat object +mapping, and detailed safety-check sequence are specified in the +[Adaptive Journal Sync design](../design_docs/adaptive_journal_sync.md). Commonlib owns the implemented +`commit-bundle-v1` wire contract. ## Decision -Adaptive WebDAV uses the same immutable pack and catalogue semantics proven by S3, with a WebDAV-specific flat object -mapping inside the configured collection. Logical writer ordering comes from the host ID, writer epoch, and dense -sequence embedded in authenticated records. Correctness does not depend on a server-provided `startAfter` order. +Adaptive WebDAV uses the same immutable Commit Bundle and external Pack semantics proven by S3, with a WebDAV-specific +flat object mapping inside the configured collection. Its Catalogue is derived locally from authenticated routes in +received Bundles rather than stored as a separate mutable remote object. Logical writer ordering comes from the host +ID, writer epoch, and dense sequence embedded in authenticated records. Correctness does not depend on a +server-provided `startAfter` order. Before a new Adaptive repository becomes writable, a non-destructive endpoint safety checker exercises random reserved probe keys and reports observed semantic capabilities: @@ -44,13 +48,153 @@ Successful mutations do not receive unconditional confirmation requests. An ambi `verify-first`, and the exact immutable key is read before retrying. The existing opaque WebDAV layout and Adaptive layout are detected separately; a mismatch requires a remote rebuild or another namespace. +## Working hypotheses and illustrative estimates + +The following values are planning assumptions for the first WebDAV experiment. They are not benchmark results, a +provider-pricing statement, or a performance guarantee. The implementation must record the corresponding measurements +before this ADR presents either Journal format as generally faster or smaller. + +### Cost and lifecycle assumptions + +The initial deployment model is a self-hosted or quota-priced WebDAV service without a direct per-request tariff. +Request count still matters because every request consumes a round trip, server work, connection capacity, and a share +of any rate limit. Commercial services, gateways, and managed hosting may use a different charging model, so the +adapter must not infer cost policy from the WebDAV label. + +Storage capacity is expected to be the more visible constraint. RFC 4331 defines optional +[`DAV:quota-used-bytes` and `DAV:quota-available-bytes`](https://www.rfc-editor.org/rfc/rfc4331.html) properties for +collections. When both are available, a connection diagnostic may report them as server-supplied evidence. Their +absence means 'unknown', not 'unlimited', and quota reporting is not an Adaptive safety requirement. + +The first experiment has no ordinary remote Garbage Collection or repacking guarantee. Commit Bundles and external +Packs remain immutable, unreachable data may remain retained, and an interrupted publication may leave an unreferenced +Pack. A remote Rebuild is therefore the only guaranteed way to reclaim all experimental history. This makes retained +bytes and object count acceptance measurements rather than later operational details. + +Adaptive and Opaque Journal consume the same local PouchDB Chunk split. A smaller average changed Chunk can reduce +retained history in either format. Adaptive adds authenticated routes and independently framed records, while Opaque +adds its own container, compression, and encryption overhead. No comparative storage saving is assumed before both +representations are measured with the same changes. + +### Ordinary request model + +The object-profile model is shared with S3. Let: + +- `B` be the number of Metadata batches published as distinct Commit Bundles; +- `X` be the number of new external Pack objects needed by those batches; +- `J` be the number of Opaque Journal objects produced for the same changes; +- `W` be the number of visible Writer streams; +- `K` be the number of those Writer descriptors not already cached by the opened process; +- `P` be the number of additional complete Pack-container reads needed for missing Chunks; and +- `M` be the number of missing Chunk frames fetched with Range. + +After the endpoint safety result and repository binding have been cached for the active configuration, ordinary +publication performs approximately `J` Opaque PUTs or `B + X` Adaptive PUTs. Every Adaptive batch creates one Bundle; +each external Pack adds one preceding PUT. A confirmed conditional create does not add a read-back. An ambiguous PUT +adds one exact-key GET before any retry and remains outside the successful-first-attempt estimate. + +The generic object-store receive path performs approximately: + +| Retrieval policy | Requests without WebDAV listing reuse | +| --- | ---: | +| `whole-pack` | `1 + W + K + B + P` | +| `range` | `1 + W + K + B + M` | + +The `1 + W` term represents one Writer listing and one Commit listing per Writer. A flat WebDAV collection normally +cannot apply those logical prefixes on the server: each call may require another Depth-one `PROPFIND` over the same +collection. The WebDAV target is therefore one complete listing at the start of a receive phase, filtered locally for +every Writer and Commit prefix in that phase. This receive-phase listing must expire before the next receive phase; it +must never become an unbounded cross-synchronisation cache. A Commit which becomes visible after the listing response +is handled by the next receive phase, which is consistent with immutable eventual replication. + +With that explicitly scoped reuse, the request estimates become `1 + K + B + P` and `1 + K + B + M`. For 100 new +inline-Pack Bundles which require no additional Pack read, the difference is illustrative rather than a benchmark: + +| Visible Writers | Descriptor state | Repeated-prefix listing | One receive-phase listing | +| ---: | --- | ---: | ---: | +| 1 | first receive (`K = 1`) | 103 | 102 | +| 1 | warm process (`K = 0`) | 102 | 101 | +| 10 | first receive (`K = 10`) | 121 | 111 | +| 10 | warm process (`K = 0`) | 111 | 101 | + +Opaque catch-up is approximately one listing plus one GET per new Journal object, or `1 + J`. Holding `B = J = 100` +would therefore give 101 operations. That comparison does not assume that the two batchers produce equal object counts +for a real editing history. + +### Listing-volume hypothesis + +Request count alone understates WebDAV discovery cost. Let `N` be the number of resources in the configured flat +collection and `L` the average XML response bytes per resource for a minimal Depth-one `PROPFIND`. Repeating the +listing for each logical prefix transfers roughly `(1 + W) * N * L`; one receive-phase listing transfers roughly +`N * L`. + +Assuming `L = 1 KiB` only to make the scale visible gives: + +| Collection resources (`N`) | One listing | Repeated listing, `W = 1` | Repeated listing, `W = 10` | +| ---: | ---: | ---: | ---: | +| 1,000 | about 1 MiB | about 2 MiB | about 11 MiB | +| 10,000 | about 10 MiB | about 20 MiB | about 107 MiB | +| 100,000 | about 98 MiB | about 195 MiB | about 1.05 GiB | + +Actual XML size, selected properties, URL length, compression, server implementation, and collection scope can change +these values materially. The adapter should request only the properties it needs and measure both response bytes and +wall time. The table explains the reuse target; it is not an estimate of every WebDAV server. + +Ignoring cleaned probe resources and abandoned Packs, the current Commit Bundle layout contains approximately +`1 + W + B + X` remote objects: one manifest, one descriptor per Writer, one object per Bundle, and one per external +Pack. Object count therefore grows with synchronisation batches even when the retained payload is small, and it feeds +back into subsequent `PROPFIND` cost. + +### Retained-byte hypothesis + +For a localised edit history, let `E` be the number of separately published changes and `D` the average bytes of newly +created encoded Chunk frames per change. Before Garbage Collection or Rebuild, retained Chunk bytes grow approximately +as `E * D`, plus Metadata, Commit, route, frame, server-metadata, and abandoned-publication overhead. + +For a 10 MiB incompressible binary file whose content-defined split resumes after one changed Chunk: + +| Published changes (`E`) | Average changed Chunk | Approximate new Chunk bytes retained | New Bundle objects when each Pack stays inline | +| ---: | ---: | ---: | ---: | +| 100 | 1 MiB | 100 MiB | 100 | +| 1,000 | 1 MiB | about 0.98 GiB | 1,000 | +| 100 | 256 KiB | 25 MiB | 100 | +| 1,000 | 256 KiB | 250 MiB | 1,000 | + +These figures exclude Metadata and protocol overhead and do not predict text splitting. They show why a 10 MiB live +file can consume much more than 10 MiB remotely after many committed edits. They also show why reducing Chunk size is a +storage-versus-record-count decision shared with Opaque Journal, not an Adaptive-only recommendation. + +### Pack-target hypothesis + +Changing the external Pack target does not normally duplicate Chunk frames: a Pack is their concatenation without +capacity padding. Smaller targets chiefly increase object count, route data, PUTs, listing work, and later whole-Pack +GETs. Larger targets increase peak buffer size, retry cost, transfer duration, and exposure to mobile watchdog or proxy +timeouts. + +For about 100 MiB of newly required encoded Chunk frames in one batch, the following serial-transfer model ignores RTT, +TLS setup, encryption, server processing, contention, and retries: + +| Pack target | External Packs (`X`) | PUTs including the Bundle | One target-sized upload at 5 / 10 Mbit/s | +| ---: | ---: | ---: | ---: | +| 8 MiB | 13 | 14 | about 13.4 / 6.7 seconds | +| 16 MiB | 7 | 8 | about 26.8 / 13.4 seconds | +| 32 MiB | 4 | 5 | about 53.7 / 26.8 seconds | +| 64 MiB | 2 | 3 | about 107.4 / 53.7 seconds | + +The total Chunk payload remains about 100 MiB in each row. The initial WebDAV experiment therefore retains the current +32 MiB preferred target as a compromise, not as a universal optimum or a user-facing recommendation. Measurements on +the disposable server must record peak memory, per-PUT duration, retry behaviour, total object count, and watchdog +survival. A later decision may lower the internal target for timeout-constrained endpoints without changing the wire +format. + ## Staged acceptance ### Adapter and Commonlib integration Unit tests own HTTP status classification, conditional-create verification, flat-name round trips, complete listing, probe isolation, cleanup reporting, Range validation, and whole-pack fallback. A disposable WebDAV integration runs -the safety checker before exercising Adaptive Metadata and Chunk synchronisation. +the safety checker before exercising Adaptive Metadata and Chunk synchronisation. The integration also proves that one +receive phase reuses a single collection listing, then invalidates it so the next phase can observe a new Commit. ### CLI end-to-end acceptance