obsidian-livesync/docs/adr/2026_08_fast_fetch_persistence_and_completion.md

Architectural Decision Record: Fast Fetch Persistence and Completion Semantics

Status

Accepted

Context

Fast Fetch accelerates Fast Setup (Simple Fetch) by reading CouchDB's continuous changes feed directly, decrypting each document, and writing batches to the local database. It then allows LiveSync to reflect the completed database into the Vault.

This path deliberately bypasses PouchDB's ordinary replication machinery. It must therefore reproduce the correctness guarantees on which the rest of the initialisation workflow relies:

• a remote document is decrypted and validated before it is written locally;
• every result from a batch write is checked;
• a checkpoint represents the last contiguous remote sequence which is durable in the local database; and
• successful completion requires CouchDB to terminate every finite changes page, every returned row to be durable, and a subsequent normal probe to report no available rows.

The existing implementation combines line parsing, decryption, persistence, and completion checks within one broad error handler. A decryption or persistence failure can consequently be reported as a malformed JSON line and skipped. A batch write can also resolve while containing individual failed results. In both cases the stream may continue, advance its checkpoint incorrectly, or wait indefinitely for a completion condition which the failed row would have satisfied.

Fast Fetch also reads the normal changes feed before opening each continuous page. A normal response's pending value counts items which remain after the response's results, so pending alone is not the available workload. With a one-row probe, the page can contain results.length + pending rows.

The documented limit=0 behaviour cannot be used as a portable zero-payload probe. CouchDB's API documentation says that limit=0 has the same effect as limit=1, while CouchDB 3.5.0 with a two-shard database was observed to return no result rows and leave the complete count in pending. Fast Fetch therefore uses an explicit one-row normal probe and includes no document bodies.

Finite continuous-feed completion differs across supported CouchDB releases. CouchDB 3.5.0 was observed to close a heartbeat-enabled feed with a { "last_seq": ... } line when its finite limit is met. CouchDB 3.2 instead continues to wait for database updates after the limit has been consumed. With a heartbeat configured, each wait emits another heartbeat and the page can remain open indefinitely, even after every requested row has arrived.

On CouchDB 3.2, an explicit timeout without a heartbeat has different semantics from a total request deadline. Shard-result waits may emit blank keep-alive lines and continue processing. Once the currently available changes have been exhausted and the feed is waiting for another database update, the timeout stops that wait and returns the feed-level last_seq. The timeout can therefore terminate a finite page without limiting the duration of an active page transfer.

The CouchDB sequence token is opaque and must be handled using CouchDB's sequence semantics, without parsing, ordering, or comparison. On clustered CouchDB, a changes row and the feed-level last_seq may encode related positions with different opaque tokens. Separate requests are also not one locked snapshot: their rows may be partially ordered, and replica failover may repeat changes. Fast Fetch must therefore be idempotent and must use each terminal last_seq only by returning it to CouchDB as the next since value.

Decision

Remote page sizing and completion

Fast Fetch obtains an approximate progress target from a normal changes-feed request with since=now, limit=1, and include_docs=false. This token is for progress reporting only. It is not compared with any other token and is not used as a completion checkpoint.

Before every bounded page, Fast Fetch requests a normal changes feed from the current durable cursor with limit=1 and include_docs=false. The probe and the following continuous page use the same since, style, and filter selection. Reading the probe does not consume rows from CouchDB; the continuous request starts again from that same cursor.

The number currently available is results.length + pending. If it is zero, Fast Fetch is caught up and completes without opening another stream. Otherwise, the next continuous request uses the smaller of that count and 10,000 as its finite limit.

Each finite page omits heartbeat and sets timeout=1000. This lets CouchDB 3.2 return the page's terminator one second after it exhausts the currently available changes, rather than keeping the request open for future writes. The client immediately reconnects from that terminator while another normal probe reports available work. This bounded cycle also preserves the intent of the earlier iOS and iPadOS heartbeat workaround: Fast Fetch no longer depends on a silent continuous request eventually closing at CouchDB's default 60-second timeout.

The probe and page are separate HTTP requests, not a transactional snapshot. New writes, replica selection, or administrative changes may alter the rows between them. A page which returns at least one row and a valid terminator may therefore be shorter than the probe's estimate. Fast Fetch persists that page and probes again. A page which terminates without making progress after a positive probe is a retryable transport failure, avoiding an unbounded busy loop.

Each continuous request ends with its own { "last_seq": ... } line. Fast Fetch treats that line separately from a changes row, flushes and validates all preceding local writes, and only then persists the opaque last_seq. The exact value is replayed as the next request's since; it is never parsed, ordered, or compared with a row's seq, another request's last_seq, or the database's update_seq.

The limit counts outer changes-result rows. A tombstone is one row and consumes one page slot even when no document body is present. With style=all_docs, multiple leaf revisions inside one row's changes array do not consume additional slots. Changing include_docs between the lightweight probe and the document-bearing continuous page changes the payload, not the row selection.

Processing and persistence

Each non-blank line from the changes feed is processed through these ordered stages:

1. parse and validate the changes-feed row;
2. decrypt and validate its document, when a document is present;
3. add the document to the pending local batch;
4. persist the batch;
5. inspect every result returned by the batch write; and
6. after the finite page ends, persist its last_seq terminator.

With new_edits: false, PouchDB follows CouchDB behaviour and may omit successful results. Fast Fetch therefore inspects every returned result and treats any error result as a failed batch. An empty result is valid when PouchDB accepted the complete batch.

The checkpoint may advance only to the last contiguous sequence for which all preceding documents are durable. A row which legitimately requires no local write may advance the checkpoint only after any preceding buffered documents have been flushed successfully. A page's last_seq is committed under the same rule before that page reports success.

If a batch is partly written, its checkpoint is not advanced. Retrying the batch with new_edits: false is expected to be idempotent, including for documents which were accepted during the first attempt.

Blank heartbeat lines are ignored. Malformed rows are failures; they are not silently skipped. Logs may describe the stage and sequence involved, but must not include the raw changes-feed line because it may be large or sensitive.

Each finite continuous changes request and its decoded reader must be terminated on every exit. Releasing a reader lock alone does not cancel the underlying request. Failure and completion paths therefore abort the request and attempt to cancel the reader before the bounded remote-activity scope ends.

Failure classification and retry

The streaming boundary returns a small structured failure with a stage and an explicit retryability decision. The initial stages are:

type StreamingFetchFailureStage = "transport" | "authentication" | "protocol" | "decryption" | "storage";

This type is an internal behavioural contract, not a user-interface status enumeration. It may carry safe diagnostic context, such as an HTTP status or a sequence token, without carrying document content.

Only explicitly recognised transient transport failures are retried automatically. Examples include an interrupted connection, HTTP 408, HTTP 429, and selected HTTP 5xx responses. Authentication, protocol, decryption, and local storage failures are terminal by default. A future implementation may recognise a narrower retryable case, but it must do so explicitly rather than retry every exception.

Each retry resumes from the last durable contiguous checkpoint. Retry exhaustion returns an actionable classified failure to the caller.

Initialisation lifecycle

Fast Fetch success is the only path which may continue with the offline scan, finish the rebuild, resume Vault reflection, clear the Fast Fetch checkpoint, remove the flag file, or forget the remembered initialisation choice.

The LiveSync initialisation boundary uses an explicit suspension policy. Ordinary Fetch and Rebuild resume Vault reflection when they finish, SCRAM keeps file watching suspended, and Fast Fetch keeps both file watching and replication result parsing suspended only when initialisation fails. Fast Fetch asserts both suspensions before it begins and owns their final state: success clears both, whereas a false result or exception sets both. This final assignment also covers a late failure after rebuild finalisation and the legacy doNotSuspendOnFetching path.

On failure:

• the local checkpoint and any durably fetched documents are retained for a later retry;
• the local database is not marked as resolved;
• Vault reflection remains suspended;
• the offline scan and rebuild finalisation are not run; and
• the flag file and remembered initialisation choice remain available so that restart recovery can offer the same operation again.

Any bounded remote-activity scope must still be released in a finally path, as defined by Bounded Remote Activity. Keeping Vault reflection suspended does not permit a wake lock or similar resource to leak.

Ownership

The responsibilities are divided at three injectable boundaries.

Streaming Fetch

The Commonlib streaming implementation owns HTTP response validation, NDJSON parsing, invocation of the decryption delegate, batch-write result validation, contiguous checkpoint advancement, finite-page completion, and classified failures. It does not know about the Vault, setup dialogues, flag files, or LiveSync settings.

Rebuilder

The Commonlib rebuilder owns the local database lifecycle, checkpoint storage, retry policy, marking a completed database as resolved, optional resumption of reflection, and final checkpoint removal. It does not parse changes-feed rows or interpret user-interface choices.

LiveSync Fast Setup

LiveSync owns the setup choices, suspension of initial Vault reflection, invocation of Fast Fetch, the success-only offline scan and rebuild finalisation, and cleanup of flag files and remembered choices. It does not reinterpret document, encryption, or storage failures as successful setup.

Non-Goals

This decision does not:

• change the Metadata and Chunks formats, encryption scheme, security seed, or path obfuscation;
• change ordinary PouchDB replication, Standard Fetch, or offline-scan semantics;
• provide a transaction spanning CouchDB and the local database;
• skip corrupt or unreadable documents and continue with a partial database;
• add an automatic fallback from Fast Fetch to Standard Fetch;
• define the detailed failure dialogue or other setup user-interface changes; or
• provide a transaction or locked snapshot across the normal probe and the following continuous page.

An explicit Standard Fetch choice remains available when a user needs the ordinary replication path. Any automatic fallback or richer recovery dialogue requires a separate decision because it changes user-visible setup behaviour.

Verification

The implementation is verified primarily with London School interaction tests, using mocks at each owned boundary to prove collaboration and call order.

Streaming Fetch unit tests

Inject the HTTP stream, decryption delegate, local batch writer, and checkpoint writer. Verify that:

• the order is decrypt, persist, inspect results, then checkpoint;
• parsing, decryption, and batch-result failures prevent checkpoint advancement and completion;
• a partly failed batch leaves the checkpoint unchanged and reports a storage failure;
• rows without a local write flush earlier buffered documents before advancing;
• a returned probe row is counted in addition to pending, including when pending is zero;
• every probe uses limit=1, excludes document bodies, and is repeated from the previous page's opaque terminator;
• each bounded page omits heartbeat, uses timeout=1000, and can complete under CouchDB 3.2 after its current rows have been delivered;
• deletion and document-less rows consume a page slot;
• a row count cannot complete a page without its last_seq terminator;
• a page terminator cannot advance the checkpoint before its batch is durable;
• a final row and last_seq with different opaque representations complete normally without a token comparison;
• a shorter valid page is persisted and followed by another probe, while a zero-row page after a positive probe fails without looping;
• workloads over 10,000 rows resume from each durable finite-page checkpoint;
• authentication and malformed-protocol responses are terminal;
• recognised transient transport failures are classified as retryable; and
• diagnostics do not log the raw changes-feed line.

Rebuilder unit tests

Inject the streaming operation and lifecycle collaborators. Verify that:

• transient failures retry from the latest durable checkpoint;
• terminal failures are attempted once;
• success marks the database as resolved, resumes reflection when requested, and clears the checkpoint in that order; and
• failure does not mark the database as resolved, resume reflection, or clear the checkpoint.

LiveSync orchestration unit tests

Inject Fast Fetch, the offline scanner, rebuild finalisation, and cleanup collaborators. Verify that failure performs none of the success-only actions and leaves initial Vault reflection suspended. Verify that success retains the existing setup sequence and cleanup.

Integration and E2E tests

Commonlib's CouchDB integration test remains responsible for the real HTTP changes feed, opaque sequence tokens, deletion rows, and local batch persistence. It should use the maintained CI CouchDB release, a two-shard database, and a data set large enough to cross a local batch boundary, and confirm that the final checkpoint can be passed back to CouchDB as since with no result rows or pending changes. The test must not compare that token's representation with a separately requested target or changes-row token. The focused regression test covers CouchDB 3.2's page-tail behaviour; compatibility with a real CouchDB 3.2 server can be confirmed manually without expanding the permanent CI matrix.

LiveSync's real Obsidian Setup URI workflow remains responsible for the actual Fast Fetch selection, E2EE passphrase, Vault reflection, ordinary file round trip, and hidden-file synchronisation. Injected parsing, decryption, and persistence failures remain unit-test responsibilities; repeating them through the real Obsidian E2E does not add coverage for an unchanged framework boundary. This follows Real Obsidian E2E.

Consequences

• A deterministic document failure which previously appeared to be skipped now fails Fast Fetch. This is an intentional safety improvement because the local database is known to be incomplete.
• Partial durable work and its contiguous checkpoint can be reused by a later attempt without exposing the partial database to the Vault.
• Retry delays are no longer spent on authentication, corrupt content, protocol, or local persistence failures which cannot repair themselves.
• Progress totals remain approximate and may grow when a later probe observes new work, without affecting correctness.
• A completed page can spend up to one second waiting for its terminator before Fast Fetch probes and reconnects. Active page transfer is not constrained to one second.
• The implementation requires coordinated changes in Commonlib and LiveSync. Commonlib remains the authoritative package for streaming and rebuilder behaviour; LiveSync consumes an immutable Commonlib release and owns its setup orchestration.
• Ordinary replication remains unchanged and continues to provide the reference correctness contract for decrypting, persisting, and checkpointing replicated documents.

References

• Apache CouchDB changes-feed API
• Apache CouchDB 2.0 upgrade notes for opaque update sequences
• Apache CouchDB replication protocol