Document bounded Fast Fetch completion

docs/adr/2026_08_fast_fetch_persistence_and_completion.md

@@ -19,8 +19,9 @@ initialisation workflow relies:
 - every result from a batch write is checked;
 - a checkpoint represents the last contiguous remote sequence which is durable
   in the local database; and
-- successful completion means that the captured remote target has been reached,
+- successful completion requires CouchDB to terminate every finite changes page,
-  rather than that an estimated number of documents has been received.
+  every returned row to be durable, and a subsequent normal probe to report no
+  available rows.
 
 The existing implementation combines line parsing, decryption, persistence, and
 completion checks within one broad error handler. A decryption or persistence
@@ -30,35 +31,71 @@ cases the stream may continue, advance its checkpoint incorrectly, or wait
 indefinitely for a completion condition which the failed row would have
 satisfied.
 
-Fast Fetch also estimates the number of documents from the changes feed's
+Fast Fetch also reads the normal changes feed before opening each continuous
-`pending` value. That value is useful for progress reporting, but it is not an
+page. A normal response's `pending` value counts items which remain after the
-authoritative completion boundary. The CouchDB sequence token is opaque and must
+response's `results`, so `pending` alone is not the available workload. With a
-be handled using CouchDB's sequence semantics, without numeric-prefix comparison
+one-row probe, the page can contain `results.length + pending` rows.
-or inferred row counts. On clustered CouchDB, database information and the
+
-changes feed may encode the same position with different opaque tokens, so an
+The documented `limit=0` behaviour cannot be used as a portable zero-payload
-`update_seq` from database information must not be compared directly with a
+probe. CouchDB's API documentation says that `limit=0` has the same effect as
-changes-feed row.
+`limit=1`, while CouchDB 3.5.0 with a two-shard database was observed to return
+no result rows and leave the complete count in `pending`. Fast Fetch therefore
+uses an explicit one-row normal probe and includes no document bodies.
+
+A continuous feed with a heartbeat remains open at the current tail. It closes
+with a `{ "last_seq": ... }` line only after its finite `limit` has been met.
+Consequently, a limit larger than the workload already known to be available
+can leave initialisation waiting for future writes.
+
+The CouchDB sequence token is opaque and must be handled using CouchDB's
+sequence semantics, without parsing, ordering, or comparison. On clustered
+CouchDB, a changes row and the feed-level `last_seq` may encode related
+positions with different opaque tokens. Separate requests are also not one
+locked snapshot: their rows may be partially ordered, and replica failover may
+repeat changes. Fast Fetch must therefore be idempotent and must use each
+terminal `last_seq` only by returning it to CouchDB as the next `since` value.
 
 ## Decision
 
-### Remote snapshot and completion
+### Remote page sizing and completion
 
-Fast Fetch must obtain an authoritative target token from a normal changes-feed
+Fast Fetch obtains an approximate progress target from a normal changes-feed
-snapshot before consuming the stream. A request from `since=now` with no result
+request with `since=now`, `limit=1`, and `include_docs=false`. This token is for
-rows provides a token in the same sequence domain as the streamed rows. If that
+progress reporting only. It is not compared with any other token and is not
-target cannot be obtained, the fetch fails instead of falling back to a database
+used as a completion checkpoint.
-information token, a document-count estimate, or another approximate sequence.
 
-The target token is treated as opaque. Fast Fetch completes only after the row
+Before every bounded page, Fast Fetch requests a normal changes feed from the
-for the captured target has been processed and all work up to that row has been
+current durable cursor with `limit=1` and `include_docs=false`. The probe and
-persisted successfully. When a status request proves that no changes exist after
+the following continuous page use the same `since`, style, and filter
-the current durable checkpoint, the captured target may be checkpointed without
+selection. Reading the probe does not consume rows from CouchDB; the continuous
-opening the continuous stream. This includes an empty remote database. Changes
+request starts again from that same cursor.
-made remotely after the target was captured are outside this Fast Fetch snapshot
-and are left for subsequent ordinary replication.
 
-The estimated document count remains available for progress reporting only. It
+The number currently available is `results.length + pending`. If it is zero,
-must not determine success.
+Fast Fetch is caught up and completes without opening another stream. Otherwise,
+the next continuous request uses the smaller of that count and 10,000 as its
+finite `limit`. This prevents a heartbeat-enabled request from waiting for
+future changes merely to fill an oversized page.
+
+The probe and page are separate HTTP requests, not a transactional snapshot.
+New writes, replica selection, or administrative changes may alter the rows
+between them. A page which returns at least one row and a valid terminator may
+therefore be shorter than the probe's estimate. Fast Fetch persists that page
+and probes again. A page which terminates without making progress after a
+positive probe is a retryable transport failure, avoiding an unbounded busy
+loop.
+
+Each continuous request ends with its own `{ "last_seq": ... }` line. Fast
+Fetch treats that line separately from a changes row, flushes and validates all
+preceding local writes, and only then persists the opaque `last_seq`. The exact
+value is replayed as the next request's `since`; it is never parsed, ordered, or
+compared with a row's `seq`, another request's `last_seq`, or the database's
+`update_seq`.
+
+The limit counts outer changes-result rows. A tombstone is one row and consumes
+one page slot even when no document body is present. With `style=all_docs`,
+multiple leaf revisions inside one row's `changes` array do not consume
+additional slots. Changing `include_docs` between the lightweight probe and the
+document-bearing continuous page changes the payload, not the row selection.
 
 ### Processing and persistence
 
@@ -68,8 +105,9 @@ stages:
 1. parse and validate the changes-feed row;
 2. decrypt and validate its document, when a document is present;
 3. add the document to the pending local batch;
-4. persist the batch; and
+4. persist the batch;
-5. inspect every result returned by the batch write.
+5. inspect every result returned by the batch write; and
+6. after the finite page ends, persist its `last_seq` terminator.
 
 With `new_edits: false`, PouchDB follows CouchDB behaviour and may omit successful
 results. Fast Fetch therefore inspects every returned result and treats any
@@ -79,8 +117,8 @@ the complete batch.
 The checkpoint may advance only to the last contiguous sequence for which all
 preceding documents are durable. A row which legitimately requires no local
 write may advance the checkpoint only after any preceding buffered documents
-have been flushed successfully. The target sequence is committed under the same
+have been flushed successfully. A page's `last_seq` is committed under the same
-rule before the operation reports success.
+rule before that page reports success.
 
 If a batch is partly written, its checkpoint is not advanced. Retrying the batch
 with `new_edits: false` is expected to be idempotent, including for documents
@@ -90,8 +128,8 @@ Blank heartbeat lines are ignored. Malformed rows are failures; they are not
 silently skipped. Logs may describe the stage and sequence involved, but must
 not include the raw changes-feed line because it may be large or sensitive.
 
-The continuous changes request and its decoded reader must be terminated on
+Each finite continuous changes request and its decoded reader must be terminated
-every exit. Releasing a reader lock alone does not cancel the underlying
+on every exit. Releasing a reader lock alone does not cancel the underlying
 request. Failure and completion paths therefore abort the request and attempt
 to cancel the reader before the bounded remote-activity scope ends.
 
@@ -156,7 +194,7 @@ The responsibilities are divided at three injectable boundaries.
 
 The Commonlib streaming implementation owns HTTP response validation, NDJSON
 parsing, invocation of the decryption delegate, batch-write result validation,
-contiguous checkpoint advancement, target-sequence completion, and classified
+contiguous checkpoint advancement, finite-page completion, and classified
 failures. It does not know about the Vault, setup dialogues, flag files, or
 LiveSync settings.
 
@@ -187,7 +225,8 @@ This decision does not:
 - add an automatic fallback from Fast Fetch to Standard Fetch;
 - define the detailed failure dialogue or other setup user-interface changes;
   or
-- require Fast Fetch to include remote changes made after its captured target.
+- provide a transaction or locked snapshot across the normal probe and the
+  following continuous page.
 
 An explicit Standard Fetch choice remains available when a user needs the
 ordinary replication path. Any automatic fallback or richer recovery dialogue
@@ -209,8 +248,18 @@ writer. Verify that:
 - a partly failed batch leaves the checkpoint unchanged and reports a storage
   failure;
 - rows without a local write flush earlier buffered documents before advancing;
-- an estimated document count cannot complete the fetch;
+- a returned probe row is counted in addition to `pending`, including when
-- the captured target cannot complete the fetch before its batch is durable;
+  `pending` is zero;
+- every probe uses `limit=1`, excludes document bodies, and is repeated from the
+  previous page's opaque terminator;
+- deletion and document-less rows consume a page slot;
+- a row count cannot complete a page without its `last_seq` terminator;
+- a page terminator cannot advance the checkpoint before its batch is durable;
+- a final row and `last_seq` with different opaque representations complete
+  normally without a token comparison;
+- a shorter valid page is persisted and followed by another probe, while a
+  zero-row page after a positive probe fails without looping;
+- workloads over 10,000 rows resume from each durable finite-page checkpoint;
 - authentication and malformed-protocol responses are terminal;
 - recognised transient transport failures are classified as retryable; and
 - diagnostics do not log the raw changes-feed line.
@@ -236,9 +285,12 @@ existing setup sequence and cleanup.
 ### Integration and E2E tests
 
 Commonlib's CouchDB integration test remains responsible for the real HTTP
-changes feed, opaque sequence tokens, and local batch persistence. It should
+changes feed, opaque sequence tokens, deletion rows, and local batch
-include a data set large enough to cross a batch boundary and confirm that the
+persistence. It should use a two-shard database, include a data set large enough
-final checkpoint equals the captured target.
+to cross a local batch boundary, and confirm that the final checkpoint can be
+passed back to CouchDB as `since` with no result rows or pending changes. The
+test must not compare that token's representation with a separately requested
+target or changes-row token.
 
 LiveSync's real Obsidian Setup URI workflow remains responsible for the actual
 Fast Fetch selection, E2EE passphrase, Vault reflection, ordinary file round
@@ -256,8 +308,8 @@ This follows [Real Obsidian E2E](2026_06_real_obsidian_e2e.md).
   attempt without exposing the partial database to the Vault.
 - Retry delays are no longer spent on authentication, corrupt content, protocol,
   or local persistence failures which cannot repair themselves.
-- Progress totals remain approximate and may change without affecting
+- Progress totals remain approximate and may grow when a later probe observes
-  correctness.
+  new work, without affecting correctness.
 - The implementation requires coordinated changes in Commonlib and LiveSync.
   Commonlib remains the authoritative package for streaming and rebuilder
   behaviour; LiveSync consumes an immutable Commonlib release and owns its setup
@@ -265,3 +317,9 @@ This follows [Real Obsidian E2E](2026_06_real_obsidian_e2e.md).
 - Ordinary replication remains unchanged and continues to provide the reference
   correctness contract for decrypting, persisting, and checkpointing replicated
   documents.
+
+## References
+
+- [Apache CouchDB changes-feed API](https://docs.couchdb.org/en/stable/api/database/changes.html)
+- [Apache CouchDB 2.0 upgrade notes for opaque update sequences](https://docs.couchdb.org/en/stable/whatsnew/2.0.html#upgrade-notes)
+- [Apache CouchDB replication protocol](https://docs.couchdb.org/en/stable/replication/protocol.html)

updates.md

@@ -12,6 +12,13 @@ Earlier releases remain available in the 0.25 release history and the legacy rel
 
 ## Unreleased
 
+### Setup and compatibility
+
+#### Fixed
+
+- Fast Setup now sizes each finite CouchDB changes page from a one-row status probe, counts the returned result together with `pending`, and resumes from the page's opaque `last_seq` without comparing token representations. Heartbeat-enabled feeds no longer wait for future writes after the currently available rows have been persisted (#1065).
+- Cancelling remote selection during a scheduled Fetch now removes the Fetch flag before restarting with file and database reflection paused, preventing the same selection dialogue from reopening on every start-up.
+
 ## 1.0.7
 
 8th August, 2026