mirror of
https://github.com/vrtmrz/obsidian-livesync.git
synced 2026-07-16 09:36:00 +00:00
fix: align chunk reads with remote delivery
Consume commonlib's delivery lifecycle, classify finite replication entry points, and document why the five-minute inactivity fuse is only a leak safety valve. Record the remote-activity counter correction and preserve continuous replication's unbounded live channel.
This commit is contained in:
vorotamoroz
@@ -31,13 +31,15 @@ The boundary has the following contract:
|
||||
- allow overlapping tasks, so transitions may be `0 → 1 → 2 → 1 → 0`; and
|
||||
- count logical operations, not physical connections, sockets, HTTP requests, queued work, or retry delays outside the bounded task.
|
||||
|
||||
`ReplicatorService` also owns a narrower `finiteReplicationActivityCount`. Callers enter it through the typed `runFiniteReplicationActivity` method; diagnostic labels do not determine behaviour. The narrower count describes operations which may still place replicated documents in the local database; it excludes rebuilds, chunk-fetch claims, and other bounded work which cannot satisfy an arbitrary missing-chunk read. It is a delivery-lifecycle capability, not a second Wake Lock policy or a connection counter.
|
||||
|
||||
The Obsidian host injects the screen wake-lock manager from the `octagonal-wheels` package in the Fancy Kit monorepo as the activity runner on both mobile and desktop. Unsupported or rejected wake-lock requests remain best effort and do not prevent the remote task from running. The manager is disposed when the plug-in unloads.
|
||||
|
||||
Finite replication enters the boundary only after readiness checks have succeeded and leaves it after `openReplication(..., continuous: false, ...)` settles. Failure handling runs afterwards so a mismatch or recovery dialogue does not retain the activity. This includes the direct start-up synchronisation path as well as manual, event-driven, and periodic calls through `ReplicationService`. Continuous replication does not enter the boundary.
|
||||
Finite replication enters both counts only after readiness checks have succeeded and leaves them after `openReplication(..., continuous: false, ...)` settles. A successful completion has reached the latest sequence in that operation's scope and is therefore an authoritative quiescence boundary for chunk retrieval. A failed operation does not prove latest state, but can no longer deliver documents from that attempt. Failure handling runs afterwards so a mismatch or recovery dialogue does not retain the activity. This includes the direct start-up synchronisation path as well as manual, event-driven, and periodic calls through `ReplicationService`. The unbounded continuous channel does not enter either boundary, but its finite initial pull-only catch-up does; the one-shot parameter fallback chain remains inside that boundary.
|
||||
|
||||
Manual P2P commands which bypass `ReplicationService` enter the same boundary. Direct P2P pull and push entry points also create finite transfer activities, covering the Obsidian panes, CLI, and Webapp. Automatic synchronisation on peer discovery, a pull requested by a remote peer, and a watched pull following a peer progress notification enter the same boundary. A normal P2P peer-selection dialogue represents one finite session: it remains inside the boundary while waiting for a peer and while the person may perform repeated synchronisations, then settles when the dialogue closes and any in-flight synchronisation has finished. Closing without synchronising returns a failed result and releases the boundary. The 'Start Sync & Close' action completes its synchronisation before closing. This deliberately protects peer discovery and selection, because display sleep can interrupt discovery or connection establishment and require the person to start detection again. It may therefore retain a Wake Lock longer than the network transfer alone. A transfer performed inside that session temporarily adds a nested activity; the count remains a logical-operation count rather than a connection total.
|
||||
Manual P2P commands which bypass `ReplicationService` enter the broad boundary. Direct P2P pull and push entry points are therefore both protected as finite remote work, covering the Obsidian panes, CLI, and Webapp. A pull or bidirectional synchronisation also enters the narrower finite-replication boundary because it can place documents in the local database. A push-only request remains broad-only: it cannot satisfy a local missing-chunk read and must not present itself as a delivery source. Automatic synchronisation on peer discovery, a pull requested by a remote peer, and a watched pull following a peer progress notification enter both boundaries because each can deliver local documents. A normal P2P peer-selection dialogue represents one broad finite session: it remains inside the boundary while waiting for a peer and while the person may perform repeated synchronisations, then settles when the dialogue closes and any in-flight synchronisation has finished. Closing without synchronising returns a failed result and releases the boundary. The 'Start Sync & Close' action completes its synchronisation before closing. This deliberately protects peer discovery and selection, because display sleep can interrupt discovery or connection establishment and require the person to start detection again. It may therefore retain a Wake Lock longer than the network transfer alone. A transfer performed inside that session temporarily adds a nested activity; the count remains a logical-operation count rather than a connection total.
|
||||
|
||||
`ChunkFetcher` enters the same boundary only around the actual `fetchRemoteChunks` request. Queue waiting, interval throttling, and local chunk validation or persistence remain outside the remote activity scope.
|
||||
`ChunkFetcher` enters the broad boundary synchronously when it accepts newly missing chunk identifiers, but it does not increment the finite-replication count. A typed per-identifier claim keeps the broad boundary active through queue waiting, interval throttling, `fetchRemoteChunks`, validation, local persistence, and terminal event delivery. Duplicate requests share the existing claim. Explicit absence, failure, cancellation, or a conservative five-minute period without fetcher-observable progress settles the affected claim. The five-minute value is only a last-resort leak fuse: it prevents a never-settling integration from retaining the per-identifier claim and waiter indefinitely and, once the activity runner has entered the claim task, lets the associated Wake Lock, lifecycle deferral, and indicator finish. It is not an arrival estimate, proof of remote absence, or a transport deadline. This scope is defined in detail by the chunk-arrival-quiescence ADR.
|
||||
|
||||
Rebuild operations use the same boundary at their destructive or remote phase:
|
||||
|
||||
@@ -58,9 +60,11 @@ If the desktop background setting applies, its existing continuous or periodic p
|
||||
|
||||
The status-bar remote-work indicator is shown when either the existing HTTP request balance is non-zero or the bounded remote activity count is non-zero. This preserves coverage from the existing counters while adding finite replication, peer waiting, and remote chunk fetching. The combined icon reports broader remote work, not an exact physical connection state; connection-level telemetry remains a separate concern.
|
||||
|
||||
Each physical HTTP attempt owns one balanced counter pair. The request counter is incremented immediately before invoking the selected fetch implementation, and the response counter is incremented in `finally`, whether the attempt returns or rejects. A web-fetch failure followed by the native fallback is two physical attempts and therefore contributes two balanced pairs. Callers must not add another pair around `performFetch`, because duplicated or missing increments leave the status indicator permanently active.
|
||||
|
||||
## Ownership
|
||||
|
||||
`ReplicatorService` is the shared ownership point because both `ReplicationService` and `ChunkFetcher` already depend on it. Placing the activity state in `ReplicationService` would make chunk fetching depend in the opposite direction and risk a service dependency cycle. Adding another Service Hub service would introduce a wider capability surface without a distinct lifecycle owner.
|
||||
`ReplicatorService` is the shared ownership point because both `ReplicationService` and `ChunkFetcher` already depend on it. It owns the broad activity count and classifies the semantic subset which represents finite replication. Placing either activity state in `ReplicationService` would make chunk fetching depend in the opposite direction and risk a service dependency cycle. Adding another Service Hub service would introduce a wider capability surface without a distinct lifecycle owner.
|
||||
|
||||
The platform activity runner remains injected. Common library and headless consumers can omit it while retaining the same bounded activity count and operation semantics.
|
||||
|
||||
@@ -83,13 +87,15 @@ Unit tests cover:
|
||||
- count cleanup after rejection;
|
||||
- entry into the boundary only after replication readiness succeeds;
|
||||
- replication failure handling occurring after the finite activity ends;
|
||||
- start-up one-shot replication entering the boundary while continuous start-up does not;
|
||||
- start-up one-shot replication and continuous start-up's finite pull-only catch-up entering the boundary, while the unbounded live channel does not;
|
||||
- start-up readiness failure avoiding the boundary;
|
||||
- direct P2P commands entering the boundary;
|
||||
- direct P2P pull and push entry points entering the boundary;
|
||||
- direct P2P pull and push entry points entering the broad boundary, while only pull and bidirectional operations enter the finite-delivery boundary;
|
||||
- automatic synchronisation on peer discovery, remote pull requests, and watched peer progress entering the boundary;
|
||||
- P2P peer-selection sessions settling on close, including cancellation, repeated synchronisation, and a close during in-flight work;
|
||||
- remote chunk fetching through the shared boundary;
|
||||
- remote chunk fetching remaining inside the shared boundary from synchronous queue acceptance through local persistence and terminal notification;
|
||||
- missing-chunk waiters rechecking local storage when observed per-identifier claims and finite replication have settled;
|
||||
- the finite-replication count excluding other bounded work;
|
||||
- standard, fast, remote, and combined rebuild activity boundaries;
|
||||
- Rebuilder-owned confirmation and completion dialogues remaining outside rebuild activity;
|
||||
- fallback from fast fetch avoiding a nested activity boundary;
|
||||
|
||||
@@ -0,0 +1,144 @@
|
||||
# Architectural Decision Record: Chunk Arrival Quiescence
|
||||
|
||||
## Status
|
||||
|
||||
Accepted
|
||||
|
||||
## Context
|
||||
|
||||
A file metadata document refers to one or more content-addressed chunk documents. LiveSync normally writes the chunks before writing the metadata, but those writes are separate database operations and are not replicated as one atomic unit. A local reader can consequently observe metadata before every referenced chunk is locally readable. This is expected when CouchDB on-demand chunk fetching is enabled, and can also occur around conflict resolution, replication event processing, and historical data.
|
||||
|
||||
Before this decision, a missing-chunk waiter used a fixed 5-second or 30-second timeout. Its budget included queueing, throttling, network transfer, validation, local persistence, and event delivery. A healthy operation could therefore time out immediately before it delivered the requested chunk. Conversely, when no operation was capable of delivering the chunk, the timer merely delayed the same unavailable result.
|
||||
|
||||
Finite replication provides a stronger boundary than elapsed wall-clock time. In the absence of an error, a finite replication does not complete until it has reached the latest sequence in its scope. Once it completes, that replication cannot deliver another chunk. An on-demand fetch is a separate finite delivery path and needs its own per-identifier boundary.
|
||||
|
||||
## Decision
|
||||
|
||||
Wait only for a delivery lifecycle which is observable when the local miss is handled. Do not guess how long an unobserved producer might take.
|
||||
|
||||
Two lifecycle signals are relevant:
|
||||
|
||||
- `finiteReplicationActivityCount` records finite replication operations which can still place database documents in the local database; and
|
||||
- a `ChunkDeliveryCoordinator` claim records the complete lifetime of an accepted on-demand request for each missing chunk identifier.
|
||||
|
||||
The finite replication count is distinct from the broader `boundedRemoteActivityCount`. Rebuilds and other bounded remote work may need Wake Lock and lifecycle protection without being able to satisfy an arbitrary missing-chunk read. A P2P push-only operation is broad for the same reason: it sends documents away but cannot place one in the local database. P2P pull and bidirectional operations are finite-delivery sources. `ReplicatorService` updates the narrower count only through the typed `runFiniteReplicationActivity` boundary; the diagnostic activity label is not used as a behavioural discriminator.
|
||||
|
||||
The waiting layer follows these rules:
|
||||
|
||||
1. Register the waiter before requesting on-demand delivery.
|
||||
2. Dispatch `missingChunks` synchronously when direct fetch is permitted. `ChunkFetcher` must claim accepted identifiers before dispatch returns, closing the scheduling gap without a timer.
|
||||
3. If a matching claim or finite replication is active, wait for that observable producer.
|
||||
4. A valid chunk arrival resolves the waiter immediately.
|
||||
5. An explicit remote-missing result resolves it as missing immediately.
|
||||
6. Once every observed producer completes, read the requested identifier from the local database once more, bypassing the cache.
|
||||
7. Return the rechecked chunk, or return unavailable. Do not add a fixed grace period after the producer has stopped.
|
||||
8. If no producer is observable after synchronous dispatch, return unavailable immediately. There is no operation for a duration to represent.
|
||||
|
||||
If new relevant activity starts while the final database recheck is pending, that result is stale. The waiter remains active until the newer producer completes and a current recheck finishes.
|
||||
|
||||
A successful finite replication completion is the authoritative ‘latest’ boundary. A failed operation does not prove that the remote lacks the chunk, but it is still terminal for that attempt: it can no longer deliver data. The same local recheck preserves any documents received before the failure, after which normal replication error and retry handling remains responsible for recovery. Missing-chunk code must not misreport that case as an explicit remote absence.
|
||||
|
||||
### On-demand fetch boundary
|
||||
|
||||
`ChunkFetcher` claims newly requested identifiers synchronously while handling the `missingChunks` event. The claim remains active through:
|
||||
|
||||
- queueing and concurrency scheduling;
|
||||
- configured interval throttling;
|
||||
- entry into the injected bounded-activity runner;
|
||||
- `fetchRemoteChunks`;
|
||||
- response validation;
|
||||
- local database persistence; and
|
||||
- fetched or missing event delivery.
|
||||
|
||||
The claim settles on every terminal path, including explicit absence, no active replicator, rejection, invalid results, destruction, and cancellation. Its completion Promise is the task passed to the bounded `chunk-fetch` activity, keeping Wake Lock, application lifecycle deferral, the remote-work indicator, and missing-chunk delivery aligned.
|
||||
|
||||
### Five-minute leak fuse
|
||||
|
||||
An accepted on-demand claim has a separate five-minute inactivity fuse. This is a last-resort leak safety valve, not a chunk-arrival budget or a remote-request timeout.
|
||||
|
||||
Its purpose is to prevent a faulty integration, a never-settling Promise, or a stalled transport from retaining logical ownership indefinitely. When it fires, the coordinator releases the per-identifier claim and its waiter. If the bounded activity callback has been entered, resolving the claim also allows the associated Wake Lock, application-lifecycle deferral, and remote-work indicator to be released. `ChunkFetcher` refreshes the fuse only at observable progress points, such as entering the activity boundary, beginning and completing throttling or transfer, and completing persistence.
|
||||
|
||||
Five minutes is deliberately a conservative operational limit, not a value derived from a network protocol, a benchmark, or evidence that a missing chunk will arrive within that period. Firing the fuse neither proves remote absence nor makes the underlying request safe to abort. The current `fetchRemoteChunks` contract has no `AbortSignal`, so a physical request may still complete after its logical claim has been released. A future cancellable transport contract should add transport-specific deadlines and explicit cancellation without changing the lifecycle-based wait rule.
|
||||
|
||||
### Continuous replication
|
||||
|
||||
The unbounded live channel is not a quiescence gate because it has no natural end. Its initial pull-only catch-up is finite, however, and must enter `runFiniteReplicationActivity`. This includes the one-shot parameter fallback chain: every retry remains within the catch-up boundary until it succeeds or stops. If continuous replication later restarts with adjusted parameters, the new initial catch-up enters a new finite boundary.
|
||||
|
||||
Once the live channel has begun, a chunk delivered through it still resolves an existing waiter immediately, but the channel itself does not keep a new waiter open. CouchDB on-demand fetching supplies its own per-identifier claim. If a future defect demonstrates a delivery race inside a live batch, that batch lifecycle should be exposed explicitly rather than approximated with another elapsed delay.
|
||||
|
||||
## Ownership
|
||||
|
||||
`ReplicatorService` owns both the broad bounded-operation count and the narrower finite-replication count. It is the common lifecycle owner for CouchDB, sequential, and P2P replicators.
|
||||
|
||||
`LayeredChunkManager` owns one `ChunkDeliveryCoordinator` and supplies it to its arrival layer and `ChunkFetcher`. `ArrivalWaitLayer` owns waiter resolution and the final local database recheck. It depends on the narrow coordinator capability rather than on `ReplicatorService` itself.
|
||||
|
||||
`ChunkFetcher` owns per-identifier claims because it knows when each identifier enters its queue and reaches a terminal result.
|
||||
|
||||
## Compatibility
|
||||
|
||||
- Preserve immediate reads when `waitForDelivery` is false or the deprecated call-site `timeout` is zero or negative.
|
||||
- Treat a positive deprecated `timeout` only as source-compatible opt-in to lifecycle waiting. Its numeric value no longer represents an arrival duration.
|
||||
- Preserve `preventRemoteRequest`: no on-demand request is dispatched, although an already-active finite replication may satisfy the waiter.
|
||||
- Preserve Promise sharing for concurrent reads of the same chunk identifier.
|
||||
- Preserve immediate explicit remote-missing results.
|
||||
- Do not change which remote types support direct on-demand fetching.
|
||||
|
||||
## Historical Evidence and Scope
|
||||
|
||||
This decision addresses the lifecycle-race class rather than treating every ‘Load failed’ report as a timeout:
|
||||
|
||||
- [Issue #166](https://github.com/vrtmrz/obsidian-livesync/issues/166) contained logs where chunk collection failed shortly before related chunk writes appeared. It is evidence for the timing class, although that issue's hidden-file start-up path was repaired separately and is not claimed as a direct regression test here.
|
||||
- The 2021 timing fixes in [commit `39e2eab0`](https://github.com/vrtmrz/obsidian-livesync/commit/39e2eab0238d9c37e3653cdec884cbeed543fc23) and the extended leaf timeout in [commit `9facb577`](https://github.com/vrtmrz/obsidian-livesync/commit/9facb577601d8aceff7df547cd2a6f9357fdaa29) show that elapsed timeout values have historically been used to absorb the same ordering uncertainty. They do not provide a protocol basis for retaining 5-second or 30-second delays.
|
||||
- Replication pacing introduced by [commit `8d66c372`](https://github.com/vrtmrz/obsidian-livesync/commit/8d66c372e15c43a2de84a223c6385077b7724eec) and commonlib [commit `051b50c`](https://github.com/vrtmrz/livesync-commonlib/commit/051b50ca38ec4c05a11e8216ac259b4488b825f0) is a direct precedent for preventing replication progress from outrunning chunk collection. The present design expresses that dependency as an explicit lifecycle and completion recheck.
|
||||
- [Issue #505](https://github.com/vrtmrz/obsidian-livesync/issues/505) was traced to chunks which were genuinely absent after the former bulk-send option broke the chunks-before-metadata guarantee. Waiting cannot recreate missing data, so this decision does not claim to fix it.
|
||||
- [Issue #771](https://github.com/vrtmrz/obsidian-livesync/issues/771) and [Issue #986](https://github.com/vrtmrz/obsidian-livesync/issues/986) contain ambiguous or version-dependent `Load failed` reports. They remain unclaimed until the original writer and database state can be reproduced.
|
||||
|
||||
The detailed setting and replicator matrix is recorded in [Chunk Retrieval and Waiting](../design_docs/chunk_retrieval_and_waiting.md).
|
||||
|
||||
## Alternatives Rejected
|
||||
|
||||
### Increase the fixed timeout constants
|
||||
|
||||
Any fixed total duration can still expire immediately before a queued or progressing operation reports its result. Larger values also make genuine failures slower without defining what the system is waiting for.
|
||||
|
||||
### Start another grace period after finite completion
|
||||
|
||||
A successful finite replication has already reached its latest sequence, and the completion recheck observes documents persisted without a waiter event. Waiting an additional 5 or 30 seconds has no identified producer to wait for and merely retains the historical approximation.
|
||||
|
||||
### Observe all bounded remote activity
|
||||
|
||||
The broad count includes operations which cannot provide the requested chunk. Using it as the delivery gate lets unrelated work delay a read and makes its completion semantically meaningless. A separate finite-replication count avoids that leak.
|
||||
|
||||
### Keep a fallback timer for unobserved delivery
|
||||
|
||||
An unobserved producer has no defined start, progress, or completion semantics. A timer would therefore be a guess rather than a safety property. Relevant delivery paths must claim their work synchronously or expose a finite replication boundary; otherwise the read returns unavailable.
|
||||
|
||||
### Remove every timer
|
||||
|
||||
The arrival wait has no elapsed timer, but an implementation fault can leave a delivery claim unresolved forever. The five-minute inactivity fuse bounds that leaked logical state without being used as a successful delivery condition.
|
||||
|
||||
## Verification
|
||||
|
||||
Unit tests use deterministic clocks and deferred Promises to cover:
|
||||
|
||||
- a finite replication which lasts well beyond the former arrival values;
|
||||
- successful finite completion causing a cache-bypassing local database recheck;
|
||||
- immediate unavailability when no producer is observable;
|
||||
- per-identifier claims covering queueing, throttling, remote fetch, validation, persistence, and event delivery;
|
||||
- explicit missing, no-replicator, rejection, invalid response, cancellation, runner rejection, and teardown paths;
|
||||
- overlapping claims and finite replications;
|
||||
- a runner which never enters the task and a request which never settles;
|
||||
- the five-minute fuse being refreshed by observable progress;
|
||||
- continuous replication's finite initial catch-up, including its parameter fallback path; and
|
||||
- the setting and replicator decision matrix.
|
||||
|
||||
Integration-style unit tests exercise `LayeredChunkManager`, `ChunkFetcher`, a memory-backed PouchDB database, and a deferred fake replicator together. A real Obsidian test is not required because the change remains behind the existing database, service, and event boundaries and does not alter platform UI or an adapter contract.
|
||||
|
||||
## Consequences
|
||||
|
||||
- A healthy finite replication or on-demand request no longer loses a race against an unrelated wall-clock estimate.
|
||||
- Successful finite replication completion provides a precise latest boundary for missing-chunk reads.
|
||||
- The local recheck closes event-delivery and cache timing gaps without extending the wait after completion.
|
||||
- Reads no longer pause for 5 or 30 seconds when no observable operation can deliver the chunk.
|
||||
- Relevant producers must expose a lifecycle and must continue to prove cleanup on every exceptional path.
|
||||
- The five-minute fuse bounds leaked logical activity, but it neither establishes remote absence nor cancels a physical request.
|
||||
@@ -0,0 +1,118 @@
|
||||
# Chunk Retrieval and Waiting
|
||||
|
||||
## Purpose
|
||||
|
||||
This document records how LiveSync retrieves chunks after file metadata has been found, which operation provides each terminal condition, and what the remaining time value means. It is an implementation specification for developers; it is not a user configuration guide.
|
||||
|
||||
The architectural decision and historical rationale are in [Chunk Arrival Quiescence](../adr/2026_07_chunk_arrival_quiescence.md).
|
||||
|
||||
## Invariants and Sources of Apparent Reordering
|
||||
|
||||
A normal local save creates and persists the chunks before it writes the metadata document which refers to them. LiveSync must preserve this invariant: publishing metadata first can expose a reference which no client can satisfy.
|
||||
|
||||
This ordering is not an atomic transaction across documents. A reader may still see metadata before a referenced chunk for these reasons:
|
||||
|
||||
- CouchDB replication transfers individual documents and does not expose the chunk and metadata writes as one atomic unit.
|
||||
- With `readChunksOnline` enabled, CouchDB pull replication deliberately excludes chunk documents. Seeing metadata first is then the intended design, and the chunk is fetched by identifier.
|
||||
- A winning metadata conflict revision may refer to chunks created by another revision or client which have not yet reached the local database.
|
||||
- Replication persists documents before every downstream change callback and file-reflection task has necessarily observed them.
|
||||
- Historical versions and removed transfer modes may have produced data which did not preserve the normal ordering invariant.
|
||||
|
||||
Waiting may resolve temporary visibility and processing gaps only when a known operation can still deliver the chunk. It cannot repair a chunk which is absent from every available source.
|
||||
|
||||
## Retrieval Capabilities
|
||||
|
||||
Direct on-demand fetch is currently available only for CouchDB when `useOnlyLocalChunk` is false. This capability deliberately does not depend on `readChunksOnline`:
|
||||
|
||||
- when `readChunksOnline` is true, direct fetch is the normal way to obtain a chunk omitted from pull replication; and
|
||||
- when `readChunksOnline` is false, direct fetch remains a recovery path if a normally replicated chunk is locally absent.
|
||||
|
||||
MinIO's sequential replicator and P2P do not implement direct `fetchRemoteChunks` delivery through this path. Their chunks must arrive through a finite replication operation. A P2P pull or bidirectional synchronisation is such a producer. A push-only P2P request remains broad remote activity for Wake Lock and lifecycle reporting, but it is deliberately excluded from `finiteReplicationActivityCount` because it cannot deliver a local document.
|
||||
|
||||
`waitForReady` is a call-site policy, not a persisted user setting. `true` permits waiting for an already-observable producer. `false` normally requests an immediate local result, except that CouchDB on-demand delivery still waits for the claim which synchronous dispatch creates.
|
||||
|
||||
## Policy Matrix
|
||||
|
||||
The matrix selects whether lifecycle waiting is permitted and whether the waiter may dispatch a direct request. It does not assign elapsed arrival budgets.
|
||||
|
||||
| Remote | `waitForReady` | `useOnlyLocalChunk` | Direct fetch | Wait for observed producer | Intended behaviour |
|
||||
| ------- | -------------: | ------------------: | -----------: | -------------------------: | --------------------------------------------------------------------------------------- |
|
||||
| CouchDB | `false` | `false` | Yes | Yes | Dispatch on-demand fetch and finish at its per-identifier claim boundary. |
|
||||
| CouchDB | `true` | `false` | Yes | Yes | Accept an active finite replication or dispatch direct fetch. |
|
||||
| CouchDB | `false` | `true` | No | No | Return immediately after the local miss. |
|
||||
| CouchDB | `true` | `true` | No | Yes | Wait for an already-active finite replication; otherwise return unavailable. |
|
||||
| MinIO | `false` | Either | No | No | Return immediately after the local miss. |
|
||||
| MinIO | `true` | Either | No | Yes | Wait for an already-active finite sequential replication; otherwise return unavailable. |
|
||||
| P2P | `false` | Either | No | No | Return immediately after the local miss. |
|
||||
| P2P | `true` | Either | No | Yes | Wait for an already-active finite P2P replication; otherwise return unavailable. |
|
||||
|
||||
For CouchDB, `readChunksOnline` changes what normal replication includes, not the direct-fetch capability or this matrix:
|
||||
|
||||
| `readChunksOnline` | CouchDB pull contains chunks | Role of direct fetch |
|
||||
| -----------------: | ---------------------------: | ------------------------------------------------------------------------ |
|
||||
| `true` | No | Primary chunk delivery after metadata arrives. |
|
||||
| `false` | Yes | Recovery fallback for a chunk which is unexpectedly unavailable locally. |
|
||||
|
||||
`concurrencyOfReadChunksOnline` and `minimumIntervalOfReadChunksOnline` affect only the scheduling of CouchDB on-demand requests. They do not change whether a request may be dispatched or which lifecycle a reader observes. Accepted identifiers remain claimed while they wait for a concurrency slot and while the configured interval is applied. A minimum interval of five minutes or more is an exceptional value: the inactivity fuse may release the logical claim before that deliberate pause completes. This safety precedence does not abort the delayed physical request.
|
||||
|
||||
## Wait State Machine
|
||||
|
||||
1. Read the cache and local database.
|
||||
2. If every requested chunk is present, return it without entering a wait.
|
||||
3. Register one shared waiter per missing identifier.
|
||||
4. If policy permits direct fetch, emit `missingChunks`. `ChunkFetcher` synchronously creates the per-identifier claim before the event dispatch returns.
|
||||
5. Observe both the matching claim and `finiteReplicationActivityCount`.
|
||||
6. Resolve immediately if a valid chunk or explicit remote-missing event arrives.
|
||||
7. If an observed producer remains active, do not charge elapsed time against an arrival budget.
|
||||
8. When all observed producers end, bypass the cache and read the identifiers from the local database once.
|
||||
9. Return the rechecked chunk, or return unavailable. Do not add another fixed grace after the authoritative boundary.
|
||||
10. If no producer is observable after synchronous dispatch, return unavailable immediately.
|
||||
|
||||
If new relevant activity starts while the final database recheck is pending, that result becomes stale. The waiter remains active until the newer producer completes and a current recheck finishes.
|
||||
|
||||
## Meaning of Finite Replication Completion
|
||||
|
||||
Finite replication enters the typed `runFiniteReplicationActivity` boundary and is represented by the narrower `finiteReplicationActivityCount`. The optional `replication` label remains diagnostic and does not control this behaviour.
|
||||
|
||||
For a successful finite operation, completion means that its replicator has reached the latest sequence in the operation's scope and processed its replication change callbacks. No more database documents can arrive from that operation. This is the primary semantic cutoff.
|
||||
|
||||
If the operation fails, it has not proved remote absence or latest state. It has nevertheless stopped being a producer. The waiting layer rechecks documents which may have arrived before the failure and then returns unavailable; the replication error and retry workflow owns further recovery.
|
||||
|
||||
Overlapping finite replications keep the count above zero until the final operation settles. The local recheck therefore occurs only after every observed finite producer is quiescent.
|
||||
|
||||
The continuous live channel is intentionally excluded because it has no completion boundary and would otherwise make a chunk read unbounded. The pull-only catch-up run before opening that channel is finite and enters the same typed boundary. Its one-shot batch-size fallback remains inside that boundary. A live-channel fallback starts another continuous attempt and therefore another bounded initial catch-up.
|
||||
|
||||
## Meaning of an On-demand Claim
|
||||
|
||||
An accepted identifier remains claimed from synchronous queue acceptance through throttling, physical fetch, validation, local persistence, and terminal event delivery. The claim is identifier-scoped because a global remote-work count cannot say whether unrelated work can provide this chunk.
|
||||
|
||||
The claim finishes when the fetcher has recorded an outcome for the identifier. A transport error, missing active replicator, or invalid result releases the claim without emitting an explicit remote-missing result unless the remote actually supplied that information.
|
||||
|
||||
## Meaning of the Five-minute Value
|
||||
|
||||
The five-minute value is an inactivity leak fuse for an accepted on-demand claim. It is the only elapsed duration in this state machine, and it is not a normal terminal condition.
|
||||
|
||||
The fuse bounds retention if a faulty activity runner never enters its task, a Promise never settles, or a transport stops making observable progress. It prevents the per-identifier claim and waiter from remaining live forever. Once the bounded activity callback has entered, releasing the claim also allows Wake Lock, application-lifecycle deferral, and the remote-work indicator associated with that callback to finish. Observable progress rearms the fuse.
|
||||
|
||||
Five minutes is a conservative operational ceiling rather than a measured chunk-arrival expectation. It must not be used to infer that the remote lacks a chunk, and it does not abort the physical request. `fetchRemoteChunks` does not yet accept an `AbortSignal`, so the request may complete after the logical state has been released. Transport cancellation and transport-specific deadlines are separate future work.
|
||||
|
||||
The old 5-second and 30-second constants remain exported for source compatibility only. A positive deprecated `ChunkReadOptions.timeout` opts into lifecycle waiting, but its numeric value is ignored. Zero or a negative value still requests an immediate result. New code uses `waitForDelivery` explicitly.
|
||||
|
||||
## Test Obligations
|
||||
|
||||
Changes to this behaviour must keep automated coverage for:
|
||||
|
||||
- chunks-before-metadata save ordering;
|
||||
- every row in the retrieval policy matrix;
|
||||
- a finite replication which remains active well beyond the former 5-second and 30-second values;
|
||||
- successful completion with the chunk already persisted but no arrival event delivered;
|
||||
- immediate unavailability when no producer is observable;
|
||||
- overlapping finite operations and overlapping per-identifier claims;
|
||||
- activity restarting while a local recheck is pending;
|
||||
- direct fetch queueing, throttling, persistence, and terminal notification;
|
||||
- explicit remote absence versus transport or replicator failure;
|
||||
- runner rejection, cancellation, teardown, and an operation which never enters its task;
|
||||
- leak-fuse refresh at observable progress points; and
|
||||
- continuous replication's finite initial catch-up and parameter fallback.
|
||||
|
||||
The service, database, and event boundaries are testable with memory-backed PouchDB and injected activity sources. A real Obsidian test is required only when a change crosses into the platform adapter, application lifecycle, or visible UI rather than for this retrieval state machine alone.
|
||||
+1
-1
Submodule src/lib updated: a58965f9cd...4639e4537f
@@ -150,7 +150,11 @@ Even if you choose to clean up, you will see this option again if you exit Obsid
|
||||
await purgeUnreferencedChunks(this.localDatabase.localDatabase, false);
|
||||
this.localDatabase.clearCaches();
|
||||
// Perform the synchronisation once.
|
||||
if (await this.core.replicator.openReplication(this.settings, false, showMessage, true)) {
|
||||
const replicated = await this.services.replicator.runFiniteReplicationActivity(
|
||||
() => this.core.replicator.openReplication(this.settings, false, showMessage, true),
|
||||
{ label: "replication" }
|
||||
);
|
||||
if (replicated) {
|
||||
await balanceChunkPurgedDBs(this.localDatabase.localDatabase, remoteDB.db);
|
||||
await purgeUnreferencedChunks(this.localDatabase.localDatabase, false);
|
||||
this.localDatabase.clearCaches();
|
||||
|
||||
@@ -69,6 +69,7 @@ describe("ModuleReplicator legacy cleanup", () => {
|
||||
activityFinished();
|
||||
}
|
||||
});
|
||||
const runFiniteReplicationActivity = vi.fn(async (task: () => unknown) => await task());
|
||||
const openReplication = vi.fn(async () => true);
|
||||
const activeReplicator = Object.assign(new LiveSyncCouchDBReplicator({} as any), {
|
||||
connectRemoteCouchDBWithSetting: vi.fn(async () => ({ db: {} })),
|
||||
@@ -90,6 +91,7 @@ describe("ModuleReplicator legacy cleanup", () => {
|
||||
replicator: {
|
||||
getActiveReplicator: vi.fn(() => activeReplicator),
|
||||
runBoundedRemoteActivity,
|
||||
runFiniteReplicationActivity,
|
||||
},
|
||||
};
|
||||
const localDatabase = {
|
||||
@@ -111,6 +113,9 @@ describe("ModuleReplicator legacy cleanup", () => {
|
||||
expect(runBoundedRemoteActivity).toHaveBeenCalledWith(expect.any(Function), {
|
||||
label: "database-cleanup",
|
||||
});
|
||||
expect(runFiniteReplicationActivity).toHaveBeenCalledWith(expect.any(Function), {
|
||||
label: "replication",
|
||||
});
|
||||
expect(openReplication).toHaveBeenCalledOnce();
|
||||
expect(openReplication.mock.invocationCallOrder[0]).toBeLessThan(activityFinished.mock.invocationCallOrder[0]);
|
||||
expect(chunkMocks.balanceChunkPurgedDBs).toHaveBeenCalledOnce();
|
||||
|
||||
@@ -33,7 +33,7 @@ export class ModuleReplicatorCouchDB extends AbstractModule {
|
||||
if (continuous) {
|
||||
void openReplication();
|
||||
} else {
|
||||
await this.services.replicator.runBoundedRemoteActivity(openReplication, {
|
||||
await this.services.replicator.runFiniteReplicationActivity(openReplication, {
|
||||
label: "replication",
|
||||
});
|
||||
}
|
||||
|
||||
@@ -3,7 +3,7 @@ import { ModuleReplicatorCouchDB } from "./ModuleReplicatorCouchDB.ts";
|
||||
|
||||
function createModule(settings: { liveSync: boolean; syncOnStart: boolean }, isReplicationReady = true) {
|
||||
const openReplication = vi.fn(async () => true);
|
||||
const runBoundedRemoteActivity = vi.fn(async (task: () => unknown) => await task());
|
||||
const runFiniteReplicationActivity = vi.fn(async (task: () => unknown) => await task());
|
||||
const services = {
|
||||
API: {
|
||||
addLog: vi.fn(),
|
||||
@@ -20,7 +20,7 @@ function createModule(settings: { liveSync: boolean; syncOnStart: boolean }, isR
|
||||
isReplicationReady: vi.fn(async () => isReplicationReady),
|
||||
},
|
||||
replicator: {
|
||||
runBoundedRemoteActivity,
|
||||
runFiniteReplicationActivity,
|
||||
},
|
||||
setting: {
|
||||
saveSettingData: vi.fn(async () => undefined),
|
||||
@@ -38,13 +38,13 @@ function createModule(settings: { liveSync: boolean; syncOnStart: boolean }, isR
|
||||
return {
|
||||
module: new ModuleReplicatorCouchDB(core),
|
||||
openReplication,
|
||||
runBoundedRemoteActivity,
|
||||
runFiniteReplicationActivity,
|
||||
};
|
||||
}
|
||||
|
||||
describe("ModuleReplicatorCouchDB resume replication activity", () => {
|
||||
it("runs start-up one-shot replication through bounded remote activity", async () => {
|
||||
const { module, openReplication, runBoundedRemoteActivity } = createModule({
|
||||
it("exposes start-up one-shot replication as finite replication activity", async () => {
|
||||
const { module, openReplication, runFiniteReplicationActivity } = createModule({
|
||||
liveSync: false,
|
||||
syncOnStart: true,
|
||||
});
|
||||
@@ -52,14 +52,14 @@ describe("ModuleReplicatorCouchDB resume replication activity", () => {
|
||||
await module._everyAfterResumeProcess();
|
||||
|
||||
await vi.waitFor(() => expect(openReplication).toHaveBeenCalledOnce());
|
||||
expect(runBoundedRemoteActivity).toHaveBeenCalledWith(expect.any(Function), {
|
||||
expect(runFiniteReplicationActivity).toHaveBeenCalledWith(expect.any(Function), {
|
||||
label: "replication",
|
||||
});
|
||||
expect(openReplication).toHaveBeenCalledWith(expect.any(Object), false, false, false);
|
||||
});
|
||||
|
||||
it("does not treat continuous replication as bounded activity", async () => {
|
||||
const { module, openReplication, runBoundedRemoteActivity } = createModule({
|
||||
it("does not wrap the unbounded continuous channel in another finite activity", async () => {
|
||||
const { module, openReplication, runFiniteReplicationActivity } = createModule({
|
||||
liveSync: true,
|
||||
syncOnStart: false,
|
||||
});
|
||||
@@ -67,12 +67,12 @@ describe("ModuleReplicatorCouchDB resume replication activity", () => {
|
||||
await module._everyAfterResumeProcess();
|
||||
|
||||
await vi.waitFor(() => expect(openReplication).toHaveBeenCalledOnce());
|
||||
expect(runBoundedRemoteActivity).not.toHaveBeenCalled();
|
||||
expect(runFiniteReplicationActivity).not.toHaveBeenCalled();
|
||||
expect(openReplication).toHaveBeenCalledWith(expect.any(Object), true, false, false);
|
||||
});
|
||||
|
||||
it("does not start a one-shot activity when start-up readiness fails", async () => {
|
||||
const { module, openReplication, runBoundedRemoteActivity } = createModule(
|
||||
const { module, openReplication, runFiniteReplicationActivity } = createModule(
|
||||
{
|
||||
liveSync: false,
|
||||
syncOnStart: true,
|
||||
@@ -83,7 +83,7 @@ describe("ModuleReplicatorCouchDB resume replication activity", () => {
|
||||
await module._everyAfterResumeProcess();
|
||||
await new Promise((resolve) => setTimeout(resolve, 0));
|
||||
|
||||
expect(runBoundedRemoteActivity).not.toHaveBeenCalled();
|
||||
expect(runFiniteReplicationActivity).not.toHaveBeenCalled();
|
||||
expect(openReplication).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -87,7 +87,7 @@ export function useP2PReplicatorUI(
|
||||
const activeReplicator = replicator.replicator;
|
||||
if (!activeReplicator) return;
|
||||
const settings = host.services.setting.currentSettings();
|
||||
void host.services.replicator.runBoundedRemoteActivity(
|
||||
void host.services.replicator.runFiniteReplicationActivity(
|
||||
() => activeReplicator.openReplication(settings, false, true, false),
|
||||
{ label: "replication" }
|
||||
);
|
||||
|
||||
@@ -12,11 +12,11 @@ vi.mock("@/features/P2PSync/P2PReplicator/P2PServerStatusPaneView", () => ({
|
||||
import { useP2PReplicatorUI } from "./useP2PReplicatorUI";
|
||||
|
||||
describe("useP2PReplicatorUI commands", () => {
|
||||
it("runs a direct modal P2P replication command through the shared activity boundary", async () => {
|
||||
it("exposes a direct modal P2P replication command as finite replication activity", async () => {
|
||||
const commands: Array<{ id: string; checkCallback?: (isChecking: boolean) => unknown }> = [];
|
||||
let initialise: (() => Promise<unknown>) | undefined;
|
||||
const openReplication = vi.fn(async () => true);
|
||||
const runBoundedRemoteActivity = vi.fn(async (task: () => unknown) => await task());
|
||||
const runFiniteReplicationActivity = vi.fn(async (task: () => unknown) => await task());
|
||||
const host = {
|
||||
services: {
|
||||
API: {
|
||||
@@ -35,7 +35,7 @@ describe("useP2PReplicatorUI commands", () => {
|
||||
onLayoutReady: { addHandler: vi.fn() },
|
||||
},
|
||||
setting: { currentSettings: vi.fn(() => ({ remoteType: "COUCHDB" })) },
|
||||
replicator: { runBoundedRemoteActivity },
|
||||
replicator: { runFiniteReplicationActivity },
|
||||
},
|
||||
} as any;
|
||||
const p2p = {
|
||||
@@ -51,7 +51,7 @@ describe("useP2PReplicatorUI commands", () => {
|
||||
commands.find((command) => command.id === "replicate-now-by-p2p")?.checkCallback?.(false);
|
||||
|
||||
await vi.waitFor(() => expect(openReplication).toHaveBeenCalledOnce());
|
||||
expect(runBoundedRemoteActivity).toHaveBeenCalledWith(expect.any(Function), {
|
||||
expect(runFiniteReplicationActivity).toHaveBeenCalledWith(expect.any(Function), {
|
||||
label: "replication",
|
||||
});
|
||||
});
|
||||
|
||||
@@ -66,6 +66,10 @@ npm run test:e2e:obsidian:local-suite:services
|
||||
|
||||
`test:e2e:obsidian:couchdb-upload` reuses the CouchDB variables from `.test.env` or the process environment. It expects a reachable CouchDB service, creates a unique database, configures Self-hosted LiveSync through `obsidian-cli eval`, creates a note in real Obsidian, commits the note into the local database, runs one-shot synchronisation, and verifies that the remote database contains both the metadata document and its chunk documents.
|
||||
|
||||
The same workflow checks the remote-activity status boundary. It holds the real one-shot replication immediately before its replicator call, confirms that the status bar contains `📲` while finite replication is active, releases it, and then requires the finite and bounded activity counts to return to zero, the HTTP request and response counts to balance, and `📲` to disappear. It then creates a remote-only chunk, holds the real on-demand fetch immediately before its remote call, makes the same active and idle assertions, and verifies that the fetched chunk is written into the local database. These gates make the active state deterministic without replacing either remote operation.
|
||||
|
||||
If this status workflow fails while Obsidian is running, it writes a full-page screenshot and a JSON snapshot of the status text and counters under `/tmp/obsidian-livesync-e2e`. Set `E2E_OBSIDIAN_DIAGNOSTICS_DIR` to use another directory.
|
||||
|
||||
`test:e2e:obsidian:cli-to-obsidian-sync` is the cross-runtime compatibility check for the official LiveSync CLI and the real Obsidian plug-in. Build the plug-in first, and build the local CLI too when no external CLI command is selected. The script uses E2EE, Path Obfuscation, and the current preferred chunk settings to create and synchronise a note through the CLI, starts real Obsidian with an isolated Vault and profile, synchronises the same CouchDB database, and verifies that the plug-in materialises identical note content. This covers the boundary that CLI-only and plug-in-only round trips do not exercise.
|
||||
|
||||
By default, the compatibility check runs `node src/apps/cli/dist/index.cjs`. Set `LIVESYNC_CLI_COMMAND` to test another CLI build or distribution. The value may be a quoted command line or a JSON array of executable and prefix arguments; the scenario arguments are appended without going through a shell.
|
||||
@@ -128,6 +132,8 @@ Useful environment variables:
|
||||
- `E2E_OBSIDIAN_CORE_READY_TIMEOUT_MS`: timeout for waiting until Self-hosted LiveSync reports that its core lifecycle and local database are ready.
|
||||
- `E2E_OBSIDIAN_LOCAL_DB_TIMEOUT_MS`: timeout for waiting until a file appears in Self-hosted LiveSync's local database.
|
||||
- `E2E_OBSIDIAN_COUCHDB_TIMEOUT_MS`: timeout for waiting until CouchDB contains uploaded E2E documents.
|
||||
- `E2E_OBSIDIAN_REMOTE_ACTIVITY_TIMEOUT_MS`: timeout for an observed remote activity to enter or leave its status boundary; default is 30 seconds.
|
||||
- `E2E_OBSIDIAN_DIAGNOSTICS_DIR`: directory for screenshots and status snapshots captured after a remote-activity failure; default is `/tmp/obsidian-livesync-e2e`.
|
||||
- `E2E_OBSIDIAN_OBJECT_STORAGE_TIMEOUT_MS`: timeout for waiting until Object Storage contains uploaded E2E objects.
|
||||
- `E2E_OBSIDIAN_KEEP_COUCHDB=true`: keep the temporary CouchDB database for inspection.
|
||||
- `E2E_OBSIDIAN_KEEP_OBJECT_STORAGE=true`: keep the temporary Object Storage prefix for inspection.
|
||||
|
||||
@@ -126,6 +126,26 @@ export async function createCouchDbDatabase(config: CouchDbConfig, dbName: strin
|
||||
}
|
||||
}
|
||||
|
||||
export async function putCouchDbDocument(
|
||||
config: CouchDbConfig,
|
||||
dbName: string,
|
||||
document: CouchDbDocument
|
||||
): Promise<void> {
|
||||
const response = await fetch(databaseUrl(config, dbName, `/${encodeURIComponent(document._id)}`), {
|
||||
method: "PUT",
|
||||
headers: {
|
||||
authorization: authHeader(config),
|
||||
"content-type": "application/json",
|
||||
},
|
||||
body: JSON.stringify(document),
|
||||
});
|
||||
if (!response.ok) {
|
||||
throw new Error(
|
||||
`Failed to write CouchDB document ${document._id}. HTTP ${response.status}: ${await response.text()}`
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
export async function deleteCouchDbDatabase(config: CouchDbConfig, dbName: string): Promise<void> {
|
||||
const response = await fetch(databaseUrl(config, dbName), {
|
||||
method: "DELETE",
|
||||
|
||||
@@ -0,0 +1,180 @@
|
||||
import { mkdir, writeFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import type { Page } from "playwright";
|
||||
import { withObsidianPage } from "./ui.ts";
|
||||
|
||||
export const REMOTE_ACTIVITY_E2E_STATE_KEY = "__livesyncE2ERemoteActivity";
|
||||
|
||||
export type RemoteActivitySnapshot = {
|
||||
boundedRemoteActivityCount: number;
|
||||
finiteReplicationActivityCount: number;
|
||||
gateDone?: boolean;
|
||||
gateEntered?: boolean;
|
||||
gateError?: string;
|
||||
gateKind?: string;
|
||||
requestCount: number;
|
||||
responseCount: number;
|
||||
statusBarFound: boolean;
|
||||
statusBarText: string;
|
||||
};
|
||||
|
||||
export type ExpectedRemoteActivityState = "chunk-fetch-active" | "finite-replication-active" | "idle";
|
||||
|
||||
type RuntimeCounter = { value?: number };
|
||||
|
||||
type RuntimeCore = {
|
||||
services?: {
|
||||
API?: {
|
||||
requestCount?: RuntimeCounter;
|
||||
responseCount?: RuntimeCounter;
|
||||
};
|
||||
replicator?: {
|
||||
boundedRemoteActivityCount?: RuntimeCounter;
|
||||
finiteReplicationActivityCount?: RuntimeCounter;
|
||||
};
|
||||
};
|
||||
};
|
||||
|
||||
type RuntimeGate = {
|
||||
done?: boolean;
|
||||
entered?: boolean;
|
||||
error?: string;
|
||||
kind?: string;
|
||||
};
|
||||
|
||||
type RendererGlobals = typeof globalThis & {
|
||||
app?: {
|
||||
plugins?: {
|
||||
plugins?: Record<string, { core?: RuntimeCore }>;
|
||||
};
|
||||
};
|
||||
};
|
||||
|
||||
async function readRemoteActivitySnapshotFromPage(page: Page): Promise<RemoteActivitySnapshot> {
|
||||
return await page.evaluate(
|
||||
({ pluginId, stateKey }) => {
|
||||
const globals = globalThis as RendererGlobals;
|
||||
const core = globals.app?.plugins?.plugins?.[pluginId]?.core;
|
||||
if (!core) throw new Error(`Obsidian plug-in is not loaded: ${pluginId}`);
|
||||
const gate = (globalThis as unknown as Record<string, RuntimeGate | undefined>)[stateKey];
|
||||
const statusBars = Array.from(document.querySelectorAll<HTMLElement>(".syncstatusbar"));
|
||||
return {
|
||||
boundedRemoteActivityCount: Number(core.services?.replicator?.boundedRemoteActivityCount?.value ?? -1),
|
||||
finiteReplicationActivityCount: Number(
|
||||
core.services?.replicator?.finiteReplicationActivityCount?.value ?? -1
|
||||
),
|
||||
gateDone: gate?.done,
|
||||
gateEntered: gate?.entered,
|
||||
gateError: gate?.error,
|
||||
gateKind: gate?.kind,
|
||||
requestCount: Number(core.services?.API?.requestCount?.value ?? -1),
|
||||
responseCount: Number(core.services?.API?.responseCount?.value ?? -1),
|
||||
statusBarFound: statusBars.length > 0,
|
||||
statusBarText: statusBars.map((element) => element.textContent ?? "").join("\n"),
|
||||
} satisfies RemoteActivitySnapshot;
|
||||
},
|
||||
{ pluginId: "obsidian-livesync", stateKey: REMOTE_ACTIVITY_E2E_STATE_KEY }
|
||||
);
|
||||
}
|
||||
|
||||
export async function readRemoteActivitySnapshot(port: number): Promise<RemoteActivitySnapshot> {
|
||||
return await withObsidianPage(port, async (page) => await readRemoteActivitySnapshotFromPage(page));
|
||||
}
|
||||
|
||||
function formatWaitFailure(
|
||||
expected: ExpectedRemoteActivityState,
|
||||
snapshot: RemoteActivitySnapshot | undefined,
|
||||
error: unknown
|
||||
): Error {
|
||||
return new Error(
|
||||
[
|
||||
`Timed out waiting for remote activity state: ${expected}`,
|
||||
snapshot ? `Last snapshot: ${JSON.stringify(snapshot)}` : undefined,
|
||||
error instanceof Error ? error.message : String(error),
|
||||
]
|
||||
.filter((line): line is string => line !== undefined)
|
||||
.join("\n")
|
||||
);
|
||||
}
|
||||
|
||||
export async function waitForRemoteActivityState(
|
||||
port: number,
|
||||
expected: ExpectedRemoteActivityState,
|
||||
timeoutMs = Number(process.env.E2E_OBSIDIAN_REMOTE_ACTIVITY_TIMEOUT_MS ?? 30000)
|
||||
): Promise<RemoteActivitySnapshot> {
|
||||
try {
|
||||
return await withObsidianPage(port, async (page) => {
|
||||
await page.waitForFunction(
|
||||
({ expectedState, pluginId, stateKey }) => {
|
||||
const globals = globalThis as RendererGlobals;
|
||||
const core = globals.app?.plugins?.plugins?.[pluginId]?.core;
|
||||
if (!core) return false;
|
||||
const gate = (globalThis as unknown as Record<string, RuntimeGate | undefined>)[stateKey];
|
||||
const statusBarText = Array.from(document.querySelectorAll<HTMLElement>(".syncstatusbar"))
|
||||
.map((element) => element.textContent ?? "")
|
||||
.join("\n");
|
||||
const bounded = Number(core.services?.replicator?.boundedRemoteActivityCount?.value ?? -1);
|
||||
const finite = Number(core.services?.replicator?.finiteReplicationActivityCount?.value ?? -1);
|
||||
const requests = Number(core.services?.API?.requestCount?.value ?? -1);
|
||||
const responses = Number(core.services?.API?.responseCount?.value ?? -1);
|
||||
const iconVisible = statusBarText.includes("📲");
|
||||
|
||||
if (expectedState === "finite-replication-active") {
|
||||
return (
|
||||
gate?.kind === "one-shot" &&
|
||||
gate.entered === true &&
|
||||
bounded > 0 &&
|
||||
finite > 0 &&
|
||||
iconVisible
|
||||
);
|
||||
}
|
||||
if (expectedState === "chunk-fetch-active") {
|
||||
return (
|
||||
gate?.kind === "chunk-fetch" &&
|
||||
gate.entered === true &&
|
||||
bounded > 0 &&
|
||||
finite === 0 &&
|
||||
iconVisible
|
||||
);
|
||||
}
|
||||
return bounded === 0 && finite === 0 && requests === responses && !iconVisible;
|
||||
},
|
||||
{
|
||||
expectedState: expected,
|
||||
pluginId: "obsidian-livesync",
|
||||
stateKey: REMOTE_ACTIVITY_E2E_STATE_KEY,
|
||||
},
|
||||
{ timeout: timeoutMs }
|
||||
);
|
||||
return await readRemoteActivitySnapshotFromPage(page);
|
||||
});
|
||||
} catch (error) {
|
||||
const snapshot = await readRemoteActivitySnapshot(port).catch(() => undefined);
|
||||
throw formatWaitFailure(expected, snapshot, error);
|
||||
}
|
||||
}
|
||||
|
||||
export type RemoteActivityDiagnostics = {
|
||||
screenshotPath: string;
|
||||
snapshot: RemoteActivitySnapshot;
|
||||
snapshotPath: string;
|
||||
};
|
||||
|
||||
export async function captureRemoteActivityDiagnostics(
|
||||
port: number,
|
||||
label: string
|
||||
): Promise<RemoteActivityDiagnostics> {
|
||||
const outputDirectory = process.env.E2E_OBSIDIAN_DIAGNOSTICS_DIR ?? "/tmp/obsidian-livesync-e2e";
|
||||
await mkdir(outputDirectory, { recursive: true });
|
||||
const safeLabel = label.replace(/[^a-z0-9_-]+/gi, "-").replace(/^-+|-+$/g, "") || "remote-activity";
|
||||
const prefix = `${safeLabel}-${new Date().toISOString().replace(/[:.]/g, "-")}`;
|
||||
const screenshotPath = join(outputDirectory, `${prefix}.png`);
|
||||
const snapshotPath = join(outputDirectory, `${prefix}.json`);
|
||||
const snapshot = await withObsidianPage(port, async (page) => {
|
||||
const current = await readRemoteActivitySnapshotFromPage(page);
|
||||
await page.screenshot({ path: screenshotPath, fullPage: true });
|
||||
return current;
|
||||
});
|
||||
await writeFile(snapshotPath, `${JSON.stringify(snapshot, undefined, 2)}\n`, "utf8");
|
||||
return { screenshotPath, snapshot, snapshotPath };
|
||||
}
|
||||
@@ -0,0 +1,187 @@
|
||||
import { evalObsidianJson } from "./cli.ts";
|
||||
import { REMOTE_ACTIVITY_E2E_STATE_KEY } from "./remoteActivity.ts";
|
||||
|
||||
export type HeldRemoteActivityResult = {
|
||||
done: boolean;
|
||||
entered: boolean;
|
||||
error?: string;
|
||||
kind: "chunk-fetch" | "one-shot";
|
||||
requestedIds?: string[];
|
||||
result?: boolean;
|
||||
resultCount?: number;
|
||||
};
|
||||
|
||||
const stateKeySource = JSON.stringify(REMOTE_ACTIVITY_E2E_STATE_KEY);
|
||||
|
||||
export async function startHeldOneShotReplication(cliBinary: string, env: NodeJS.ProcessEnv): Promise<void> {
|
||||
await evalObsidianJson<{ started: boolean }>(
|
||||
cliBinary,
|
||||
[
|
||||
"(async()=>{",
|
||||
`const stateKey=${stateKeySource};`,
|
||||
"const host=globalThis;",
|
||||
"if(host[stateKey]) throw new Error('A remote activity E2E gate is already installed.');",
|
||||
"const core=app.plugins.plugins['obsidian-livesync'].core;",
|
||||
"const replicator=core.services.replicator.getActiveReplicator();",
|
||||
"if(!replicator) throw new Error('No active replicator is available.');",
|
||||
"const original=replicator.openReplication;",
|
||||
"let releaseGate;",
|
||||
"const gate=new Promise((resolve)=>{releaseGate=resolve;});",
|
||||
"const state={kind:'one-shot',entered:false,done:false,released:false,error:undefined,result:undefined,promise:undefined,release:undefined,restore:undefined};",
|
||||
"state.release=()=>{if(!state.released){state.released=true;releaseGate();}};",
|
||||
"state.restore=()=>{replicator.openReplication=original;};",
|
||||
"host[stateKey]=state;",
|
||||
"replicator.openReplication=async function(...args){",
|
||||
"state.entered=true;",
|
||||
"await gate;",
|
||||
"return await original.apply(this,args);",
|
||||
"};",
|
||||
"state.promise=(async()=>{",
|
||||
"try{",
|
||||
"if(!(await core.services.fileProcessing.commitPendingFileEvents())) throw new Error('Pending file events could not be committed.');",
|
||||
"state.result=!!(await core.services.replication.replicate(true));",
|
||||
"}catch(error){",
|
||||
"state.error=error instanceof Error?error.message:String(error);",
|
||||
"}finally{",
|
||||
"state.restore();",
|
||||
"state.done=true;",
|
||||
"}",
|
||||
"})();",
|
||||
"return JSON.stringify({started:true});",
|
||||
"})()",
|
||||
].join(""),
|
||||
env
|
||||
);
|
||||
}
|
||||
|
||||
export async function startHeldChunkFetch(cliBinary: string, env: NodeJS.ProcessEnv, chunkId: string): Promise<void> {
|
||||
await evalObsidianJson<{ started: boolean }>(
|
||||
cliBinary,
|
||||
[
|
||||
"(async()=>{",
|
||||
`const stateKey=${stateKeySource};`,
|
||||
`const chunkId=${JSON.stringify(chunkId)};`,
|
||||
"const host=globalThis;",
|
||||
"if(host[stateKey]) throw new Error('A remote activity E2E gate is already installed.');",
|
||||
"const core=app.plugins.plugins['obsidian-livesync'].core;",
|
||||
"const replicator=core.services.replicator.getActiveReplicator();",
|
||||
"if(!replicator) throw new Error('No active replicator is available.');",
|
||||
"const localDb=core.localDatabase.localDatabase;",
|
||||
"const existing=await localDb.get(chunkId).catch(()=>undefined);",
|
||||
"if(existing&&!existing._deleted) throw new Error(`The remote-only chunk already exists locally: ${chunkId}`);",
|
||||
"const original=replicator.fetchRemoteChunks;",
|
||||
"let releaseGate;",
|
||||
"let resolveDone;",
|
||||
"const gate=new Promise((resolve)=>{releaseGate=resolve;});",
|
||||
"const donePromise=new Promise((resolve)=>{resolveDone=resolve;});",
|
||||
"const state={kind:'chunk-fetch',entered:false,done:false,released:false,error:undefined,resultCount:undefined,requestedIds:undefined,promise:donePromise,release:undefined,restore:undefined};",
|
||||
"state.release=()=>{if(!state.released){state.released=true;releaseGate();}};",
|
||||
"state.restore=()=>{replicator.fetchRemoteChunks=original;};",
|
||||
"host[stateKey]=state;",
|
||||
"replicator.fetchRemoteChunks=async function(...args){",
|
||||
"state.entered=true;",
|
||||
"state.requestedIds=Array.isArray(args[0])?[...args[0]]:[];",
|
||||
"await gate;",
|
||||
"try{",
|
||||
"const result=await original.apply(this,args);",
|
||||
"state.resultCount=Array.isArray(result)?result.length:0;",
|
||||
"return result;",
|
||||
"}catch(error){",
|
||||
"state.error=error instanceof Error?error.message:String(error);",
|
||||
"throw error;",
|
||||
"}finally{",
|
||||
"state.restore();",
|
||||
"state.done=true;",
|
||||
"resolveDone();",
|
||||
"}",
|
||||
"};",
|
||||
"core.localDatabase.managers.chunkFetcher.onEvent([chunkId]);",
|
||||
"return JSON.stringify({started:true});",
|
||||
"})()",
|
||||
].join(""),
|
||||
env
|
||||
);
|
||||
}
|
||||
|
||||
export async function finishHeldRemoteActivity(
|
||||
cliBinary: string,
|
||||
env: NodeJS.ProcessEnv
|
||||
): Promise<HeldRemoteActivityResult> {
|
||||
return await evalObsidianJson<HeldRemoteActivityResult>(
|
||||
cliBinary,
|
||||
[
|
||||
"(async()=>{",
|
||||
`const stateKey=${stateKeySource};`,
|
||||
"const state=globalThis[stateKey];",
|
||||
"if(!state) throw new Error('No remote activity E2E gate is installed.');",
|
||||
"state.release();",
|
||||
"await state.promise;",
|
||||
"return JSON.stringify({kind:state.kind,entered:state.entered,done:state.done,error:state.error,result:state.result,resultCount:state.resultCount,requestedIds:state.requestedIds});",
|
||||
"})()",
|
||||
].join(""),
|
||||
env
|
||||
);
|
||||
}
|
||||
|
||||
export async function waitForRestoredChunk(
|
||||
cliBinary: string,
|
||||
env: NodeJS.ProcessEnv,
|
||||
chunkId: string,
|
||||
timeoutMs = Number(process.env.E2E_OBSIDIAN_REMOTE_ACTIVITY_TIMEOUT_MS ?? 30000)
|
||||
): Promise<{ id: string; type: string }> {
|
||||
return await evalObsidianJson<{ id: string; type: string }>(
|
||||
cliBinary,
|
||||
[
|
||||
"(async()=>{",
|
||||
`const chunkId=${JSON.stringify(chunkId)};`,
|
||||
`const deadline=Date.now()+${JSON.stringify(timeoutMs)};`,
|
||||
"const core=app.plugins.plugins['obsidian-livesync'].core;",
|
||||
"const localDb=core.localDatabase.localDatabase;",
|
||||
"const sleep=(ms)=>new Promise((resolve)=>setTimeout(resolve,ms));",
|
||||
"while(Date.now()<deadline){",
|
||||
"const chunk=await localDb.get(chunkId).catch(()=>undefined);",
|
||||
"if(chunk&&!chunk._deleted&&chunk.type==='leaf'&&typeof chunk.data==='string') return JSON.stringify({id:chunk._id,type:chunk.type});",
|
||||
"await sleep(100);",
|
||||
"}",
|
||||
"throw new Error(`Timed out waiting for the fetched chunk to return: ${chunkId}`);",
|
||||
"})()",
|
||||
].join(""),
|
||||
env
|
||||
);
|
||||
}
|
||||
|
||||
export async function clearHeldRemoteActivity(cliBinary: string, env: NodeJS.ProcessEnv): Promise<void> {
|
||||
await evalObsidianJson<{ cleared: boolean }>(
|
||||
cliBinary,
|
||||
[
|
||||
"(async()=>{",
|
||||
`const stateKey=${stateKeySource};`,
|
||||
"const state=globalThis[stateKey];",
|
||||
"if(!state) return JSON.stringify({cleared:false});",
|
||||
"if(!state.done) throw new Error('The remote activity E2E gate is still running.');",
|
||||
"delete globalThis[stateKey];",
|
||||
"return JSON.stringify({cleared:true});",
|
||||
"})()",
|
||||
].join(""),
|
||||
env
|
||||
);
|
||||
}
|
||||
|
||||
export async function cleanUpHeldRemoteActivity(cliBinary: string, env: NodeJS.ProcessEnv): Promise<void> {
|
||||
await evalObsidianJson<{ cleared: boolean }>(
|
||||
cliBinary,
|
||||
[
|
||||
"(async()=>{",
|
||||
`const stateKey=${stateKeySource};`,
|
||||
"const state=globalThis[stateKey];",
|
||||
"if(!state) return JSON.stringify({cleared:false});",
|
||||
"state.release?.();",
|
||||
"await Promise.race([Promise.resolve(state.promise),new Promise((resolve)=>setTimeout(resolve,5000))]);",
|
||||
"state.restore?.();",
|
||||
"delete globalThis[stateKey];",
|
||||
"return JSON.stringify({cleared:true});",
|
||||
"})()",
|
||||
].join(""),
|
||||
env
|
||||
);
|
||||
}
|
||||
@@ -5,6 +5,7 @@ import {
|
||||
deleteCouchDbDatabase,
|
||||
loadCouchDbConfig,
|
||||
makeUniqueDatabaseName,
|
||||
putCouchDbDocument,
|
||||
waitForCouchDbDocs,
|
||||
} from "../runner/couchdb.ts";
|
||||
import { discoverObsidianCli, requireObsidianBinary } from "../runner/environment.ts";
|
||||
@@ -12,10 +13,18 @@ import {
|
||||
assertEqual,
|
||||
configureCouchDb,
|
||||
prepareRemote,
|
||||
pushLocalChanges,
|
||||
waitForLiveSyncCoreReady,
|
||||
type LocalDatabaseEntry,
|
||||
} from "../runner/liveSyncWorkflow.ts";
|
||||
import { captureRemoteActivityDiagnostics, waitForRemoteActivityState } from "../runner/remoteActivity.ts";
|
||||
import {
|
||||
cleanUpHeldRemoteActivity,
|
||||
clearHeldRemoteActivity,
|
||||
finishHeldRemoteActivity,
|
||||
startHeldChunkFetch,
|
||||
startHeldOneShotReplication,
|
||||
waitForRestoredChunk,
|
||||
} from "../runner/remoteActivityWorkflow.ts";
|
||||
import { startObsidianLiveSyncSession, type ObsidianLiveSyncSession } from "../runner/session.ts";
|
||||
import { createTemporaryVault } from "../runner/vault.ts";
|
||||
|
||||
@@ -72,6 +81,7 @@ async function main(): Promise<void> {
|
||||
const dbName = makeUniqueDatabaseName(couchDb.dbPrefix, "obsidian-upload");
|
||||
const vault = await createTemporaryVault();
|
||||
let session: ObsidianLiveSyncSession | undefined;
|
||||
let activityStage = "session-startup";
|
||||
|
||||
try {
|
||||
await assertCouchDbReachable(couchDb);
|
||||
@@ -104,8 +114,25 @@ async function main(): Promise<void> {
|
||||
assertEqual(configured.syncOnSave, false, "Sync on save should remain disabled during this workflow.");
|
||||
|
||||
await prepareRemote(cli.binary, session.cliEnv);
|
||||
activityStage = "initial-idle";
|
||||
const initialIdle = await waitForRemoteActivityState(session.remoteDebuggingPort, "idle");
|
||||
const localEntry = await createNoteAndWaitForLocalDb(cli.binary, session.cliEnv);
|
||||
await pushLocalChanges(cli.binary, session.cliEnv);
|
||||
|
||||
activityStage = "one-shot-active";
|
||||
await startHeldOneShotReplication(cli.binary, session.cliEnv);
|
||||
const oneShotActive = await waitForRemoteActivityState(
|
||||
session.remoteDebuggingPort,
|
||||
"finite-replication-active"
|
||||
);
|
||||
const oneShotResult = await finishHeldRemoteActivity(cli.binary, session.cliEnv);
|
||||
assertEqual(oneShotResult.error, undefined, "One-shot replication failed while its activity was observed.");
|
||||
assertEqual(oneShotResult.result, true, "One-shot replication did not report success.");
|
||||
activityStage = "one-shot-idle";
|
||||
const oneShotIdle = await waitForRemoteActivityState(session.remoteDebuggingPort, "idle");
|
||||
if (oneShotIdle.requestCount <= initialIdle.requestCount) {
|
||||
throw new Error("One-shot replication did not make an observed remote request.");
|
||||
}
|
||||
await clearHeldRemoteActivity(cli.binary, session.cliEnv);
|
||||
|
||||
const remoteDocs = await waitForCouchDbDocs(couchDb, dbName, (docs) => {
|
||||
const ids = new Set(docs.map((doc) => doc._id));
|
||||
@@ -118,11 +145,73 @@ async function main(): Promise<void> {
|
||||
"Remote metadata path did not match the local database entry."
|
||||
);
|
||||
|
||||
const sourceChunkId = localEntry.children[0];
|
||||
if (!sourceChunkId) throw new Error("The uploaded note did not produce a chunk for the fetch workflow.");
|
||||
const sourceChunk = remoteDocs.find((document) => document._id === sourceChunkId);
|
||||
if (!sourceChunk || sourceChunk.type !== "leaf") {
|
||||
throw new Error(`The uploaded source chunk was not found in CouchDB: ${sourceChunkId}`);
|
||||
}
|
||||
const { _rev: _sourceRevision, ...remoteOnlyChunk } = sourceChunk;
|
||||
const chunkId = `h:e2e-remote-activity-${Date.now().toString(36)}`;
|
||||
await putCouchDbDocument(couchDb, dbName, { ...remoteOnlyChunk, _id: chunkId });
|
||||
activityStage = "chunk-fetch-active";
|
||||
await startHeldChunkFetch(cli.binary, session.cliEnv, chunkId);
|
||||
const chunkFetchActive = await waitForRemoteActivityState(session.remoteDebuggingPort, "chunk-fetch-active");
|
||||
const chunkFetchResult = await finishHeldRemoteActivity(cli.binary, session.cliEnv);
|
||||
assertEqual(
|
||||
chunkFetchResult.error,
|
||||
undefined,
|
||||
"On-demand chunk fetching failed while its activity was observed."
|
||||
);
|
||||
if (!chunkFetchResult.requestedIds?.includes(chunkId)) {
|
||||
throw new Error(`The on-demand chunk request did not include the selected chunk: ${chunkId}`);
|
||||
}
|
||||
if ((chunkFetchResult.resultCount ?? 0) < 1) {
|
||||
throw new Error(`The remote did not return the selected chunk: ${chunkId}`);
|
||||
}
|
||||
const restoredChunk = await waitForRestoredChunk(cli.binary, session.cliEnv, chunkId);
|
||||
assertEqual(restoredChunk.id, chunkId, "The restored chunk ID did not match the requested chunk.");
|
||||
activityStage = "chunk-fetch-idle";
|
||||
const chunkFetchIdle = await waitForRemoteActivityState(session.remoteDebuggingPort, "idle");
|
||||
if (chunkFetchIdle.requestCount <= oneShotIdle.requestCount) {
|
||||
throw new Error("On-demand chunk fetching did not make an observed remote request.");
|
||||
}
|
||||
await clearHeldRemoteActivity(cli.binary, session.cliEnv);
|
||||
|
||||
console.log(
|
||||
`Uploaded metadata ${localEntry.id} and ${localEntry.children.length} chunk(s) to CouchDB database ${dbName}`
|
||||
);
|
||||
console.log(
|
||||
[
|
||||
`One-shot activity: ${oneShotActive.statusBarText.trim()} -> idle`,
|
||||
`Chunk-fetch activity: ${chunkFetchActive.statusBarText.trim()} -> idle`,
|
||||
`Balanced remote requests: ${chunkFetchIdle.requestCount}/${chunkFetchIdle.responseCount}`,
|
||||
].join("\n")
|
||||
);
|
||||
} catch (error) {
|
||||
if (session) {
|
||||
const diagnostics = await captureRemoteActivityDiagnostics(
|
||||
session.remoteDebuggingPort,
|
||||
`couchdb-upload-${activityStage}`
|
||||
).catch((diagnosticError: unknown) => {
|
||||
console.warn(
|
||||
`Could not capture remote activity diagnostics: ${
|
||||
diagnosticError instanceof Error ? diagnosticError.message : String(diagnosticError)
|
||||
}`
|
||||
);
|
||||
return undefined;
|
||||
});
|
||||
if (diagnostics) {
|
||||
console.error(`Remote activity screenshot: ${diagnostics.screenshotPath}`);
|
||||
console.error(`Remote activity snapshot: ${diagnostics.snapshotPath}`);
|
||||
}
|
||||
}
|
||||
throw error;
|
||||
} finally {
|
||||
if (session) {
|
||||
await cleanUpHeldRemoteActivity(cli.binary, session.cliEnv).catch((error: unknown) => {
|
||||
console.warn(error instanceof Error ? error.message : error);
|
||||
});
|
||||
await session.app.stop();
|
||||
}
|
||||
await vault.dispose();
|
||||
|
||||
@@ -5,6 +5,11 @@ The head note of 0.25 is now in [updates_old.md](https://github.com/vrtmrz/obsid
|
||||
|
||||
## Unreleased
|
||||
|
||||
### Fixed
|
||||
|
||||
- Fixed the 📲 remote-activity indicator remaining visible after CouchDB requests had completed.
|
||||
- Fixed missing chunks being reported unavailable while an in-progress on-demand fetch or finite replication could still deliver them. Reads now follow the actual delivery lifecycle and recheck the local database when it finishes.
|
||||
|
||||
## 0.25.82
|
||||
|
||||
15th July, 2026
|
||||
|
||||
Reference in New Issue
Block a user