mirror of
https://github.com/vrtmrz/obsidian-livesync.git
synced 2026-08-08 20:55:46 +00:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
9765569bb6 | ||
|
|
c5835a9da6 | ||
|
|
8311060f77 | ||
|
|
13d9624407 | ||
|
|
76560e3bf2 | ||
|
|
1e190d042c | ||
|
|
40215032dd | ||
|
|
fd9a9175dd | ||
|
|
1dfdb72fbd | ||
|
|
23d9fa360d | ||
|
|
070b63c952 | ||
|
|
d37af53858 | ||
|
|
b05309ad9c | ||
|
|
92e01a17e7 | ||
|
|
b3030bba53 | ||
|
|
cf5181bb28 | ||
|
|
002cf57116 | ||
|
|
15abe344bb | ||
|
|
a9e64860d5 |
+2
-1
@@ -33,7 +33,8 @@
|
||||
const id = params.get('id');
|
||||
const total = parseInt(params.get('n') || '0');
|
||||
const index = parseInt(params.get('i') || '-1');
|
||||
const data = params.get('d');
|
||||
// Keep the chunk percent-encoded so URI delimiters remain part of the settings payload.
|
||||
const data = hash.match(/(?:^|&)d=([^&]*)/)?.[1];
|
||||
|
||||
const app = document.getElementById('app');
|
||||
|
||||
|
||||
@@ -0,0 +1,351 @@
|
||||
# Architectural Decision Record: Fast Fetch Persistence and Completion Semantics
|
||||
|
||||
## Status
|
||||
|
||||
Accepted
|
||||
|
||||
## Context
|
||||
|
||||
Fast Fetch accelerates Fast Setup (Simple Fetch) by reading CouchDB's continuous
|
||||
changes feed directly, decrypting each document, and writing batches to the local
|
||||
database. It then allows LiveSync to reflect the completed database into the
|
||||
Vault.
|
||||
|
||||
This path deliberately bypasses PouchDB's ordinary replication machinery. It
|
||||
must therefore reproduce the correctness guarantees on which the rest of the
|
||||
initialisation workflow relies:
|
||||
|
||||
- a remote document is decrypted and validated before it is written locally;
|
||||
- every result from a batch write is checked;
|
||||
- a checkpoint represents the last contiguous remote sequence which is durable
|
||||
in the local database; and
|
||||
- successful completion requires CouchDB to terminate every finite changes page,
|
||||
every returned row to be durable, and a subsequent normal probe to report no
|
||||
available rows.
|
||||
|
||||
The existing implementation combines line parsing, decryption, persistence, and
|
||||
completion checks within one broad error handler. A decryption or persistence
|
||||
failure can consequently be reported as a malformed JSON line and skipped. A
|
||||
batch write can also resolve while containing individual failed results. In both
|
||||
cases the stream may continue, advance its checkpoint incorrectly, or wait
|
||||
indefinitely for a completion condition which the failed row would have
|
||||
satisfied.
|
||||
|
||||
Fast Fetch also reads the normal changes feed before opening each continuous
|
||||
page. A normal response's `pending` value counts items which remain after the
|
||||
response's `results`, so `pending` alone is not the available workload. With a
|
||||
one-row probe, the page can contain `results.length + pending` rows.
|
||||
|
||||
The documented `limit=0` behaviour cannot be used as a portable zero-payload
|
||||
probe. CouchDB's API documentation says that `limit=0` has the same effect as
|
||||
`limit=1`, while CouchDB 3.5.0 with a two-shard database was observed to return
|
||||
no result rows and leave the complete count in `pending`. Fast Fetch therefore
|
||||
uses an explicit one-row normal probe and includes no document bodies.
|
||||
|
||||
Finite continuous-feed completion differs across supported CouchDB releases.
|
||||
CouchDB 3.5.0 was observed to close a heartbeat-enabled feed with a
|
||||
`{ "last_seq": ... }` line when its finite `limit` is met. CouchDB 3.2 instead
|
||||
continues to wait for database updates after the limit has been consumed. With
|
||||
a heartbeat configured, each wait emits another heartbeat and the page can
|
||||
remain open indefinitely, even after every requested row has arrived.
|
||||
|
||||
On CouchDB 3.2, an explicit `timeout` without a heartbeat has different
|
||||
semantics from a total request deadline. Shard-result waits may emit blank
|
||||
keep-alive lines and continue processing. Once the currently available changes
|
||||
have been exhausted and the feed is waiting for another database update, the
|
||||
timeout stops that wait and returns the feed-level `last_seq`. The timeout can
|
||||
therefore terminate a finite page without limiting the duration of an active
|
||||
page transfer.
|
||||
|
||||
The CouchDB sequence token is opaque and must be handled using CouchDB's
|
||||
sequence semantics, without parsing, ordering, or comparison. On clustered
|
||||
CouchDB, a changes row and the feed-level `last_seq` may encode related
|
||||
positions with different opaque tokens. Separate requests are also not one
|
||||
locked snapshot: their rows may be partially ordered, and replica failover may
|
||||
repeat changes. Fast Fetch must therefore be idempotent and must use each
|
||||
terminal `last_seq` only by returning it to CouchDB as the next `since` value.
|
||||
|
||||
## Decision
|
||||
|
||||
### Remote page sizing and completion
|
||||
|
||||
Fast Fetch obtains an approximate progress target from a normal changes-feed
|
||||
request with `since=now`, `limit=1`, and `include_docs=false`. This token is for
|
||||
progress reporting only. It is not compared with any other token and is not
|
||||
used as a completion checkpoint.
|
||||
|
||||
Before every bounded page, Fast Fetch requests a normal changes feed from the
|
||||
current durable cursor with `limit=1` and `include_docs=false`. The probe and
|
||||
the following continuous page use the same `since`, style, and filter
|
||||
selection. Reading the probe does not consume rows from CouchDB; the continuous
|
||||
request starts again from that same cursor.
|
||||
|
||||
The number currently available is `results.length + pending`. If it is zero,
|
||||
Fast Fetch is caught up and completes without opening another stream. Otherwise,
|
||||
the next continuous request uses the smaller of that count and 10,000 as its
|
||||
finite `limit`.
|
||||
|
||||
Each finite page omits `heartbeat` and sets `timeout=1000`. This lets CouchDB
|
||||
3.2 return the page's terminator one second after it exhausts the currently
|
||||
available changes, rather than keeping the request open for future writes. The
|
||||
client immediately reconnects from that terminator while another normal probe
|
||||
reports available work. This bounded cycle also preserves the intent of the
|
||||
earlier iOS and iPadOS heartbeat workaround: Fast Fetch no longer depends on a
|
||||
silent continuous request eventually closing at CouchDB's default 60-second
|
||||
timeout.
|
||||
|
||||
The probe and page are separate HTTP requests, not a transactional snapshot.
|
||||
New writes, replica selection, or administrative changes may alter the rows
|
||||
between them. A page which returns at least one row and a valid terminator may
|
||||
therefore be shorter than the probe's estimate. Fast Fetch persists that page
|
||||
and probes again. A page which terminates without making progress after a
|
||||
positive probe is a retryable transport failure, avoiding an unbounded busy
|
||||
loop.
|
||||
|
||||
Each continuous request ends with its own `{ "last_seq": ... }` line. Fast
|
||||
Fetch treats that line separately from a changes row, flushes and validates all
|
||||
preceding local writes, and only then persists the opaque `last_seq`. The exact
|
||||
value is replayed as the next request's `since`; it is never parsed, ordered, or
|
||||
compared with a row's `seq`, another request's `last_seq`, or the database's
|
||||
`update_seq`.
|
||||
|
||||
The limit counts outer changes-result rows. A tombstone is one row and consumes
|
||||
one page slot even when no document body is present. With `style=all_docs`,
|
||||
multiple leaf revisions inside one row's `changes` array do not consume
|
||||
additional slots. Changing `include_docs` between the lightweight probe and the
|
||||
document-bearing continuous page changes the payload, not the row selection.
|
||||
|
||||
### Processing and persistence
|
||||
|
||||
Each non-blank line from the changes feed is processed through these ordered
|
||||
stages:
|
||||
|
||||
1. parse and validate the changes-feed row;
|
||||
2. decrypt and validate its document, when a document is present;
|
||||
3. add the document to the pending local batch;
|
||||
4. persist the batch;
|
||||
5. inspect every result returned by the batch write; and
|
||||
6. after the finite page ends, persist its `last_seq` terminator.
|
||||
|
||||
With `new_edits: false`, PouchDB follows CouchDB behaviour and may omit successful
|
||||
results. Fast Fetch therefore inspects every returned result and treats any
|
||||
error result as a failed batch. An empty result is valid when PouchDB accepted
|
||||
the complete batch.
|
||||
|
||||
The checkpoint may advance only to the last contiguous sequence for which all
|
||||
preceding documents are durable. A row which legitimately requires no local
|
||||
write may advance the checkpoint only after any preceding buffered documents
|
||||
have been flushed successfully. A page's `last_seq` is committed under the same
|
||||
rule before that page reports success.
|
||||
|
||||
If a batch is partly written, its checkpoint is not advanced. Retrying the batch
|
||||
with `new_edits: false` is expected to be idempotent, including for documents
|
||||
which were accepted during the first attempt.
|
||||
|
||||
Blank heartbeat lines are ignored. Malformed rows are failures; they are not
|
||||
silently skipped. Logs may describe the stage and sequence involved, but must
|
||||
not include the raw changes-feed line because it may be large or sensitive.
|
||||
|
||||
Each finite continuous changes request and its decoded reader must be terminated
|
||||
on every exit. Releasing a reader lock alone does not cancel the underlying
|
||||
request. Failure and completion paths therefore abort the request and attempt
|
||||
to cancel the reader before the bounded remote-activity scope ends.
|
||||
|
||||
### Failure classification and retry
|
||||
|
||||
The streaming boundary returns a small structured failure with a stage and an
|
||||
explicit retryability decision. The initial stages are:
|
||||
|
||||
```typescript
|
||||
type StreamingFetchFailureStage = "transport" | "authentication" | "protocol" | "decryption" | "storage";
|
||||
```
|
||||
|
||||
This type is an internal behavioural contract, not a user-interface status
|
||||
enumeration. It may carry safe diagnostic context, such as an HTTP status or a
|
||||
sequence token, without carrying document content.
|
||||
|
||||
Only explicitly recognised transient transport failures are retried
|
||||
automatically. Examples include an interrupted connection, HTTP 408, HTTP 429,
|
||||
and selected HTTP 5xx responses. Authentication, protocol, decryption, and local
|
||||
storage failures are terminal by default. A future implementation may recognise
|
||||
a narrower retryable case, but it must do so explicitly rather than retry every
|
||||
exception.
|
||||
|
||||
Each retry resumes from the last durable contiguous checkpoint. Retry exhaustion
|
||||
returns an actionable classified failure to the caller.
|
||||
|
||||
### Initialisation lifecycle
|
||||
|
||||
Fast Fetch success is the only path which may continue with the offline scan,
|
||||
finish the rebuild, resume Vault reflection, clear the Fast Fetch checkpoint,
|
||||
remove the flag file, or forget the remembered initialisation choice.
|
||||
|
||||
The LiveSync initialisation boundary uses an explicit suspension policy.
|
||||
Ordinary Fetch and Rebuild resume Vault reflection when they finish, SCRAM keeps
|
||||
file watching suspended, and Fast Fetch keeps both file watching and replication
|
||||
result parsing suspended only when initialisation fails. Fast Fetch asserts both
|
||||
suspensions before it begins and owns their final state: success clears both,
|
||||
whereas a false result or exception sets both. This final assignment also covers
|
||||
a late failure after rebuild finalisation and the legacy
|
||||
`doNotSuspendOnFetching` path.
|
||||
|
||||
On failure:
|
||||
|
||||
- the local checkpoint and any durably fetched documents are retained for a
|
||||
later retry;
|
||||
- the local database is not marked as resolved;
|
||||
- Vault reflection remains suspended;
|
||||
- the offline scan and rebuild finalisation are not run; and
|
||||
- the flag file and remembered initialisation choice remain available so that
|
||||
restart recovery can offer the same operation again.
|
||||
|
||||
Any bounded remote-activity scope must still be released in a `finally` path, as
|
||||
defined by [Bounded Remote Activity](2026_07_bounded_remote_activity.md). Keeping
|
||||
Vault reflection suspended does not permit a wake lock or similar resource to
|
||||
leak.
|
||||
|
||||
## Ownership
|
||||
|
||||
The responsibilities are divided at three injectable boundaries.
|
||||
|
||||
### Streaming Fetch
|
||||
|
||||
The Commonlib streaming implementation owns HTTP response validation, NDJSON
|
||||
parsing, invocation of the decryption delegate, batch-write result validation,
|
||||
contiguous checkpoint advancement, finite-page completion, and classified
|
||||
failures. It does not know about the Vault, setup dialogues, flag files, or
|
||||
LiveSync settings.
|
||||
|
||||
### Rebuilder
|
||||
|
||||
The Commonlib rebuilder owns the local database lifecycle, checkpoint storage,
|
||||
retry policy, marking a completed database as resolved, optional resumption of
|
||||
reflection, and final checkpoint removal. It does not parse changes-feed rows or
|
||||
interpret user-interface choices.
|
||||
|
||||
### LiveSync Fast Setup
|
||||
|
||||
LiveSync owns the setup choices, suspension of initial Vault reflection,
|
||||
invocation of Fast Fetch, the success-only offline scan and rebuild finalisation,
|
||||
and cleanup of flag files and remembered choices. It does not reinterpret
|
||||
document, encryption, or storage failures as successful setup.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
This decision does not:
|
||||
|
||||
- change the Metadata and Chunks formats, encryption scheme, security seed, or
|
||||
path obfuscation;
|
||||
- change ordinary PouchDB replication, Standard Fetch, or offline-scan
|
||||
semantics;
|
||||
- provide a transaction spanning CouchDB and the local database;
|
||||
- skip corrupt or unreadable documents and continue with a partial database;
|
||||
- add an automatic fallback from Fast Fetch to Standard Fetch;
|
||||
- define the detailed failure dialogue or other setup user-interface changes;
|
||||
or
|
||||
- provide a transaction or locked snapshot across the normal probe and the
|
||||
following continuous page.
|
||||
|
||||
An explicit Standard Fetch choice remains available when a user needs the
|
||||
ordinary replication path. Any automatic fallback or richer recovery dialogue
|
||||
requires a separate decision because it changes user-visible setup behaviour.
|
||||
|
||||
## Verification
|
||||
|
||||
The implementation is verified primarily with London School interaction tests,
|
||||
using mocks at each owned boundary to prove collaboration and call order.
|
||||
|
||||
### Streaming Fetch unit tests
|
||||
|
||||
Inject the HTTP stream, decryption delegate, local batch writer, and checkpoint
|
||||
writer. Verify that:
|
||||
|
||||
- the order is decrypt, persist, inspect results, then checkpoint;
|
||||
- parsing, decryption, and batch-result failures prevent checkpoint advancement
|
||||
and completion;
|
||||
- a partly failed batch leaves the checkpoint unchanged and reports a storage
|
||||
failure;
|
||||
- rows without a local write flush earlier buffered documents before advancing;
|
||||
- a returned probe row is counted in addition to `pending`, including when
|
||||
`pending` is zero;
|
||||
- every probe uses `limit=1`, excludes document bodies, and is repeated from the
|
||||
previous page's opaque terminator;
|
||||
- each bounded page omits `heartbeat`, uses `timeout=1000`, and can complete
|
||||
under CouchDB 3.2 after its current rows have been delivered;
|
||||
- deletion and document-less rows consume a page slot;
|
||||
- a row count cannot complete a page without its `last_seq` terminator;
|
||||
- a page terminator cannot advance the checkpoint before its batch is durable;
|
||||
- a final row and `last_seq` with different opaque representations complete
|
||||
normally without a token comparison;
|
||||
- a shorter valid page is persisted and followed by another probe, while a
|
||||
zero-row page after a positive probe fails without looping;
|
||||
- workloads over 10,000 rows resume from each durable finite-page checkpoint;
|
||||
- authentication and malformed-protocol responses are terminal;
|
||||
- recognised transient transport failures are classified as retryable; and
|
||||
- diagnostics do not log the raw changes-feed line.
|
||||
|
||||
### Rebuilder unit tests
|
||||
|
||||
Inject the streaming operation and lifecycle collaborators. Verify that:
|
||||
|
||||
- transient failures retry from the latest durable checkpoint;
|
||||
- terminal failures are attempted once;
|
||||
- success marks the database as resolved, resumes reflection when requested, and
|
||||
clears the checkpoint in that order; and
|
||||
- failure does not mark the database as resolved, resume reflection, or clear
|
||||
the checkpoint.
|
||||
|
||||
### LiveSync orchestration unit tests
|
||||
|
||||
Inject Fast Fetch, the offline scanner, rebuild finalisation, and cleanup
|
||||
collaborators. Verify that failure performs none of the success-only actions and
|
||||
leaves initial Vault reflection suspended. Verify that success retains the
|
||||
existing setup sequence and cleanup.
|
||||
|
||||
### Integration and E2E tests
|
||||
|
||||
Commonlib's CouchDB integration test remains responsible for the real HTTP
|
||||
changes feed, opaque sequence tokens, deletion rows, and local batch
|
||||
persistence. It should use the maintained CI CouchDB release, a two-shard
|
||||
database, and a data set large enough to cross a local batch boundary, and
|
||||
confirm that the final checkpoint can be passed back to CouchDB as `since` with
|
||||
no result rows or pending changes. The test must not compare that token's
|
||||
representation with a separately requested target or changes-row token. The
|
||||
focused regression test covers CouchDB 3.2's page-tail behaviour; compatibility
|
||||
with a real CouchDB 3.2 server can be confirmed manually without expanding the
|
||||
permanent CI matrix.
|
||||
|
||||
LiveSync's real Obsidian Setup URI workflow remains responsible for the actual
|
||||
Fast Fetch selection, E2EE passphrase, Vault reflection, ordinary file round
|
||||
trip, and hidden-file synchronisation. Injected parsing, decryption, and
|
||||
persistence failures remain unit-test responsibilities; repeating them through
|
||||
the real Obsidian E2E does not add coverage for an unchanged framework boundary.
|
||||
This follows [Real Obsidian E2E](2026_06_real_obsidian_e2e.md).
|
||||
|
||||
## Consequences
|
||||
|
||||
- A deterministic document failure which previously appeared to be skipped now
|
||||
fails Fast Fetch. This is an intentional safety improvement because the local
|
||||
database is known to be incomplete.
|
||||
- Partial durable work and its contiguous checkpoint can be reused by a later
|
||||
attempt without exposing the partial database to the Vault.
|
||||
- Retry delays are no longer spent on authentication, corrupt content, protocol,
|
||||
or local persistence failures which cannot repair themselves.
|
||||
- Progress totals remain approximate and may grow when a later probe observes
|
||||
new work, without affecting correctness.
|
||||
- A completed page can spend up to one second waiting for its terminator before
|
||||
Fast Fetch probes and reconnects. Active page transfer is not constrained to
|
||||
one second.
|
||||
- The implementation requires coordinated changes in Commonlib and LiveSync.
|
||||
Commonlib remains the authoritative package for streaming and rebuilder
|
||||
behaviour; LiveSync consumes an immutable Commonlib release and owns its setup
|
||||
orchestration.
|
||||
- Ordinary replication remains unchanged and continues to provide the reference
|
||||
correctness contract for decrypting, persisting, and checkpointing replicated
|
||||
documents.
|
||||
|
||||
## References
|
||||
|
||||
- [Apache CouchDB changes-feed API](https://docs.couchdb.org/en/stable/api/database/changes.html)
|
||||
- [Apache CouchDB 2.0 upgrade notes for opaque update sequences](https://docs.couchdb.org/en/stable/whatsnew/2.0.html#upgrade-notes)
|
||||
- [Apache CouchDB replication protocol](https://docs.couchdb.org/en/stable/replication/protocol.html)
|
||||
+3
-1
@@ -68,6 +68,8 @@ On the next start, LiveSync:
|
||||
4. discards and reconstructs the local LiveSync database from the selected remote; and
|
||||
5. resumes only after the scheduled operation has completed or been cancelled safely.
|
||||
|
||||
Fast Setup retains its fetch flag, last successfully stored remote position, and selected data-processing method when an error while decrypting data, reading the remote response, or writing to the local database stops the reconstruction. File watching and database reflection remain suspended. Review the first specific error in **Show log**, correct its cause, then restart Obsidian to retry from the retained state. Do not remove the fetch flag or resume the Scram switches while you intend to continue the operation. If the same error remains, leave LiveSync suspended and [collect a report](troubleshooting.md#collect-a-report).
|
||||
|
||||
For P2P, a source peer must be online, discovered, and selected in `P2P Rebuild`. Merely opening an empty signalling room does not complete Fetch. Closing the rebuild dialogue without selecting a peer reports failure and does not treat the local database as restored.
|
||||
|
||||
Review the [Fast Setup guide](tips/fast-setup.md) before using this operation on a Vault which contains unsynchronised local work.
|
||||
@@ -105,7 +107,7 @@ Create only the flag required for the chosen operation.
|
||||
| `flag_fetch.md` or `redflag3.md` | Schedule **Reset Synchronisation on This Device** from the selected remote. |
|
||||
| `flag_rebuild.md` or `redflag2.md` | Schedule **Overwrite Server Data with This Device's Files**, or local P2P preparation when no central remote exists. |
|
||||
|
||||
Flag files themselves are excluded from synchronisation. Fetch and rebuild flags are removed by the scheduled workflow after completion or cancellation; `redflag.md` is a manual emergency stop.
|
||||
Flag files themselves are excluded from synchronisation. Fetch and rebuild flags are removed by the scheduled workflow after completion or safe cancellation. A failed Fast Setup retains its fetch flag so that a later start can retry it; `redflag.md` is a manual emergency stop.
|
||||
|
||||
## When the warning continues
|
||||
|
||||
|
||||
@@ -63,3 +63,11 @@ Once you confirm your choices:
|
||||
1. The plug-in performs a fast download of the remote database (`fetchLocalDBFast`).
|
||||
2. It automatically runs a full scan (`synchroniseAllFilesBetweenDBandStorage`) in the foreground to reflect database changes in your local vault files immediately.
|
||||
3. The plug-in finalises the process and resumes normal operational status.
|
||||
|
||||
### If Fast Setup Stops
|
||||
|
||||
Fast Setup records the last successfully stored remote position as it saves documents. A transient connection interruption is retried automatically from that position. The operation reports completion only after the captured remote state has been stored successfully.
|
||||
|
||||
If an error while decrypting data, reading the remote response, or writing to the local database stops the operation, LiveSync does not run the Vault scan or finalise the reconstructed local database. It retains the saved position, the selected data-processing method, and the fetch flag. File watching and database reflection also remain suspended so that a partly reconstructed database cannot be applied to the Vault or combined with new local changes.
|
||||
|
||||
Review the first specific error in **Show log**, correct its cause, then restart Obsidian to retry from the retained state. While you intend to continue the operation, do not remove the fetch flag or manually resume the Scram switches. If the same error remains after a restart, leave LiveSync suspended, preserve the available data, and [collect a report](../troubleshooting.md#collect-a-report).
|
||||
|
||||
+1
-1
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"id": "obsidian-livesync",
|
||||
"name": "Self-hosted LiveSync",
|
||||
"version": "1.0.6",
|
||||
"version": "1.0.9",
|
||||
"minAppVersion": "1.7.2",
|
||||
"description": "Community implementation of self-hosted livesync. Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.",
|
||||
"author": "vorotamoroz",
|
||||
|
||||
Generated
+9
-9
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"name": "obsidian-livesync",
|
||||
"version": "1.0.6",
|
||||
"version": "1.0.9",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "obsidian-livesync",
|
||||
"version": "1.0.6",
|
||||
"version": "1.0.9",
|
||||
"license": "MIT",
|
||||
"workspaces": [
|
||||
"src/apps/cli",
|
||||
@@ -23,7 +23,7 @@
|
||||
"@smithy/types": "^4.14.3",
|
||||
"@smithy/util-retry": "^4.4.5",
|
||||
"@vrtmrz/browser-ui-kit": "0.1.0",
|
||||
"@vrtmrz/livesync-commonlib": "0.1.5",
|
||||
"@vrtmrz/livesync-commonlib": "0.1.8",
|
||||
"@vrtmrz/obsidian-plugin-kit": "0.1.3",
|
||||
"@vrtmrz/ui-interactions": "0.1.2",
|
||||
"diff-match-patch": "^1.0.5",
|
||||
@@ -4775,9 +4775,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/@vrtmrz/livesync-commonlib": {
|
||||
"version": "0.1.5",
|
||||
"resolved": "https://registry.npmjs.org/@vrtmrz/livesync-commonlib/-/livesync-commonlib-0.1.5.tgz",
|
||||
"integrity": "sha512-DJBzVWevZ/8ZLTPmweMxRZAvNU9Aaxz6Ld3Def2uioBv6grUmpebey7+yT93ddR6v3aRiva2+yNf96xq4qttQA==",
|
||||
"version": "0.1.8",
|
||||
"resolved": "https://registry.npmjs.org/@vrtmrz/livesync-commonlib/-/livesync-commonlib-0.1.8.tgz",
|
||||
"integrity": "sha512-Kn1AF41h2Dog37ThU7KgLcKxItCCerLEBWg1eSGAUoTk3TyPBYynvtmVFwWhy6LCePcuwB/+x7EQNTL3Sgzkyg==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@aws-sdk/client-s3": "^3.808.0",
|
||||
@@ -15924,7 +15924,7 @@
|
||||
},
|
||||
"src/apps/cli": {
|
||||
"name": "self-hosted-livesync-cli",
|
||||
"version": "1.0.6-cli",
|
||||
"version": "1.0.9-cli",
|
||||
"dependencies": {
|
||||
"chokidar": "^4.0.0",
|
||||
"minimatch": "^10.2.5",
|
||||
@@ -15949,7 +15949,7 @@
|
||||
},
|
||||
"src/apps/webapp": {
|
||||
"name": "livesync-webapp",
|
||||
"version": "1.0.6-webapp",
|
||||
"version": "1.0.9-webapp",
|
||||
"dependencies": {
|
||||
"octagonal-wheels": "^0.1.52"
|
||||
},
|
||||
@@ -15961,7 +15961,7 @@
|
||||
}
|
||||
},
|
||||
"src/apps/webpeer": {
|
||||
"version": "1.0.6-webpeer",
|
||||
"version": "1.0.9-webpeer",
|
||||
"dependencies": {
|
||||
"octagonal-wheels": "^0.1.52"
|
||||
},
|
||||
|
||||
+2
-2
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "obsidian-livesync",
|
||||
"version": "1.0.6",
|
||||
"version": "1.0.9",
|
||||
"description": "Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.",
|
||||
"main": "main.js",
|
||||
"type": "module",
|
||||
@@ -177,7 +177,7 @@
|
||||
"@smithy/types": "^4.14.3",
|
||||
"@smithy/util-retry": "^4.4.5",
|
||||
"@vrtmrz/browser-ui-kit": "0.1.0",
|
||||
"@vrtmrz/livesync-commonlib": "0.1.5",
|
||||
"@vrtmrz/livesync-commonlib": "0.1.8",
|
||||
"@vrtmrz/obsidian-plugin-kit": "0.1.3",
|
||||
"@vrtmrz/ui-interactions": "0.1.2",
|
||||
"diff-match-patch": "^1.0.5",
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"name": "self-hosted-livesync-cli",
|
||||
"private": true,
|
||||
"version": "1.0.6-cli",
|
||||
"version": "1.0.9-cli",
|
||||
"main": "dist/index.cjs",
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"name": "livesync-webapp",
|
||||
"private": true,
|
||||
"version": "1.0.6-webapp",
|
||||
"version": "1.0.9-webapp",
|
||||
"type": "module",
|
||||
"description": "Browser-based Self-hosted LiveSync using FileSystem API",
|
||||
"scripts": {
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"name": "webpeer",
|
||||
"private": true,
|
||||
"version": "1.0.6-webpeer",
|
||||
"version": "1.0.9-webpeer",
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "vite",
|
||||
|
||||
@@ -10,11 +10,7 @@ import {
|
||||
synchroniseAllFilesBetweenDBandStorage,
|
||||
type FullScanOptions,
|
||||
} from "@vrtmrz/livesync-commonlib/compat/serviceFeatures/offlineScanner";
|
||||
import {
|
||||
adjustSettingToRemoteIfNeeded,
|
||||
cancelScheduledInitialisation,
|
||||
processVaultInitialisation,
|
||||
} from "./redFlag";
|
||||
import { adjustSettingToRemoteIfNeeded, cancelScheduledInitialisation, processVaultInitialisation } from "./redFlag";
|
||||
|
||||
export const SIMPLE_FETCH_STAGE1_REMOTE_WINS = "Overwrite all with remote files";
|
||||
export const SIMPLE_FETCH_STAGE1_NEWER_WINS = "Compare time and take newer";
|
||||
@@ -217,7 +213,7 @@ export async function askAndPerformFastSetupOnScheduledFetchAll(
|
||||
return await cancelScheduledInitialisation(host, cleanupFlag);
|
||||
}
|
||||
|
||||
return await processVaultInitialisation(host, log, async () => {
|
||||
const performFastSetup = async () => {
|
||||
// 1. Perform fast DB fetch (download remote DB content to local DB)
|
||||
await host.serviceModules.rebuilder.$fetchLocalDBFast(false);
|
||||
|
||||
@@ -253,5 +249,6 @@ export async function askAndPerformFastSetupOnScheduledFetchAll(
|
||||
clearRememberedSimpleFetchMode(host);
|
||||
log("Simple fetch and scan operation completed.", LOG_LEVEL_NOTICE);
|
||||
return true;
|
||||
});
|
||||
};
|
||||
return await processVaultInitialisation(host, log, performFastSetup, "keep-on-failure");
|
||||
}
|
||||
|
||||
@@ -150,7 +150,7 @@ export function createFetchAllFlagHandler(
|
||||
// Select the remote database if there are multiple remotes configured.
|
||||
const isRemoteActivated = await askAndActivateRemoteDatabase(host, log);
|
||||
if (!isRemoteActivated) {
|
||||
return false;
|
||||
return await cancelScheduledInitialisation(host, cleanupFlag);
|
||||
}
|
||||
|
||||
// Ask user for use Fast Setup
|
||||
@@ -372,26 +372,35 @@ export async function cancelScheduledInitialisation(
|
||||
return false;
|
||||
}
|
||||
|
||||
type InitialisationSuspensionPolicy = "resume" | "keep" | "keep-on-failure";
|
||||
|
||||
/**
|
||||
* Process vault initialisation with suspending file watching and sync.
|
||||
* @param proc process to be executed during initialisation, should return true if can be continued, false if app is unable to continue the process.
|
||||
* @param keepSuspending whether to keep suspending file watching after the process.
|
||||
* @returns result of the process, or false if error occurs.
|
||||
* Process Vault initialisation with file watching and synchronisation suspended.
|
||||
* @param proc Process to execute during initialisation. It returns true only when normal operation may resume.
|
||||
* @param suspensionPolicy Final file-reflection state. `keep-on-failure` controls both reflection directions so a partly completed Fast Setup remains isolated.
|
||||
* @returns The result of the process, or false if an error occurs.
|
||||
*/
|
||||
export async function processVaultInitialisation(
|
||||
host: NecessaryServices<"setting", never>,
|
||||
log: LogFunction,
|
||||
proc: () => Promise<boolean>,
|
||||
keepSuspending = false
|
||||
suspensionPolicy: InitialisationSuspensionPolicy = "resume"
|
||||
) {
|
||||
let completed = false;
|
||||
try {
|
||||
// Disable batch saving and file watching during initialisation.
|
||||
await host.services.setting.applyPartial({ batchSave: false }, false);
|
||||
await host.services.setting.suspendAllSync();
|
||||
await host.services.setting.suspendExtraSync();
|
||||
await host.services.setting.applyPartial({ suspendFileWatching: true }, true);
|
||||
await host.services.setting.applyPartial(
|
||||
suspensionPolicy === "keep-on-failure"
|
||||
? { suspendFileWatching: true, suspendParseReplicationResult: true }
|
||||
: { suspendFileWatching: true },
|
||||
true
|
||||
);
|
||||
try {
|
||||
const result = await proc();
|
||||
completed = result;
|
||||
return result;
|
||||
} catch (ex) {
|
||||
log("Error during vault initialisation process.", LOG_LEVEL_NOTICE);
|
||||
@@ -403,9 +412,21 @@ export async function processVaultInitialisation(
|
||||
log(ex, LOG_LEVEL_VERBOSE);
|
||||
return false;
|
||||
} finally {
|
||||
if (!keepSuspending) {
|
||||
// Re-enable file watching after initialisation.
|
||||
if (suspensionPolicy === "resume") {
|
||||
await host.services.setting.applyPartial({ suspendFileWatching: false }, true);
|
||||
} else if (suspensionPolicy === "keep") {
|
||||
await host.services.setting.applyPartial({ suspendFileWatching: true }, true);
|
||||
} else {
|
||||
// Fast Setup owns both directions at this boundary. Reasserting the
|
||||
// outcome also covers a late failure after finishRebuild started to
|
||||
// resume reflection, and the legacy doNotSuspendOnFetching path.
|
||||
await host.services.setting.applyPartial(
|
||||
{
|
||||
suspendFileWatching: !completed,
|
||||
suspendParseReplicationResult: !completed,
|
||||
},
|
||||
true
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -512,7 +533,7 @@ export function createSuspendFlagHandler(
|
||||
await host.services.setting.applyPartial({ writeLogToTheFile: true }, true);
|
||||
return Promise.resolve(false);
|
||||
},
|
||||
true
|
||||
"keep"
|
||||
);
|
||||
};
|
||||
|
||||
|
||||
@@ -325,13 +325,13 @@ describe("Red Flag Feature", () => {
|
||||
() => {
|
||||
return Promise.resolve(true);
|
||||
},
|
||||
false
|
||||
"resume"
|
||||
);
|
||||
|
||||
expect(host.mocks.setting.currentSettings().suspendFileWatching).toBe(false);
|
||||
});
|
||||
|
||||
it("should keep suspending when keepSuspending is true", async () => {
|
||||
it("should keep suspending when the policy is keep", async () => {
|
||||
const host = createHostMock();
|
||||
const log = createLoggerMock();
|
||||
|
||||
@@ -341,7 +341,7 @@ describe("Red Flag Feature", () => {
|
||||
() => {
|
||||
return Promise.resolve(true);
|
||||
},
|
||||
true
|
||||
"keep"
|
||||
);
|
||||
|
||||
expect(host.mocks.setting.currentSettings().suspendFileWatching).toBe(true);
|
||||
@@ -357,7 +357,7 @@ describe("Red Flag Feature", () => {
|
||||
() => {
|
||||
throw new Error("Process failed");
|
||||
},
|
||||
false
|
||||
"resume"
|
||||
);
|
||||
|
||||
expect(result).toBe(false);
|
||||
@@ -581,6 +581,16 @@ describe("Red Flag Feature", () => {
|
||||
|
||||
expect(result).toBe(false);
|
||||
expect(host.mocks.ui.confirm.confirmWithMessage).not.toHaveBeenCalled();
|
||||
expect(host.mocks.storageAccess.files.has(FlagFilesOriginal.FETCH_ALL)).toBe(false);
|
||||
await expect(handler.check()).resolves.toBe(false);
|
||||
expect(host.mocks.setting.applyPartial).toHaveBeenCalledWith(
|
||||
{
|
||||
suspendFileWatching: true,
|
||||
suspendParseReplicationResult: true,
|
||||
},
|
||||
true
|
||||
);
|
||||
expect(host.mocks.appLifecycle.performRestart).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
it("should activate selected remote configuration", async () => {
|
||||
@@ -759,6 +769,79 @@ describe("Red Flag Feature", () => {
|
||||
});
|
||||
|
||||
describe("askAndPerformFastSetupOnScheduledFetchAll", () => {
|
||||
it("releases both reflection suspensions after Fast Setup succeeds", async () => {
|
||||
const host = createHostMock();
|
||||
const log = createLoggerMock();
|
||||
const cleanupFlag = vi.fn().mockResolvedValue(undefined);
|
||||
|
||||
Object.assign(host.mocks.setting.settings, {
|
||||
doNotSuspendOnFetching: true,
|
||||
suspendParseReplicationResult: true,
|
||||
});
|
||||
host.mocks.ui.confirm.confirmWithMessage
|
||||
.mockResolvedValueOnce(SIMPLE_FETCH_STAGE1_NEWER_WINS)
|
||||
.mockResolvedValueOnce(SIMPLE_FETCH_STAGE2_NEWER_CLEANUP);
|
||||
host.mocks.tweakValue.fetchRemotePreferred.mockResolvedValue(availableRemoteTweaks({ batchSave: false }));
|
||||
|
||||
await expect(askAndPerformFastSetupOnScheduledFetchAll(host as any, log, cleanupFlag)).resolves.toBe(true);
|
||||
|
||||
expect(host.mocks.setting.currentSettings()).toMatchObject({
|
||||
suspendFileWatching: false,
|
||||
suspendParseReplicationResult: false,
|
||||
});
|
||||
});
|
||||
|
||||
it("keeps Vault reflection suspended and preserves recovery state when Fast Fetch fails", async () => {
|
||||
const host = createHostMock();
|
||||
const log = createLoggerMock();
|
||||
const cleanupFlag = vi.fn().mockResolvedValue(undefined);
|
||||
|
||||
host.mocks.ui.confirm.confirmWithMessage
|
||||
.mockResolvedValueOnce(SIMPLE_FETCH_STAGE1_NEWER_WINS)
|
||||
.mockResolvedValueOnce(SIMPLE_FETCH_STAGE2_NEWER_CLEANUP);
|
||||
host.mocks.tweakValue.fetchRemotePreferred.mockResolvedValue(availableRemoteTweaks({ batchSave: false }));
|
||||
host.mocks.rebuilder.$fetchLocalDBFast.mockRejectedValueOnce(new Error("cannot decrypt remote document"));
|
||||
|
||||
await expect(askAndPerformFastSetupOnScheduledFetchAll(host as any, log, cleanupFlag)).resolves.toBe(false);
|
||||
|
||||
expect(host.mocks.setting.currentSettings()).toMatchObject({
|
||||
suspendFileWatching: true,
|
||||
suspendParseReplicationResult: true,
|
||||
});
|
||||
expect(host.mocks.rebuilder.finishRebuild).not.toHaveBeenCalled();
|
||||
expect(cleanupFlag).not.toHaveBeenCalled();
|
||||
expect(host.mocks.setting.deleteSmallConfig).not.toHaveBeenCalledWith("simple-fetch-mode");
|
||||
});
|
||||
|
||||
it("re-suspends both reflection directions when finalisation fails after releasing them", async () => {
|
||||
const host = createHostMock();
|
||||
const log = createLoggerMock();
|
||||
const cleanupFlag = vi.fn().mockResolvedValue(undefined);
|
||||
|
||||
host.mocks.ui.confirm.confirmWithMessage
|
||||
.mockResolvedValueOnce(SIMPLE_FETCH_STAGE1_NEWER_WINS)
|
||||
.mockResolvedValueOnce(SIMPLE_FETCH_STAGE2_NEWER_CLEANUP);
|
||||
host.mocks.tweakValue.fetchRemotePreferred.mockResolvedValue(availableRemoteTweaks({ batchSave: false }));
|
||||
host.mocks.rebuilder.finishRebuild.mockImplementationOnce(async () => {
|
||||
await host.mocks.setting.applyPartial(
|
||||
{
|
||||
suspendFileWatching: false,
|
||||
suspendParseReplicationResult: false,
|
||||
},
|
||||
true
|
||||
);
|
||||
throw new Error("Vault scan failed after reflection resumed");
|
||||
});
|
||||
|
||||
await expect(askAndPerformFastSetupOnScheduledFetchAll(host as any, log, cleanupFlag)).resolves.toBe(false);
|
||||
|
||||
expect(host.mocks.setting.currentSettings()).toMatchObject({
|
||||
suspendFileWatching: true,
|
||||
suspendParseReplicationResult: true,
|
||||
});
|
||||
expect(cleanupFlag).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("should remember quick flow choices while the scheduled fetch is pending", async () => {
|
||||
const host = createHostMock();
|
||||
const log = createLoggerMock();
|
||||
@@ -1352,7 +1435,7 @@ describe("Red Flag Feature", () => {
|
||||
() => {
|
||||
return Promise.resolve(false);
|
||||
},
|
||||
true
|
||||
"keep"
|
||||
);
|
||||
|
||||
expect(host.mocks.setting.currentSettings().suspendFileWatching).toBe(true);
|
||||
|
||||
@@ -30,12 +30,15 @@ Deno.test({
|
||||
const aggregator = await browser.newPage();
|
||||
const assertNoAggregatorFailures = observePageFailures(aggregator);
|
||||
const assertNoAggregatorNetworkFailures = observeNetworkFailures(aggregator);
|
||||
await aggregator.goto(new URL("aggregator.html#id=pages-smoke&n=2&i=0&d=first-", server.baseUrl).href);
|
||||
await aggregator.goto(new URL("aggregator.html#id=pages-smoke&n=2&i=0&d=before%2", server.baseUrl).href);
|
||||
await aggregator.getByText("1 / 2 Loaded", { exact: true }).waitFor();
|
||||
await aggregator.goto(new URL("aggregator.html#id=pages-smoke&n=2&i=1&d=second", server.baseUrl).href);
|
||||
await aggregator.goto(
|
||||
new URL("aggregator.html#id=pages-smoke&n=2&i=1&d=3after%26amp%2Bplus%25percent", server.baseUrl)
|
||||
.href
|
||||
);
|
||||
assertEquals(
|
||||
await aggregator.getByRole("link", { name: "Open Obsidian to complete setup" }).getAttribute("href"),
|
||||
"obsidian://setuplivesync?settingsQR=first-second"
|
||||
"obsidian://setuplivesync?settingsQR=before%23after%26amp%2Bplus%25percent"
|
||||
);
|
||||
assertNoAggregatorFailures();
|
||||
assertNoAggregatorNetworkFailures();
|
||||
|
||||
+36
@@ -12,6 +12,42 @@ Earlier releases remain available in the 0.25 release history and the legacy rel
|
||||
|
||||
## Unreleased
|
||||
|
||||
## 1.0.9
|
||||
|
||||
8th August, 2026
|
||||
|
||||
For the first time in a while, I published a release that could not be promoted to a stable release. Sorry about that! I am glad that we caught it while it was still a pre-release.
|
||||
|
||||
### Setup and compatibility
|
||||
|
||||
#### Fixed
|
||||
|
||||
- Multi-part settings QR codes now preserve special characters in passwords, passphrases, and other settings (PR #1083). Thank you to @calvinbui for the improvement!
|
||||
- Fast Setup now sizes each finite CouchDB changes page from a one-row status probe, counts the returned result together with `pending`, and resumes from the page's opaque `last_seq` without comparing token representations. Each page uses a one-second idle timeout instead of a heartbeat, allowing CouchDB 3.2 to return its terminator after the currently available rows have been persisted.
|
||||
|
||||
## 1.0.8
|
||||
|
||||
8th August, 2026
|
||||
|
||||
This version was published for pre-release validation only and was not promoted to a stable release.
|
||||
|
||||
### Setup and compatibility
|
||||
|
||||
#### Fixed
|
||||
|
||||
- Fast Setup now sizes each finite CouchDB changes page from a one-row status probe, counts the returned result together with `pending`, and resumes from the page's opaque `last_seq` without comparing token representations. Heartbeat-enabled feeds no longer wait for future writes after the currently available rows have been persisted (#1065).
|
||||
- Cancelling remote selection during a scheduled Fetch now removes the Fetch flag before restarting with file and database reflection paused, preventing the same selection dialogue from reopening on every start-up.
|
||||
|
||||
## 1.0.7
|
||||
|
||||
8th August, 2026
|
||||
|
||||
### Setup and compatibility
|
||||
|
||||
#### Fixed
|
||||
|
||||
- Fast Setup now completes only after the captured CouchDB changes target has been persisted. Decryption, protocol, and local write failures stop the operation without finalising an incomplete database, while transient interruptions resume from the last durable checkpoint (#1065).
|
||||
|
||||
## 1.0.6
|
||||
|
||||
6th August, 2026
|
||||
|
||||
+4
-1
@@ -18,5 +18,8 @@
|
||||
"1.0.3": "1.7.2",
|
||||
"1.0.4": "1.7.2",
|
||||
"1.0.5": "1.7.2",
|
||||
"1.0.6": "1.7.2"
|
||||
"1.0.6": "1.7.2",
|
||||
"1.0.7": "1.7.2",
|
||||
"1.0.8": "1.7.2",
|
||||
"1.0.9": "1.7.2"
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user