diff --git a/docs/adr/2026_10_local_file_save_completion_and_batching.md b/docs/adr/2026_10_local_file_save_completion_and_batching.md new file mode 100644 index 00000000..8abef973 --- /dev/null +++ b/docs/adr/2026_10_local_file_save_completion_and_batching.md @@ -0,0 +1,356 @@ +--- +date: 2026-10-24 +commonlib-version: "0.2.0" +self-hosted-livesync-version: "1.0.40" +status: accepted-unreleased +--- + +# Architectural Decision Record: Local File Save Completion and Batching + +## Status + +Accepted on 2026-10-25. The package versions above identify the investigated +baseline, rather than a released implementation of this decision. + +## Context + +[Issue #1207](https://github.com/vrtmrz/obsidian-livesync/issues/1207) includes +conflicts around automated note creation, editing, and renaming. Batching local +file updates can reduce the intermediate revisions which another device receives. +It also extends the period during which the Vault contains an edit which the +local database has not saved. Recovery and incoming-file protection must remain +correct throughout that period. + +Two-device Obsidian E2E tests using Self-hosted LiveSync 1.0.40 and Commonlib +0.2.0 found that a test-only removal of the LiveSync mode restriction reduced +six consecutive edits from six additional revisions to one, in both directions. +Incoming identical content produced no extra conflict, while genuinely different +edits or a deletion preserved the necessary branches. A synthetic +create–edit–rename sequence still delivered intermediate content. These tests +used Vault operations, without Templater or Modal Forms themselves. + +Restart tests exposed a separate recovery gap: + +1. The event manager removes an event from its waiting map when its timer is + released. It also removes a buffered event before dispatching it. Its saved + snapshot covers buffered and waiting events, but omits work in progress. +2. `waitForIdle()` waits for the waiting gates to open, rather than for the file + handlers to finish. `ServiceFileAccessBase` reports that pending events have + been committed after this gate-only wait. Some handler failures also settle + without propagating a failure to that caller. +3. If the process stops before the database write, the file can retain the new + content while the database retains the old content and the saved event + snapshot is empty. +4. At start-up, the Offline Scanner's `EVEN` branch tries to record missing + provenance, then reports completion. It ignores that helper's Boolean result. + Different content can consequently remain unsaved when both modification + times fall in the same two-second comparison bucket. + +The last condition was reproduced by pausing the target operation immediately +before database processing, then restarting the same isolated profile. A +different timestamp bucket recovered the edit; the same bucket did not. The +changed-condition success demonstrates the boundary, rather than a fix. This +does not establish the cause of the reporter's historical conflicts. + +The runtime evidence covers Linux aarch64, Obsidian 1.12.7, and CouchDB 3.5.0. +Windows, mobile lifecycles, and the reporter's full automation remain outside +that evidence. The inspected Commonlib source is commit +`aa54282937c20d3db7f43778d240feeb1586cced`. + +## Decision + +Make local file processing recoverable until its result is established, use +content and provenance to reconcile ambiguous timestamps, and enable batching +in LiveSync mode only through an explicit choice after those contracts pass +verification. Apply the recovery contract with batching enabled or disabled. + +The content-reconciliation change targets selected ordinary files in the +Offline Scanner's `NEWER_WINS` / `sync-newer` action. Keep the existing action +matrix, explicit `DB_APPLY` policy, selection and size limits, conflict guards, +and recovery suspensions. Hidden and internal files retain their owning +feature's recovery contract; an ordinary file scan cannot recover those paths +on its behalf. + +### 1. Distinguish local persistence from replication + +The following boundaries have different meanings: + +| Boundary | Meaning | +| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Batch wait released | The file operation may start. Its content can still be absent from the local database. | +| Local processing completed | The intended change is represented durably by the local database and its required content, or has been proved already represented or safely superseded. | +| Replication completed | A finite replication reports its own outcome. A local save alone does not prove remote receipt or reflection into another Vault. | + +Ordinary file saving by Obsidian remains independent of database saving. Pending +and processing indicators must include work in progress and unresolved failures, +so an empty timer map cannot imply that local processing is complete. + +### 2. Keep a recoverable record through processing + +The event manager owns each operation from queue admission through completion. +Its existing persistent snapshot must represent buffered, waiting, and +in-progress work in an order which preserves deletion and rename dependencies. +Moving work between those states must not create a persistent gap. + +An event can leave the recoverable set only after its effect is durable, current +state proves that the effect is already represented, or a durably recorded +successor safely subsumes it. Finishing an older operation must not remove a +newer operation for the same path. Cancellation of a batch timer alone is not +evidence that its content is safe to forget. + +Snapshot writes must preserve mutation order. An older asynchronous write must +not replace a newer snapshot, and a failed snapshot write must remain visible +to the lifecycle caller. Reuse the existing device-local snapshot store; a +second journal and a transaction spanning the file system and database are not +part of this proposal. + +The first implementation should retain the existing event snapshot format, +including event keys and rename intent, and add pending ownership within the +event manager. The persistence adapter remains responsible for storing the +ordered snapshots. Handler outcomes must distinguish a failed attempt from a +policy-approved skip before the manager can acknowledge either safely. + +A crash after the database write but before snapshot removal can replay an +operation. Replay must check current storage and revision provenance, retain +the existing protection against stale restored events, and avoid duplicate +revisions or deletion of a newer same-path file. It must be safe to interrupt +replay again. Existing saved snapshots remain readable. + +Boolean failure, an exception, and an unavailable comparison are not successful +completion. Keep unresolved work recoverable and report the failure through the +existing service and diagnostic boundaries. Explicit exclusions or a verified +superseding change may finish work without a new revision; a generic handler +failure may not. Recovery may continue with independent paths, while retaining +the failed path and its dependencies. This proposal adds no unbounded retry loop. + +The recoverable snapshot is an operation record, not a backup of every transient +editor buffer. File content already saved by the host remains available for +start-up reconciliation if the process stops before the first snapshot write. +Crash guarantees cannot include bytes or events which the host never persisted +or delivered, or repair arbitrary storage failure. + +### 3. Give flush a finite, truthful completion boundary + +An explicit commit of pending file events captures the work accepted before +that request, releases its batch delays, and waits for the actual processing +results, including buffered and already-running work. Later edits cannot extend +that boundary indefinitely. A newer coalesced operation may satisfy earlier work +only when its successful result covers that work. + +The internal ordering barrier before a delete or rename waits only for its +predecessors. It must not wait for itself or later events in the same dispatch +loop. Replacing every gate wait with a wait for global idleness would risk a +dependency cycle or starvation. + +Manual synchronisation and replication readiness use the completed local +boundary. Settings application must observe its result before reporting success +or resuming work which depends on it. Persisting a settings file is a separate +operation: `saveSettingData()` returning is not a file-processing completion +signal, and making every settings-saved notification synchronous is unnecessary. + +When a host permits orderly shutdown, stop admitting watcher work before +draining accepted operations or preserving their pending records. Keep their +storage available until that work settles. Suspension, forced termination, and +Obsidian's synchronous unload callback cannot guarantee that asynchronous work +finishes; correctness must also follow from the persistent record and start-up +reconciliation. Remote availability must not determine whether local pending +work survives shutdown. + +### 4. Reconcile content when timestamps are inconclusive + +Retain the existing timestamp tolerance for platforms with coarse modification +times. Timestamp equality, including exactly equal raw timestamps, does not +prove content equality. The Offline Scanner must establish one of the following +outcomes through the ordinary non-forced file handler and its provenance rules: + +| Evidence | Required outcome | +| ----------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | +| Vault content already matches an eligible current revision | Record or retain valid provenance as appropriate; create no duplicate revision. | +| Vault content matches its recorded revision or saved content evidence, while the current database revision has advanced | Apply the database state through the existing protected reflection path. Do not resurrect the unchanged older content as an edit. | +| Vault content differs from its recorded revision | Preserve the edit on that recorded branch. Keep independent concurrent branches available for conflict resolution. | +| Vault content has no provable origin and matches no current live leaf | Preserve it according to the existing independent-branch rules. Do not attach it to an arbitrary database winner. | +| Required content or provenance cannot be read, or state changes during comparison | Retain unresolved work and report or re-evaluate it. Do not mark it reconciled merely because the timestamp comparison is `EVEN`. | + +Revalidate the relevant file and database state before acting on a comparison. +Reuse the existing content and provenance checks; avoid a separate conflict +algorithm in the scanner. Metadata continues to reference Chunks containing the +file content. + +`tryRecordUntrackedFileRevision()` answers whether it could safely record +provenance. Its `false` also covers an already-populated record, conflicting +history, unavailable content, and a concurrent change. It is not an 'edited +file' predicate. A fallback which treats every `false` as permission for a +forced save would violate the required outcomes. + +Propagate the file handler's actual outcome into the pair result, including for +storage-to-database writes. Preserve the scanner's `completed`, `skipped`, and +`failed` distinctions and its existing `continueOnFileFailure` policy. Ordinary +Obsidian start-up can report `completed-with-file-failures` and allow unaffected +files to proceed; an unresolved pair must not become `completed` to achieve +that result. Keep the stricter callers' existing completion requirements. + +This deliberately extends the current equal-time scanner contract. Existing +provenance tests expect no database write during that scan, including for an +edited file. Preserve their unchanged-file and later incoming-protection +scenarios; change the edited-file expectation to require preservation during +the scan itself. Test both identical timestamps and distinct raw timestamps +within one comparison bucket. + +### 5. Make batch timing and activation explicit + +Batching is a bounded delay before local database processing. The quiet-period +timer follows the latest eligible content update; the maximum intentional delay +starts with the first uncommitted update and is not reset by later edits or time +spent awaiting a processing slot. Once due, work must become eligible to run. +The maximum delay is not a guarantee of database completion, network delivery, +or another device's reflection time. Waiting timers should not consume the +capacity needed to process unrelated ready files. + +For ordinary local content updates, queue the changed path and operation intent +without reading the file body merely to wait. Acquire the current content when +the operation becomes due, after coalescing eligible events. Reuse that content +snapshot within the operation for comparison, provenance evidence, and Chunk +creation. This retains the current ordinary-save boundary; it does not require +reading and hashing the file for every keystroke or queued notification. + +Content reuse within one operation is distinct from keeping a body cache across +edits. A cached body would need a proven invalidation and freshness contract; +mtime and size alone cannot establish that it is current. A saved provenance +hash records evidence about earlier content, rather than replacing acquisition +of the current bytes. Restart recovery, an incoming update, or a comparison +which needs revalidation can still require additional reads. Preserve those +checks rather than making a global promise of one read per path. + +Batching therefore aims to reduce repeated content acquisition, comparison, +Chunk processing, and local database updates, as well as intermediate revisions +and subsequent replication and reflection work. Event filtering, bookkeeping, +and pending-operation persistence still occur while waiting. The revision +reduction measured by the existing E2E does not quantify file reads, CPU cost, +physical disk I/O, or network traffic; measure those costs separately. Host or +operating-system caching also means that a read API call is not a physical disk +read count. + +Manual flush and required deletion or rename ordering can end a batch early. +Ordinary incoming replication continues, using the existing protection for +unsaved local content. It must not require all unrelated local batches to flush +for every incoming document. + +Add an independent `batchSaveInLiveSync` Boolean, defaulting to `false` +for both existing settings and new Vaults. In LiveSync mode, this value selects +batching; other modes continue to use `batchSave`. Both choices use the existing +minimum and maximum delay settings. The interface presents the choice for the +current mode. An ordinary mode switch preserves each mode's choice. + +This adds one persisted setting to distinguish a new explicit choice from old +`liveSync: true, batchSave: true` configurations, where batching was ignored. +Neither upgrade nor import of that legacy pair may activate batching. An older +client continues to apply its existing LiveSync restriction. Include the new +property in Commonlib's setting policy and import/export tests; do not infer +consent from the presence of the old `batchSave` property. + +The setting and scheduling changes follow the recovery work. Recovery in the +currently supported modes must pass verification before the new mode is enabled. + +### 6. Keep batching separate from automation completion + +The intended benefit is fewer short-lived intermediate revisions. Batching +cannot discover when a third-party template has finished, particularly while a +dialogue is open or another device also runs automation. A maximum delay or a +rename boundary can legitimately publish intermediate content. + +Coalescing `CREATE` followed by `CHANGED` for the same still-existing file is a +useful follow-up, but it needs its own ordering and interruption tests. Preserve +the existing cross-path rename rule: store the destination before deleting the +proven source branch. Do not retarget or discard pending source operations +without proving the effect is already covered. + +A guarantee that a multi-step template appears remotely only after completion +would need an explicit producer completion contract and its failure behaviour. +That is a separate design. Enabling batching alone must not be described as +eliminating genuine concurrent-edit conflicts or as a confirmed fix for #1207. + +## Ownership and implementation order + +1. **Commonlib recovery:** event ownership, ordered snapshot persistence, + successful completion and failure propagation, finite flush boundaries, and + same-time content reconciliation. Add focused failing regressions before + changing production code. Reuse the existing event manager, file handler, + scanner, and provenance store. +2. **Host integration:** keep the Obsidian persistence adapter and lifecycle + wiring aligned with those contracts. Verify callers of + `commitPendingFileEvents`, including settings application and database + preparation, and preserve their failure outcomes. Validate the packed + Commonlib artefact and update the consumer's exact dependency. +3. **LiveSync mode batching:** add the explicit setting and its primary-language + presentation only after the recovery regressions pass. Establish the stated + timing semantics and repeat the two-device comparisons with the actual + package, including start-up with batching already enabled. +4. **Further coalescing:** assess same-path creation/editing and any proposed + rename optimisation separately, against the original automation problem. + +Keep these changes independently reviewable. Shared behaviour belongs in +Commonlib; the plug-in must not acquire a source mirror or patch installed +package files. Snapshot format changes, a second persistent store, or a new +automation transaction API require a further design decision. + +## Verification required + +Use controlled promises at the manager and persistence boundaries for focused +regressions, and real PouchDB tests for content and ancestry. The acceptance +matrix is: + +| Scenario | Required observation | +| -------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | +| Stop after dequeue, timer release, or during a database write | Pending work remains recoverable; restart eventually represents the saved Vault edit. | +| Database write succeeds before snapshot cleanup; restart twice | Replay creates no duplicate revision and preserves later edits. | +| Old and new events share a path; snapshot writes overlap | Completing old work cannot remove new work or regress the durable snapshot. | +| Handler returns false, throws, or snapshot persistence fails | Flush reports failure; pending work is retained; unrelated recoverable work can continue. | +| Buffered or in-progress work exists with no waiting timer | Flush still waits for actual completion. | +| A flush overlaps new typing or a rename/delete barrier | The captured work completes without waiting for later work or deadlocking. | +| Equal timestamps: unchanged, edited, untracked, pruned provenance, unavailable Chunks, or concurrent recheck changes | Preserve correct ancestry and bytes, avoid duplicate revisions, and distinguish unresolved comparisons from successful reconciliation. | +| Settings save followed by stop; ordinary unload; interruption during replay | No operation is forgotten because a settings callback or shutdown released its timer. | +| Sustained edits and more paths than available processing slots | Observe the intentional-delay deadline and progress of unrelated ready files. | +| Old settings, mode changes, explicit opt-in, and settings import | Existing LiveSync mode users retain immediate saving until they choose batching. | +| Incoming identical content, an edit, a deletion, or delete/recreate while waiting | Preserve genuine branches without extra conflicts or loss of local work. | + +Keep the existing E2E comparison between batching disabled, an ignored legacy +setting, and explicitly enabled batching. Repeat both directions, manual +synchronisation, saved-snapshot restart, and controlled in-progress restart with +the fix. Record the original failure separately from repaired results; a pass +with a different timestamp bucket or a test-only bundle is not verification of +the production fix. Run platform-specific lifecycle checks before extending the +evidence to Windows or mobile. + +## Alternatives and consequences + +- **Remove only the LiveSync guard:** leaves recovery gaps and can activate a + previously ignored setting. The explicit choice follows correctness work. +- **Wait only at unload:** cannot cover abrupt termination and hosts which do + not await asynchronous unload. Continuous recoverability is still required. +- **Repair only the scanner:** improves surviving-file recovery, but does not + establish event ordering, deletion intent, or truthful flush completion. +- **Retain only in-progress events:** closes a persistence gap, but still needs + success/failure handling, replay checks, and recovery before the first snapshot. +- **Remove timestamp rounding or force the Vault copy into the current winner:** + fails for exactly equal timestamps and can destroy branch intent. Content and + provenance determine the safe action. + +Keeping work recoverable adds snapshot bookkeeping and can make local flush +wait longer because it now waits for real processing. Same-time reconciliation +can require additional content reads. Batching trades prompt remote visibility +for fewer database updates and may increase the window for genuine concurrent +edits. Measure these costs without weakening the content-preservation contract. + +## References + +- [Conflict resolution and revision provenance](../specs_conflict_resolution.md) +- [Batch database update settings](../settings.md#batch-database-update) +- [Common library package boundary](2026_07_common_library_package_boundary.md) +- [Fast Fetch persistence and completion](2026_08_fast_fetch_persistence_and_completion.md) +- [Two-device Batch save reproduction](../../test/e2e-obsidian/scripts/issue1207-livesync-batch.ts) +- [Commonlib event manager at the investigated revision](https://github.com/vrtmrz/livesync-commonlib/blob/aa54282937c20d3db7f43778d240feeb1586cced/src/managers/StorageEventManager.ts) +- [Commonlib Offline Scanner at the investigated revision](https://github.com/vrtmrz/livesync-commonlib/blob/aa54282937c20d3db7f43778d240feeb1586cced/src/serviceFeatures/offlineScanner.ts) +- [Commonlib file handler at the investigated revision](https://github.com/vrtmrz/livesync-commonlib/blob/aa54282937c20d3db7f43778d240feeb1586cced/src/serviceModules/ServiceFileHandlerBase.ts) +- [Commonlib scanner policy and stored event interpretation](https://github.com/vrtmrz/livesync-commonlib/blob/aa54282937c20d3db7f43778d240feeb1586cced/docs/offline-scanner.md) +- [Commonlib event admission and reflection contract](https://github.com/vrtmrz/livesync-commonlib/blob/aa54282937c20d3db7f43778d240feeb1586cced/docs/storage-events-and-reflection.md) +- [Existing equal-time provenance regression](https://github.com/vrtmrz/livesync-commonlib/blob/aa54282937c20d3db7f43778d240feeb1586cced/src/serviceFeatures/offlineScanner.provenance.unit.spec.ts) diff --git a/docs/settings.md b/docs/settings.md index c65b2949..b7592846 100644 --- a/docs/settings.md +++ b/docs/settings.md @@ -669,17 +669,28 @@ Setup URIs preserve both sleep preferences. Older Setup URIs which do not contai #### Batch database update Setting key: batchSave -Reducing the frequency with which on-disk changes are reflected into the DB + +Combine repeated local file updates before saving them to the local database in modes other than LiveSync mode. This reduces intermediate database revisions and delays their availability to other devices. + +#### Batch database updates in LiveSync mode + +Setting key: batchSaveInLiveSync + +Apply the same bounded delay in LiveSync mode. This separate choice is disabled by default for both existing settings and new Vaults. An older configuration containing `liveSync: true` and `batchSave: true` does not enable it. The settings interface shows the choice for the current mode, and switching modes preserves both choices. Older clients continue to ignore batching in LiveSync mode. + +Both choices share the delays below. Manual synchronisation, deletion, or renaming can end a batch early. Batching reduces intermediate updates; it does not identify when a template has finished or prevent genuine concurrent edits on different devices. #### Minimum delay for batch database updating Setting key: batchSaveMinimumDelay -Seconds. Saving to the local database will be delayed until this value after we stop typing or saving. + +Seconds of quiet after the latest eligible edit before saving to the local database, subject to the maximum delay below. #### Maximum delay for batch database updating Setting key: batchSaveMaximumDelay -Saving will be performed forcefully after this number of seconds. + +Maximum intentional delay in seconds from the first pending update to a file. Later edits do not restart this limit, and waiting for a processing slot does not add another batch delay. This limit makes the work eligible to run; it does not guarantee database completion or receipt by another device within that time. ### 4. Deletion Propagation (Advanced) diff --git a/src/main.ts b/src/main.ts index 8aebf8ad..d4e7d346 100644 --- a/src/main.ts +++ b/src/main.ts @@ -1,4 +1,5 @@ import { useP2PSettingsPreparation } from "@/serviceFeatures/useP2PSettingsPreparation"; +import { LOG_LEVEL_URGENT } from "octagonal-wheels/common/logger"; import { getLanguage, Notice, Plugin, type App, type PluginManifest } from "./deps"; import { setGetLanguage } from "@vrtmrz/livesync-commonlib/compat/common/coreEnvFunctions"; setGetLanguage(getLanguage); @@ -233,6 +234,12 @@ export default class ObsidianLiveSyncPlugin extends Plugin { void this._startUp(); } override onunload(): void { - return void this.core.services.control.onUnload(); + void this.core.services.control.onUnload().catch((error) => { + this.core.services.API.addLog( + "Failed to unload Self-hosted LiveSync: " + String(error), + LOG_LEVEL_URGENT, + "shutdown" + ); + }); } } diff --git a/src/modules/essentialObsidian/ModuleObsidianEvents.ts b/src/modules/essentialObsidian/ModuleObsidianEvents.ts index 5f689bc6..10001ca2 100644 --- a/src/modules/essentialObsidian/ModuleObsidianEvents.ts +++ b/src/modules/essentialObsidian/ModuleObsidianEvents.ts @@ -230,7 +230,13 @@ export class ModuleObsidianEvents extends AbstractObsidianModule { } this.isLastHidden = isHidden; - await this.services.fileProcessing.commitPendingFileEvents(); + const pendingFileEventsCommitted = await this.services.fileProcessing.commitPendingFileEvents(); + if (!pendingFileEventsCommitted) { + this._log( + "Pending file events remain because their commit did not complete before the window visibility changed.", + LOG_LEVEL_NOTICE + ); + } // Desktop opt-in (LiveSync/Periodic only): keep the background channel running while the // window is hidden, instead of suspending on hide. On hide we skip the suspend for both @@ -287,7 +293,10 @@ export class ModuleObsidianEvents extends AbstractObsidianModule { if (this.settings.suspendFileWatching) return; if (!this.settings.isConfigured) return; if (!this.services.appLifecycle.isReady()) return; - await this.services.fileProcessing.commitPendingFileEvents(); + const pendingFileEventsCommitted = await this.services.fileProcessing.commitPendingFileEvents(); + if (!pendingFileEventsCommitted) { + return; + } if (file == null) { return; } diff --git a/src/modules/essentialObsidian/ModuleObsidianEvents.unit.spec.ts b/src/modules/essentialObsidian/ModuleObsidianEvents.unit.spec.ts index 64b92b2b..5b61464b 100644 --- a/src/modules/essentialObsidian/ModuleObsidianEvents.unit.spec.ts +++ b/src/modules/essentialObsidian/ModuleObsidianEvents.unit.spec.ts @@ -3,6 +3,7 @@ import { describe, it, expect, vi, afterEach } from "vitest"; import { ModuleObsidianEvents } from "./ModuleObsidianEvents"; import { DEFAULT_SETTINGS, REMOTE_COUCHDB } from "@vrtmrz/livesync-commonlib/compat/common/types"; import { reactiveSource } from "octagonal-wheels/dataobject/reactive"; +import { LOG_LEVEL_NOTICE } from "octagonal-wheels/common/logger"; type SetupOptions = { settings?: Partial; @@ -23,6 +24,8 @@ function setup(opts: SetupOptions) { onResumed: vi.fn(async () => true), }; const fileProcessing = { commitPendingFileEvents: vi.fn(async () => true) }; + const replication = { replicateUnattendedByEvent: vi.fn(async () => undefined) }; + const conflict = { queueCheckForIfOpen: vi.fn(async () => undefined) }; const boundedRemoteActivityCount = reactiveSource(0); const boundedLocalApplicationActivityCount = reactiveSource(0); @@ -39,6 +42,8 @@ function setup(opts: SetupOptions) { setting: { saveSettingData: vi.fn(async () => undefined) }, appLifecycle, fileProcessing, + replication, + conflict, replicator: { boundedRemoteActivityCount, boundedLocalApplicationActivityCount }, }, settings: { @@ -61,6 +66,9 @@ function setup(opts: SetupOptions) { module, appLifecycle, fileProcessing, + replication, + conflict, + api: core._services.API, boundedRemoteActivityCount, boundedLocalApplicationActivityCount, }; @@ -303,6 +311,23 @@ describe("watchWindowVisibilityAsync — keepReplicationActiveInBackground", () expect(appLifecycle.onResumed).toHaveBeenCalledTimes(1); }); + it("logs a failed pending-file commit and still suspends on hide", async () => { + const { module, appLifecycle, fileProcessing, api } = setup({ + settings: { keepReplicationActiveInBackground: false, liveSync: true }, + hidden: true, + }); + fileProcessing.commitPendingFileEvents.mockResolvedValue(false); + + await module.watchWindowVisibilityAsync(); + + expect(appLifecycle.onSuspending).toHaveBeenCalledTimes(1); + expect(api.addLog).toHaveBeenCalledWith( + expect.stringContaining("Pending file events remain"), + LOG_LEVEL_NOTICE, + expect.any(String) + ); + }); + it("does not apply on mobile even if the flag is set", async () => { const { module, appLifecycle } = setup({ settings: { keepReplicationActiveInBackground: true, liveSync: true }, @@ -313,3 +338,20 @@ describe("watchWindowVisibilityAsync — keepReplicationActiveInBackground", () expect(appLifecycle.onSuspending).toHaveBeenCalledTimes(1); }); }); + +describe("watchWorkspaceOpenAsync — pending file event commit", () => { + it("skips replication and conflict checks when pending file events fail to commit", async () => { + const { module, fileProcessing, replication, conflict } = setup({ + settings: { syncOnFileOpen: true }, + hidden: false, + }); + fileProcessing.commitPendingFileEvents.mockResolvedValue(false); + + await module.watchWorkspaceOpenAsync({ path: "note.md" } as Parameters< + typeof module.watchWorkspaceOpenAsync + >[0]); + + expect(replication.replicateUnattendedByEvent).not.toHaveBeenCalled(); + expect(conflict.queueCheckForIfOpen).not.toHaveBeenCalled(); + }); +}); diff --git a/src/modules/features/SettingDialogue/PaneSyncSettings.ts b/src/modules/features/SettingDialogue/PaneSyncSettings.ts index 70a4b566..76c5b1a9 100644 --- a/src/modules/features/SettingDialogue/PaneSyncSettings.ts +++ b/src/modules/features/SettingDialogue/PaneSyncSettings.ts @@ -171,20 +171,18 @@ export function paneSyncSettings( } }); - void addPanel( - paneEl, - $msg("obsidianLiveSyncSettingTab.titleUpdateThinning"), - undefined, - visibleOnly(() => !this.isConfiguredAs("syncMode", "LIVESYNC")) - ).then((paneEl) => { - new Setting(paneEl).autoWireToggle("batchSave"); + void addPanel(paneEl, $msg("obsidianLiveSyncSettingTab.titleUpdateThinning")).then((paneEl) => { + const inLiveSyncMode = () => this.isConfiguredAs("syncMode", "LIVESYNC"); + const batchEnabled = () => this.isConfiguredAs(inLiveSyncMode() ? "batchSaveInLiveSync" : "batchSave", true); + new Setting(paneEl).autoWireToggle("batchSave", { onUpdate: visibleOnly(() => !inLiveSyncMode()) }); + new Setting(paneEl).autoWireToggle("batchSaveInLiveSync", { onUpdate: visibleOnly(inLiveSyncMode) }); new Setting(paneEl).autoWireNumeric("batchSaveMinimumDelay", { acceptZero: true, - onUpdate: visibleOnly(() => this.isConfiguredAs("batchSave", true)), + onUpdate: visibleOnly(batchEnabled), }); new Setting(paneEl).autoWireNumeric("batchSaveMaximumDelay", { acceptZero: true, - onUpdate: visibleOnly(() => this.isConfiguredAs("batchSave", true)), + onUpdate: visibleOnly(batchEnabled), }); }); diff --git a/src/modules/features/SettingDialogue/PaneSyncSettings.unit.spec.ts b/src/modules/features/SettingDialogue/PaneSyncSettings.unit.spec.ts new file mode 100644 index 00000000..ca88eaee --- /dev/null +++ b/src/modules/features/SettingDialogue/PaneSyncSettings.unit.spec.ts @@ -0,0 +1,98 @@ +import { beforeEach, describe, expect, it, vi } from "vitest"; +import { DEFAULT_SETTINGS } from "@vrtmrz/livesync-commonlib/compat/common/types"; +import type { ObsidianLiveSyncSettingTab } from "./ObsidianLiveSyncSettingTab"; +import type { OnUpdateFunc, PageFunctions } from "./SettingPane"; + +const controls = vi.hoisted(() => new Map { visibility?: boolean } }>()); +vi.mock("@/common/translation", () => ({ $msg: (key: string) => key })); +vi.mock("@vrtmrz/livesync-commonlib/compat/common/logger", () => ({ Logger: vi.fn() })); +vi.mock("./LiveSyncSetting.ts", () => ({ + LiveSyncSetting: class { + autoWireDropDown(key: string, options = {}) { + controls.set(key, options); + return this; + } + autoWireToggle(key: string, options = {}) { + controls.set(key, options); + return this; + } + autoWireNumeric(key: string, options = {}) { + controls.set(key, options); + return this; + } + }, +})); + +import { paneSyncSettings } from "./PaneSyncSettings"; + +function setup(liveSync: boolean, batchSave: boolean, batchSaveInLiveSync: boolean) { + const callbacks = new Map Promise>(); + const panels = new Map(); + const tab = { + editingSettings: { ...DEFAULT_SETTINGS, liveSync, batchSave, batchSaveInLiveSync }, + services: { API: { isMobile: () => false }, control: { applySettings: vi.fn(async () => {}) } }, + addOnSaved: (key: string, callback: (value: string) => Promise) => callbacks.set(key, callback), + isConfiguredAs(key: string, value: unknown) { + if (key === "syncMode") { + return ( + (this.editingSettings.liveSync + ? "LIVESYNC" + : this.editingSettings.periodicReplication + ? "PERIODIC" + : "ONEVENTS") === value + ); + } + return this.editingSettings[key as keyof typeof this.editingSettings] === value; + }, + saveSettings: vi.fn(async () => {}), + requestCatalogueRefresh: vi.fn(), + }; + const functions = { + addPanel: (_element: HTMLElement, title: string, _callback: unknown, onUpdate?: OnUpdateFunc) => ({ + then(callback: (element: HTMLElement) => void) { + if (!title.endsWith("titleSynchronizationMethod") && !title.endsWith("titleUpdateThinning")) return; + panels.set(title, onUpdate); + callback({} as HTMLElement); + }, + }), + } as unknown as PageFunctions; + paneSyncSettings.call(tab as unknown as ObsidianLiveSyncSettingTab, {} as HTMLElement, functions); + return { tab, callbacks, panels }; +} + +function visible(key: string): boolean { + const control = controls.get(key); + if (!control) throw new Error(`Missing setting control: ${key}`); + return control.onUpdate?.().visibility !== false; +} + +beforeEach(() => controls.clear()); + +describe("mode-specific database batching settings", () => { + it.each([ + { liveSync: true, batchSave: true, batchSaveInLiveSync: false, delays: false }, + { liveSync: true, batchSave: false, batchSaveInLiveSync: true, delays: true }, + { liveSync: false, batchSave: true, batchSaveInLiveSync: false, delays: true }, + { liveSync: false, batchSave: false, batchSaveInLiveSync: true, delays: false }, + ])("shows the current mode choice and its shared delays: %j", ({ delays, ...settings }) => { + const { panels } = setup(settings.liveSync, settings.batchSave, settings.batchSaveInLiveSync); + expect(panels.get("obsidianLiveSyncSettingTab.titleUpdateThinning")?.().visibility).not.toBe(false); + expect(visible("batchSave")).toBe(!settings.liveSync); + expect(visible("batchSaveInLiveSync")).toBe(settings.liveSync); + expect(visible("batchSaveMinimumDelay")).toBe(delays); + expect(visible("batchSaveMaximumDelay")).toBe(delays); + }); + + it.each([ + [true, false], + [false, true], + ])("retains both batching choices across mode changes: %j", async (batchSave, batchSaveInLiveSync) => { + const { tab, callbacks } = setup(false, batchSave, batchSaveInLiveSync); + const switchMode = callbacks.get("syncMode")!; + for (const mode of ["LIVESYNC", "PERIODIC", "ONEVENTS", "LIVESYNC"]) { + await switchMode(mode); + expect(tab.editingSettings).toMatchObject({ batchSave, batchSaveInLiveSync }); + } + expect(tab.services.control.applySettings).toHaveBeenCalledTimes(4); + }); +}); diff --git a/test/e2e-obsidian/scripts/issue1207-livesync-batch.ts b/test/e2e-obsidian/scripts/issue1207-livesync-batch.ts new file mode 100644 index 00000000..10f6f6a6 --- /dev/null +++ b/test/e2e-obsidian/scripts/issue1207-livesync-batch.ts @@ -0,0 +1,961 @@ +import { createHash } from "node:crypto"; +import { readFileSync } from "node:fs"; +import { mkdir, readFile, writeFile } from "node:fs/promises"; +import { createServer } from "node:net"; +import { join } from "node:path"; +import type { Page } from "playwright"; +import { + assertCouchDbReachable, + createCouchDbDatabase, + deleteCouchDbDatabase, + loadCouchDbConfig, + makeUniqueDatabaseName, +} from "../runner/couchdb.ts"; +import { discoverObsidianCli, requireObsidianBinary } from "../runner/environment.ts"; +import { + assertE2eCompatibilityUnpaused, + configureCouchDb, + createE2eCouchDbPluginData, + createE2eObsidianDeviceLocalState, + prepareRemote, + waitForLiveSyncCoreReady, +} from "../runner/liveSyncWorkflow.ts"; +import { startObsidianLiveSyncSession, type ObsidianLiveSyncSession } from "../runner/session.ts"; +import { withObsidianPage } from "../runner/ui.ts"; +import { createTemporaryVault, type TemporaryVault } from "../runner/vault.ts"; + +type Mode = "normal" | "ignored-batch-setting" | "enabled-batch"; +type Slot = { name: string; vault: TemporaryVault; session?: ObsidianLiveSyncSession }; +type Leaf = { rev: string; ancestry: string[]; logicalDeleted: boolean; pouchDeleted: boolean; body: string | null }; +type Snapshot = { + path: string; + rev: string | null; + content: string | null; + fileMtime: number | null; + databaseMtime: number | null; + exists: boolean; + leaves: Leaf[]; + waiting: Array<{ path: string; type: string; since: number }>; + queued: Array<{ path: string | null; type: string }>; + shouldBatchSave: boolean; +}; +type Pair = { a: Page; b: Page }; + +const diagnostics = process.env.E2E_OBSIDIAN_DIAGNOSTICS_DIR ?? "/tmp/issue1207-livesync-batch"; +const artifactRoot = process.env.ISSUE1207_ARTIFACT_ROOT ?? process.cwd(); +const evidence: Record = { cases: [], observations: [] }; +const cases = evidence.cases as Array>; +const observations = evidence.observations as unknown[]; +const pause = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)); +const prefix = "issue1207-batch-"; + +function check(condition: unknown, message: string): asserts condition { + if (!condition) throw new Error(message); +} + +function hashes() { + return Object.fromEntries( + ["main.js", "manifest.json", "styles.css"].map((name) => [ + name, + createHash("sha256") + .update(new Uint8Array(readFileSync(join(artifactRoot, name)))) + .digest("hex"), + ]) + ); +} + +async function availablePort(): Promise { + const server = createServer(); + await new Promise((resolve, reject) => { + server.once("error", reject); + server.listen(0, "127.0.0.1", resolve); + }); + const address = server.address(); + check(address && typeof address !== "string", "Missing debug port"); + await new Promise((resolve, reject) => server.close((error) => (error ? reject(error) : resolve()))); + return address.port; +} + +async function evaluate(page: Page, code: string): Promise { + return await page.evaluate(`(async()=>{ + const core=app.plugins.plugins['obsidian-livesync'].core; + const manager=core.serviceModules.storageAccess.vaultManager; + ${code} + })()`); +} + +async function installProbe(page: Page): Promise { + await evaluate( + page, + ` + if(globalThis.__issue1207Batch) return; + const probe={events:[],changes:[]}; + globalThis.__issue1207Batch=probe; + for(const type of ['create','modify','rename','delete']) { + app.vault.on(type,(file,oldPath)=>{ + const path=file.path; + if(!path.startsWith(${JSON.stringify(prefix)})) return; + const item={type,path,oldPath:oldPath??null,at:Date.now(),content:null}; + probe.events.push(item); + if(type!=='delete' && file.stat) { + void app.vault.read(file).then(body=>item.content=body).catch(error=>item.error=String(error)); + } + }); + } + probe.feed=core.localDatabase.localDatabase.changes({live:true,since:'now',include_docs:true}) + .on('change',change=>{ + if(typeof change.doc?.path==='string' && change.doc.path.startsWith(${JSON.stringify(prefix)})) { + probe.changes.push({path:change.doc.path,rev:change.doc._rev, + logicalDeleted:!!change.doc.deleted,pouchDeleted:!!change.doc._deleted,at:Date.now()}); + } + }).on('error',error=>probe.changes.push({error:String(error),at:Date.now()})); + ` + ); +} + +async function options(page: Page, mode: Mode, minimum = 2, maximum = 6): Promise { + await evaluate( + page, + ` + const settings=core.services.setting.currentSettings(); + Object.assign(settings,{liveSync:true,periodicReplication:false, + batchSave:${mode === "ignored-batch-setting"},batchSaveInLiveSync:${mode === "enabled-batch"}, + batchSaveMinimumDelay:${minimum},batchSaveMaximumDelay:${maximum},deleteMetadataOfDeletedFiles:false}); + return {liveSync:settings.liveSync,batchSave:settings.batchSave, + batchSaveInLiveSync:settings.batchSaveInLiveSync,shouldBatchSave:manager.shouldBatchSave}; + ` + ); +} + +async function snapshot(page: Page, path: string): Promise { + return await evaluate( + page, + ` + const path=${JSON.stringify(path)}; + const id=await core.services.path.path2id(path); + const row=(await core.localDatabase.allDocsRaw({keys:[id],include_docs:true})).rows[0]; + let leaves=[]; + if(row?.value?.rev) { + const rawLeaves=await core.localDatabase.getRaw(id,{open_revs:'all',revs:true}); + for(const item of rawLeaves) { + if(!item.ok) continue; + const raw=item.ok; + let body=null; + if(!raw._deleted) { + const entry=await core.localDatabase.getDBEntryFromMeta(raw,false,false,true); + if(entry!==false) body=Array.isArray(entry.data)?entry.data.join(''):entry.data; + } + leaves.push({rev:raw._rev,logicalDeleted:!!raw.deleted,pouchDeleted:!!raw._deleted,body, + ancestry:raw._revisions?.ids.map((hash,i)=>(raw._revisions.start-i)+'-'+hash)??[]}); + } + } + const file=app.vault.getAbstractFileByPath(path); + return {path,rev:row?.value?.rev??null,exists:!!file?.stat, + fileMtime:file?.stat?.mtime??null,databaseMtime:row?.doc?.mtime??null, + content:file?.stat?await app.vault.read(file):null,leaves, + waiting:[...manager._waitingMap].map(([key,value])=>({path:key,type:value.type,since:value.since})), + queued:manager.bufferedQueuedItems.map(item=>({path:item.args?.file?.path??null,type:item.type})), + shouldBatchSave:!!manager.shouldBatchSave}; + ` + ); +} + +function liveLeaves(state: Snapshot) { + return state.leaves.filter((leaf) => !leaf.logicalDeleted && !leaf.pouchDeleted); +} + +async function waitFor( + page: Page, + path: string, + predicate: (state: Snapshot) => boolean, + timeout = 20000 +): Promise { + const deadline = Date.now() + timeout; + let state: Snapshot; + do { + state = await snapshot(page, path); + if (predicate(state)) return state; + await pause(100); + } while (Date.now() < deadline); + throw new Error(`State did not settle: ${JSON.stringify(state)}`); +} + +async function settled(pair: Pair, path: string, body: string): Promise<[Snapshot, Snapshot]> { + const ready = (state: Snapshot) => + state.content === body && + liveLeaves(state).length === 1 && + liveLeaves(state)[0].body === body && + state.leaves.length === 1 && + !state.waiting.some((item) => item.path === path); + const deadline = Date.now() + 20000; + let states: [Snapshot, Snapshot]; + do { + states = await Promise.all([snapshot(pair.a, path), snapshot(pair.b, path)]); + // The row, leaves, and storage are read asynchronously. Require each + // sample to agree internally, then compare both current devices. + if ( + states.every((state) => ready(state) && state.rev === liveLeaves(state)[0].rev) && + states[0].rev === states[1].rev + ) + return states; + await pause(100); + } while (Date.now() < deadline); + throw new Error(`Devices did not converge: ${JSON.stringify(states)}`); +} + +async function create(page: Page, path: string, body: string): Promise { + await evaluate(page, `await app.vault.create(${JSON.stringify(path)},${JSON.stringify(body)});`); +} + +async function modify(page: Page, path: string, body: string): Promise { + await evaluate( + page, + `await app.vault.modify(app.vault.getFileByPath(${JSON.stringify(path)}),${JSON.stringify(body)});` + ); +} + +async function trace(page: Page, path: string) { + return await evaluate( + page, + `return { + events:globalThis.__issue1207Batch.events.filter(item=>item.path===${JSON.stringify(path)}||item.oldPath===${JSON.stringify(path)}), + changes:globalThis.__issue1207Batch.changes.filter(item=>item.path===${JSON.stringify(path)}||item.error)};` + ); +} + +async function main(): Promise { + await mkdir(diagnostics, { recursive: true }); + const binary = requireObsidianBinary(); + const cli = discoverObsidianCli(); + check(cli.binary, "Missing Obsidian CLI"); + const cliBinary = cli.binary; + const config = await loadCouchDbConfig(process.env.ISSUE1207_COUCHDB_ENV); + check(["127.0.0.1", "localhost"].includes(new URL(config.uri).hostname), "Use a disposable local CouchDB"); + await assertCouchDbReachable(config); + const dbName = makeUniqueDatabaseName(config.dbPrefix, "live-batch"); + const couch = { ...config, dbName }; + const overrides = { + doctorProcessedVersion: "1.0.0", + notifyThresholdOfRemoteStorageSize: -1, + liveSync: false, + periodicReplication: false, + batchSave: false, + batchSaveInLiveSync: false, + syncOnStart: false, + syncOnSave: false, + syncOnFileOpen: false, + syncOnEditorSave: false, + syncAfterMerge: false, + checkConflictOnlyOnOpen: true, + showMergeDialogOnlyOnActive: true, + disableMarkdownAutoMerge: true, + resolveConflictsByNewerFile: false, + writeDocumentsIfConflicted: false, + writeLogToTheFile: false, + }; + const before = hashes(); + evidence.artefactsBefore = before; + evidence.manifest = JSON.parse(await readFile(join(artifactRoot, "manifest.json"), "utf8")); + evidence.commonlibVersion = JSON.parse( + await readFile("node_modules/@vrtmrz/livesync-commonlib/package.json", "utf8") + ).version; + evidence.sourceBaseRevision = "73a14d4e4b0b414a1acfe170d01500abb3dca2e9"; + evidence.unreleasedImplementation = true; + if (process.env.ISSUE1207_COMMONLIB_ARTEFACT) { + evidence.commonlibArtefactSha256 = createHash("sha256") + .update(new Uint8Array(readFileSync(process.env.ISSUE1207_COMMONLIB_ARTEFACT))) + .digest("hex"); + } + evidence.timing = { minimumSeconds: 2, maximumSeconds: 6, incomingAndManualSeconds: { minimum: 5, maximum: 60 } }; + const slots: Slot[] = []; + let ownedDatabase = false; + + function redact(value: unknown): string { + return JSON.stringify(value, null, 2) + .split(config.password) + .join("") + .split(Buffer.from(`${config.username}:${config.password}`).toString("base64")) + .join(""); + } + + async function start(slot: Slot, restart = false) { + console.log(JSON.stringify({ phase: "starting", device: slot.name, restart })); + slot.session = await startObsidianLiveSyncSession({ + binary, + cliBinary, + artifactRoot, + vault: slot.vault, + startupGraceMs: 1000, + pluginData: restart ? undefined : createE2eCouchDbPluginData(couch, overrides), + localStorageEntries: createE2eObsidianDeviceLocalState(slot.vault.name), + env: { E2E_OBSIDIAN_REMOTE_DEBUGGING_PORT: String(await availablePort()) }, + lifecycle: { + beforeLaunch: async () => { + for (const registry of [ + join(slot.vault.homePath, ".config", "obsidian", "obsidian.json"), + join(slot.vault.homePath, "Library", "Application Support", "obsidian", "obsidian.json"), + join(slot.vault.xdgConfigPath, "obsidian", "obsidian.json"), + join(slot.vault.userDataPath, "obsidian.json"), + ]) { + const state = JSON.parse(await readFile(registry, "utf8")); + state.updateDisabled = true; + await writeFile(registry, JSON.stringify(state)); + } + }, + }, + }); + console.log( + JSON.stringify({ + phase: "device-started", + device: slot.name, + port: slot.session.remoteDebuggingPort, + vault: slot.vault.path, + }) + ); + await waitForLiveSyncCoreReady(cliBinary, slot.session.cliEnv); + await assertE2eCompatibilityUnpaused(cliBinary, slot.session.cliEnv, slot.session.remoteDebuggingPort); + if (!restart) { + await configureCouchDb(cliBinary, slot.session.cliEnv, couch, overrides); + await waitForLiveSyncCoreReady(cliBinary, slot.session.cliEnv); + await prepareRemote(cliBinary, slot.session.cliEnv); + } + } + + async function stop(slot: Slot) { + if (!slot.session) return; + await slot.session.app.stop(); + await writeFile( + join(diagnostics, `process-${slot.name}-${slot.session.remoteDebuggingPort}.json`), + redact(slot.session.app.output()) + ); + slot.session = undefined; + } + + async function withPair(callback: (pair: Pair) => Promise): Promise { + check(slots[0].session && slots[1].session, "Both devices must be running"); + return await withObsidianPage( + slots[0].session.remoteDebuggingPort, + async (a) => + await withObsidianPage(slots[1].session!.remoteDebuggingPort, async (b) => { + await installProbe(a); + await installProbe(b); + return await callback({ a, b }); + }) + ); + } + + async function run(name: string, body: () => Promise) { + if (process.env.ISSUE1207_CASE_FILTER && !new RegExp(process.env.ISSUE1207_CASE_FILTER).test(name)) return; + console.log(JSON.stringify({ phase: "case-start", name })); + try { + const result = await body(); + cases.push({ name, passed: true, result }); + console.log(JSON.stringify({ phase: "case-passed", name })); + } catch (error) { + const failure = error instanceof Error ? error.stack : String(error); + cases.push({ name, passed: false, error: failure }); + console.log(JSON.stringify({ phase: "case-failed", name, error: failure })); + await withPair(async (pair) => { + let flushFailure: unknown; + for (const [device, page] of Object.entries(pair)) { + observations.push({ + name, + device, + probe: await evaluate( + page, + `return { + events:globalThis.__issue1207Batch.events,changes:globalThis.__issue1207Batch.changes, + pending:(await manager.adapter.persistence.loadSnapshot()).map(item=>({ + type:item.type,path:item.args?.file?.path,oldPath:item.args?.oldPath}))};` + ), + }); + await page.screenshot({ path: join(diagnostics, `${name}-${device}-failure.png`) }); + try { + await evaluate(page, "await manager.waitForIdle();"); + } catch (error) { + observations.push({ name, device, flushFailure: String(error) }); + flushFailure ??= error; + } + } + if (flushFailure) throw flushFailure; + }); + await pause(500); + } + await writeFile(join(diagnostics, "evidence.json"), redact(evidence)); + } + + try { + await createCouchDbDatabase(config, dbName); + ownedDatabase = true; + for (const name of ["a", "b"]) { + const slot = { name, vault: await createTemporaryVault(`issue1207-live-batch-${name}-`) }; + slots.push(slot); + await start(slot); + } + evidence.runtime = await withPair(async (pair) => { + const versions = []; + for (const page of [pair.a, pair.b]) { + versions.push( + await evaluate( + page, + `let appVersion=null; + try {appVersion=JSON.parse(require('fs').readFileSync(require('path').join(process.resourcesPath,'app.asar','package.json'),'utf8')).version;} catch {} + return {appVersion,electronVersion:process.versions.electron, + pluginVersion:app.plugins.plugins['obsidian-livesync'].manifest.version};` + ) + ); + await options(page, "normal"); + const outcome = await evaluate( + page, + `return await core.services.replication.startContinuous({ + trigger:'daemon',interaction:{kind:'forbidden'}});` + ); + check( + (outcome as { status?: string })?.status === "completed", + `Continuous start failed: ${JSON.stringify(outcome)}` + ); + } + return versions; + }); + + evidence.connectivity = await withPair(async (pair) => { + const path = `${prefix}connectivity.md`; + await create(pair.a, path, "Connectivity probe\n"); + const first = await settled(pair, path, "Connectivity probe\n"); + await modify(pair.b, path, "Bidirectional connectivity probe\n"); + const second = await settled(pair, path, "Bidirectional connectivity probe\n"); + return { first, second }; + }); + + for (const mode of ["normal", "ignored-batch-setting", "enabled-batch"] as const) { + for (let repeat = 0; repeat < 2; repeat++) { + await run(`${mode}-coalesce-${repeat}`, () => + withPair(async (pair) => { + for (const page of [pair.a, pair.b]) await options(page, mode); + const source = repeat === 0 ? pair.a : pair.b; + const receiver = repeat === 0 ? pair.b : pair.a; + const path = `${prefix}${mode}-edits-${repeat}.md`; + await create(source, path, "Initial body\n"); + const initial = (await settled(pair, path, "Initial body\n"))[0]; + for (let edit = 0; edit < 6; edit++) { + await modify(source, path, `Edit ${edit}\n`); + await pause(150); + } + const during = await snapshot(source, path); + const final = await settled(pair, path, "Edit 5\n"); + const finalSource = repeat === 0 ? final[0] : final[1]; + const delta = Number(finalSource.rev?.split("-")[0]) - Number(initial.rev?.split("-")[0]); + if (mode === "enabled-batch") { + check( + during.waiting.some((item) => item.path === path), + "Batch did not retain the pending edit" + ); + check(delta === 1, `Repeated edits created ${delta} revisions`); + } else { + check(!during.shouldBatchSave, "LiveSync batching was enabled without an explicit choice"); + check(delta >= 3, `The control did not expose enough edits: ${delta}`); + } + return { + path, + initial, + during, + final, + revisionDelta: delta, + source: await trace(source, path), + receiver: await trace(receiver, path), + }; + }) + ); + } + + await run(`${mode}-template-lifecycle`, () => + withPair(async (pair) => { + for (const page of [pair.a, pair.b]) await options(page, mode); + const oldPath = `${prefix}${mode}-untitled.md`; + const path = `${prefix}${mode}-completed.md`; + await create(pair.a, oldPath, ""); + await pause(300); + await modify(pair.a, oldPath, "Template in progress\n"); + await pause(300); + await evaluate( + pair.a, + `await app.fileManager.renameFile(app.vault.getFileByPath(${JSON.stringify(oldPath)}),${JSON.stringify(path)});` + ); + await pause(300); + await modify(pair.a, path, "Completed template\n"); + const during = await snapshot(pair.a, path); + const final = await settled(pair, path, "Completed template\n"); + await waitFor(pair.b, oldPath, (state) => !state.exists && liveLeaves(state).length === 0); + return { + oldPath, + path, + during, + final, + oldSource: await snapshot(pair.a, oldPath), + oldReceiver: await snapshot(pair.b, oldPath), + source: await trace(pair.a, path), + receiver: await trace(pair.b, path), + oldSourceTrace: await trace(pair.a, oldPath), + oldReceiverTrace: await trace(pair.b, oldPath), + }; + }) + ); + + for (const physical of [false, true]) { + await run(`${mode}-recreate-${physical}`, () => + withPair(async (pair) => { + for (const page of [pair.a, pair.b]) { + await options(page, mode); + await evaluate( + page, + `core.services.setting.currentSettings().deleteMetadataOfDeletedFiles=${physical};` + ); + } + const path = `${prefix}${mode}-recreate-${physical}.md`; + await create(pair.a, path, "Transient body\n"); + await pause(400); + await evaluate( + pair.a, + `await app.vault.delete(app.vault.getFileByPath(${JSON.stringify(path)}),true);` + ); + await pause(400); + await create(pair.a, path, "Recreated body\n"); + const final = await settled(pair, path, "Recreated body\n"); + return { + physical, + final, + source: await trace(pair.a, path), + receiver: await trace(pair.b, path), + }; + }) + ); + } + } + + for (const incoming of ["identical", "different", "delete"] as const) { + await run(`enabled-pending-incoming-${incoming}`, () => + withPair(async (pair) => { + await options(pair.a, "enabled-batch", 5, 60); + await options(pair.b, "normal", 5, 60); + const path = `${prefix}pending-${incoming}.md`; + await create(pair.b, path, "Shared base\n"); + const initial = await settled(pair, path, "Shared base\n"); + const local = "Local edit\n"; + await modify(pair.a, path, local); + const pending = await waitFor(pair.a, path, (state) => + state.waiting.some((item) => item.path === path) + ); + check(pending.rev === initial[0].rev, "Local edit reached the DB before incoming delivery"); + const sentAt = Date.now(); + if (incoming === "delete") { + await evaluate( + pair.b, + `await app.vault.delete(app.vault.getFileByPath(${JSON.stringify(path)}),true);` + ); + } else await modify(pair.b, path, incoming === "identical" ? local : "Remote edit\n"); + const arrived = await waitFor( + pair.a, + path, + (state) => + incoming === "identical" + ? state.rev !== initial[0].rev && liveLeaves(state).some((leaf) => leaf.body === local) + : state.leaves.some((leaf) => leaf.body === local) && + state.leaves.some((leaf) => + incoming === "delete" ? leaf.logicalDeleted : leaf.body === "Remote edit\n" + ), + 4000 + ); + check( + arrived.waiting.some((item) => item.path === path), + "The incoming assertion did not occur during Batch waiting" + ); + const leavesBefore = arrived.leaves.map((leaf) => leaf.rev).sort(); + if (incoming !== "identical") { + check(arrived.content === local, "Incoming reflection overwrote the pending local edit"); + check(arrived.leaves.length === 2, "Expected exactly two genuine concurrent branches"); + check( + arrived.leaves.every((leaf) => leaf.ancestry[1] === initial[0].rev), + "Concurrent branches lost their shared base" + ); + } + await pause(5600); + const after = await snapshot(pair.a, path); + const peer = await snapshot(pair.b, path); + check(!after.waiting.some((item) => item.path === path), "Batch timer did not complete"); + check( + JSON.stringify(after.leaves.map((leaf) => leaf.rev).sort()) === JSON.stringify(leavesBefore), + "The delayed event created another revision" + ); + check( + JSON.stringify(peer.leaves.map((leaf) => leaf.rev).sort()) === JSON.stringify(leavesBefore), + "Both branches did not converge to the peer" + ); + if (incoming === "identical") + check(after.leaves.length === 1, "Identical content created a branch"); + return { initial, pending, arrived, after, peer, incomingLatencyMs: Date.now() - sentAt - 5600 }; + }) + ); + } + + await run("enabled-maximum-delay", () => + withPair(async (pair) => { + await options(pair.a, "enabled-batch", 2, 6); + await options(pair.b, "normal"); + const path = `${prefix}maximum.md`; + await create(pair.b, path, "Initial maximum body\n"); + const initial = await settled(pair, path, "Initial maximum body\n"); + const started = Date.now(); + let finalBody = ""; + for (let edit = 0; edit < 26; edit++) { + finalBody = `Continuous edit ${edit}\n`; + await modify(pair.a, path, finalBody); + await pause(350); + } + const stopped = Date.now(); + const final = await settled(pair, path, finalBody); + const changes = await evaluate>( + pair.a, + `return globalThis.__issue1207Batch.changes.filter(item=>item.path===${JSON.stringify(path)}&&item.rev!==${JSON.stringify(initial[0].rev)});` + ); + check(changes.length >= 2, "Continuous edits did not produce bounded intermediate and final saves"); + const first = changes[0].at - started; + check(first >= 5000 && first <= 8000, `Unexpected first maximum-delay flush: ${first}ms`); + check(changes[0].at < stopped - 1000, "The maximum flush only occurred after typing stopped"); + return { started, stopped, firstSaveMs: first, changes, final }; + }) + ); + + await run("enabled-many-paths", () => + withPair(async (pair) => { + await options(pair.a, "enabled-batch", 2, 6); + await options(pair.b, "normal"); + const started = Date.now(); + for (let index = 0; index < 12; index++) + await create(pair.a, `${prefix}many-${index}.md`, `File ${index}\n`); + const final = []; + for (let index = 0; index < 12; index++) + final.push(await settled(pair, `${prefix}many-${index}.md`, `File ${index}\n`)); + return { elapsedMs: Date.now() - started, final }; + }) + ); + + await run("enabled-manual-sync", () => + withPair(async (pair) => { + await options(pair.a, "enabled-batch", 5, 60); + await options(pair.b, "normal"); + const path = `${prefix}manual.md`; + await create(pair.b, path, "Before manual sync\n"); + await settled(pair, path, "Before manual sync\n"); + await modify(pair.a, path, "After manual sync\n"); + const pending = await waitFor(pair.a, path, (state) => + state.waiting.some((item) => item.path === path) + ); + const started = Date.now(); + const outcome = await evaluate(pair.a, "return await core.services.replication.replicate(true);"); + const atReturn = await snapshot(pair.a, path); + const final = await settled(pair, path, "After manual sync\n"); + const elapsedMs = Date.now() - started; + check(elapsedMs < 4000, `Manual sync waited for the Batch timer: ${elapsedMs}ms`); + return { pending, outcome, atReturn, final, elapsedMs }; + }) + ); + + const saveAfterEdit = process.env.ISSUE1207_RESTART_SAVE_AFTER_EDIT === "true"; + const restartBucket = process.env.ISSUE1207_RESTART_BUCKET; + check( + !restartBucket || ["same", "exact", "different"].includes(restartBucket), + "Unknown restart timestamp condition" + ); + const restartName = restartBucket + ? `enabled-inflight-restart-${restartBucket}` + : saveAfterEdit + ? "enabled-settings-save-restart" + : "enabled-restart"; + await run(restartName, async () => { + const path = `${prefix}restart.md`; + const beforeRestart = await withPair(async (pair) => { + await options(pair.a, "enabled-batch", 5, 60); + await options(pair.b, "normal"); + if (!saveAfterEdit) + await evaluate( + pair.a, + "await core.services.setting.saveSettingData(); await core.services.control.applySettings();" + ); + await create(pair.b, path, "Before restart\n"); + const initial = await settled(pair, path, "Before restart\n"); + if (restartBucket) { + check(initial[0].databaseMtime !== null, "Missing initial database mtime"); + const mtime = + restartBucket === "exact" + ? initial[0].databaseMtime + : Math.floor(initial[0].databaseMtime / 2000) * 2000 + + (restartBucket === "same" ? 1000 : 3000); + await evaluate( + pair.a, + `const original=manager.requestProcessQueue.bind(manager); + manager.requestProcessQueue=async item=>{ + if(item.args?.file?.path===${JSON.stringify(path)}) { + globalThis.__issue1207Batch.inflightPaused=true; + await new Promise(resolve=>setTimeout(resolve,30000)); + } + return await original(item); + }; + await app.vault.modify(app.vault.getFileByPath(${JSON.stringify(path)}),"Retained after restart\\n",{mtime:${mtime}});` + ); + } else await modify(pair.a, path, "Retained after restart\n"); + const pending = await waitFor(pair.a, path, (state) => + state.waiting.some((item) => item.path === path) + ); + if (saveAfterEdit) await evaluate(pair.a, "await core.services.setting.saveSettingData();"); + if (restartBucket) + await evaluate( + pair.a, + `const deadline=Date.now()+3000; + while(!globalThis.__issue1207Batch.inflightPaused&&Date.now()setTimeout(resolve,10)); + if(!globalThis.__issue1207Batch.inflightPaused) throw new Error('The in-flight test gate did not start');` + ); + await evaluate(pair.a, "await manager._takeSnapshot();"); + check(pending.rev === initial[0].rev, "Restart test did not retain a pending edit"); + if (restartBucket) { + const inFlight = await snapshot(pair.a, path); + const saved = await evaluate>( + pair.a, + `return (await manager.adapter.persistence.loadSnapshot()).map(item=>({path:item.args?.file?.path}));` + ); + check( + inFlight.rev === initial[0].rev && !inFlight.waiting.some((item) => item.path === path), + "The in-flight fixture did not stop between gate release and database storage" + ); + check( + saved.some((item) => item.path === path), + "The in-flight event was lost from the recoverable snapshot" + ); + check(inFlight.fileMtime !== null && inFlight.databaseMtime !== null, "Missing in-flight mtimes"); + const sameBucket = + Math.floor(inFlight.fileMtime / 2000) === Math.floor(inFlight.databaseMtime / 2000); + check(sameBucket === (restartBucket !== "different"), "The timestamp condition was not applied"); + if (restartBucket === "exact") + check(inFlight.fileMtime === inFlight.databaseMtime, "Raw mtimes differ"); + observations.push({ + phase: "inflight-before-stop", + restartBucket, + pauseBeforeDatabaseMs: 30000, + inFlight, + saved, + sameBucket, + }); + } + if (!saveAfterEdit && !restartBucket) { + const strictState = await snapshot(pair.a, path); + const saved = await evaluate>( + pair.a, + `return (await manager.adapter.persistence.loadSnapshot()).map(item=>({path:item.args?.file?.path}));` + ); + check( + strictState.rev === initial[0].rev && strictState.waiting.some((item) => item.path === path), + "Restart fixture flushed before shutdown" + ); + check( + saved.some((item) => item.path === path), + "Restart fixture did not persist the pending event" + ); + observations.push({ + phase: "restart-verified-pending-snapshot", + strictState, + saved, + initialized: await evaluate(pair.a, "return await core.kvDB.get('initialized');"), + }); + } + if (!saveAfterEdit) + observations.push({ + phase: "restart-before-stop", + initial, + pending, + persisted: await evaluate( + pair.a, + `const events=await manager.adapter.persistence.loadSnapshot(); + const settings=core.services.setting.currentSettings(); + return {initialized:await core.kvDB.get('initialized'),restorationRequested:manager.snapShotRestored!==null, + settings:{isConfigured:settings.isConfigured,liveSync:settings.liveSync,batchSave:settings.batchSave,batchSaveInLiveSync:settings.batchSaveInLiveSync, + suspendFileWatching:settings.suspendFileWatching}, + events:events?.map(item=>({type:item.type,path:item.args?.file?.path, + stat:item.args?.file?.stat,skipBatchWait:item.skipBatchWait}))};` + ), + }); + return { initial, pending }; + }); + await stop(slots[0]); + const persistedSettings = JSON.parse( + await readFile(join(slots[0].vault.path, ".obsidian/plugins/obsidian-livesync/data.json"), "utf8") + ); + observations.push({ + phase: "restart-on-disk-settings", + settings: Object.fromEntries( + [ + "isConfigured", + "liveSync", + "batchSave", + "batchSaveInLiveSync", + "suspendFileWatching", + "settingVersion", + ].map((key) => [key, persistedSettings[key]]) + ), + }); + await start(slots[0], true); + const afterRestart = await withPair(async (pair) => { + observations.push({ + phase: "restart-after-boot", + state: await snapshot(pair.a, path), + persisted: await evaluate( + pair.a, + `const events=await manager.adapter.persistence.loadSnapshot(); + const settings=core.services.setting.currentSettings(); + return {initialized:await core.kvDB.get('initialized'),restorationRequested:manager.snapShotRestored!==null, + settings:{isConfigured:settings.isConfigured,liveSync:settings.liveSync,batchSave:settings.batchSave,batchSaveInLiveSync:settings.batchSaveInLiveSync, + suspendFileWatching:settings.suspendFileWatching}, + events:events?.map(item=>({type:item.type,path:item.args?.file?.path,stat:item.args?.file?.stat}))};` + ), + }); + const result = await settled(pair, path, "Retained after restart\n"); + check(result[0].shouldBatchSave, "The saved LiveSync batching choice was not restored"); + return { result, restoredGuard: result[0].shouldBatchSave }; + }); + return { beforeRestart, afterRestart }; + }); + + await run("enabled-after-write-restart", async () => { + const path = `${prefix}after-write.md`; + const saved = await withPair(async (pair) => { + await options(pair.a, "enabled-batch", 1, 3); + await options(pair.b, "normal"); + await evaluate( + pair.a, + "await core.services.setting.saveSettingData(); await core.services.control.applySettings();" + ); + await create(pair.b, path, "Before the durable write\n"); + const initial = await settled(pair, path, "Before the durable write\n"); + await evaluate( + pair.a, + ` + const original=manager.requestProcessQueue.bind(manager); + manager.requestProcessQueue=async item=>{ + await original(item); + if(item.args?.file?.path===${JSON.stringify(path)}) { + globalThis.__issue1207Batch.afterWritePaused=true; + await new Promise(resolve=>setTimeout(resolve,30000)); + } + }; + await app.vault.modify(app.vault.getFileByPath(${JSON.stringify(path)}),"Durable before restart\\n"); + const deadline=Date.now()+6000; + while(!globalThis.__issue1207Batch.afterWritePaused&&Date.now()setTimeout(resolve,20)); + if(!globalThis.__issue1207Batch.afterWritePaused) throw new Error('The after-write gate did not start'); + await manager._takeSnapshot(); + ` + ); + const durable = await snapshot(pair.a, path); + const pending = await evaluate>( + pair.a, + "return (await manager.adapter.persistence.loadSnapshot()).map(item=>({path:item.args?.file?.path}));" + ); + check(durable.rev !== initial[0].rev, "The database write did not complete before interruption"); + check( + pending.some((item) => item.path === path), + "The after-write operation was not recoverable" + ); + return { initial, durable, pending }; + }); + const restarts = []; + for (let attempt = 0; attempt < 2; attempt++) { + await stop(slots[0]); + await start(slots[0], true); + const final = await withPair((pair) => settled(pair, path, "Durable before restart\n")); + check(final[0].rev === saved.durable.rev, "Replay created a duplicate database revision"); + restarts.push(final); + } + const later = await withPair(async (pair) => { + await modify(pair.a, path, "A later edit survives\n"); + return await settled(pair, path, "A later edit survives\n"); + }); + return { saved, restarts, later }; + }); + + await run("enabled-before-snapshot-restart-exact", async () => { + const path = `${prefix}before-snapshot.md`; + const beforeStop = await withPair(async (pair) => { + await options(pair.a, "enabled-batch", 5, 60); + await options(pair.b, "normal"); + await evaluate( + pair.a, + "await core.services.setting.saveSettingData(); await core.services.control.applySettings();" + ); + await create(pair.b, path, "Before the first snapshot\n"); + const initial = await settled(pair, path, "Before the first snapshot\n"); + check(initial[0].databaseMtime !== null, "Missing initial database mtime"); + await evaluate( + pair.a, + ` + await manager.waitForIdle(); + const persistence=manager.adapter.persistence; + const original=persistence.saveSnapshot.bind(persistence); + persistence.saveSnapshot=async events=>{ + if(events.some(item=>item.type==='CHANGED'&&item.args?.file?.path===${JSON.stringify(path)})) { + globalThis.__issue1207Batch.beforeSnapshotPaused=true; + await new Promise(resolve=>setTimeout(resolve,30000)); + } + return await original(events); + }; + await app.vault.modify(app.vault.getFileByPath(${JSON.stringify(path)}), + "Recovered without an event snapshot\\n",{mtime:${initial[0].databaseMtime}}); + const deadline=Date.now()+3000; + while(!globalThis.__issue1207Batch.beforeSnapshotPaused&&Date.now()setTimeout(resolve,10)); + if(!globalThis.__issue1207Batch.beforeSnapshotPaused) + throw new Error('The snapshot persistence gate did not start'); + ` + ); + const state = await snapshot(pair.a, path); + const persisted = await evaluate>( + pair.a, + "return (await manager.adapter.persistence.loadSnapshot()??[]).map(item=>({path:item.args?.file?.path}));" + ); + check(state.rev === initial[0].rev, "The unsaved edit reached the database before interruption"); + check(state.fileMtime === state.databaseMtime, "Raw mtimes differ before interruption"); + check(!persisted.some((item) => item.path === path), "The interrupted edit already has a saved event"); + return { initial, state, persisted }; + }); + await stop(slots[0]); + await start(slots[0], true); + const recovered = await withPair((pair) => settled(pair, path, "Recovered without an event snapshot\n")); + check(recovered[0].shouldBatchSave, "The saved LiveSync batching choice was not restored"); + return { beforeStop, recovered }; + }); + + evidence.artefactsAfter = hashes(); + check( + JSON.stringify(before) === JSON.stringify(evidence.artefactsAfter), + "Plug-in artefacts changed during testing" + ); + evidence.passed = cases.every((item) => item.passed); + await writeFile(join(diagnostics, "evidence.json"), redact(evidence)); + console.log( + JSON.stringify({ + phase: "completed", + passed: evidence.passed, + cases: cases.length, + failed: cases.filter((item) => !item.passed).map((item) => item.name), + }) + ); + if (!evidence.passed) process.exitCode = 1; + } finally { + for (const slot of slots) await stop(slot); + for (const slot of slots) await slot.vault.dispose(); + if (ownedDatabase) await deleteCouchDbDatabase(config, dbName); + evidence.cleanup = { devicesStopped: slots.every((slot) => !slot.session), databaseRemoved: ownedDatabase }; + await writeFile(join(diagnostics, "evidence.json"), redact(evidence)); + } +} + +main().catch((error: unknown) => { + console.error(error instanceof Error ? error.stack : error); + process.exitCode = 1; +}); diff --git a/updates.md b/updates.md index cefb2bce..e3e425e5 100644 --- a/updates.md +++ b/updates.md @@ -12,6 +12,21 @@ Earlier releases remain available in the 1.0 release history, the 1.0 preview hi ## Unreleased +### Synchronisation + +#### New Feature + +- We can enable **Batch database updates in LiveSync mode** to combine repeated local edits before sending them to other devices. + - The choice starts disabled, including when older settings contain the previously ignored batch option. Each mode retains its own choice and uses the same minimum and maximum delays. + - Batching delays remote visibility and reduces intermediate updates. It does not establish when a template has finished or prevent genuine concurrent edits. + +#### Fixed + +- Local changes remain available for recovery while their database save is running, including when the application stops after a batch delay ends. + - Manual synchronisation and settings application now wait for the accepted local saves to finish and report failures. Pending events retain the existing snapshot format. +- Start-up reconciliation checks content when modification times are equal, so an edit within the same timestamp interval is not mistaken for an unchanged file. + - Existing revision provenance preserves the correct edit branch. Unreadable or continually changing content remains unresolved rather than being reported as successfully reconciled. + ## 1.0.40 24th October, 2026