diff --git a/docs/adr/2026_09_customisation_and_hidden_file_sync_ownership.md b/docs/adr/2026_09_customisation_and_hidden_file_sync_ownership.md index 0233b7b0..01393388 100644 --- a/docs/adr/2026_09_customisation_and_hidden_file_sync_ownership.md +++ b/docs/adr/2026_09_customisation_and_hidden_file_sync_ownership.md @@ -20,12 +20,13 @@ one writer for raw events, scheduled scans, and automatic database-to-local reflection. The constructor-name add-on lookup and the `ConfigSync` and `HiddenFileSync` add-on identities have been retired. -The Customisation Sync private context owns its catalogue, progress, manifest, -queue, and periodic state. The Hidden File Sync private context owns its -processed-state caches, reconciliation and conflict queues, activity counts, -and periodic state. Each receives live settings and database projections, -focused storage, path, and exact-revision capabilities, and explicit host -effects rather than `LiveSyncCore`. +The Customisation Sync private context coordinates its snapshot operations, +scan queues, periodic work, and focused transient-state owners. The Hidden File +Sync private context coordinates reconciliation, periodic work, and the +lifetimes of focused processed-state, change-processing, and +conflict-resolution owners. Each receives live settings and database +projections, focused storage, path, and exact-revision capabilities, and +explicit host effects rather than `LiveSyncCore`. The corresponding implemented topology is documented in [Optional-file synchronisation architecture](../design_docs/optional_file_sync_architecture.md). @@ -105,7 +106,7 @@ The Customisation Sync runtime will own: operations; and - the state required to serialise and report those operations. -The Hidden File Sync runtime will own: +The Hidden File Sync runtime and its composed focused owners will own: - the device-local processed-state records; - one-file storage and database transfer, including exact-revision repair; @@ -194,15 +195,44 @@ caches, processed-state records, recent-event records, locks, semaphores, queues, periodic processors, Notices, and event subscriptions must be created and disposed by their feature context or a focused resource owner. +The contexts are orchestration roots, rather than containers for every mutable +detail. `HiddenFileSyncProcessedState` owns the three device-local maps, their +autosave initialisation, exact key formats, retained known mtime, reset rules, +and settlement effects on `IPathService`. Database write and extraction +operations receive this capability through one narrow `processedState` port; +they do not receive bundles of individual state callbacks or implement state +keys themselves. + +`CustomisationSyncCatalogueState` separately owns the transient catalogue +rows, manifest lookup and mtime cache, their reactive stores, and catalogue +update progress. A small recent-event deduplicator owns raw-event admission +history. These Customisation Sync owners are deliberately not implementations +of a shared Hidden File Sync state abstraction: the former state is a derived, +in-memory projection, while Hidden File Sync markers are persisted operational +reconciliation state with different identity, invalidation, and deletion +rules. + +`HiddenFileSyncChangeProcessor` owns storage and database change processing, +the bounded semaphore, same-path event serialisation, activity counts, and the +inherited order in which processed-state markers and transfer results settle. +This boundary keeps event concurrency and settlement directly testable without +giving the processor full scan, initialisation, notification, or host +responsibilities. + The joint composition may return several views backed by those contexts: - a Customisation Sync catalogue and operation view for its dialogue; -- a Hidden File Sync initialisation view for settings workflows; and -- a Hidden File Sync repair view for the Hatch pane. +- a Hidden File Sync initialisation view for settings workflows; +- a Hidden File Sync repair view for the Hatch pane; +- immutable semantic handler views for registration by the joint composition; + and +- explicitly internal testing views for maintained real-Obsidian workflows. Several views over one context do not create several owners. Views expose -stable application data and operations rather than PouchDB entries, queue -objects, mutable settings records, or the complete core. +stable application data and named operations rather than PouchDB entries, +queue objects, mutable settings records, dependency objects, or the complete +core. The testing views also avoid exposing context instances or writable +internal state; time-sensitive E2E work uses a scoped operation interceptor. Obsidian commands, ribbon actions, dialogues, Notices, plug-in reloads, and restart scheduling will remain in host-owned composition. The UI will receive @@ -231,10 +261,10 @@ serviceFeature. `LiveSyncBaseCore.getAddOn()` and its constructor-name lookup have been removed. Production code consumes focused views and does not use the broad context -surface. Maintained real-Obsidian E2E workflows use an explicitly internal -test view exposed by the composed feature. This is a transitional test seam, -not a production service locator, and it should be narrowed as those workflows -move to public operations or commands. +surface. Maintained real-Obsidian E2E workflows use explicitly internal, +immutable test views exposed by the composed feature. These are transitional +test seams, not production service locators, and they should be narrowed as +those workflows move to public operations or commands. The retirement was gated on: @@ -328,11 +358,13 @@ to the non-owner. - Replace the complete-core dependency with narrow dependencies. The private context, path module, codec module, focused presentation view, and -resource teardown are implemented. Catalogue, enumeration, migration, scan, -and manifest state is owned per context instance. The context accepts only -narrow, live projections and explicit effects; an Obsidian adapter at the -composition edge owns dialogues, Notices, plug-in reload, restart, lifecycle, -Vault access, and compatibility scan telemetry. +resource teardown are implemented. A focused catalogue-state owner holds the +rows, manifests, manifest mtime cache, reactive stores, and update progress, +while a bounded deduplicator owns recent raw-event keys. Enumeration and scan +queues remain with the orchestrating context. The context accepts only narrow, +live projections and explicit effects; an Obsidian adapter at the composition +edge owns dialogues, Notices, plug-in reload, restart, lifecycle, Vault access, +and compatibility scan telemetry. ### Stage 5: extract the Hidden File Sync runtime — implemented @@ -343,16 +375,26 @@ Vault access, and compatibility scan telemetry. The private context, focused initialisation, repair, and command views, host-owned command registration, and processor, cache, subscription, and -Notice teardown are implemented. The context owns transfer, reconciliation, -conflict, and processed-state behaviour through narrow live dependencies. -An Obsidian adapter owns JSON conflict dialogues, progress presentation, -grouped Notices, plug-in reload, restart scheduling, Vault enumeration, and -compatibility activity publication. +Notice teardown are implemented. A focused processed-state owner holds all +three persisted maps, their key and mtime rules, reset operations, and +cross-side settlement. Database write and extraction operations consume one +narrow state port. A focused change processor owns storage and database event +processing, bounded concurrency, per-path serialisation, activity publication, +and compatibility settlement order. The context retains scan, initialisation, +notification, and reconciliation orchestration. A focused conflict-resolution +owner owns pending-path admission, +the parallel classification and serial interaction queues, automatic merge, +newer-revision selection, interactive JSON application, settlement, and queue +disposal. An Obsidian adapter owns JSON conflict dialogue instances, progress +presentation, grouped Notices, plug-in reload, restart scheduling, Vault +enumeration, and compatibility activity publication. ### Stage 6: move synchronisation composition — implemented - Register the overlapping Service handlers once through the joint serviceFeature. +- Consume immutable semantic handler views rather than exposing registry-style + methods on either context. - Preserve the characterised lifecycle callback order and Commonlib aggregation semantics through focused tests. @@ -438,11 +480,13 @@ migration without first establishing narrow dependencies. - The composition root gains several focused views but does not gain another runtime service locator. - The legacy add-on identity and constructor-name lookup are removed. -- The two contexts remain sizeable because each owns one cohesive persisted - synchronisation model, but their dependency surfaces are explicit and do - not include the complete core. Further extraction should follow a concrete - behavioural boundary rather than create additional serviceFeatures for - private operations. +- The two contexts coordinate separate synchronisation workflows, but their + dependency surfaces are explicit and do not include the complete core. + Customisation Sync delegates its derived catalogue and recent-event state, + while Hidden File Sync delegates processed-state, change-processing, and + conflict lifecycles, to focused owners. Further extraction should follow a + concrete behavioural boundary rather than create additional serviceFeatures + for private operations. ## References diff --git a/docs/design_docs/optional_file_sync_architecture.md b/docs/design_docs/optional_file_sync_architecture.md index 239a104e..3ed03e42 100644 --- a/docs/design_docs/optional_file_sync_architecture.md +++ b/docs/design_docs/optional_file_sync_architecture.md @@ -31,6 +31,9 @@ Obsidian composition (`main.ts`) | +--> pure local-path and document routing policy | | | +--> `CustomisationSyncContext` + | | +-- `CustomisationSyncCatalogueState` + | | +-- recent-event deduplicator + | | +-- immutable service-handler and testing views | | ^ | | +-- narrow dependencies from | | `customisationSyncObsidianAdapter` @@ -39,6 +42,20 @@ Obsidian composition (`main.ts`) | ^ | +-- narrow dependencies from | `hiddenFileSyncObsidianAdapter` + | | + | +--> `HiddenFileSyncProcessedState` + | | +-- three autosaved reconciliation maps + | | +-- key, mtime, reset, and settlement rules + | | + | +--> `HiddenFileSyncChangeProcessor` + | | +-- storage and database change processing + | | +-- per-path serialisation and activity state + | | + | +--> `HiddenFileSyncConflictResolution` + | | +-- pending paths and two-stage conflict queue + | | +-- automatic and interactive JSON resolution + | | + | +-- immutable service-handler, command, repair, and testing views | +--> `useCustomisationSyncUI` | +-- catalogue and operation view @@ -58,22 +75,37 @@ application core. ## Ownership -| Owner | Owns | Does not own | -| --- | --- | --- | -| `useOptionalFileSync` | Construction of both contexts, handler registration and removal, local-owner selection, namespace dispatch, compatibility callback order, and context disposal order. | Persisted feature state, synchronisation algorithms, commands, dialogues, or Notices. | -| `CustomisationSyncContext` | The `ix:` codec and path rules, catalogue, manifest cache, scan queues, migration progress, snapshot storage and application, and periodic scan state. | Obsidian dialogues, plug-in lifecycle APIs, ribbon actions, or handler registration. | -| `HiddenFileSyncContext` | The `i:` transfer rules, device-local processed-state caches, reconciliation, exact-revision repair, conflict queues, notification batching, activity counts, and periodic scan state. | Obsidian conflict dialogues, grouped Notices, plug-in lifecycle APIs, or handler registration. | -| Customisation Sync Obsidian adapter | Obsidian conflict selection, Notice presentation, plug-in reload, restart requests, Vault enumeration, progress telemetry, and platform-derived fallback device names. | Catalogue state, routing, or persisted document operations. | -| Hidden File Sync Obsidian adapter | JSON conflict dialogue lifetime, progress presentation, grouped change Notices, plug-in reload actions, restart scheduling, Vault enumeration, and compatibility activity publication. | Transfer, reconciliation, processed-state, or conflict decisions. | -| `useCustomisationSyncUI` | Command, ribbon, dialogue, open-request subscription, and their unload teardown. | Synchronisation state or Hidden File Sync initialisation behaviour. | -| `useHiddenFileSyncCommands` | Hidden File Sync command registration, setting-change subscription, and their unload teardown. | Synchronisation state or command implementation. | +| Owner | Owns | Does not own | +| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | +| `useOptionalFileSync` | Construction of both contexts, handler registration and removal, local-owner selection, namespace dispatch, compatibility callback order, and context disposal order. | Persisted feature state, synchronisation algorithms, commands, dialogues, or Notices. | +| `CustomisationSyncContext` | The `ix:` codec and path rules, scan queues, snapshot storage and application, periodic scan state, and the lifetimes of its transient-state owners. | Catalogue mutations, recent-event history, Obsidian dialogues, ribbon actions, or handler registration. | +| `CustomisationSyncCatalogueState` | Transient catalogue rows, manifest lookup and mtime cache, reactive catalogue and manifest stores, and catalogue update progress. | Database or storage I/O, scan scheduling, routing, or persistence. | +| Customisation Sync event deduplicator | The bounded, newest-first keys used to admit raw configuration events once. | Scheduling, path ownership, catalogue state, or persisted data. | +| `HiddenFileSyncContext` | The `i:` scan and reconciliation workflow, exact-revision repair composition, notification batching, periodic scan state, and focused-owner lifetimes. | Change-event serialisation, processed-state representation, conflict queue state, Obsidian dialogues, or service handler registration. | +| `HiddenFileSyncProcessedState` | Three device-local autosaved maps, exact storage and database keys, retained known mtime, reset behaviour, and storage/database settlement effects. | File transfer, scan scheduling, conflict handling, or presentation. | +| `HiddenFileSyncChangeProcessor` | Storage and database change processing, same-path event serialisation, bounded concurrency, activity counts, and the inherited event-consumption and settlement order. | Full scans, initialisation policy, notification presentation, or conflict interaction. | +| `HiddenFileSyncConflictResolution` | Conflict admission and deduplication, pending paths, the parallel classification and serial interaction queues, automatic merge, newer-revision policy, interactive JSON resolution, and conflict settlement. | Obsidian dialogue instances, general Hidden File Sync scans, processed-state caches, or Service handler registration. | +| Customisation Sync Obsidian adapter | Obsidian conflict selection, Notice presentation, plug-in reload, restart requests, Vault enumeration, progress telemetry, and platform-derived fallback device names. | Catalogue state, routing, or persisted document operations. | +| Hidden File Sync Obsidian adapter | JSON conflict dialogue lifetime, progress presentation, grouped change Notices, plug-in reload actions, restart scheduling, Vault enumeration, and compatibility activity publication. | Transfer, reconciliation, processed-state, or conflict decisions. | +| `useCustomisationSyncUI` | Command, ribbon, dialogue, open-request subscription, and their unload teardown. | Synchronisation state or Hidden File Sync initialisation behaviour. | +| `useHiddenFileSyncCommands` | Hidden File Sync command registration, setting-change subscription, and their unload teardown. | Synchronisation state or command implementation. | -The two domain contexts remain sizeable because they each own one cohesive -persisted synchronisation model. Their private operations are not additional -serviceFeatures: they do not independently register host integration or have -separate application lifetimes. Extract a further ordinary module or focused -state owner when a concrete invariant, replacement lifecycle, or independently -testable operation justifies that boundary. +The two domain contexts coordinate one cohesive synchronisation workflow each. +Their private operations and focused owners are not additional serviceFeatures: +they do not independently register host integration or have separate +application lifetimes. `HiddenFileSyncChangeProcessor` is a focused resource +owner because its semaphore, per-path serialisation, activity counters, and +event settlement form one independently testable lifecycle. +`HiddenFileSyncConflictResolution` is another focused resource owner because +conflict admission, pending-path identity, two serialisation stages, and +disposal form a separate lifecycle. + +The two state owners intentionally do not implement a common generic state +contract. Customisation Sync projects transient catalogue and presentation +state from `ix:` documents. Hidden File Sync persists operational markers used +for reconciliation, with distinct path identity, deletion, reset, and retained +mtime rules. Their common boundary is lifecycle ownership by a context, rather +than interchangeable state semantics. ## Routing and handler contracts @@ -85,21 +117,24 @@ static policy selects Hidden File Sync. The maintained local ownership is: -| Path mode | Owner | -| --- | --- | -| Customisation path in Selective or Flagged Selective mode | Customisation Sync | -| Customisation path in Automatic mode, with Hidden File Sync enabled | Hidden File Sync | -| Customisation path in Ignore mode | Neither context | -| Other eligible hidden path | Hidden File Sync | -| Disabled, excluded, ignored, or ordinary Vault path | Neither context | +| Path mode | Owner | +| ------------------------------------------------------------------- | ------------------ | +| Customisation path in Selective or Flagged Selective mode | Customisation Sync | +| Customisation path in Automatic mode, with Hidden File Sync enabled | Hidden File Sync | +| Customisation path in Ignore mode | Neither context | +| Other eligible hidden path | Hidden File Sync | +| Disabled, excluded, ignored, or ordinary Vault path | Neither context | Local ownership is distinct from persisted-document recognition. The composition dispatches `ps:` and `ix:` conflict documents to Customisation -Sync and `i:` conflict documents to Hidden File Sync. Existing documents remain -recognisable after a local mode changes. +Sync and sends `i:` conflict documents through the Hidden File Sync semantic +handler view. Existing documents remain recognisable after a local mode +changes. -The composition adapts the contexts to the existing Commonlib handler -contracts: +Each context exposes an immutable semantic handler view. The composition +registers these operations and adapts them to the existing Commonlib handler +contracts; registry aggregation names and binding concerns do not leak back +into either context: - raw optional-file events are offered to exactly one selected owner; - a selected handler which skips or fails does not fall through to the other @@ -148,23 +183,35 @@ owners: operations to the Hatch pane. Production consumers receive these views directly. `OptionalFileSyncFeature` -also exposes an explicitly internal `testing` view for maintained real-Obsidian -contract tests. That test seam is not a production service locator and should -not be used by application features. +also exposes immutable, explicitly internal testing views for maintained +real-Obsidian contract tests. They provide named operations, including a scoped +rebuild interceptor, without exposing context instances, dependency objects, +queues, or writable stores. These test seams are not production service +locators and should not be used by application features. ## Lifecycle and disposal The composition is created after the Service Hub and required ServiceModules exist, and before lifecycle-driven feature work begins. Each context creates -its own queues, caches, semaphores, activity state, and periodic processor. +its own periodic processor and focused resource owners. +`CustomisationSyncContext` creates one catalogue-state owner and one +recent-event deduplicator. `HiddenFileSyncContext` creates one processed-state +owner before composing database write and extraction operations around its +narrow port, then creates one change processor and one conflict-resolution +owner. The change processor owns its semaphore and activity state. +Full conflict scans admit discovered paths into the same queue without +suspending it, so ordinary database conflict notifications continue during a +slow scan. After enumeration, the operation waits for both classification and +interaction stages to drain. On application unload, `useOptionalFileSync` first removes every Service handler registration. It then disposes Customisation Sync followed by Hidden File Sync, preserving the former compatibility order. Disposal disables -periodic admission, terminates queues, clears transient caches and pending -sets, cancels scheduled notification work, resets compatibility telemetry, and -hides owned Notices. The two presentation serviceFeatures independently remove -their commands, event subscriptions, ribbon state, and dialogue instances. +periodic admission, disposes the conflict-resolution owner, terminates queues, +clears transient caches and pending sets, cancels scheduled notification work, +resets compatibility telemetry, and hides owned Notices. The two presentation +serviceFeatures independently remove their commands, event subscriptions, +ribbon state, and dialogue instances. ## Persisted compatibility @@ -184,9 +231,11 @@ appropriate, a migration decision. ## Verification boundaries -Focused unit tests cover routing, handler aggregation, context state -isolation, teardown, initial cache selection, exact-revision repair, conflict -dialogue adaptation, grouped Notices, and compatibility activity publication. +Focused unit tests cover routing, semantic handler views, context state +isolation, teardown, initial cache selection, exact-revision repair, change +event serialisation and settlement, conflict queue admission, revision +selection, automatic and interactive merge effect ordering, conflict dialogue +adaptation, grouped Notices, and compatibility activity publication. The boundary test prevents either domain context from regaining core or Obsidian dependencies. diff --git a/src/features/ConfigSync/customisationSyncCatalogueState.ts b/src/features/ConfigSync/customisationSyncCatalogueState.ts new file mode 100644 index 00000000..3fd4aed8 --- /dev/null +++ b/src/features/ConfigSync/customisationSyncCatalogueState.ts @@ -0,0 +1,152 @@ +import type { PluginManifest } from "@/deps.ts"; +import type { FilePathWithPrefix } from "@vrtmrz/livesync-commonlib/compat/common/types"; +import { isObjectDifferent } from "@vrtmrz/livesync-commonlib/compat/common/utils"; +import { writable } from "svelte/store"; + +import { PluginDataExDisplayV2 } from "./customisationSyncModel.ts"; +import type { IPluginDataExDisplay } from "./customisationSyncView.ts"; + +/** + * Owns the transient catalogue projection used by Customisation Sync. + * + * The database and storage operations remain in the context. This owner only + * coordinates the in-memory rows, their reactive publications, manifest + * lookup, and the update counter which is derived from those operations. + */ +export class CustomisationSyncCatalogueState { + private catalogueRows: IPluginDataExDisplay[] = []; + private readonly manifestByKey = new Map(); + private readonly loadedManifestMTimeByKey = new Map(); + private activeUpdateCount = 0; + + readonly catalogue = writable([]); + readonly migrationProgress = writable(0); + readonly manifests = writable(this.manifestByKey); + + /** The current manifest lookup passed to V2 display rows. */ + get manifestLookup(): ReadonlyMap { + return this.manifestByKey; + } + + /** The current loaded-manifest cache, exposed read-only for diagnostics. */ + get loadedManifestMTime(): ReadonlyMap { + return this.loadedManifestMTimeByKey; + } + + /** Returns the authoritative row for a V1 document path, when present. */ + findPlugin(documentPath: FilePathWithPrefix | string): IPluginDataExDisplay | undefined { + return this.catalogueRows.find((entry) => entry.documentPath == documentPath); + } + + /** Returns every row matching a document path, preserving legacy duplicates. */ + findPlugins(documentPath: FilePathWithPrefix | string): readonly IPluginDataExDisplay[] { + return this.catalogueRows.filter((entry) => entry.documentPath == documentPath); + } + + /** Replaces a V1 row and publishes it immediately, preserving legacy order. */ + replacePlugin(plugin: IPluginDataExDisplay): void { + const newList = this.catalogueRows.filter((entry) => entry.documentPath != plugin.documentPath); + newList.push(plugin); + this.catalogueRows = newList; + this.catalogue.set(newList); + } + + /** + * Replaces a V2 row without publishing it. V2 callers publish through the + * existing delayed task after a cohesive row update has completed. + */ + private replaceV2Plugin(plugin: PluginDataExDisplayV2): void { + const newList = this.catalogueRows.filter((entry) => entry.documentPath != plugin.documentPath); + newList.push(plugin); + this.catalogueRows = newList; + } + + /** Applies one loaded or removed V2 file and replaces its catalogue row. */ + async updateV2Plugin( + plugin: PluginDataExDisplayV2, + file: Parameters[0] | false, + missingFilePath: string + ): Promise { + if (file) { + await plugin.setFile(file); + } else { + plugin.deleteFile(missingFilePath); + } + this.replaceV2Plugin(plugin); + } + + /** Publishes the current V2 row set when the legacy delayed task fires. */ + publishCatalogue(): void { + this.catalogue.set(this.catalogueRows); + } + + /** + * Clears rows and loaded manifest mtimes for an explicit reload. The + * manifest map intentionally survives this narrower refresh. + */ + clearForReload(): void { + this.catalogueRows = []; + this.loadedManifestMTimeByKey.clear(); + this.catalogue.set(this.catalogueRows); + } + + /** Clears only the catalogue rows for a disabled refresh. */ + clearForDisabledRefresh(): void { + this.catalogueRows = []; + this.catalogue.set(this.catalogueRows); + } + + /** Begins one catalogue update and publishes its progress count. */ + beginUpdate(): void { + this.activeUpdateCount++; + this.migrationProgress.set(this.activeUpdateCount); + } + + /** Ends one catalogue update and publishes its progress count. */ + endUpdate(): void { + this.activeUpdateCount--; + this.migrationProgress.set(this.activeUpdateCount); + } + + /** + * Applies a manifest according to the inherited cache rules. A manifest is + * parsed only when no manifest has previously been accepted for the key; + * failed parses still record their mtime, while a later mtime never + * replaces a successfully parsed first manifest. + */ + processManifest( + confKey: string, + mtime: number, + parseManifest: () => PluginManifest, + onParseError: (error: unknown) => void = () => undefined + ): void { + let publishCatalogue = false; + if (this.loadedManifestMTimeByKey.get(confKey) != mtime && this.manifestByKey.get(confKey) == undefined) { + try { + this.setManifest(confKey, parseManifest()); + this.applyLoadedManifest(confKey); + publishCatalogue = true; + } catch (error) { + onParseError(error); + } + this.loadedManifestMTimeByKey.set(confKey, mtime); + } else { + this.applyLoadedManifest(confKey); + publishCatalogue = true; + } + if (publishCatalogue) this.catalogue.set(this.catalogueRows); + } + + private setManifest(key: string, manifest: PluginManifest): void { + const old = this.manifestByKey.get(key); + if (old && !isObjectDifferent(manifest, old)) return; + this.manifestByKey.set(key, manifest); + this.manifests.set(this.manifestByKey); + } + + private applyLoadedManifest(confKey: string): void { + this.catalogueRows + .filter((entry) => entry instanceof PluginDataExDisplayV2 && entry.confKey == confKey) + .forEach((entry) => (entry as PluginDataExDisplayV2).applyLoadedManifest()); + } +} diff --git a/src/features/ConfigSync/customisationSyncCatalogueState.unit.spec.ts b/src/features/ConfigSync/customisationSyncCatalogueState.unit.spec.ts new file mode 100644 index 00000000..f57e34db --- /dev/null +++ b/src/features/ConfigSync/customisationSyncCatalogueState.unit.spec.ts @@ -0,0 +1,127 @@ +import { get } from "svelte/store"; +import { describe, expect, it, vi } from "vitest"; +import type { PluginManifest } from "@/deps.ts"; +import type { FilePathWithPrefix } from "@vrtmrz/livesync-commonlib/compat/common/types"; + +import { CustomisationSyncCatalogueState } from "./customisationSyncCatalogueState.ts"; +import { PluginDataExDisplayV2 } from "./customisationSyncModel.ts"; +import type { IPluginDataExDisplay, LoadedEntryPluginDataExFile } from "./customisationSyncView.ts"; + +function display(documentPath = "ix:device-a/PLUGIN_MAIN/example.md"): IPluginDataExDisplay { + return { + documentPath: documentPath as FilePathWithPrefix, + category: "PLUGIN_MAIN", + name: "example", + term: "device-a", + files: [], + mtime: 1, + }; +} + +function file(filename: string, mtime: number): LoadedEntryPluginDataExFile { + return { + path: `ix:device-a/PLUGIN_MAIN/example%${filename}` as FilePathWithPrefix, + filename, + mtime, + data: [filename], + size: filename.length, + } as LoadedEntryPluginDataExFile; +} + +describe("Customisation Sync catalogue state", () => { + it("publishes V1 replacement and keeps V2 replacement delayed", async () => { + const state = new CustomisationSyncCatalogueState(); + const setCatalogue = vi.spyOn(state.catalogue, "set"); + const row = display(); + + state.replacePlugin(row); + expect(get(state.catalogue)).toEqual([row]); + expect(setCatalogue).toHaveBeenCalledOnce(); + + const v2 = new PluginDataExDisplayV2( + { + ...display(), + files: [file("main.js", 1)], + }, + state.manifestLookup + ); + await state.updateV2Plugin(v2, file("main.js", 2), "main.js"); + + expect(get(state.catalogue)).toEqual([row]); + expect(state.findPlugin(row.documentPath)).toBe(v2); + state.publishCatalogue(); + expect(get(state.catalogue)).toEqual([v2]); + }); + + it("retains the first parsed manifest and records failed mtimes", () => { + const state = new CustomisationSyncCatalogueState(); + const first = { name: "First", version: "1.0.0" } as PluginManifest; + const parseManifest = vi.fn(() => first); + + state.processManifest("device-a/plugins/example", 20, parseManifest); + state.processManifest( + "device-a/plugins/example", + 30, + () => ({ name: "Second", version: "2.0.0" }) as PluginManifest + ); + + expect(state.manifestLookup.get("device-a/plugins/example")).toBe(first); + expect(state.loadedManifestMTime.get("device-a/plugins/example")).toBe(20); + expect(parseManifest).toHaveBeenCalledOnce(); + + const failedState = new CustomisationSyncCatalogueState(); + const onParseError = vi.fn(); + const failure = new SyntaxError("invalid"); + failedState.processManifest( + "device-a/plugins/failure", + 40, + () => { + throw failure; + }, + onParseError + ); + failedState.processManifest("device-a/plugins/failure", 40, () => first, onParseError); + + expect(onParseError).toHaveBeenCalledWith(failure); + expect(failedState.loadedManifestMTime.get("device-a/plugins/failure")).toBe(40); + expect(failedState.manifestLookup.has("device-a/plugins/failure")).toBe(false); + }); + + it("clears rows and loaded mtimes on reload while retaining manifest lookup", () => { + const state = new CustomisationSyncCatalogueState(); + const key = "device-a/plugins/example"; + state.processManifest(key, 20, () => ({ name: "Example" }) as PluginManifest); + state.replacePlugin(display()); + + state.clearForReload(); + + expect(get(state.catalogue)).toEqual([]); + expect(state.loadedManifestMTime.size).toBe(0); + expect(state.manifestLookup.get(key)).toEqual({ name: "Example" }); + expect(get(state.catalogue)).toEqual([]); + }); + + it("keeps manifest caches through the narrower disabled refresh", () => { + const state = new CustomisationSyncCatalogueState(); + const key = "device-a/plugins/example"; + state.processManifest(key, 20, () => ({ name: "Example" }) as PluginManifest); + state.replacePlugin(display()); + + state.clearForDisabledRefresh(); + + expect(get(state.catalogue)).toEqual([]); + expect(state.loadedManifestMTime.get(key)).toBe(20); + expect(state.manifestLookup.has(key)).toBe(true); + }); + + it("tracks V2 updates through migration progress", () => { + const state = new CustomisationSyncCatalogueState(); + + state.beginUpdate(); + state.beginUpdate(); + expect(get(state.migrationProgress)).toBe(2); + state.endUpdate(); + state.endUpdate(); + expect(get(state.migrationProgress)).toBe(0); + }); +}); diff --git a/src/features/ConfigSync/customisationSyncContext.command.unit.spec.ts b/src/features/ConfigSync/customisationSyncContext.command.unit.spec.ts index 0a93cae8..4774fd66 100644 --- a/src/features/ConfigSync/customisationSyncContext.command.unit.spec.ts +++ b/src/features/ConfigSync/customisationSyncContext.command.unit.spec.ts @@ -26,29 +26,42 @@ vi.mock("@/common/translation", () => ({ vi.mock("@/common/obsidianCommunityPlugins.ts", () => ({ getObsidianCommunityPluginManager: vi.fn(), })); -import { cancelTask } from "@/common/utils.ts"; +import { cancelTask, scheduleTask } from "@/common/utils.ts"; import { CustomisationSyncContext } from "./customisationSyncContext"; import { createCustomisationSyncTestDependencies } from "./customisationSyncContext.unit.fixture.ts"; describe("CustomisationSyncContext commands", () => { - it("keeps the legacy dialogue methods as delegates to the host-owned UI", () => { + it("opens the host-owned dialogue from a scheduled configuration Notice", async () => { const control = { open: vi.fn(), close: vi.fn(), - isOpen: vi.fn(), + isOpen: vi.fn(() => false), }; + const showConfigurationNotice = vi.fn(); + const updatePluginList = vi.fn(async () => undefined); const configSync = Object.create(CustomisationSyncContext.prototype) as CustomisationSyncContext; Object.assign(configSync, { dependencies: createCustomisationSyncTestDependencies({ getUIControl: () => control, + getSettings: () => ({ usePluginSync: true, notifyPluginOrSettingUpdated: true }) as never, + showConfigurationNotice, }), + updatePluginList, }); - configSync.showPluginSyncModal(); - configSync.hidePluginSyncModal(); + await configSync.serviceHandlers.processVirtualDocument({ + _id: "ix:example", + path: "ix:example", + } as never); + const scheduledNotice = vi.mocked(scheduleTask).mock.calls[0]?.[2] as (() => void) | undefined; + expect(scheduledNotice).toBeTypeOf("function"); + scheduledNotice?.(); + const openDialogue = showConfigurationNotice.mock.calls[0]?.[0] as (() => void) | undefined; + expect(openDialogue).toBeTypeOf("function"); + openDialogue?.(); expect(control.open).toHaveBeenCalledOnce(); - expect(control.close).toHaveBeenCalledOnce(); + expect(updatePluginList).toHaveBeenCalledWith(false, "ix:example"); }); it("releases every owned processor and reactive subscription", () => { @@ -85,4 +98,25 @@ describe("CustomisationSyncContext commands", () => { expect(setEnumerationActive).toHaveBeenCalledWith(false); expect(publishScanCount).toHaveBeenCalledWith(0); }); + + it("characterises the inherited setting-realisation gates pending separate review", async () => { + const isReady = vi.fn(() => false); + const isSuspended = vi.fn(() => false); + const periodicPluginSweepProcessor = { disable: vi.fn(), enable: vi.fn() }; + const scanAllConfigFiles = vi.fn(async () => undefined); + const configSync = Object.create(CustomisationSyncContext.prototype) as CustomisationSyncContext; + Object.assign(configSync, { + dependencies: createCustomisationSyncTestDependencies({ isReady, isSuspended }), + periodicPluginSweepProcessor, + scanAllConfigFiles, + }); + + await expect(configSync.serviceHandlers.onRealiseSetting()).resolves.toBe(true); + + expect(periodicPluginSweepProcessor.disable).toHaveBeenCalledOnce(); + expect(isReady).not.toHaveBeenCalled(); + expect(isSuspended).toHaveBeenCalledOnce(); + expect(scanAllConfigFiles).not.toHaveBeenCalled(); + expect(periodicPluginSweepProcessor.enable).not.toHaveBeenCalled(); + }); }); diff --git a/src/features/ConfigSync/customisationSyncContext.path.unit.spec.ts b/src/features/ConfigSync/customisationSyncContext.path.unit.spec.ts index 112a7107..df25460e 100644 --- a/src/features/ConfigSync/customisationSyncContext.path.unit.spec.ts +++ b/src/features/ConfigSync/customisationSyncContext.path.unit.spec.ts @@ -1,5 +1,4 @@ import { describe, expect, it, vi } from "vitest"; -import type { FilePathWithPrefix } from "@vrtmrz/livesync-commonlib/compat/common/types"; vi.mock("@/deps.ts", () => ({ diff_match_patch: class DiffMatchPatch {}, @@ -65,7 +64,7 @@ describe("compatibility: Customisation Sync paths", () => { [".obsidian/workspace", ""], ["notes/example.json", "CONFIG"], ])("classifies %s as %s", (path, expected) => { - expect(createConfigSync().getFileCategory(path)).toBe(expected); + expect(createConfigSync().testing.getFileCategory(path)).toBe(expected); }); it("keeps other plug-in files outside V1 and disabled plug-in-extra synchronisation", () => { @@ -73,17 +72,17 @@ describe("compatibility: Customisation Sync paths", () => { const withoutPluginEtc = createConfigSync({ usePluginEtc: false }); const path = ".obsidian/plugins/example/other.json"; - expect(v1.getFileCategory(path)).toBe(""); - expect(withoutPluginEtc.getFileCategory(path)).toBe(""); + expect(v1.testing.getFileCategory(path)).toBe(""); + expect(withoutPluginEtc.testing.getFileCategory(path)).toBe(""); }); it("recognises only classified files below the Obsidian configuration directory", () => { const configSync = createConfigSync(); - expect(configSync.isTargetPath(".obsidian/app.json")).toBe(true); - expect(configSync.isTargetPath(".obsidian/plugins/example/main.js")).toBe(true); - expect(configSync.isTargetPath(".obsidian/workspace")).toBe(false); - expect(configSync.isTargetPath("notes/example.json")).toBe(false); + expect(configSync.testing.isTargetPath(".obsidian/app.json")).toBe(true); + expect(configSync.testing.isTargetPath(".obsidian/plugins/example/main.js")).toBe(true); + expect(configSync.testing.isTargetPath(".obsidian/workspace")).toBe(false); + expect(configSync.testing.isTargetPath("notes/example.json")).toBe(false); }); it.each([ @@ -92,7 +91,7 @@ describe("compatibility: Customisation Sync paths", () => { [".obsidian/plugins/example/main.js", "ix:device-a/PLUGIN_MAIN/example.md"], [".obsidian/plugins/example/data.json", "ix:device-a/PLUGIN_DATA/example.md"], ])("creates the V1 document path for %s", (path, expected) => { - expect(createConfigSync().filenameToUnifiedKey(path)).toBe(expected); + expect(createConfigSync().testing.filenameToUnifiedKey(path)).toBe(expected); }); it.each([ @@ -101,30 +100,18 @@ describe("compatibility: Customisation Sync paths", () => { [".obsidian/plugins/example/main.js", "ix:device-a/PLUGIN_MAIN/example%main.js"], [".obsidian/plugins/example/data.json", "ix:device-a/PLUGIN_DATA/example%data.json"], ])("creates the V2 document path for %s", (path, expected) => { - expect(createConfigSync().filenameWithUnifiedKey(path)).toBe(expected); + expect(createConfigSync().testing.filenameWithUnifiedKey(path)).toBe(expected); }); it("uses an explicit device name when supplied", () => { const configSync = createConfigSync(); - expect(configSync.filenameToUnifiedKey(".obsidian/app.json", "device-b")).toBe( + expect(configSync.testing.filenameToUnifiedKey(".obsidian/app.json", "device-b")).toBe( "ix:device-b/CONFIG/app.json.md" ); - expect(configSync.filenameWithUnifiedKey(".obsidian/app.json", "device-b")).toBe( + expect(configSync.testing.filenameWithUnifiedKey(".obsidian/app.json", "device-b")).toBe( "ix:device-b/CONFIG/app.json%app.json" ); - expect(configSync.unifiedKeyPrefixOfTerminal("device-b")).toBe("ix:device-b/"); - }); - - it("parses a V2 document path and derives its V1 compatibility path", () => { - expect( - createConfigSync().parseUnifiedPath("ix:device-a/PLUGIN_MAIN/example%main.js" as FilePathWithPrefix) - ).toEqual({ - device: "device-a", - category: "PLUGIN_MAIN", - key: "example", - filename: "main.js", - pathV1: "ix:device-a/PLUGIN_MAIN/example.md", - }); + expect(configSync.testing.unifiedKeyPrefixOfTerminal("device-b")).toBe("ix:device-b/"); }); }); diff --git a/src/features/ConfigSync/customisationSyncContext.routing.unit.spec.ts b/src/features/ConfigSync/customisationSyncContext.routing.unit.spec.ts index 66a21771..0c28b799 100644 --- a/src/features/ConfigSync/customisationSyncContext.routing.unit.spec.ts +++ b/src/features/ConfigSync/customisationSyncContext.routing.unit.spec.ts @@ -31,6 +31,7 @@ vi.mock("@/common/obsidianCommunityPlugins.ts", () => ({ import { scheduleTask } from "@/common/utils.ts"; import { CustomisationSyncContext } from "./customisationSyncContext.ts"; import { createCustomisationSyncTestDependencies } from "./customisationSyncContext.unit.fixture.ts"; +import { CustomisationSyncRecentEventDeduplicator } from "./customisationSyncRecentEventDeduplicator.ts"; const PATH = ".obsidian/plugins/example/data.json" as FilePath; @@ -43,9 +44,7 @@ function createConfigSync(options: { ready?: boolean; suspended?: boolean; enabl }; const ownsLocalFile = vi.fn(() => options.owned ?? true); const statHidden = vi.fn(async () => ({ type: "file", mtime: 1 })); - const recentProcessedInternalFiles = Object.assign([] as string[], { - contains: vi.fn(() => false), - }); + const recentProcessedInternalFiles = new CustomisationSyncRecentEventDeduplicator(); const configSync = Object.create(CustomisationSyncContext.prototype) as CustomisationSyncContext; Object.assign(configSync, { dependencies: createCustomisationSyncTestDependencies({ @@ -71,7 +70,7 @@ describe("Customisation Sync raw-event admission", () => { it("schedules a recognised path granted by the composition owner", async () => { const { configSync, ownsLocalFile } = createConfigSync(); - await expect(configSync._anyProcessOptionalFileEvent(PATH)).resolves.toBe(true); + await expect(configSync.serviceHandlers.processOptionalFileEvent(PATH)).resolves.toBe(true); expect(ownsLocalFile).toHaveBeenCalledWith(PATH); expect(scheduleTask).toHaveBeenCalledOnce(); }); @@ -79,7 +78,7 @@ describe("Customisation Sync raw-event admission", () => { it("rejects an event while the host is not ready", async () => { const { configSync, statHidden } = createConfigSync({ ready: false }); - await expect(configSync._anyProcessOptionalFileEvent(PATH)).resolves.toBe(false); + await expect(configSync.serviceHandlers.processOptionalFileEvent(PATH)).resolves.toBe(false); expect(statHidden).not.toHaveBeenCalled(); expect(scheduleTask).not.toHaveBeenCalled(); }); @@ -87,7 +86,9 @@ describe("Customisation Sync raw-event admission", () => { it("rejects a path outside the recognised Customisation Sync categories", async () => { const { configSync, ownsLocalFile } = createConfigSync(); - await expect(configSync._anyProcessOptionalFileEvent(".obsidian/workspace" as FilePath)).resolves.toBe(false); + await expect(configSync.serviceHandlers.processOptionalFileEvent(".obsidian/workspace" as FilePath)).resolves.toBe( + false + ); expect(ownsLocalFile).not.toHaveBeenCalled(); expect(scheduleTask).not.toHaveBeenCalled(); }); @@ -95,7 +96,7 @@ describe("Customisation Sync raw-event admission", () => { it("rejects a recognised path assigned to another owner", async () => { const { configSync, statHidden } = createConfigSync({ owned: false }); - await expect(configSync._anyProcessOptionalFileEvent(PATH)).resolves.toBe(false); + await expect(configSync.serviceHandlers.processOptionalFileEvent(PATH)).resolves.toBe(false); expect(statHidden).not.toHaveBeenCalled(); expect(scheduleTask).not.toHaveBeenCalled(); }); @@ -106,7 +107,7 @@ describe("Customisation Sync raw-event admission", () => { ] as const)("rejects an event while %s", async (_label, options) => { const { configSync } = createConfigSync(options); - await expect(configSync._anyProcessOptionalFileEvent(PATH)).resolves.toBe(false); + await expect(configSync.serviceHandlers.processOptionalFileEvent(PATH)).resolves.toBe(false); expect(scheduleTask).not.toHaveBeenCalled(); }); }); diff --git a/src/features/ConfigSync/customisationSyncContext.state.unit.spec.ts b/src/features/ConfigSync/customisationSyncContext.state.unit.spec.ts index ce2958af..bb7cad84 100644 --- a/src/features/ConfigSync/customisationSyncContext.state.unit.spec.ts +++ b/src/features/ConfigSync/customisationSyncContext.state.unit.spec.ts @@ -40,4 +40,46 @@ describe("CustomisationSyncContext state ownership", () => { expect(first.manifests).not.toBe(second.manifests); expect(get(first.manifests)).not.toBe(get(second.manifests)); }); + + it("exposes frozen semantic service and testing views without writable state", () => { + const context = new CustomisationSyncContext(createCustomisationSyncTestDependencies()); + + expect(Object.isFrozen(context.serviceHandlers)).toBe(true); + expect(Object.keys(context.serviceHandlers).sort()).toEqual( + [ + "enableOptionalFeature", + "onBeforeReplicate", + "onDatabaseInitialised", + "onRealiseSetting", + "onResuming", + "processOptionalFileEvent", + "processVirtualDocument", + "suspendExtraSync", + ].sort() + ); + + expect(Object.isFrozen(context.testing)).toBe(true); + expect(Object.keys(context.testing).sort()).toEqual( + [ + "applyDataV2", + "configDir", + "createPluginDataExFileV2", + "createPluginDataFromV2", + "deleteConfigOnDatabase", + "filenameToUnifiedKey", + "filenameWithUnifiedKey", + "getFileCategory", + "isTargetPath", + "scanAllConfigFiles", + "scanInternalFiles", + "storeCustomizationFiles", + "unifiedKeyPrefixOfTerminal", + ].sort() + ); + expect("catalogue" in context.testing).toBe(false); + expect("enumerationActive" in context.testing).toBe(false); + expect("manifests" in context.testing).toBe(false); + + context.dispose(); + }); }); diff --git a/src/features/ConfigSync/customisationSyncContext.ts b/src/features/ConfigSync/customisationSyncContext.ts index 1524d11a..42dd3346 100644 --- a/src/features/ConfigSync/customisationSyncContext.ts +++ b/src/features/ConfigSync/customisationSyncContext.ts @@ -1,6 +1,6 @@ import { writable } from "svelte/store"; import type PouchDB from "pouchdb-core"; -import { type PluginManifest, parseYaml, normalizePath, type ListedFiles, diff_match_patch } from "@/deps.ts"; +import { type PluginManifest, parseYaml, normalizePath, diff_match_patch } from "@/deps.ts"; import type { EntryDoc, @@ -25,25 +25,18 @@ import { import { ICXHeader, PERIODIC_PLUGIN_SWEEP } from "@/common/types.ts"; import { createBlob, - createSavingEntryFromLoadedEntry, createTextBlob, delay, fireAndForget, getDocData, getDocDataAsArray, isDocContentSame, - isLoadedEntry, - isObjectDifferent, } from "@vrtmrz/livesync-commonlib/compat/common/utils"; import { digestHash } from "@vrtmrz/livesync-commonlib/compat/string_and_binary/hash"; -import { - arrayBufferToBase64, - decodeBinary, - readString, -} from "@vrtmrz/livesync-commonlib/compat/string_and_binary/convert"; +import { arrayBufferToBase64, decodeBinary } from "@vrtmrz/livesync-commonlib/compat/string_and_binary/convert"; import { serialized, shareRunningResult } from "octagonal-wheels/concurrency/lock"; import { stripAllPrefixes } from "@vrtmrz/livesync-commonlib/compat/string_and_binary/path"; -import { cancelTask, EVEN, isCustomisationSyncMetadata, isPluginMetadata, scheduleTask } from "@/common/utils.ts"; +import { cancelTask, EVEN, scheduleTask } from "@/common/utils.ts"; import { QueueProcessor } from "octagonal-wheels/concurrency/processor"; import { reactiveSource, type ReactiveSource } from "octagonal-wheels/dataobject/reactive"; import { base64ToArrayBuffer, base64ToString } from "octagonal-wheels/binary/base64"; @@ -59,13 +52,15 @@ import { isCustomisationSyncTargetPath, parseCustomisationSyncV2DocumentPath, } from "./customisationSyncPaths.ts"; -import { createCustomisationSyncCodec, type PluginDataEx, type PluginDataExFile } from "./customisationSyncCodec.ts"; +import { createCustomisationSyncCodec, type PluginDataEx } from "./customisationSyncCodec.ts"; import type { CustomisationSyncDialogView, CustomisationSyncUIControl, + CustomisationSyncFileCategory, + CustomisationSyncServiceHandlers, + CustomisationSyncTestingView, IPluginDataExDisplay, LoadedEntryPluginDataExFile, - PluginDataExDisplay, } from "./customisationSyncView.ts"; import { REPLICATION_PROGRESS_PRESENTATIONS, @@ -75,9 +70,29 @@ import type { LiveSyncLocalDB } from "@vrtmrz/livesync-commonlib/compat/pouchdb/ import type { StorageAccess } from "@vrtmrz/livesync-commonlib/compat/interfaces/StorageAccess"; import type { IPathService, IReplicationService } from "@vrtmrz/livesync-commonlib/compat/services/base/IService"; import type { LogFunction } from "@vrtmrz/livesync-commonlib/compat/services/lib/logUtils"; +import { + collectOptionalFileSyncFiles, + type OptionalFileSyncFileTreeDependencies, +} from "@/features/optionalFileSyncFileTree.ts"; +import { PluginDataExDisplayV2 } from "./customisationSyncModel.ts"; +import { + decodeCustomisationSyncV2File, + loadCustomisationDisplayData, + loadCustomisationV2Entry, + readCustomisationFile, +} from "./customisationSyncReadOperations.ts"; +import { CustomisationSyncCatalogueState } from "./customisationSyncCatalogueState.ts"; +import { CustomisationSyncRecentEventDeduplicator } from "./customisationSyncRecentEventDeduplicator.ts"; export type { PluginDataEx, PluginDataExFile } from "./customisationSyncCodec.ts"; -export type { IPluginDataExDisplay, PluginDataExDisplay } from "./customisationSyncView.ts"; +export type { + CustomisationSyncFileCategory, + CustomisationSyncServiceHandlers, + CustomisationSyncTestingView, + IPluginDataExDisplay, + PluginDataExDisplay, +} from "./customisationSyncView.ts"; +export { PluginDataExDisplayV2 } from "./customisationSyncModel.ts"; const UPDATED_CONFIGURATION_NOTICE_KEY = "config-sync:updated-configuration"; @@ -87,87 +102,8 @@ const { dummyHead: DUMMY_HEAD, dummyEnd: DUMMY_END, } = createCustomisationSyncCodec({ digestHash, parseYaml }); +const CUSTOMISATION_SYNC_READ_CODEC = { deserialize, serialize }; -function categoryToFolder(category: string, configDir: string = ""): string { - switch (category) { - case "CONFIG": - return `${configDir}/`; - case "THEME": - return `${configDir}/themes/`; - case "SNIPPET": - return `${configDir}/snippets/`; - case "PLUGIN_MAIN": - return `${configDir}/plugins/`; - case "PLUGIN_DATA": - return `${configDir}/plugins/`; - case "PLUGIN_ETC": - return `${configDir}/plugins/`; - default: - return ""; - } -} - -export class PluginDataExDisplayV2 { - documentPath: FilePathWithPrefix; - category: string; - - term: string; - - files = [] as LoadedEntryPluginDataExFile[]; - - name: string; - confKey: string; - constructor( - data: IPluginDataExDisplay, - private readonly manifestLookup: ReadonlyMap - ) { - this.documentPath = `${data.documentPath}` as FilePathWithPrefix; - this.category = `${data.category}`; - this.name = `${data.name}`; - this.term = `${data.term}`; - this.files = [...(data.files as LoadedEntryPluginDataExFile[])]; - this.confKey = `${categoryToFolder(this.category, this.term)}${this.name}`; - this.applyLoadedManifest(); - } - async setFile(file: LoadedEntryPluginDataExFile) { - const old = this.files.find((e) => e.filename == file.filename); - if (old) { - if (old.mtime == file.mtime && (await isDocContentSame(old.data, file.data))) return; - this.files = this.files.filter((e) => e.filename != file.filename); - } - this.files.push(file); - if (file.filename == "manifest.json") { - this.applyLoadedManifest(); - } - } - deleteFile(filename: string) { - this.files = this.files.filter((e) => e.filename != filename); - } - - _displayName: string | undefined; - _version: string | undefined; - - applyLoadedManifest() { - const manifest = this.manifestLookup.get(this.confKey); - if (manifest) { - this._displayName = manifest.name; - if (this.category == "PLUGIN_MAIN" || this.category == "THEME") { - this._version = manifest?.version; - } - } - } - get displayName(): string { - // if (this._displayNameBuffer !== symbolUnInitialised) return this._displayNameBuffer; - // return this._bufferManifest().displayName; - return this._displayName || this.name; - } - get version(): string | undefined { - return this._version; - } - get mtime(): number { - return ~~this.files.reduce((a, b) => a + b.mtime, 0) / this.files.length; - } -} type CustomisationSyncSettings = Pick< ObsidianLiveSyncSettings, | "usePluginSync" @@ -195,7 +131,7 @@ export type CustomisationSyncPeriodicProcessor = { disable(): void; }; -export type CustomisationSyncContextDependencies = { +export type CustomisationSyncContextDependencies = OptionalFileSyncFileTreeDependencies & { getSettings(): CustomisationSyncSettings; getLocalDatabase(): CustomisationSyncDatabase; storageAccess: CustomisationSyncStorage; @@ -212,7 +148,6 @@ export type CustomisationSyncContextDependencies = { isSuspended(): boolean; askRestart(): void; createPeriodicProcessor(process: () => Promise): CustomisationSyncPeriodicProcessor; - listFiles(path: string): Promise; resolveJsonConflict( path: FilePath, files: [LoadedEntryPluginDataExFile, LoadedEntryPluginDataExFile], @@ -232,19 +167,22 @@ export type CustomisationSyncContextDependencies = { export class CustomisationSyncContext implements CustomisationSyncDialogView { private readonly dependencies: CustomisationSyncContextDependencies; + private readonly catalogueState = new CustomisationSyncCatalogueState(); + private readonly recentProcessedInternalFiles = new CustomisationSyncRecentEventDeduplicator(); + private serviceHandlersView: CustomisationSyncServiceHandlers | undefined; + private testingView: CustomisationSyncTestingView | undefined; private readonly scanProgress = reactiveSource(0); private readonly pluginScanningChanged: Parameters["onChanged"]>[0] = (event) => { this.enumerationActive.set(event.value != 0); this.dependencies.publishScanCount(event.value); }; - readonly catalogue = writable([]); readonly enumerationActive = writable(false); - readonly migrationProgress = writable(0); - private readonly pluginManifests = new Map(); - readonly manifests = writable(this.pluginManifests); + readonly catalogue = this.catalogueState.catalogue; + readonly migrationProgress = this.catalogueState.migrationProgress; + readonly manifests = this.catalogueState.manifests; - readonly periodicPluginSweepProcessor: CustomisationSyncPeriodicProcessor; + private readonly periodicPluginSweepProcessor: CustomisationSyncPeriodicProcessor; constructor(dependencies: CustomisationSyncContextDependencies) { this.dependencies = dependencies; @@ -253,7 +191,62 @@ export class CustomisationSyncContext implements CustomisationSyncDialogView { ); this.scanProgress.onChanged(this.pluginScanningChanged); } - get configDir() { + + /** + * Semantic callbacks for registration by the optional-file composition + * feature. The returned object is immutable, and each callback retains its + * context without requiring callers to bind a concrete implementation. + */ + get serviceHandlers(): CustomisationSyncServiceHandlers { + if (!this.serviceHandlersView) { + this.serviceHandlersView = Object.freeze({ + processOptionalFileEvent: (path: FilePath) => this.processOptionalFileEvent(path), + processVirtualDocument: (docs: PouchDB.Core.ExistingDocument) => + this.processVirtualDocument(docs), + onRealiseSetting: () => this.realiseSettingSyncMode(), + onResuming: () => this.onResumeProcess(), + onBeforeReplicate: (showMessage: boolean) => this.beforeReplicate(showMessage), + onDatabaseInitialised: (showNotice: boolean) => this.onDatabaseInitialised(showNotice), + suspendExtraSync: () => this.suspendExtraSync(), + enableOptionalFeature: (mode: OptionalSyncFeatureMode) => this.enableOptionalFeature(mode), + }); + } + return this.serviceHandlersView; + } + + /** + * Narrow internal surface used by maintained real-Obsidian contract tests. + * It intentionally omits the context, queues, and writable stores. + */ + get testing(): CustomisationSyncTestingView { + if (!this.testingView) { + this.testingView = Object.freeze({ + configDir: this.configDir, + scanInternalFiles: async () => await this.scanInternalFiles(), + scanAllConfigFiles: async (showMessage: boolean) => await this.scanAllConfigFiles(showMessage), + getFileCategory: (filePath: string) => this.getFileCategory(filePath), + isTargetPath: (filePath: string) => this.isTargetPath(filePath), + filenameToUnifiedKey: (path: string, termOverride?: string) => + this.filenameToUnifiedKey(path, termOverride), + filenameWithUnifiedKey: (path: string, termOverride?: string) => + this.filenameWithUnifiedKey(path, termOverride), + unifiedKeyPrefixOfTerminal: (termOverride?: string) => + this.unifiedKeyPrefixOfTerminal(termOverride), + storeCustomizationFiles: async (path: FilePath, termOverride?: string) => + await this.storeCustomizationFiles(path, termOverride), + deleteConfigOnDatabase: async (path: FilePathWithPrefix, forceWrite?: boolean) => + await this.deleteConfigOnDatabase(path, forceWrite), + createPluginDataFromV2: (path: FilePathWithPrefix) => this.createPluginDataFromV2(path), + createPluginDataExFileV2: async (path: FilePathWithPrefix, loaded?: LoadedEntry) => + await this.createPluginDataExFileV2(path, loaded), + applyDataV2: async (data: PluginDataExDisplayV2, content?: string) => + await this.applyDataV2(data, content), + }); + } + return this.testingView; + } + + private get configDir() { return this.dependencies.getConfigDir(); } @@ -277,11 +270,11 @@ export class CustomisationSyncContext implements CustomisationSyncDialogView { return this.dependencies.path.getPath(entry); } - _isMainReady() { + private _isMainReady() { return this.dependencies.isReady(); } - _isMainSuspended() { + private _isMainSuspended() { return this.dependencies.isSuspended(); } @@ -289,13 +282,13 @@ export class CustomisationSyncContext implements CustomisationSyncDialogView { this.dependencies.log(message, level, key); } - get useV2() { + private get useV2() { return this.settings.usePluginSyncV2; } - get useSyncPluginEtc() { + private get useSyncPluginEtc() { return this.settings.usePluginEtc; } - isThisModuleEnabled() { + private isThisModuleEnabled() { return this.settings.usePluginSync; } @@ -355,11 +348,11 @@ export class CustomisationSyncContext implements CustomisationSyncDialogView { ): Promise { const dataACopy = dataA instanceof PluginDataExDisplayV2 - ? new PluginDataExDisplayV2(dataA, this.pluginManifests) + ? new PluginDataExDisplayV2(dataA, this.catalogueState.manifestLookup) : { ...dataA }; const dataBCopy = dataB instanceof PluginDataExDisplayV2 - ? new PluginDataExDisplayV2(dataB, this.pluginManifests) + ? new PluginDataExDisplayV2(dataB, this.catalogueState.manifestLookup) : { ...dataB }; dataACopy.files = dataACopy.files.filter((file) => file.filename == filename); dataBCopy.files = dataBCopy.files.filter((file) => file.filename == filename); @@ -372,14 +365,6 @@ export class CustomisationSyncContext implements CustomisationSyncDialogView { await this.updatePluginList(false, this.filenameToUnifiedKey(path, deviceName)); } - pluginList: IPluginDataExDisplay[] = []; - showPluginSyncModal() { - this.dependencies.getUIControl()?.open(); - } - - hidePluginSyncModal() { - this.dependencies.getUIControl()?.close(); - } dispose() { cancelTask(UPDATED_CONFIGURATION_NOTICE_KEY); this.periodicPluginSweepProcessor?.disable(); @@ -391,30 +376,21 @@ export class CustomisationSyncContext implements CustomisationSyncDialogView { this.dependencies.hideConfigurationNotice(); } - private setManifest(key: string, manifest: PluginManifest) { - const old = this.pluginManifests.get(key); - if (old && !isObjectDifferent(manifest, old)) return; - this.pluginManifests.set(key, manifest); - this.manifests.set(this.pluginManifests); - } - - getFileCategory( - filePath: string - ): "CONFIG" | "THEME" | "SNIPPET" | "PLUGIN_MAIN" | "PLUGIN_ETC" | "PLUGIN_DATA" | "" { + private getFileCategory(filePath: string): CustomisationSyncFileCategory { return getCustomisationSyncFileCategory(filePath, { configDir: this.configDir, useV2: this.useV2, usePluginEtc: this.useSyncPluginEtc, }); } - isTargetPath(filePath: string): boolean { + private isTargetPath(filePath: string): boolean { return isCustomisationSyncTargetPath(filePath, { configDir: this.configDir, useV2: this.useV2, usePluginEtc: this.useSyncPluginEtc, }); } - async _everyOnDatabaseInitialized(showNotice: boolean) { + private async onDatabaseInitialised(showNotice: boolean) { if (!this.isThisModuleEnabled()) return true; try { this._log("Scanning customizations..."); @@ -426,7 +402,7 @@ export class CustomisationSyncContext implements CustomisationSyncDialogView { } return true; } - async _everyBeforeReplicate(showNotice: boolean) { + private async beforeReplicate(showNotice: boolean) { if (!this.isThisModuleEnabled()) return true; if (this.settings.autoSweepPlugins) { await this.scanAllConfigFiles(showNotice); @@ -434,7 +410,7 @@ export class CustomisationSyncContext implements CustomisationSyncDialogView { } return true; } - async _everyOnResumeProcess(): Promise { + private async onResumeProcess(): Promise { if (!this.isThisModuleEnabled()) return true; if (this._isMainSuspended()) { return true; @@ -450,44 +426,10 @@ export class CustomisationSyncContext implements CustomisationSyncDialogView { return true; } async reloadPluginList(showMessage: boolean) { - this.pluginList = []; - this.loadedManifest_mTime.clear(); - this.catalogue.set(this.pluginList); + this.catalogueState.clearForReload(); await this.updatePluginList(showMessage); } - async loadPluginData(path: FilePathWithPrefix): Promise { - const wx = await this.localDatabase.getDBEntry(path, undefined, false, false); - if (wx) { - const data = deserialize(getDocDataAsArray(wx.data), {}) as PluginDataEx; - const xFiles = [] as PluginDataExFile[]; - let missingHash = false; - for (const file of data.files) { - const work = { ...file, data: [] as string[] }; - if (!file.hash) { - // debugger; - const tempStr = getDocDataAsArray(work.data); - const hash = digestHash(tempStr); - file.hash = hash; - missingHash = true; - } - work.data = [file.hash]; - xFiles.push(work); - } - if (missingHash) { - this._log(`Digest created for ${path} to improve checking`, LOG_LEVEL_VERBOSE); - wx.data = serialize(data); - fireAndForget(() => this.localDatabase.putDBEntry(createSavingEntryFromLoadedEntry(wx))); - } - return { - ...data, - documentPath: this.getPath(wx), - files: xFiles, - } satisfies PluginDataExDisplay; - } - return false; - } - - pluginScanProcessor = new QueueProcessor( + private pluginScanProcessor = new QueueProcessor( async (v: AnyEntry[]) => { const plugin = v[0]; if (this.useV2) { @@ -495,16 +437,16 @@ export class CustomisationSyncContext implements CustomisationSyncDialogView { return []; } const path = plugin.path || this.getPath(plugin); - const oldEntry = this.pluginList.find((e) => e.documentPath == path); + const oldEntry = this.catalogueState.findPlugin(path); if (oldEntry && oldEntry.mtime == plugin.mtime) return []; try { - const pluginData = await this.loadPluginData(path); + const pluginData = await loadCustomisationDisplayData( + this.dependencies, + path, + CUSTOMISATION_SYNC_READ_CODEC + ); if (pluginData) { - let newList = [...this.pluginList]; - newList = newList.filter((x) => x.documentPath != pluginData.documentPath); - newList.push(pluginData); - this.pluginList = newList; - this.catalogue.set(newList); + this.catalogueState.replacePlugin(pluginData); } // Failed to load return []; @@ -525,20 +467,23 @@ export class CustomisationSyncContext implements CustomisationSyncDialogView { } ).startPipeline(); - pluginScanProcessorV2 = new QueueProcessor( + // Compatibility question: no production path currently enqueues work into + // this second processor. Preserve its construction and disposal until the + // intended V2 scan path, or its safe removal, has focused coverage. + private pluginScanProcessorV2 = new QueueProcessor( async (v: AnyEntry[]) => { const plugin = v[0]; const path = plugin.path || this.getPath(plugin); - const oldEntry = this.pluginList.find((e) => e.documentPath == path); + const oldEntry = this.catalogueState.findPlugin(path); if (oldEntry && oldEntry.mtime == plugin.mtime) return []; try { - const pluginData = await this.loadPluginData(path); + const pluginData = await loadCustomisationDisplayData( + this.dependencies, + path, + CUSTOMISATION_SYNC_READ_CODEC + ); if (pluginData) { - let newList = [...this.pluginList]; - newList = newList.filter((x) => x.documentPath != pluginData.documentPath); - newList.push(pluginData); - this.pluginList = newList; - this.catalogue.set(newList); + this.catalogueState.replacePlugin(pluginData); } // Failed to load return []; @@ -559,7 +504,7 @@ export class CustomisationSyncContext implements CustomisationSyncDialogView { } ).startPipeline(); - filenameToUnifiedKey(path: string, termOverRide?: string) { + private filenameToUnifiedKey(path: string, termOverRide?: string): FilePathWithPrefix { const term = termOverRide || this.dependencies.getDeviceAndVaultName(); return createCustomisationSyncV1DocumentPath(path, term, { configDir: this.configDir, @@ -568,7 +513,7 @@ export class CustomisationSyncContext implements CustomisationSyncDialogView { }); } - filenameWithUnifiedKey(path: string, termOverRide?: string) { + private filenameWithUnifiedKey(path: string, termOverRide?: string): FilePathWithPrefix { const term = termOverRide || this.dependencies.getDeviceAndVaultName(); return createCustomisationSyncV2DocumentPath(path, term, { configDir: this.configDir, @@ -577,88 +522,38 @@ export class CustomisationSyncContext implements CustomisationSyncDialogView { }); } - unifiedKeyPrefixOfTerminal(termOverRide?: string) { + private unifiedKeyPrefixOfTerminal(termOverRide?: string): string { const term = termOverRide || this.dependencies.getDeviceAndVaultName(); return createCustomisationSyncDevicePrefix(term); } - parseUnifiedPath(unifiedPath: FilePathWithPrefix): { - category: string; - device: string; - key: string; - filename: string; - pathV1: FilePathWithPrefix; - } { - return parseCustomisationSyncV2DocumentPath(unifiedPath); - } - - loadedManifest_mTime = new Map(); - - async createPluginDataExFileV2( + private async createPluginDataExFileV2( unifiedPathV2: FilePathWithPrefix, loaded?: LoadedEntry ): Promise { - const { category, key, filename, device } = this.parseUnifiedPath(unifiedPathV2); - if (!loaded) { - const d = await this.localDatabase.getDBEntry(unifiedPathV2); - if (!d) { - this._log(`The file ${unifiedPathV2} is not found`, LOG_LEVEL_VERBOSE); - return false; - } - if (!isLoadedEntry(d)) { - this._log(`The file ${unifiedPathV2} is not a note`, LOG_LEVEL_VERBOSE); - return false; - } - loaded = d; - } - const confKey = `${categoryToFolder(category, device)}${key}`; - const relativeFilename = - `${categoryToFolder(category, "")}${category == "CONFIG" || category == "SNIPPET" ? "" : key + "/"}${filename}`.substring( - 1 - ); - const dataSrc = getDocData(loaded.data); - const dataStart = dataSrc.indexOf(DUMMY_END); - const data = dataSrc.substring(dataStart + DUMMY_END.length); - const file: LoadedEntryPluginDataExFile = { - ...loaded, - hash: "", - data: [base64ToString(data)], - filename: relativeFilename, - displayName: filename, - }; - if (filename == "manifest.json") { - // Same as previously loaded - if ( - this.loadedManifest_mTime.get(confKey) != file.mtime && - this.pluginManifests.get(confKey) == undefined - ) { - try { - const parsedManifest = JSON.parse(base64ToString(data)) as PluginManifest; - this.setManifest(confKey, parsedManifest); - this.pluginList - .filter((e) => e instanceof PluginDataExDisplayV2 && e.confKey == confKey) - .forEach((e) => (e as PluginDataExDisplayV2).applyLoadedManifest()); - this.catalogue.set(this.pluginList); - } catch (ex) { + // Compatibility: a caller-supplied entry bypasses the database lookup + // and the isLoadedEntry check performed by loadCustomisationV2Entry. + const loadedEntry = loaded ?? (await loadCustomisationV2Entry(this.dependencies, unifiedPathV2)); + if (!loadedEntry) return false; + const { confKey, file, isManifest } = decodeCustomisationSyncV2File(unifiedPathV2, loadedEntry, DUMMY_END); + if (isManifest) { + this.catalogueState.processManifest( + confKey, + file.mtime, + () => JSON.parse(file.data[0]) as PluginManifest, + (error) => { this._log( - `The file ${loaded.path} seems to manifest, but could not be decoded as JSON`, + `The file ${loadedEntry.path} seems to manifest, but could not be decoded as JSON`, LOG_LEVEL_VERBOSE ); - this._log(ex, LOG_LEVEL_VERBOSE); + this._log(error, LOG_LEVEL_VERBOSE); } - this.loadedManifest_mTime.set(confKey, file.mtime); - } else { - this.pluginList - .filter((e) => e instanceof PluginDataExDisplayV2 && e.confKey == confKey) - .forEach((e) => (e as PluginDataExDisplayV2).applyLoadedManifest()); - this.catalogue.set(this.pluginList); - } - // } + ); } return file; } - createPluginDataFromV2(unifiedPathV2: FilePathWithPrefix) { - const { category, device, key, pathV1 } = this.parseUnifiedPath(unifiedPathV2); + private createPluginDataFromV2(unifiedPathV2: FilePathWithPrefix) { + const { category, device, key, pathV1 } = parseCustomisationSyncV2DocumentPath(unifiedPathV2); if (category == "") return; const ret: PluginDataExDisplayV2 = new PluginDataExDisplayV2( @@ -670,21 +565,18 @@ export class CustomisationSyncContext implements CustomisationSyncDialogView { files: [], mtime: 0, }, - this.pluginManifests + this.catalogueState.manifestLookup ); return ret; } - updatingV2Count = 0; - - async updatePluginListV2(showMessage: boolean, unifiedFilenameWithKey: FilePathWithPrefix): Promise { + private async updatePluginListV2(showMessage: boolean, unifiedFilenameWithKey: FilePathWithPrefix): Promise { try { - this.updatingV2Count++; - this.migrationProgress.set(this.updatingV2Count); + this.catalogueState.beginUpdate(); // const unifiedFilenameWithKey = this.filenameWithUnifiedKey(updatedDocumentPath); - const { pathV1 } = this.parseUnifiedPath(unifiedFilenameWithKey); + const { pathV1 } = parseCustomisationSyncV2DocumentPath(unifiedFilenameWithKey); - const oldEntry = this.pluginList.find((e) => e.documentPath == pathV1); + const oldEntry = this.catalogueState.findPlugin(pathV1); let entry: PluginDataExDisplayV2 | undefined = undefined; if (!oldEntry || !(oldEntry instanceof PluginDataExDisplayV2)) { @@ -697,28 +589,19 @@ export class CustomisationSyncContext implements CustomisationSyncDialogView { } if (!entry) return; const file = await this.createPluginDataExFileV2(unifiedFilenameWithKey); - if (file) { - await entry.setFile(file); - } else { - entry.deleteFile(unifiedFilenameWithKey); - if (entry.files.length == 0) { - this.pluginList = this.pluginList.filter((e) => e.documentPath != pathV1); - } - } - const newList = this.pluginList.filter((e) => e.documentPath != entry.documentPath); - newList.push(entry); - this.pluginList = newList; + // Compatibility: the inherited update always re-adds an empty V2 + // row after deleting its final file. + await this.catalogueState.updateV2Plugin(entry, file, unifiedFilenameWithKey); scheduleTask("updatePluginListV2", 100, () => { - this.catalogue.set(this.pluginList); + this.catalogueState.publishCatalogue(); }); } finally { - this.updatingV2Count--; - this.migrationProgress.set(this.updatingV2Count); + this.catalogueState.endUpdate(); } } - async migrateV1ToV2(showMessage: boolean, entry: AnyEntry): Promise { + private async migrateV1ToV2(showMessage: boolean, entry: AnyEntry): Promise { const v1Path = entry.path; this._log(`Migrating ${entry.path} to V2`, showMessage ? LOG_LEVEL_NOTICE : LOG_LEVEL_INFO); if (entry.deleted) { @@ -789,13 +672,11 @@ export class CustomisationSyncContext implements CustomisationSyncDialogView { async updatePluginList(showMessage: boolean, updatedDocumentPath?: FilePathWithPrefix): Promise { if (!this.isThisModuleEnabled()) { this.pluginScanProcessor.clearQueue(); - this.pluginList = []; - this.catalogue.set(this.pluginList); + this.catalogueState.clearForDisabledRefresh(); return; } try { - this.updatingV2Count++; - this.migrationProgress.set(this.updatingV2Count); + this.catalogueState.beginUpdate(); const updatedDocumentId = updatedDocumentPath ? await this.path2id(updatedDocumentPath) : ""; const plugins = updatedDocumentPath ? this.localDatabase.findEntries(updatedDocumentId, updatedDocumentId + "\u{10ffff}", { @@ -817,8 +698,7 @@ export class CustomisationSyncContext implements CustomisationSyncDialogView { } } finally { this.enumerationActive.set(false); - this.updatingV2Count--; - this.migrationProgress.set(this.updatingV2Count); + this.catalogueState.endUpdate(); } this.enumerationActive.set(false); // return entries; @@ -892,7 +772,7 @@ export class CustomisationSyncContext implements CustomisationSyncDialogView { return false; } } - async applyDataV2(data: PluginDataExDisplayV2, content?: string): Promise { + private async applyDataV2(data: PluginDataExDisplayV2, content?: string): Promise { const baseDir = this.configDir; try { if (content) { @@ -1010,10 +890,10 @@ export class CustomisationSyncContext implements CustomisationSyncDialogView { if (data.documentPath) { const delList = []; if (this.useV2) { - const deleteList = this.pluginList - .filter((e) => e.documentPath == data.documentPath) - .filter((e) => e instanceof PluginDataExDisplayV2) - .map((e) => e.files) + const deleteList = this.catalogueState + .findPlugins(data.documentPath) + .filter((entry) => entry instanceof PluginDataExDisplayV2) + .map((entry) => entry.files) .flat(); for (const e of deleteList) { delList.push(e.path); @@ -1039,7 +919,7 @@ export class CustomisationSyncContext implements CustomisationSyncDialogView { return false; } } - async _anyModuleParsedReplicationResultItem(docs: PouchDB.Core.ExistingDocument) { + private async processVirtualDocument(docs: PouchDB.Core.ExistingDocument) { if (!docs._id.startsWith(ICXHeader)) return false; if (this.isThisModuleEnabled()) { await this.updatePluginList( @@ -1050,14 +930,18 @@ export class CustomisationSyncContext implements CustomisationSyncDialogView { if (this.isThisModuleEnabled() && this.settings.notifyPluginOrSettingUpdated) { if (!this.dependencies.getUIControl()?.isOpen()) { scheduleTask(UPDATED_CONFIGURATION_NOTICE_KEY, 1000, () => { - this.dependencies.showConfigurationNotice(() => this.showPluginSyncModal()); + this.dependencies.showConfigurationNotice(() => this.dependencies.getUIControl()?.open()); }); } } return true; } - async _everyRealizeSettingSyncMode(): Promise { + private async realiseSettingSyncMode(): Promise { this.periodicPluginSweepProcessor?.disable(); + // Compatibility question: this inherited callback checks the method + // reference rather than invoking it, then proceeds only while the host is + // suspended. Preserve both gates until their intended lifecycle semantics + // are verified and corrected under a separate behavioural test. if (!this._isMainReady) return true; if (!this._isMainSuspended()) return true; if (!this.isThisModuleEnabled()) return true; @@ -1072,55 +956,7 @@ export class CustomisationSyncContext implements CustomisationSyncDialogView { return true; } - recentProcessedInternalFiles = [] as string[]; - async makeEntryFromFile(path: FilePath): Promise { - const stat = await this.storageAccess.statHidden(path); - let version: string | undefined; - let displayName: string | undefined; - if (!stat) { - return false; - } - const contentBin = await this.storageAccess.readHiddenFileBinary(path); - let content: string[]; - try { - content = await arrayBufferToBase64(contentBin); - if (path.toLowerCase().endsWith("/manifest.json")) { - const v = readString(new Uint8Array(contentBin)); - try { - const json: unknown = JSON.parse(v); - if (typeof json === "object" && json !== null) { - if ("version" in json) { - version = String(json.version); - } - if ("name" in json) { - displayName = String(json.name); - } - } - } catch (ex) { - this._log( - `Configuration sync data: ${path} looks like manifest, but could not read the version`, - LOG_LEVEL_INFO - ); - this._log(ex, LOG_LEVEL_VERBOSE); - } - } - } catch (ex) { - this._log(`The file ${path} could not be encoded`); - this._log(ex, LOG_LEVEL_VERBOSE); - return false; - } - const mtime = stat.mtime; - return { - filename: path.substring(this.configDir.length + 1), - data: content, - mtime, - size: stat.size, - version, - displayName: displayName, - }; - } - - async storeCustomisationFileV2(path: FilePath, term: string, force = false) { + private async storeCustomisationFileV2(path: FilePath, term: string, force = false) { const vf = this.filenameWithUnifiedKey(path, term); return await serialized(`plugin-${vf}`, async () => { const prefixedFileName = vf; @@ -1199,7 +1035,7 @@ export class CustomisationSyncContext implements CustomisationSyncDialogView { } }); } - async storeCustomizationFiles(path: FilePath, termOverRide?: string) { + private async storeCustomizationFiles(path: FilePath, termOverRide?: string) { const term = termOverRide || this.dependencies.getDeviceAndVaultName(); if (term == "") { this._log($msg("We have to configure the device name"), LOG_LEVEL_NOTICE); @@ -1247,7 +1083,7 @@ export class CustomisationSyncContext implements CustomisationSyncDialogView { fileTargets = ["manifest.json", "theme.css"].map((e) => `${parentPath}/${e}` as FilePath); } for (const target of fileTargets) { - const data = await this.makeEntryFromFile(target); + const data = await readCustomisationFile(this.dependencies, target, this.configDir); if (data == false) { this._log(`Config: skipped (Possibly is not exist): ${target} `, LOG_LEVEL_VERBOSE); continue; @@ -1343,11 +1179,11 @@ export class CustomisationSyncContext implements CustomisationSyncDialogView { } }); } - async _anyProcessOptionalFileEvent(path: FilePath): Promise { + private async processOptionalFileEvent(path: FilePath): Promise { return await this.watchVaultRawEventsAsync(path); } - async watchVaultRawEventsAsync(path: FilePath) { + private async watchVaultRawEventsAsync(path: FilePath) { if (!this._isMainReady()) return false; if (this._isMainSuspended()) return false; if (!this.isThisModuleEnabled()) return false; @@ -1360,12 +1196,11 @@ export class CustomisationSyncContext implements CustomisationSyncDialogView { // this._log(`Customization file detected: ${path}`, LOG_LEVEL_VERBOSE); const storageMTime = ~~(((stat && stat.mtime) || 0) / 1000); const key = `${path}-${storageMTime}`; - if (this.recentProcessedInternalFiles.contains(key)) { + if (!this.recentProcessedInternalFiles.admit(key)) { // If recently processed, it may caused by self. // return true to prevent pass the event to the next. return true; } - this.recentProcessedInternalFiles = [key, ...this.recentProcessedInternalFiles].slice(0, 100); // To prevent saving half-collected file sets. const keySchedule = this.filenameToUnifiedKey(path); scheduleTask(keySchedule, 100, async () => { @@ -1480,7 +1315,7 @@ export class CustomisationSyncContext implements CustomisationSyncDialogView { }); } - async deleteConfigOnDatabase(prefixedFileName: FilePathWithPrefix, forceWrite = false) { + private async deleteConfigOnDatabase(prefixedFileName: FilePathWithPrefix, forceWrite = false): Promise { // const id = await this.path2id(prefixedFileName); const mtime = new Date().getTime(); return await serialized("file-x-" + prefixedFileName, async () => { @@ -1518,24 +1353,22 @@ export class CustomisationSyncContext implements CustomisationSyncDialogView { }); } - async scanInternalFiles(): Promise { - const filenames = (await this.getFiles(this.configDir, 2)) + private async scanInternalFiles(): Promise { + const filenames = ( + await collectOptionalFileSyncFiles(this.dependencies, this.configDir, { + maxDepth: 2, + onError: (path, error) => { + this._log(`Could not traverse(CustomisationSync):${path}`, LOG_LEVEL_INFO); + this._log(error, LOG_LEVEL_VERBOSE); + }, + }) + ) .filter((e) => e.startsWith(".")) .filter((e) => !e.startsWith(".trash")); return filenames as FilePath[]; } - _anyGetOptionalConflictCheckMethod(path: FilePathWithPrefix): Promise { - if (isPluginMetadata(path)) { - return Promise.resolve("newer"); - } - if (isCustomisationSyncMetadata(path)) { - return Promise.resolve("newer"); - } - return Promise.resolve(false); - } - - _allSuspendExtraSync(): Promise { + private suspendExtraSync(): Promise { if (this.settings.usePluginSync || this.settings.autoSweepPlugins) { this._log( "Customisation sync have been temporarily disabled. Please enable them after the fetching, if you need them.", @@ -1547,11 +1380,11 @@ export class CustomisationSyncContext implements CustomisationSyncDialogView { return Promise.resolve(true); } - async _allConfigureOptionalSyncFeature(mode: OptionalSyncFeatureMode) { - await this.configureHiddenFileSync(mode); + private async enableOptionalFeature(mode: OptionalSyncFeatureMode): Promise { + await this.configureCustomisationSync(mode); return true; } - async configureHiddenFileSync(mode: OptionalSyncFeatureMode) { + private async configureCustomisationSync(mode: OptionalSyncFeatureMode) { if (mode == "DISABLE") { await this.dependencies.applySettings( { @@ -1584,21 +1417,4 @@ export class CustomisationSyncContext implements CustomisationSyncDialogView { await this.scanAllConfigFiles(true); } } - - async getFiles(path: string, lastDepth: number) { - if (lastDepth == -1) return []; - let w: ListedFiles; - try { - w = await this.dependencies.listFiles(path); - } catch (ex) { - this._log(`Could not traverse(CustomisationSync):${path}`, LOG_LEVEL_INFO); - this._log(ex, LOG_LEVEL_VERBOSE); - return []; - } - let files = [...w.files]; - for (const v of w.folders) { - files = files.concat(await this.getFiles(v, lastDepth - 1)); - } - return files; - } } diff --git a/src/features/ConfigSync/customisationSyncContext.v2Manifest.unit.spec.ts b/src/features/ConfigSync/customisationSyncContext.v2Manifest.unit.spec.ts new file mode 100644 index 00000000..4e835735 --- /dev/null +++ b/src/features/ConfigSync/customisationSyncContext.v2Manifest.unit.spec.ts @@ -0,0 +1,129 @@ +import { describe, expect, it, vi } from "vitest"; + +vi.mock("@/deps.ts", () => ({ + diff_match_patch: class DiffMatchPatch {}, + normalizePath: vi.fn((path: string) => path), + parseYaml: vi.fn(), + Platform: {}, +})); +vi.mock("@/common/types.ts", () => ({ + ICXHeader: "ix:", + PERIODIC_PLUGIN_SWEEP: 60, +})); +vi.mock("@/common/utils.ts", () => ({ + cancelTask: vi.fn(), + EVEN: Symbol("even"), + isCustomisationSyncMetadata: vi.fn(), + isPluginMetadata: vi.fn(), + scheduleTask: vi.fn(), +})); +vi.mock("@/common/PeriodicProcessor.ts", () => ({ + PeriodicProcessor: class PeriodicProcessor {}, +})); +vi.mock("@/common/translation", () => ({ + $msg: vi.fn((message: string) => message), +})); +vi.mock("@/common/obsidianCommunityPlugins.ts", () => ({ + getObsidianCommunityPluginManager: vi.fn(), +})); + +import { + LOG_LEVEL_VERBOSE, + type FilePathWithPrefix, + type LoadedEntry, +} from "@vrtmrz/livesync-commonlib/compat/common/types"; +import { digestHash } from "@vrtmrz/livesync-commonlib/compat/string_and_binary/hash"; +import { createCustomisationSyncCodec } from "./customisationSyncCodec.ts"; +import { CustomisationSyncCatalogueState } from "./customisationSyncCatalogueState.ts"; +import { CustomisationSyncContext } from "./customisationSyncContext.ts"; +import { createCustomisationSyncTestDependencies } from "./customisationSyncContext.unit.fixture.ts"; + +const path = "ix:device-a/PLUGIN_MAIN/example%manifest.json" as FilePathWithPrefix; +const confKey = "device-a/plugins/example"; +const codec = createCustomisationSyncCodec({ + digestHash, + parseYaml: () => undefined, +}); + +function loadedManifest(manifestSource: string, mtime: number): LoadedEntry { + const data = `${codec.dummyHead}${codec.dummyEnd}${btoa(manifestSource)}`; + return { + _id: "entry-id", + _rev: "1-a", + path, + type: "plain", + datatype: "plain", + data, + ctime: 10, + mtime, + size: data.length, + children: [], + eden: {}, + } as unknown as LoadedEntry; +} + +function createContext() { + const log = vi.fn(); + const catalogueState = new CustomisationSyncCatalogueState(); + const pluginManifests = catalogueState.manifestLookup; + const loadedManifest_mTime = catalogueState.loadedManifestMTime; + const setManifests = vi.spyOn(catalogueState.manifests, "set"); + const setCatalogue = vi.spyOn(catalogueState.catalogue, "set"); + const context = Object.create(CustomisationSyncContext.prototype) as CustomisationSyncContext; + Object.assign(context, { + dependencies: createCustomisationSyncTestDependencies({ + log, + getLocalDatabase: () => ({ getDBEntry: async () => false }) as never, + }), + catalogueState, + }); + return { + context, + loadedManifest_mTime, + log, + pluginManifests, + setCatalogue, + setManifests, + }; +} + +describe("compatibility: Customisation Sync V2 manifest state", () => { + it("keeps the first parsed manifest when a later file has a different mtime", async () => { + const { context, loadedManifest_mTime, pluginManifests, setManifests } = createContext(); + + await context.testing.createPluginDataExFileV2( + path, + loadedManifest(JSON.stringify({ id: "example", name: "First", version: "1.0.0" }), 20) + ); + await context.testing.createPluginDataExFileV2( + path, + loadedManifest(JSON.stringify({ id: "example", name: "Second", version: "2.0.0" }), 30) + ); + + expect(pluginManifests.get(confKey)).toMatchObject({ name: "First", version: "1.0.0" }); + expect(loadedManifest_mTime.get(confKey)).toBe(20); + expect(setManifests).toHaveBeenCalledOnce(); + }); + + it("records a failed manifest mtime and does not retry the same revision", async () => { + const { context, loadedManifest_mTime, log, pluginManifests, setCatalogue } = createContext(); + const invalid = loadedManifest("{invalid", 20); + + await expect(context.testing.createPluginDataExFileV2(path, invalid)).resolves.toMatchObject({ + filename: "plugins/example/manifest.json", + }); + await context.testing.createPluginDataExFileV2(path, invalid); + + expect(pluginManifests.has(confKey)).toBe(false); + expect(loadedManifest_mTime.get(confKey)).toBe(20); + expect(log).toHaveBeenCalledTimes(2); + expect(log).toHaveBeenNthCalledWith( + 1, + `The file ${path} seems to manifest, but could not be decoded as JSON`, + LOG_LEVEL_VERBOSE, + undefined + ); + expect(log).toHaveBeenNthCalledWith(2, expect.any(SyntaxError), LOG_LEVEL_VERBOSE, undefined); + expect(setCatalogue).toHaveBeenCalledOnce(); + }); +}); diff --git a/src/features/ConfigSync/customisationSyncModel.ts b/src/features/ConfigSync/customisationSyncModel.ts new file mode 100644 index 00000000..a7e18ec3 --- /dev/null +++ b/src/features/ConfigSync/customisationSyncModel.ts @@ -0,0 +1,67 @@ +import type { PluginManifest } from "@/deps.ts"; +import type { FilePathWithPrefix } from "@vrtmrz/livesync-commonlib/compat/common/types"; +import { isDocContentSame } from "@vrtmrz/livesync-commonlib/compat/common/utils"; +import { getCustomisationSyncCategoryFolder } from "./customisationSyncPaths.ts"; +import type { IPluginDataExDisplay, LoadedEntryPluginDataExFile } from "./customisationSyncView.ts"; + +export class PluginDataExDisplayV2 { + documentPath: FilePathWithPrefix; + category: string; + term: string; + files: LoadedEntryPluginDataExFile[]; + name: string; + confKey: string; + _displayName: string | undefined; + _version: string | undefined; + + constructor( + data: IPluginDataExDisplay, + private readonly manifestLookup: ReadonlyMap + ) { + this.documentPath = `${data.documentPath}` as FilePathWithPrefix; + this.category = `${data.category}`; + this.name = `${data.name}`; + this.term = `${data.term}`; + this.files = [...(data.files as LoadedEntryPluginDataExFile[])]; + this.confKey = `${getCustomisationSyncCategoryFolder(this.category, this.term)}${this.name}`; + this.applyLoadedManifest(); + } + + async setFile(file: LoadedEntryPluginDataExFile): Promise { + const old = this.files.find((entry) => entry.filename == file.filename); + if (old) { + if (old.mtime == file.mtime && (await isDocContentSame(old.data, file.data))) return; + this.files = this.files.filter((entry) => entry.filename != file.filename); + } + this.files.push(file); + if (file.filename == "manifest.json") { + this.applyLoadedManifest(); + } + } + + deleteFile(filename: string): void { + this.files = this.files.filter((entry) => entry.filename != filename); + } + + applyLoadedManifest(): void { + const manifest = this.manifestLookup.get(this.confKey); + if (manifest) { + this._displayName = manifest.name; + if (this.category == "PLUGIN_MAIN" || this.category == "THEME") { + this._version = manifest.version; + } + } + } + + get displayName(): string { + return this._displayName || this.name; + } + + get version(): string | undefined { + return this._version; + } + + get mtime(): number { + return ~~this.files.reduce((sum, file) => sum + file.mtime, 0) / this.files.length; + } +} diff --git a/src/features/ConfigSync/customisationSyncModel.unit.spec.ts b/src/features/ConfigSync/customisationSyncModel.unit.spec.ts new file mode 100644 index 00000000..5850e702 --- /dev/null +++ b/src/features/ConfigSync/customisationSyncModel.unit.spec.ts @@ -0,0 +1,58 @@ +import { describe, expect, it } from "vitest"; +import type { PluginManifest } from "@/deps.ts"; +import type { FilePathWithPrefix } from "@vrtmrz/livesync-commonlib/compat/common/types"; +import { PluginDataExDisplayV2 } from "./customisationSyncModel.ts"; +import type { IPluginDataExDisplay, LoadedEntryPluginDataExFile } from "./customisationSyncView.ts"; + +function file(filename: string, mtime: number, data: string[]): LoadedEntryPluginDataExFile { + return { filename, mtime, data, size: data.join("").length } as LoadedEntryPluginDataExFile; +} + +function display(files: LoadedEntryPluginDataExFile[] = []): IPluginDataExDisplay { + return { + documentPath: "ix:device-a/PLUGIN_MAIN/example%main.js" as FilePathWithPrefix, + category: "PLUGIN_MAIN", + name: "example", + term: "device-a", + files, + mtime: 0, + }; +} + +describe("PluginDataExDisplayV2", () => { + it("projects manifest identity and file modification time", () => { + const manifests = new Map([ + ["device-a/plugins/example", { name: "Example plug-in", version: "1.2.3" } as PluginManifest], + ]); + const model = new PluginDataExDisplayV2( + display([file("main.js", 10, ["main"]), file("data.json", 20, ["data"])]), + manifests + ); + + expect(model.confKey).toBe("device-a/plugins/example"); + expect(model.displayName).toBe("Example plug-in"); + expect(model.version).toBe("1.2.3"); + expect(model.mtime).toBe(15); + }); + + it("retains an unchanged file and replaces changed content", async () => { + const original = file("main.js", 10, ["same"]); + const model = new PluginDataExDisplayV2(display([original]), new Map()); + + await model.setFile(file("main.js", 10, ["same"])); + expect(model.files[0]).toBe(original); + + const changed = file("main.js", 10, ["changed"]); + await model.setFile(changed); + expect(model.files).toEqual([changed]); + }); + + it("deletes only the named file", () => { + const retained = file("styles.css", 20, ["css"]); + const model = new PluginDataExDisplayV2(display([file("main.js", 10, ["main"]), retained]), new Map()); + + model.deleteFile("main.js"); + + expect(model.files).toEqual([retained]); + }); +}); diff --git a/src/features/ConfigSync/customisationSyncPaths.ts b/src/features/ConfigSync/customisationSyncPaths.ts index 71a5137a..ac92fa5e 100644 --- a/src/features/ConfigSync/customisationSyncPaths.ts +++ b/src/features/ConfigSync/customisationSyncPaths.ts @@ -17,6 +17,23 @@ export type CustomisationSyncPathOptions = { usePluginEtc: boolean; }; +export function getCustomisationSyncCategoryFolder(category: string, configDir: string = ""): string { + switch (category) { + case "CONFIG": + return `${configDir}/`; + case "THEME": + return `${configDir}/themes/`; + case "SNIPPET": + return `${configDir}/snippets/`; + case "PLUGIN_MAIN": + case "PLUGIN_DATA": + case "PLUGIN_ETC": + return `${configDir}/plugins/`; + default: + return ""; + } +} + export function getCustomisationSyncFileCategory( filePath: string, options: CustomisationSyncPathOptions diff --git a/src/features/ConfigSync/customisationSyncPaths.unit.spec.ts b/src/features/ConfigSync/customisationSyncPaths.unit.spec.ts index ecd2719d..63cb81f3 100644 --- a/src/features/ConfigSync/customisationSyncPaths.unit.spec.ts +++ b/src/features/ConfigSync/customisationSyncPaths.unit.spec.ts @@ -5,6 +5,7 @@ import { createCustomisationSyncDevicePrefix, createCustomisationSyncV1DocumentPath, createCustomisationSyncV2DocumentPath, + getCustomisationSyncCategoryFolder, getCustomisationSyncFileCategory, isCustomisationSyncTargetPath, getCustomisationSyncSettingKey, @@ -20,6 +21,18 @@ const currentOptions: CustomisationSyncPathOptions = { }; describe("compatibility: Customisation Sync path operations", () => { + it.each([ + ["CONFIG", ".obsidian/"], + ["THEME", ".obsidian/themes/"], + ["SNIPPET", ".obsidian/snippets/"], + ["PLUGIN_MAIN", ".obsidian/plugins/"], + ["PLUGIN_DATA", ".obsidian/plugins/"], + ["PLUGIN_ETC", ".obsidian/plugins/"], + ["UNKNOWN", ""], + ])("maps category %s to folder %s", (category, expected) => { + expect(getCustomisationSyncCategoryFolder(category, ".obsidian")).toBe(expected); + }); + it.each([ [".obsidian/app.json", "CONFIG"], [".obsidian/themes/Minimal/theme.css", "THEME"], diff --git a/src/features/ConfigSync/customisationSyncReadOperations.ts b/src/features/ConfigSync/customisationSyncReadOperations.ts new file mode 100644 index 00000000..f34eff83 --- /dev/null +++ b/src/features/ConfigSync/customisationSyncReadOperations.ts @@ -0,0 +1,208 @@ +import type { + FilePath, + FilePathWithPrefix, + LoadedEntry, + LOG_LEVEL, +} from "@vrtmrz/livesync-commonlib/compat/common/types"; +import { LOG_LEVEL_INFO, LOG_LEVEL_VERBOSE } from "@vrtmrz/livesync-commonlib/compat/common/types"; +import type { StorageAccess } from "@vrtmrz/livesync-commonlib/compat/interfaces/StorageAccess"; +import type { LiveSyncLocalDB } from "@vrtmrz/livesync-commonlib/compat/pouchdb/LiveSyncLocalDB"; +import type { IPathService } from "@vrtmrz/livesync-commonlib/compat/services/base/IService"; +import type { LogFunction } from "@vrtmrz/livesync-commonlib/compat/services/lib/logUtils"; +import { + createSavingEntryFromLoadedEntry, + fireAndForget, + getDocData, + getDocDataAsArray, + isLoadedEntry, +} from "@vrtmrz/livesync-commonlib/compat/common/utils"; +import { digestHash } from "@vrtmrz/livesync-commonlib/compat/string_and_binary/hash"; +import { arrayBufferToBase64, readString } from "@vrtmrz/livesync-commonlib/compat/string_and_binary/convert"; +import { base64ToString } from "octagonal-wheels/binary/base64"; + +import type { PluginDataEx, PluginDataExFile } from "./customisationSyncCodec.ts"; +import { getCustomisationSyncCategoryFolder, parseCustomisationSyncV2DocumentPath } from "./customisationSyncPaths.ts"; +import type { LoadedEntryPluginDataExFile, PluginDataExDisplay } from "./customisationSyncView.ts"; + +type CustomisationSyncLogDependency = { + log: LogFunction; +}; + +type CustomisationSyncStorageMethods = { + storageAccess: Pick; +}; + +type CustomisationSyncDatabaseMethods = { + getLocalDatabase(): Pick; +}; + +type CustomisationSyncPathMethods = { + path: Pick; +}; + +export type CustomisationSyncFileReaderDependencies = CustomisationSyncStorageMethods< + "readHiddenFileBinary" | "statHidden" +> & + CustomisationSyncLogDependency; + +export type CustomisationSyncDisplayLoaderDependencies = CustomisationSyncDatabaseMethods<"getDBEntry" | "putDBEntry"> & + CustomisationSyncPathMethods<"getPath"> & + CustomisationSyncLogDependency; + +export type CustomisationSyncV2EntryLoaderDependencies = CustomisationSyncDatabaseMethods<"getDBEntry"> & + CustomisationSyncLogDependency; + +export type CustomisationSyncReadCodec = { + deserialize(source: string[], defaultValue: T): T; + serialize(data: PluginDataEx): string; +}; + +export type DecodedCustomisationSyncV2File = { + confKey: string; + file: LoadedEntryPluginDataExFile; + isManifest: boolean; +}; + +function log(dependencies: CustomisationSyncLogDependency, message: unknown, level?: LOG_LEVEL, key?: string): void { + dependencies.log(message, level, key); +} + +export async function readCustomisationFile( + dependencies: CustomisationSyncFileReaderDependencies, + path: FilePath, + configDir: string +): Promise { + const stat = await dependencies.storageAccess.statHidden(path); + let version: string | undefined; + let displayName: string | undefined; + if (!stat) { + return false; + } + const contentBin = await dependencies.storageAccess.readHiddenFileBinary(path); + let content: string[]; + try { + content = await arrayBufferToBase64(contentBin); + if (path.toLowerCase().endsWith("/manifest.json")) { + const manifestSource = readString(new Uint8Array(contentBin)); + try { + const manifest: unknown = JSON.parse(manifestSource); + if (typeof manifest === "object" && manifest !== null) { + if ("version" in manifest) { + version = String(manifest.version); + } + if ("name" in manifest) { + displayName = String(manifest.name); + } + } + } catch (error) { + log( + dependencies, + `Configuration sync data: ${path} looks like manifest, but could not read the version`, + LOG_LEVEL_INFO + ); + log(dependencies, error, LOG_LEVEL_VERBOSE); + } + } + } catch (error) { + log(dependencies, `The file ${path} could not be encoded`); + log(dependencies, error, LOG_LEVEL_VERBOSE); + return false; + } + return { + // Compatibility: target validation belongs to the caller. The legacy + // reader derives this name positionally without checking the prefix. + filename: path.substring(configDir.length + 1), + data: content, + mtime: stat.mtime, + size: stat.size, + version, + displayName, + }; +} + +export async function loadCustomisationDisplayData( + dependencies: CustomisationSyncDisplayLoaderDependencies, + path: FilePathWithPrefix, + codec: CustomisationSyncReadCodec +): Promise { + const loaded = await dependencies.getLocalDatabase().getDBEntry(path, undefined, false, false); + if (!loaded) { + return false; + } + + const data = codec.deserialize(getDocDataAsArray(loaded.data), {}) as PluginDataEx; + const displayFiles: PluginDataExFile[] = []; + let missingHash = false; + for (const file of data.files) { + const displayFile = { ...file, data: [] as string[] }; + if (!file.hash) { + // Compatibility question: the inherited implementation clears the + // display copy before calculating this temporary hash, so callers + // see digestHash([]) until the asynchronously repaired document is + // loaded again. The serialiser still writes the real content hash. + const temporaryHashSource = getDocDataAsArray(displayFile.data); + file.hash = digestHash(temporaryHashSource); + missingHash = true; + } + displayFile.data = [file.hash]; + displayFiles.push(displayFile); + } + if (missingHash) { + log(dependencies, `Digest created for ${path} to improve checking`, LOG_LEVEL_VERBOSE); + loaded.data = codec.serialize(data); + // Compatibility: catalogue loading does not wait for the repair write. + fireAndForget(() => dependencies.getLocalDatabase().putDBEntry(createSavingEntryFromLoadedEntry(loaded))); + } + return { + ...data, + documentPath: dependencies.path.getPath(loaded), + files: displayFiles, + } satisfies PluginDataExDisplay; +} + +export async function loadCustomisationV2Entry( + dependencies: CustomisationSyncV2EntryLoaderDependencies, + path: FilePathWithPrefix +): Promise { + const loaded = await dependencies.getLocalDatabase().getDBEntry(path); + if (!loaded) { + log(dependencies, `The file ${path} is not found`, LOG_LEVEL_VERBOSE); + return false; + } + if (!isLoadedEntry(loaded)) { + log(dependencies, `The file ${path} is not a note`, LOG_LEVEL_VERBOSE); + return false; + } + return loaded; +} + +export function decodeCustomisationSyncV2File( + path: FilePathWithPrefix, + loaded: LoadedEntry, + dummyEnd: string +): DecodedCustomisationSyncV2File { + const { category, key, filename, device } = parseCustomisationSyncV2DocumentPath(path); + const categoryFolder = getCustomisationSyncCategoryFolder(category, device); + const confKey = `${categoryFolder}${key}`; + const relativeFilename = + `${getCustomisationSyncCategoryFolder(category, "")}${category == "CONFIG" || category == "SNIPPET" ? "" : key + "/"}${filename}`.substring( + 1 + ); + const source = getDocData(loaded.data); + const dataStart = source.indexOf(dummyEnd); + // Compatibility question: a missing marker is not rejected. substring() + // starts at dummyEnd.length - 1, preserving the old best-effort decode. + const encodedData = source.substring(dataStart + dummyEnd.length); + const file: LoadedEntryPluginDataExFile = { + ...loaded, + hash: "", + data: [base64ToString(encodedData)], + filename: relativeFilename, + displayName: filename, + }; + return { + confKey, + file, + isManifest: filename == "manifest.json", + }; +} diff --git a/src/features/ConfigSync/customisationSyncReadOperations.unit.spec.ts b/src/features/ConfigSync/customisationSyncReadOperations.unit.spec.ts new file mode 100644 index 00000000..5b2238db --- /dev/null +++ b/src/features/ConfigSync/customisationSyncReadOperations.unit.spec.ts @@ -0,0 +1,222 @@ +import { describe, expect, it, vi } from "vitest"; +import { + LOG_LEVEL_INFO, + LOG_LEVEL_VERBOSE, + type FilePath, + type FilePathWithPrefix, + type LoadedEntry, + type UXStat, +} from "@vrtmrz/livesync-commonlib/compat/common/types"; +import { digestHash } from "@vrtmrz/livesync-commonlib/compat/string_and_binary/hash"; + +import { createCustomisationSyncCodec, type PluginDataEx } from "./customisationSyncCodec.ts"; +import { + decodeCustomisationSyncV2File, + loadCustomisationDisplayData, + loadCustomisationV2Entry, + readCustomisationFile, +} from "./customisationSyncReadOperations.ts"; + +const configDir = ".obsidian"; +const filePath = ".obsidian/plugins/example/manifest.json" as FilePath; +const documentPath = "ix:device-a/PLUGIN_MAIN/example.md" as FilePathWithPrefix; +const stat = { ctime: 10, mtime: 20, size: 42, type: "file" } as UXStat; +const codec = createCustomisationSyncCodec({ + digestHash, + parseYaml: () => undefined, +}); + +function createDependencies() { + const localDatabase = { + getDBEntry: vi.fn(), + putDBEntry: vi.fn(async (_entry: unknown) => ({ ok: true, id: "id", rev: "1-a" })), + }; + const storageAccess = { + statHidden: vi.fn(async () => stat as UXStat | null), + readHiddenFileBinary: vi.fn(), + }; + const log = vi.fn(); + const dependencies = { + getLocalDatabase: () => localDatabase as never, + storageAccess: storageAccess as never, + path: { + getPath: vi.fn((entry: { path: FilePathWithPrefix }) => entry.path), + } as never, + log, + }; + return { dependencies, localDatabase, log, storageAccess }; +} + +function loadedEntry(path: FilePathWithPrefix, data: string): LoadedEntry { + return { + _id: "entry-id", + _rev: "1-a", + path, + type: "plain", + datatype: "plain", + data, + ctime: 10, + mtime: 20, + size: data.length, + children: [], + eden: {}, + } as unknown as LoadedEntry; +} + +function pluginData(hash?: string): PluginDataEx { + return { + category: "PLUGIN_MAIN", + name: "example", + term: "device-a", + mtime: 20, + files: [ + { + filename: "plugins/example/main.js", + data: ["payload"], + mtime: 20, + size: 7, + hash, + }, + ], + }; +} + +describe("Customisation Sync read operations", () => { + it("does not read content when a local file is missing", async () => { + const { dependencies, storageAccess } = createDependencies(); + storageAccess.statHidden.mockResolvedValue(null); + + await expect(readCustomisationFile(dependencies, filePath, configDir)).resolves.toBe(false); + expect(storageAccess.readHiddenFileBinary).not.toHaveBeenCalled(); + }); + + it("propagates a storage read failure without converting it to an encoding failure", async () => { + const { dependencies, log, storageAccess } = createDependencies(); + const error = new Error("read failed"); + storageAccess.readHiddenFileBinary.mockRejectedValue(error); + + await expect(readCustomisationFile(dependencies, filePath, configDir)).rejects.toBe(error); + expect(log).not.toHaveBeenCalled(); + }); + + it("encodes a manifest and extracts its display metadata", async () => { + const { dependencies, storageAccess } = createDependencies(); + const source = JSON.stringify({ name: "Example plug-in", version: "1.2.3" }); + storageAccess.readHiddenFileBinary.mockResolvedValue(new TextEncoder().encode(source).buffer); + + await expect(readCustomisationFile(dependencies, filePath, configDir)).resolves.toEqual({ + filename: "plugins/example/manifest.json", + data: [btoa(source)], + mtime: 20, + size: 42, + version: "1.2.3", + displayName: "Example plug-in", + }); + }); + + it("keeps an unreadable manifest as file data and reports only the metadata failure", async () => { + const { dependencies, log, storageAccess } = createDependencies(); + const errorSource = "{invalid"; + storageAccess.readHiddenFileBinary.mockResolvedValue(new TextEncoder().encode(errorSource).buffer); + + await expect(readCustomisationFile(dependencies, filePath, configDir)).resolves.toMatchObject({ + filename: "plugins/example/manifest.json", + data: [btoa(errorSource)], + version: undefined, + displayName: undefined, + }); + expect(log).toHaveBeenNthCalledWith( + 1, + `Configuration sync data: ${filePath} looks like manifest, but could not read the version`, + LOG_LEVEL_INFO, + undefined + ); + expect(log).toHaveBeenNthCalledWith(2, expect.any(SyntaxError), LOG_LEVEL_VERBOSE, undefined); + }); + + it("loads V1 display data without retaining file content", async () => { + const { dependencies, localDatabase } = createDependencies(); + const data = pluginData("known-hash"); + localDatabase.getDBEntry.mockResolvedValue(loadedEntry(documentPath, JSON.stringify(data))); + + await expect(loadCustomisationDisplayData(dependencies, documentPath, codec)).resolves.toEqual({ + ...data, + documentPath, + files: [{ ...data.files[0], data: ["known-hash"] }], + }); + expect(localDatabase.getDBEntry).toHaveBeenCalledWith(documentPath, undefined, false, false); + expect(localDatabase.putDBEntry).not.toHaveBeenCalled(); + }); + + it("preserves the inherited transient empty-data hash while repairing a V1 document", async () => { + const { dependencies, localDatabase, log } = createDependencies(); + const data = pluginData(); + localDatabase.getDBEntry.mockResolvedValue(loadedEntry(documentPath, JSON.stringify(data))); + + const result = await loadCustomisationDisplayData(dependencies, documentPath, codec); + + expect(result).toMatchObject({ files: [{ data: [digestHash([])] }] }); + expect(localDatabase.putDBEntry).toHaveBeenCalledOnce(); + const saving = localDatabase.putDBEntry.mock.calls[0][0] as { data: Blob }; + expect(await saving.data.text()).toContain(digestHash(["payload"])); + expect(log).toHaveBeenCalledWith( + `Digest created for ${documentPath} to improve checking`, + LOG_LEVEL_VERBOSE, + undefined + ); + }); + + it("returns false when a V1 document is absent", async () => { + const { dependencies, localDatabase } = createDependencies(); + localDatabase.getDBEntry.mockResolvedValue(false); + + await expect(loadCustomisationDisplayData(dependencies, documentPath, codec)).resolves.toBe(false); + }); + + it("distinguishes an absent V2 entry from a non-note database entry", async () => { + const { dependencies, localDatabase, log } = createDependencies(); + const path = "ix:device-a/PLUGIN_MAIN/example%main.js" as FilePathWithPrefix; + + localDatabase.getDBEntry.mockResolvedValueOnce(false); + await expect(loadCustomisationV2Entry(dependencies, path)).resolves.toBe(false); + expect(log).toHaveBeenLastCalledWith(`The file ${path} is not found`, LOG_LEVEL_VERBOSE, undefined); + + localDatabase.getDBEntry.mockResolvedValueOnce({ path, type: "leaf" }); + await expect(loadCustomisationV2Entry(dependencies, path)).resolves.toBe(false); + expect(log).toHaveBeenLastCalledWith(`The file ${path} is not a note`, LOG_LEVEL_VERBOSE, undefined); + }); + + it("returns a loaded V2 entry after the exact single-argument database lookup", async () => { + const { dependencies, localDatabase } = createDependencies(); + const path = "ix:device-a/PLUGIN_MAIN/example%main.js" as FilePathWithPrefix; + const loaded = loadedEntry(path, "data"); + localDatabase.getDBEntry.mockResolvedValue(loaded); + + await expect(loadCustomisationV2Entry(dependencies, path)).resolves.toBe(loaded); + expect(localDatabase.getDBEntry).toHaveBeenCalledWith(path); + }); + + it("decodes a V2 payload into its relative Customisation Sync filename", () => { + const path = "ix:device-a/PLUGIN_MAIN/example%main.js" as FilePathWithPrefix; + const loaded = loadedEntry(path, `${codec.dummyHead}${codec.dummyEnd}${btoa("console.log('example');")}`); + + expect(decodeCustomisationSyncV2File(path, loaded, codec.dummyEnd)).toEqual({ + confKey: "device-a/plugins/example", + isManifest: false, + file: { + ...loaded, + filename: "plugins/example/main.js", + displayName: "main.js", + hash: "", + data: ["console.log('example');"], + }, + }); + }); + + it("preserves the inherited best-effort offset when a V2 marker is missing", () => { + const path = "ix:device-a/CONFIG/app.json%app.json" as FilePathWithPrefix; + const loaded = loadedEntry(path, `00${btoa("hello")}`); + + expect(decodeCustomisationSyncV2File(path, loaded, "END").file.data).toEqual(["hello"]); + }); +}); diff --git a/src/features/ConfigSync/customisationSyncRecentEventDeduplicator.ts b/src/features/ConfigSync/customisationSyncRecentEventDeduplicator.ts new file mode 100644 index 00000000..85511580 --- /dev/null +++ b/src/features/ConfigSync/customisationSyncRecentEventDeduplicator.ts @@ -0,0 +1,17 @@ +const MAX_RECENT_CUSTOMISATION_EVENTS = 100; + +/** Keeps the bounded newest-first raw-event keys used by Customisation Sync. */ +export class CustomisationSyncRecentEventDeduplicator { + private keys: string[] = []; + + /** + * Records a key when it is new and returns whether the caller should act. + * Native Array#includes is intentional: the old `.contains` extension is + * not available in every runtime where the feature is exercised. + */ + admit(key: string): boolean { + if (this.keys.includes(key)) return false; + this.keys = [key, ...this.keys].slice(0, MAX_RECENT_CUSTOMISATION_EVENTS); + return true; + } +} diff --git a/src/features/ConfigSync/customisationSyncRecentEventDeduplicator.unit.spec.ts b/src/features/ConfigSync/customisationSyncRecentEventDeduplicator.unit.spec.ts new file mode 100644 index 00000000..63efc78e --- /dev/null +++ b/src/features/ConfigSync/customisationSyncRecentEventDeduplicator.unit.spec.ts @@ -0,0 +1,24 @@ +import { describe, expect, it } from "vitest"; + +import { CustomisationSyncRecentEventDeduplicator } from "./customisationSyncRecentEventDeduplicator.ts"; + +describe("Customisation Sync recent raw-event keys", () => { + it("admits a key once and keeps newer keys first", () => { + const history = new CustomisationSyncRecentEventDeduplicator(); + + expect(history.admit("old")).toBe(true); + expect(history.admit("new")).toBe(true); + expect(history.admit("old")).toBe(false); + }); + + it("evicts the oldest key when the newest-first history exceeds 100 entries", () => { + const history = new CustomisationSyncRecentEventDeduplicator(); + + for (let index = 0; index < 101; index++) { + expect(history.admit(`key-${index}`)).toBe(true); + } + + expect(history.admit("key-0")).toBe(true); + expect(history.admit("key-100")).toBe(false); + }); +}); diff --git a/src/features/ConfigSync/customisationSyncView.ts b/src/features/ConfigSync/customisationSyncView.ts index d0c664c1..9089771e 100644 --- a/src/features/ConfigSync/customisationSyncView.ts +++ b/src/features/ConfigSync/customisationSyncView.ts @@ -1,15 +1,22 @@ import type { PluginManifest } from "@/deps.ts"; import type { + EntryDoc, FilePathWithPrefix, + FilePath, LoadedEntry, PluginSyncSettingEntry, SYNC_MODE, } from "@vrtmrz/livesync-commonlib/compat/common/types"; +import type PouchDB from "pouchdb-core"; import type { Readable } from "svelte/store"; import type { PluginDataExFile } from "./customisationSyncCodec.ts"; +import type { CustomisationSyncFileCategory } from "./customisationSyncPaths.ts"; +import type { OptionalSyncFeatureMode } from "@/features/optionalSyncFeatures.ts"; +import type { PluginDataExDisplayV2 } from "./customisationSyncModel.ts"; export type LoadedEntryPluginDataExFile = LoadedEntry & PluginDataExFile; +export type { CustomisationSyncFileCategory } from "./customisationSyncPaths.ts"; export interface IPluginDataExDisplay { documentPath: FilePathWithPrefix; @@ -33,6 +40,48 @@ export type PluginDataExDisplay = { mtime: number; }; +/** + * Semantic callbacks registered by the optional-file composition feature. + * + * The context owns the implementations, while the optional-file composition + * adapts these operations to Commonlib's aggregation contracts. Consumers + * receive only callable operations, not the context or its private state. + */ +export interface CustomisationSyncServiceHandlers { + readonly processOptionalFileEvent: (path: FilePath) => Promise; + readonly processVirtualDocument: (docs: PouchDB.Core.ExistingDocument) => Promise; + readonly onRealiseSetting: () => Promise; + readonly onResuming: () => Promise; + readonly onBeforeReplicate: (showMessage: boolean) => Promise; + readonly onDatabaseInitialised: (showNotice: boolean) => Promise; + readonly suspendExtraSync: () => Promise; + readonly enableOptionalFeature: (mode: OptionalSyncFeatureMode) => Promise; +} + +/** + * Explicit internal operations used by maintained real-Obsidian contract + * tests. This is deliberately narrower than the concrete context and does + * not expose reactive stores, queues, or host dependencies. + */ +export interface CustomisationSyncTestingView { + readonly configDir: string; + scanInternalFiles(): Promise; + scanAllConfigFiles(showMessage: boolean): Promise; + getFileCategory(filePath: string): CustomisationSyncFileCategory; + isTargetPath(filePath: string): boolean; + filenameToUnifiedKey(path: string, termOverride?: string): FilePathWithPrefix; + filenameWithUnifiedKey(path: string, termOverride?: string): FilePathWithPrefix; + unifiedKeyPrefixOfTerminal(termOverride?: string): string; + storeCustomizationFiles(path: FilePath, termOverride?: string): Promise; + deleteConfigOnDatabase(prefixedFileName: FilePathWithPrefix, forceWrite?: boolean): Promise; + createPluginDataFromV2(unifiedPathV2: FilePathWithPrefix): PluginDataExDisplayV2 | undefined; + createPluginDataExFileV2( + unifiedPathV2: FilePathWithPrefix, + loaded?: LoadedEntry + ): Promise; + applyDataV2(data: PluginDataExDisplayV2, content?: string): Promise; +} + /** Stable catalogue and operation surface consumed by the Obsidian dialogue. */ export interface CustomisationSyncDialogView { readonly catalogue: Readable; diff --git a/src/features/HiddenFileSync/hiddenFileSyncChangeProcessor.ts b/src/features/HiddenFileSync/hiddenFileSyncChangeProcessor.ts new file mode 100644 index 00000000..c9ffe024 --- /dev/null +++ b/src/features/HiddenFileSync/hiddenFileSyncChangeProcessor.ts @@ -0,0 +1,260 @@ +import { + LOG_LEVEL_DEBUG, + LOG_LEVEL_VERBOSE, + type FilePath, + type FilePathWithPrefix, + type LoadedEntry, + type LOG_LEVEL, + type MetaEntry, + type UXFileInfo, +} from "@vrtmrz/livesync-commonlib/compat/common/types"; +import type { StorageAccess } from "@vrtmrz/livesync-commonlib/compat/interfaces/StorageAccess"; +import type { LogFunction } from "@vrtmrz/livesync-commonlib/compat/services/lib/logUtils"; +import { addPrefix } from "@vrtmrz/livesync-commonlib/compat/string_and_binary/path"; +import { serialized } from "octagonal-wheels/concurrency/lock"; +import { Semaphore } from "octagonal-wheels/concurrency/semaphore"; + +import { ICHeader } from "@/common/types.ts"; +import type { HiddenFileSyncConflictResolution } from "./hiddenFileSyncConflictResolution.ts"; +import type { HiddenFileSyncDatabaseExtractionOperations } from "./hiddenFileSyncDatabaseExtractionOperations.ts"; +import type { HiddenFileSyncDatabaseWriteOperations } from "./hiddenFileSyncDatabaseWriteOperations.ts"; +import type { HiddenFileSyncProcessedState } from "./hiddenFileSyncProcessedState.ts"; +import { getHiddenFileSyncComparisonMTime } from "./hiddenFileSyncState.ts"; +import { compareMTime, TARGET_IS_NEW } from "@/common/utils.ts"; + +type HiddenFileSyncStorageChangeAccess = Pick; + +export type HiddenFileSyncChangeProcessorDependencies = { + storageAccess: HiddenFileSyncStorageChangeAccess; + readFileWithInfo(path: FilePath): Promise; + loadDatabaseMetadata(path: FilePathWithPrefix): Promise; + databaseWriteOperations: Pick; + databaseExtractionOperations: Pick; + processedState: Pick< + HiddenFileSyncProcessedState, + | "fileToStatKey" + | "getLastProcessedFileKey" + | "getLastProcessedFileMTime" + | "updateLastProcessedFile" + | "updateLastProcessed" + >; + conflictResolution: Pick; + log: LogFunction; + publishActivity(eventCount: number, processingCount: number): void; +}; + +export type HiddenFileSyncDatabaseChangeOptions = Readonly<{ + preventDoubleProcess?: boolean; + onlyNew?: boolean; + metaEntry?: MetaEntry | false; + includeDeletion?: boolean; +}>; + +export type HiddenFileSyncChangeProcessor = { + processStorageChange( + path: FilePath, + onlyNew?: boolean, + forceWrite?: boolean, + includeDeleted?: boolean + ): Promise; + processDatabaseChange( + path: FilePath, + headerLine: string, + options?: HiddenFileSyncDatabaseChangeOptions + ): Promise; + dispose(): void; +}; + +class HiddenFileSyncChangeProcessorOwner implements HiddenFileSyncChangeProcessor { + private readonly semaphore = Semaphore(10); + private eventCount = 0; + private processingCount = 0; + private disposed = false; + + constructor(private readonly dependencies: HiddenFileSyncChangeProcessorDependencies) {} + + async processStorageChange( + path: FilePath, + onlyNew = false, + forceWrite = false, + includeDeleted = true + ): Promise { + try { + return await this.serialiseForEvent(path, async () => { + let stat = await this.dependencies.storageAccess.statHidden(path); + // Sometimes a folder is delivered as a file event. + if (stat != null && stat.type != "file") { + return false; + } + const key = await this.dependencies.processedState.fileToStatKey(path, stat); + // A raw event can occur while the file is being read. Scans + // still enumerate every path, but event admission skips this + // exact already-settled key. + const lastKey = this.dependencies.processedState.getLastProcessedFileKey(path); + if (lastKey == key) { + this.log(`${path} Already processed.`, LOG_LEVEL_DEBUG); + return true; + } + + // Read the stat and content as one operation. The stat is + // deliberately compared again below: a file can change while + // the first stat is in flight. + const fileInfo = await this.dependencies.readFileWithInfo(path); + const cacheMTime = getHiddenFileSyncComparisonMTime(fileInfo.stat); + const statMtime = getHiddenFileSyncComparisonMTime(stat); + if (cacheMTime != statMtime) { + this.log(`Hidden file:${path} is changed.`, LOG_LEVEL_VERBOSE); + stat = fileInfo.stat; + } + + // Compatibility: the storage marker advances before the + // database operation. A later write failure can therefore + // leave this event marked as processed until a scan or state + // change causes it to be reconsidered. + this.dependencies.processedState.updateLastProcessedFile(path, stat!); + const lastIsNotFound = !lastKey || lastKey.endsWith("-0-0"); + const nowIsNotFound = fileInfo.deleted; + const type = lastIsNotFound && nowIsNotFound ? "invalid" : nowIsNotFound ? "delete" : "modified"; + + if (type == "invalid") { + // Maybe the folder was deleted. + return false; + } + + const storageMTimeActual = getHiddenFileSyncComparisonMTime(stat); + const storageMTime = + storageMTimeActual == 0 + ? this.dependencies.processedState.getLastProcessedFileMTime(path) + : storageMTimeActual; + + if (onlyNew) { + const prefixedFileName = addPrefix(path, ICHeader); + const fileOnDatabase = await this.dependencies.loadDatabaseMetadata(prefixedFileName); + const databaseMTime = getHiddenFileSyncComparisonMTime(fileOnDatabase, includeDeleted); + const difference = compareMTime(storageMTime, databaseMTime); + if (difference != TARGET_IS_NEW) { + this.log(`Hidden file:${path} is not new.`, LOG_LEVEL_VERBOSE); + // OnlyNew does not handle a deletion. Preserve the + // inherited partial settlement when both values exist. + if (fileOnDatabase && stat) { + this.dependencies.processedState.updateLastProcessed(path, fileOnDatabase, stat); + } + return true; + } + } + + if (type == "delete") { + this.log(`Deletion detected: ${path}`); + return await this.dependencies.databaseWriteOperations.delete(path, forceWrite); + } + if (type == "modified") { + this.log(`Modification detected:${path}`, LOG_LEVEL_VERBOSE); + const result = await this.dependencies.databaseWriteOperations.store(fileInfo, forceWrite); + const resultText = result === undefined ? "Nothing changed" : result ? "Updated" : "Failed"; + this.log(`${resultText}: ${path} ${resultText}`, LOG_LEVEL_VERBOSE); + return result; + } + return false; + }); + } catch (error) { + this.log(`Failed to process hidden file:${path}`); + this.log(error, LOG_LEVEL_VERBOSE); + } + // Could not be processed, but it was this operation's event. Return + // true to prevent a later handler from claiming it. + return true; + } + + async processDatabaseChange( + path: FilePath, + headerLine: string, + options: HiddenFileSyncDatabaseChangeOptions = {} + ): Promise { + const { + preventDoubleProcess = false, + onlyNew = false, + metaEntry = false, + includeDeletion = true, + } = options; + return await this.serialiseForEvent(path, async () => { + try { + const prefixedPath = addPrefix(path, ICHeader); + const docMeta = metaEntry + ? metaEntry + : await this.dependencies.loadDatabaseMetadata(prefixedPath); + if (docMeta === false) { + this.log(`${headerLine}: Failed to read detail of ${path}`); + throw new Error(`Failed to read detail ${path}`); + } + if (docMeta._conflicts && docMeta._conflicts.length > 0) { + this.dependencies.conflictResolution.queue(path); + this.log(`${headerLine} Hidden file conflicted, enqueued to resolve`); + return true; + } + const extracted = await this.dependencies.databaseExtractionOperations.extract(path, { + metaEntry: docMeta, + preventDoubleProcess, + onlyNew, + includeDeletion, + }); + if (extracted) { + this.log(`${headerLine} Hidden file processed`); + } + } catch (error) { + this.log(`${headerLine} Failed to process hidden file`); + this.log(error, LOG_LEVEL_VERBOSE); + } + // Compatibility: recognition consumes the database event even when + // extraction returned false or threw. A later scan or state change, + // rather than handler fall-through, is responsible for retrying it. + return true; + }); + } + + dispose(): void { + if (this.disposed) return; + this.disposed = true; + this.eventCount = 0; + this.processingCount = 0; + this.publishActivity(); + } + + private async serialiseForEvent(file: FilePath, operation: () => Promise): Promise { + this.eventCount++; + this.publishActivity(); + const release = await this.semaphore.acquire(); + try { + return await serialized(`hidden-file-event:${file}`, async () => { + this.processingCount++; + this.publishActivity(); + try { + return await operation(); + } finally { + this.processingCount = Math.max(0, this.processingCount - 1); + this.publishActivity(); + } + }); + } finally { + release(); + this.eventCount = Math.max(0, this.eventCount - 1); + this.publishActivity(); + } + } + + private publishActivity(): void { + this.dependencies.publishActivity( + this.disposed ? 0 : this.eventCount, + this.disposed ? 0 : this.processingCount + ); + } + + private log(message: unknown, level?: LOG_LEVEL, key?: string): void { + this.dependencies.log(message, level, key); + } +} + +export function createHiddenFileSyncChangeProcessor( + dependencies: HiddenFileSyncChangeProcessorDependencies +): HiddenFileSyncChangeProcessor { + return new HiddenFileSyncChangeProcessorOwner(dependencies); +} diff --git a/src/features/HiddenFileSync/hiddenFileSyncChangeProcessor.unit.spec.ts b/src/features/HiddenFileSync/hiddenFileSyncChangeProcessor.unit.spec.ts new file mode 100644 index 00000000..8ff112d7 --- /dev/null +++ b/src/features/HiddenFileSync/hiddenFileSyncChangeProcessor.unit.spec.ts @@ -0,0 +1,159 @@ +import { describe, expect, it, vi } from "vitest"; +import type { FilePath, MetaEntry, UXFileInfo, UXStat } from "@vrtmrz/livesync-commonlib/compat/common/types"; + +vi.mock("@/deps.ts", () => ({})); + +import { + createHiddenFileSyncChangeProcessor, + type HiddenFileSyncChangeProcessorDependencies, +} from "./hiddenFileSyncChangeProcessor.ts"; + +const path = ".obsidian/app.json" as FilePath; +const stat = { ctime: 1, mtime: 2, size: 3, type: "file" } as UXStat; + +function fileInfo(): UXFileInfo { + return { + path, + name: "app.json", + isInternal: true, + deleted: false, + body: new Blob(["{}"]), + stat, + } as UXFileInfo; +} + +function metadata(): MetaEntry { + return { + _id: "i:app", + _rev: "2-current", + path: `i:${path}`, + type: "plain", + datatype: "plain", + ctime: 1, + mtime: 2, + size: 3, + children: [], + eden: {}, + deleted: false, + } as unknown as MetaEntry; +} + +function createDependencies( + overrides: Partial = {} +): HiddenFileSyncChangeProcessorDependencies { + const state = { + fileToStatKey: vi.fn(async () => "2-3"), + getLastProcessedFileKey: vi.fn(() => undefined), + getLastProcessedFileMTime: vi.fn(() => 0), + databaseStateKey: vi.fn(() => "2-3-2-current--1"), + getLastProcessedDatabaseKey: vi.fn(() => undefined), + updateLastProcessedFile: vi.fn(), + updateLastProcessedDatabase: vi.fn(), + updateLastProcessed: vi.fn(), + }; + return { + storageAccess: { + statHidden: vi.fn(async () => stat), + }, + readFileWithInfo: vi.fn(async () => fileInfo()), + loadDatabaseMetadata: vi.fn(async () => metadata()), + databaseWriteOperations: { + store: vi.fn(async () => true), + delete: vi.fn(async () => true), + }, + databaseExtractionOperations: { + extract: vi.fn(async () => true), + }, + processedState: state, + conflictResolution: { queue: vi.fn() }, + log: vi.fn(), + publishActivity: vi.fn(), + ...overrides, + } as HiddenFileSyncChangeProcessorDependencies; +} + +describe("HiddenFileSyncChangeProcessor activity and serialisation", () => { + it("publishes admission, processing, and release transitions", async () => { + const dependencies = createDependencies(); + const processor = createHiddenFileSyncChangeProcessor(dependencies); + + await expect(processor.processStorageChange(path)).resolves.toBe(true); + + const publishActivity = vi.mocked(dependencies.publishActivity); + expect(publishActivity.mock.calls).toEqual([ + [1, 0], + [1, 1], + [1, 0], + [0, 0], + ]); + processor.dispose(); + }); + + it("serialises same-path storage changes while allowing each event to settle", async () => { + let active = 0; + let maximumActive = 0; + let releaseFirst!: () => void; + const firstStarted = new Promise((resolve) => { + const write = resolve; + releaseFirst = write; + }); + const dependencies = createDependencies({ + databaseWriteOperations: { + store: vi.fn(async () => { + active++; + maximumActive = Math.max(maximumActive, active); + if (active == 1) await firstStarted; + active--; + return true; + }), + delete: vi.fn(async () => true), + }, + }); + const processor = createHiddenFileSyncChangeProcessor(dependencies); + const first = processor.processStorageChange(path); + await vi.waitFor(() => expect(dependencies.databaseWriteOperations.store).toHaveBeenCalledOnce()); + const second = processor.processStorageChange(path); + await new Promise((resolve) => setTimeout(resolve, 0)); + expect(dependencies.databaseWriteOperations.store).toHaveBeenCalledOnce(); + releaseFirst(); + await expect(first).resolves.toBe(true); + await expect(second).resolves.toBe(true); + expect(maximumActive).toBe(1); + expect(dependencies.databaseWriteOperations.store).toHaveBeenCalledTimes(2); + processor.dispose(); + }); +}); + +describe("HiddenFileSyncChangeProcessor compatibility settlement", () => { + it("consumes database events when metadata loading fails", async () => { + const error = new Error("metadata unavailable"); + const dependencies = createDependencies({ + loadDatabaseMetadata: vi.fn(async () => { + throw error; + }), + }); + const processor = createHiddenFileSyncChangeProcessor(dependencies); + + await expect(processor.processDatabaseChange(path, "[Replication]")).resolves.toBe(true); + + expect(dependencies.log).toHaveBeenCalledWith("[Replication] Failed to process hidden file", undefined, undefined); + expect(dependencies.log).toHaveBeenCalledWith(error, expect.any(Number), undefined); + processor.dispose(); + }); + + it("advances the storage marker before a failed database write", async () => { + const dependencies = createDependencies({ + databaseWriteOperations: { + store: vi.fn(async () => false), + delete: vi.fn(async () => true), + }, + }); + const processor = createHiddenFileSyncChangeProcessor(dependencies); + + await expect(processor.processStorageChange(path)).resolves.toBe(false); + + expect(dependencies.processedState.updateLastProcessedFile).toHaveBeenCalledWith(path, stat); + expect(dependencies.databaseWriteOperations.store).toHaveBeenCalledOnce(); + processor.dispose(); + }); +}); diff --git a/src/features/HiddenFileSync/hiddenFileSyncConflictResolution.ts b/src/features/HiddenFileSync/hiddenFileSyncConflictResolution.ts new file mode 100644 index 00000000..239bf5cb --- /dev/null +++ b/src/features/HiddenFileSync/hiddenFileSyncConflictResolution.ts @@ -0,0 +1,426 @@ +import { + LOG_LEVEL_INFO, + LOG_LEVEL_VERBOSE, + type DocumentID, + type FilePath, + type FilePathWithPrefix, + type LoadedEntry, + type LOG_LEVEL, + type MetaEntry, + type UXStat, +} from "@vrtmrz/livesync-commonlib/compat/common/types"; +import type { LogFunction } from "@vrtmrz/livesync-commonlib/compat/services/lib/logUtils"; +import { isInternalMetadata } from "@vrtmrz/livesync-commonlib/compat/common/typeUtils"; +import { stripAllPrefixes } from "@vrtmrz/livesync-commonlib/compat/string_and_binary/path"; +import { QueueProcessor } from "octagonal-wheels/concurrency/processor"; + +import type { InternalFileInfo } from "@/common/types.ts"; +import { getHiddenFileSyncComparisonMTime } from "./hiddenFileSyncState.ts"; + +export type HiddenFileSyncConflictPath = FilePath | FilePathWithPrefix; + +export type HiddenFileSyncRevisionInfo = { + rev: string; + status: string; +}; + +export type HiddenFileSyncRevisionHistory = MetaEntry & { + _revs_info?: HiddenFileSyncRevisionInfo[]; +}; + +export type HiddenFileSyncJsonResolution = { + keepRevision?: string; + mergedText?: string; +}; + +export type HiddenFileSyncConflictDatabase = { + scanConflictedEntries(): AsyncIterable; + getDocumentId(path: HiddenFileSyncConflictPath): Promise; + loadCurrentMetadata(id: DocumentID): Promise; + loadConflictingMetadata(id: DocumentID, revision: string): Promise; + loadRevisionHistory(id: DocumentID): Promise; + loadRevisionEntry(path: HiddenFileSyncConflictPath, revision: string): Promise; + mergeJson( + path: FilePathWithPrefix, + baseRevision: string, + currentRevision: string, + conflictedRevision: string + ): Promise; + removeRevision(id: DocumentID, revision: string): Promise; + deleteRevision(entry: LoadedEntry): Promise; +}; + +export type HiddenFileSyncConflictStorage = { + ensureDirectory(path: FilePath): Promise; + writeFile(path: FilePath, data: string): Promise; + triggerEvent(path: FilePath): Promise; +}; + +export type HiddenFileSyncConflictReconciliation = { + storeFile(file: InternalFileInfo, forceWrite?: boolean): Promise; + extractFile(path: FilePath): Promise; +}; + +export type HiddenFileSyncConflictInteraction = { + resolveJsonConflict( + path: FilePath, + docs: [LoadedEntry, LoadedEntry], + apply: (resolution: HiddenFileSyncJsonResolution) => Promise + ): Promise; +}; + +/** Read-only queue counters retained for the real-Obsidian contract tests. */ +export type HiddenFileSyncConflictProcessorTestingView = { + readonly remaining: number; + readonly totalRemaining: number; + readonly nowProcessing: number; +}; + +/** Focused conflict operations exposed through the Hidden File Sync test view. */ +export interface HiddenFileSyncConflictTestingView { + resolveAll(): Promise; + resolveJson(docA: LoadedEntry, docB: LoadedEntry): Promise; + readonly pendingPaths: readonly HiddenFileSyncConflictPath[]; + readonly processor: HiddenFileSyncConflictProcessorTestingView; +} + +export type HiddenFileSyncConflictResolutionDependencies = { + database: HiddenFileSyncConflictDatabase; + storage: HiddenFileSyncConflictStorage; + reconciliation: HiddenFileSyncConflictReconciliation; + interaction: HiddenFileSyncConflictInteraction; + shouldOverwrite(path: FilePath): boolean; + log: LogFunction; +}; + +export interface HiddenFileSyncConflictResolution { + queue(path: HiddenFileSyncConflictPath): void; + resolveAll(): Promise; + resolveJson(docA: LoadedEntry, docB: LoadedEntry): Promise; + dispose(): void; + readonly testing: HiddenFileSyncConflictTestingView; +} + +type PendingJsonConflict = { + id: DocumentID; + doc: MetaEntry; + path: HiddenFileSyncConflictPath; + revA: string; + revB: string; +}; + +export function selectHiddenFileSyncRevisionToDelete( + currentDoc: MetaEntry, + currentRevision: string, + conflictedDoc: MetaEntry, + conflictedRevision: string +): string { + const currentMTime = getHiddenFileSyncComparisonMTime(currentDoc, true); + const conflictedMTime = getHiddenFileSyncComparisonMTime(conflictedDoc, true); + // Compatibility: an equal mtime keeps the current leaf and deletes the + // conflicted leaf. A different tie-breaker would alter existing winners. + return currentMTime < conflictedMTime ? currentRevision : conflictedRevision; +} + +export function findHiddenFileSyncMergeBase( + revisions: readonly HiddenFileSyncRevisionInfo[] | undefined, + conflictedRevision: string +): string { + const conflictedGeneration = Number(conflictedRevision.split("-")[0]); + // Compatibility question: this is the first available lower generation + // from the current branch, not a proven nearest shared ancestor. Changing + // it requires a separate conflict-history decision. + return ( + revisions?.find(({ rev, status }) => status == "available" && Number(rev.split("-")[0]) < conflictedGeneration) + ?.rev ?? "" + ); +} + +class HiddenFileSyncConflictResolutionOwner implements HiddenFileSyncConflictResolution { + private readonly pendingPaths = new Set(); + private readonly processor: QueueProcessor; + private disposed = false; + readonly testing: HiddenFileSyncConflictTestingView; + + constructor(private readonly dependencies: HiddenFileSyncConflictResolutionDependencies) { + const interactionProcessor = new QueueProcessor( + async (results) => { + const { id, doc, path, revA, revB } = results[0]; + // Compatibility question: these reads intentionally remain + // outside the catch below. A rejected read can leave the path + // pending until another lifecycle event reconstructs the owner. + const docAMerge = await this.dependencies.database.loadRevisionEntry(path, revA); + const docBMerge = await this.dependencies.database.loadRevisionEntry(path, revB); + try { + if (docAMerge != false && docBMerge != false) { + if (await this.resolveJson(docAMerge, docBMerge)) { + this.requeue(path); + } else { + this.finish(path); + } + return; + } + await this.resolveByNewerEntry(id, path, doc, revA, revB); + } catch (error) { + this.finish(path); + throw error; + } + }, + { + suspended: false, + batchSize: 1, + concurrentLimit: 1, + delay: 10, + keepResultUntilDownstreamConnected: false, + yieldThreshold: 10, + } + ); + this.processor = new QueueProcessor( + async (paths) => await this.processPath(paths[0]), + { + suspended: false, + batchSize: 1, + concurrentLimit: 5, + delay: 10, + keepResultUntilDownstreamConnected: true, + yieldThreshold: 10, + pipeTo: interactionProcessor, + } + ); + const pendingPaths = () => [...this.pendingPaths]; + const processor = this.processor; + const processorView = Object.freeze({ + get remaining() { + return processor.remaining; + }, + get totalRemaining() { + return processor.totalRemaining; + }, + get nowProcessing() { + return processor.nowProcessing; + }, + }); + this.testing = Object.freeze({ + resolveAll: async () => await this.resolveAll(), + resolveJson: async (docA: LoadedEntry, docB: LoadedEntry) => await this.resolveJson(docA, docB), + get pendingPaths() { + return pendingPaths(); + }, + processor: processorView, + }); + } + + queue(path: HiddenFileSyncConflictPath): void { + if (this.disposed) return; + // Compatibility: this deliberately deduplicates exact strings only. + // Prefixed and unprefixed forms of one path can therefore coexist. + if (this.pendingPaths.has(path)) return; + this.pendingPaths.add(path); + // Compatibility question: if QueueProcessor throws during this + // synchronous admission, the pending marker is retained. No current + // caller expects enqueue to throw. + this.processor.enqueue(path); + } + + async resolveAll(): Promise { + // Creating the iterator and awaiting the completed pipeline remain + // outside the catch. Only iteration failures are logged and swallowed + // by this operation. + const conflicted = this.dependencies.database.scanConflictedEntries(); + // Do not suspend ordinary conflict admission during the scan. + // QueueProcessor v2 can lose its resume event when scan completion + // races with the suspended pump, leaving every admitted path pending. + try { + for await (const doc of conflicted) { + if (!("_conflicts" in doc)) continue; + if (isInternalMetadata(doc._id)) { + this.queue(doc.path); + } + } + } catch (error) { + this.log("something went wrong on resolving all conflicted internal files"); + this.log(error, LOG_LEVEL_VERBOSE); + } + await this.processor.waitForAllProcessed(); + } + + async resolveJson(docA: LoadedEntry, docB: LoadedEntry): Promise { + this.log("Opening data-merging dialog", LOG_LEVEL_VERBOSE); + const docs: [LoadedEntry, LoadedEntry] = [docA, docB]; + const storageFilePath = stripAllPrefixes(docA.path); + const displayFilename = `${storageFilePath}`; + return await this.dependencies.interaction.resolveJsonConflict( + storageFilePath, + docs, + async ({ keepRevision: keep, mergedText: result }) => { + try { + let needFlush = false; + if (!result && !keep) { + this.log(`Skipped merging: ${displayFilename}`); + return false; + } + // Compatibility question: the selected revision is not + // validated against these two documents. An unknown value + // consequently deletes both revisions without writing a + // merged result. The sequential effects are also not + // transactional, so an earlier deletion survives a later + // failure. + for (const doc of docs) { + if (doc._rev != keep) { + if (await this.dependencies.database.deleteRevision(doc)) { + this.log(`Conflicted revision has been deleted: ${displayFilename}`); + needFlush = true; + } + } + } + if (!keep && result) { + await this.dependencies.storage.ensureDirectory(storageFilePath); + const stat = await this.dependencies.storage.writeFile(storageFilePath, result); + if (!stat) { + throw new Error("Stat failed"); + } + const mtime = getHiddenFileSyncComparisonMTime(stat); + // Compatibility: interactive merged text forces the + // database write, whereas automatic merge below uses + // the writer's default admission policy. + await this.dependencies.reconciliation.storeFile( + { + path: storageFilePath, + mtime, + ctime: stat.ctime ?? mtime, + size: stat.size ?? 0, + }, + true + ); + await this.dependencies.storage.triggerEvent(storageFilePath); + this.log(`STORAGE <-- DB:${displayFilename}: written (hidden,merged)`); + } + if (needFlush) { + if (await this.dependencies.reconciliation.extractFile(storageFilePath)) { + this.log(`STORAGE --> DB:${displayFilename}: extracted (hidden,merged)`); + } else { + this.log(`STORAGE --> DB:${displayFilename}: extracted (hidden,merged) Failed`); + } + } + return true; + } catch (error) { + this.log("Could not merge conflicted json"); + this.log(error, LOG_LEVEL_VERBOSE); + return false; + } + } + ); + } + + dispose(): void { + if (this.disposed) return; + this.disposed = true; + // QueueProcessor termination cascades downstream, but cannot cancel an + // already-running database operation or dialogue callback. + this.processor.terminate(); + this.pendingPaths.clear(); + } + + private async processPath(path: HiddenFileSyncConflictPath): Promise { + try { + const id = await this.dependencies.database.getDocumentId(path); + const doc = await this.dependencies.database.loadCurrentMetadata(id); + if (doc._conflicts === undefined || doc._conflicts.length == 0) { + this.finish(path); + return []; + } + this.log(`Hidden file conflicted:${path}`); + // Compatibility: sorting mutates the loaded Metadata object before + // it is forwarded to the manual-resolution stage. + const conflicts = doc._conflicts.sort((a, b) => Number(a.split("-")[0]) - Number(b.split("-")[0])); + const revA = doc._rev!; + const revB = conflicts[0]; + + if (path.endsWith(".json")) { + const revisionHistory = await this.dependencies.database.loadRevisionHistory(id); + const commonBase = findHiddenFileSyncMergeBase(revisionHistory._revs_info, revB); + const result = await this.dependencies.database.mergeJson(doc.path, commonBase, revA, revB); + if (result) { + this.log(`Object merge:${path}`, LOG_LEVEL_INFO); + const filename = stripAllPrefixes(path); + await this.dependencies.storage.ensureDirectory(filename); + const stat = await this.dependencies.storage.writeFile(filename, result); + if (!stat) { + throw new Error(`HiddenFileSyncConflictResolution: Failed to stat file ${filename}`); + } + await this.dependencies.reconciliation.storeFile({ path: filename, ...stat }); + // Compatibility question: extraction is attempted before + // the conflicted branch is removed, so its conflict guard + // normally refuses it. Requeueing eventually reflects the + // winner; changing the order needs a separate decision. + await this.dependencies.reconciliation.extractFile(filename); + await this.dependencies.database.removeRevision(id, revB); + this.requeue(path); + return []; + } + this.log(`Object merge is not applicable.`, LOG_LEVEL_VERBOSE); + if (this.dependencies.shouldOverwrite(stripAllPrefixes(path))) { + this.log(`Overwrite rule applied for conflicted hidden file: ${path}`, LOG_LEVEL_INFO); + await this.resolveByNewerEntry(id, path, doc, revA, revB); + return []; + } + return [{ path, revA, revB, id, doc }]; + } + await this.resolveByNewerEntry(id, path, doc, revA, revB); + return []; + } catch (error) { + this.finish(path); + this.log(`Failed to resolve conflict (Hidden): ${path}`); + this.log(error, LOG_LEVEL_VERBOSE); + return []; + } + } + + private async resolveByNewerEntry( + id: DocumentID, + path: HiddenFileSyncConflictPath, + currentDoc: MetaEntry, + currentRevision: string, + conflictedRevision: string + ): Promise { + const conflictedDoc = await this.dependencies.database.loadConflictingMetadata(id, conflictedRevision); + const revisionToDelete = selectHiddenFileSyncRevisionToDelete( + currentDoc, + currentRevision, + conflictedDoc, + conflictedRevision + ); + // Compatibility: the database result is ignored. The following conflict + // read, rather than the deletion response, decides settlement. + await this.dependencies.database.removeRevision(id, revisionToDelete); + this.log(`Older one has been deleted:${path}`); + const current = await this.dependencies.database.loadCurrentMetadata(id); + if (current._conflicts?.length === 0) { + await this.dependencies.reconciliation.extractFile(stripAllPrefixes(path)); + this.finish(path); + } else { + // Compatibility: an absent _conflicts field is not considered + // settled here, although the main path treats it as conflict-free. + this.requeue(path); + } + } + + private finish(path: HiddenFileSyncConflictPath): void { + this.pendingPaths.delete(path); + } + + private requeue(path: HiddenFileSyncConflictPath): void { + this.finish(path); + this.queue(path); + } + + private log(message: unknown, level?: LOG_LEVEL, key?: string): void { + this.dependencies.log(message, level, key); + } +} + +export function createHiddenFileSyncConflictResolution( + dependencies: HiddenFileSyncConflictResolutionDependencies +): HiddenFileSyncConflictResolution { + return new HiddenFileSyncConflictResolutionOwner(dependencies); +} diff --git a/src/features/HiddenFileSync/hiddenFileSyncConflictResolution.unit.spec.ts b/src/features/HiddenFileSync/hiddenFileSyncConflictResolution.unit.spec.ts new file mode 100644 index 00000000..8c42c549 --- /dev/null +++ b/src/features/HiddenFileSync/hiddenFileSyncConflictResolution.unit.spec.ts @@ -0,0 +1,422 @@ +import { describe, expect, it, vi } from "vitest"; +import { + LOG_LEVEL_VERBOSE, + type DocumentID, + type FilePath, + type FilePathWithPrefix, + type LoadedEntry, + type MetaEntry, + type UXStat, +} from "@vrtmrz/livesync-commonlib/compat/common/types"; + +import { + createHiddenFileSyncConflictResolution, + findHiddenFileSyncMergeBase, + selectHiddenFileSyncRevisionToDelete, + type HiddenFileSyncConflictDatabase, + type HiddenFileSyncConflictInteraction, + type HiddenFileSyncConflictReconciliation, + type HiddenFileSyncConflictResolutionDependencies, + type HiddenFileSyncConflictStorage, + type HiddenFileSyncJsonResolution, + type HiddenFileSyncRevisionHistory, +} from "./hiddenFileSyncConflictResolution.ts"; + +const path = ".obsidian/plugins/example/data.json" as FilePath; +const prefixedPath = `i:${path}` as FilePathWithPrefix; +const id = "i:hidden-entry-id" as DocumentID; + +function metadata( + revision: string, + mtime: number, + overrides: Partial = {} +): HiddenFileSyncRevisionHistory { + return { + _id: id, + _rev: revision, + path: prefixedPath, + type: "plain", + datatype: "plain", + ctime: 10, + mtime, + size: 20, + children: [], + eden: {}, + deleted: false, + ...overrides, + } as unknown as MetaEntry; +} + +function loadedEntry(revision: string, content: string): LoadedEntry { + return { + ...metadata(revision, 20), + data: content, + } as LoadedEntry; +} + +function entries(...values: MetaEntry[]): AsyncIterable { + return { + async *[Symbol.asyncIterator]() { + yield* values; + }, + }; +} + +type DependencyOverrides = { + database?: Partial; + storage?: Partial; + reconciliation?: Partial; + interaction?: Partial; + shouldOverwrite?: HiddenFileSyncConflictResolutionDependencies["shouldOverwrite"]; + log?: HiddenFileSyncConflictResolutionDependencies["log"]; +}; + +function createDependencies(overrides: DependencyOverrides = {}): HiddenFileSyncConflictResolutionDependencies { + const database: HiddenFileSyncConflictDatabase = { + scanConflictedEntries: () => entries(), + getDocumentId: vi.fn(async () => id), + loadCurrentMetadata: vi.fn(async () => metadata("1-current", 10)), + loadConflictingMetadata: vi.fn(async () => metadata("1-conflict", 10)), + loadRevisionHistory: vi.fn(async () => metadata("1-current", 10, { _revs_info: [] })), + loadRevisionEntry: vi.fn(async (): Promise => false), + mergeJson: vi.fn(async (): Promise => false), + removeRevision: vi.fn(async () => true), + deleteRevision: vi.fn(async () => true), + ...overrides.database, + }; + const storage: HiddenFileSyncConflictStorage = { + ensureDirectory: vi.fn(async () => undefined), + writeFile: vi.fn(async () => null), + triggerEvent: vi.fn(async () => undefined), + ...overrides.storage, + }; + const reconciliation: HiddenFileSyncConflictReconciliation = { + storeFile: vi.fn(async () => true), + extractFile: vi.fn(async () => true), + ...overrides.reconciliation, + }; + const interaction: HiddenFileSyncConflictInteraction = { + resolveJsonConflict: vi.fn(async () => false), + ...overrides.interaction, + }; + return { + database, + storage, + reconciliation, + interaction, + shouldOverwrite: overrides.shouldOverwrite ?? (() => false), + log: overrides.log ?? vi.fn(), + }; +} + +describe("Hidden File Sync conflict policy", () => { + it("keeps the current revision when both mtimes are equal", () => { + const current = metadata("3-current", 20); + const conflicted = metadata("2-conflict", 20); + + expect(selectHiddenFileSyncRevisionToDelete(current, current._rev!, conflicted, conflicted._rev!)).toBe( + conflicted._rev + ); + }); + + it("selects the first available lower-generation revision as the merge base", () => { + expect( + findHiddenFileSyncMergeBase( + [ + { rev: "4-current", status: "available" }, + { rev: "3-missing", status: "missing" }, + { rev: "2-base", status: "available" }, + { rev: "1-older", status: "available" }, + ], + "3-conflict" + ) + ).toBe("2-base"); + expect(findHiddenFileSyncMergeBase(undefined, "3-conflict")).toBe(""); + }); +}); + +describe("Hidden File Sync conflict queue", () => { + it("continues processing queued conflict notifications while a full scan is in progress", async () => { + let releaseScan!: () => void; + let markScanStarted!: () => void; + const scanGate = new Promise((resolve) => { + releaseScan = resolve; + }); + const scanStarted = new Promise((resolve) => { + markScanStarted = resolve; + }); + const loadCurrentMetadata = vi.fn(async () => metadata("1-current", 10)); + const dependencies = createDependencies({ + database: { + scanConflictedEntries: () => ({ + async *[Symbol.asyncIterator]() { + markScanStarted(); + await scanGate; + }, + }), + loadCurrentMetadata, + }, + }); + const resolution = createHiddenFileSyncConflictResolution(dependencies); + const resolvingAll = resolution.resolveAll(); + await scanStarted; + // Cross the macrotask boundary which allowed the legacy suspended + // processor to stop before a database notification arrived. + await new Promise((resolve) => setTimeout(resolve, 0)); + + resolution.queue(prefixedPath); + try { + await vi.waitFor(() => expect(loadCurrentMetadata).toHaveBeenCalledOnce(), { + interval: 10, + timeout: 250, + }); + } finally { + releaseScan(); + await resolvingAll; + resolution.dispose(); + } + }); + + it("deduplicates exact paths and does not accept work after disposal", async () => { + const loadCurrentMetadata = vi.fn(async () => metadata("1-current", 10)); + const dependencies = createDependencies({ database: { loadCurrentMetadata } }); + const resolution = createHiddenFileSyncConflictResolution(dependencies); + + resolution.queue(prefixedPath); + resolution.queue(prefixedPath); + await resolution.resolveAll(); + + expect(loadCurrentMetadata).toHaveBeenCalledOnce(); + resolution.dispose(); + resolution.queue(`i:${path}.other` as FilePathWithPrefix); + expect(loadCurrentMetadata).toHaveBeenCalledOnce(); + }); + + it("retains prefixed and unprefixed path forms as separate compatibility keys", async () => { + const loadCurrentMetadata = vi.fn(async () => metadata("1-current", 10)); + const dependencies = createDependencies({ database: { loadCurrentMetadata } }); + const resolution = createHiddenFileSyncConflictResolution(dependencies); + + resolution.queue(path); + resolution.queue(prefixedPath); + await resolution.resolveAll(); + + expect(loadCurrentMetadata).toHaveBeenCalledTimes(2); + resolution.dispose(); + }); + + it("deletes the conflicted revision on an mtime tie, then extracts", async () => { + const events: string[] = []; + const current = metadata("3-current", 20, { _conflicts: ["2-conflict"] }); + const settled = metadata("3-current", 20, { _conflicts: [] }); + const dependencies = createDependencies({ + database: { + scanConflictedEntries: () => entries(current), + loadCurrentMetadata: vi.fn().mockResolvedValueOnce(current).mockResolvedValueOnce(settled), + loadConflictingMetadata: vi.fn(async () => metadata("2-conflict", 20)), + removeRevision: vi.fn(async (_id, revision) => { + events.push(`remove:${revision}`); + return true; + }), + }, + reconciliation: { + extractFile: vi.fn(async () => { + events.push("extract"); + return true; + }), + }, + }); + const resolution = createHiddenFileSyncConflictResolution(dependencies); + + await resolution.resolveAll(); + + expect(events).toEqual(["remove:2-conflict", "extract"]); + resolution.dispose(); + }); + + it("stores and extracts an automatic merge before removing the conflicted revision", async () => { + const events: string[] = []; + const current = metadata("3-current", 30, { _conflicts: ["2-conflict"] }); + const settled = metadata("4-merged", 40, { _conflicts: [] }); + const stat = { ctime: 10, mtime: 20, size: 30, type: "file" } as UXStat; + const mergeJson = vi.fn(async () => '{"merged":true}'); + const dependencies = createDependencies({ + database: { + scanConflictedEntries: () => entries(current), + loadCurrentMetadata: vi.fn().mockResolvedValueOnce(current).mockResolvedValueOnce(settled), + loadRevisionHistory: vi.fn(async () => + metadata("3-current", 30, { + _revs_info: [ + { rev: "3-current", status: "available" }, + { rev: "1-base", status: "available" }, + ], + }) + ), + mergeJson, + removeRevision: vi.fn(async () => { + events.push("remove"); + return true; + }), + }, + storage: { + ensureDirectory: vi.fn(async () => { + events.push("ensure"); + }), + writeFile: vi.fn(async () => { + events.push("write"); + return stat; + }), + }, + reconciliation: { + storeFile: vi.fn(async () => { + events.push("store"); + return true; + }), + extractFile: vi.fn(async () => { + events.push("extract"); + return false; + }), + }, + }); + const resolution = createHiddenFileSyncConflictResolution(dependencies); + + await resolution.resolveAll(); + + expect(mergeJson).toHaveBeenCalledWith(prefixedPath, "1-base", "3-current", "2-conflict"); + expect(events).toEqual(["ensure", "write", "store", "extract", "remove"]); + resolution.dispose(); + }); +}); + +describe("Hidden File Sync JSON conflict application", () => { + function createJsonResolutionFixture( + jsonResolution: HiddenFileSyncJsonResolution, + deletionResult: boolean | Error = true + ) { + const events: string[] = []; + const docA = loadedEntry("3-current", '{"current":true}'); + const docB = loadedEntry("2-conflict", '{"conflict":true}'); + const deleteRevision = vi.fn(async (entry: LoadedEntry) => { + events.push(`delete:${entry._rev}`); + if (deletionResult instanceof Error) { + if (entry._rev === docB._rev) throw deletionResult; + return true; + } + return deletionResult; + }); + const extractFile = vi.fn(async () => { + events.push("extract"); + return false; + }); + const storeFile = vi.fn(async () => { + events.push("store"); + return true; + }); + const stat = { ctime: 11, mtime: 21, size: 22, type: "file" } as UXStat; + const log = vi.fn(); + const dependencies = createDependencies({ + database: { deleteRevision }, + storage: { + ensureDirectory: vi.fn(async () => { + events.push("ensure"); + }), + writeFile: vi.fn(async () => { + events.push("write"); + return stat; + }), + triggerEvent: vi.fn(async () => { + events.push("trigger"); + }), + }, + reconciliation: { extractFile, storeFile }, + interaction: { + resolveJsonConflict: vi.fn(async (_path, _docs, apply) => await apply(jsonResolution)), + }, + log, + }); + return { + deleteRevision, + docA, + docB, + events, + extractFile, + log, + resolution: createHiddenFileSyncConflictResolution(dependencies), + stat, + storeFile, + }; + } + + it("returns false without changing data when no resolution is selected", async () => { + const fixture = createJsonResolutionFixture({}); + + await expect(fixture.resolution.resolveJson(fixture.docA, fixture.docB)).resolves.toBe(false); + + expect(fixture.events).toEqual([]); + fixture.resolution.dispose(); + }); + + it("keeps the selected revision but reports success when follow-up extraction fails", async () => { + const fixture = createJsonResolutionFixture({ keepRevision: "3-current" }); + + await expect(fixture.resolution.resolveJson(fixture.docA, fixture.docB)).resolves.toBe(true); + + expect(fixture.deleteRevision).toHaveBeenCalledTimes(1); + expect(fixture.deleteRevision).toHaveBeenCalledWith(fixture.docB); + expect(fixture.events).toEqual([`delete:${fixture.docB._rev}`, "extract"]); + expect(fixture.log).toHaveBeenCalledWith( + `STORAGE --> DB:${path}: extracted (hidden,merged) Failed`, + undefined, + undefined + ); + fixture.resolution.dispose(); + }); + + it("deletes both supplied revisions when the selected revision is unknown", async () => { + const fixture = createJsonResolutionFixture({ keepRevision: "9-unknown" }); + + await expect(fixture.resolution.resolveJson(fixture.docA, fixture.docB)).resolves.toBe(true); + + expect(fixture.events).toEqual([`delete:${fixture.docA._rev}`, `delete:${fixture.docB._rev}`, "extract"]); + expect(fixture.storeFile).not.toHaveBeenCalled(); + fixture.resolution.dispose(); + }); + + it("deletes both revisions before writing and storing a merged result", async () => { + const fixture = createJsonResolutionFixture({ mergedText: '{"merged":true}' }); + + await expect(fixture.resolution.resolveJson(fixture.docA, fixture.docB)).resolves.toBe(true); + + expect(fixture.storeFile).toHaveBeenCalledWith( + { + path, + ctime: fixture.stat.ctime, + mtime: fixture.stat.mtime, + size: fixture.stat.size, + }, + true + ); + expect(fixture.events).toEqual([ + `delete:${fixture.docA._rev}`, + `delete:${fixture.docB._rev}`, + "ensure", + "write", + "store", + "trigger", + "extract", + ]); + fixture.resolution.dispose(); + }); + + it("keeps an earlier successful deletion when a later deletion throws", async () => { + const error = new Error("second deletion failed"); + const fixture = createJsonResolutionFixture({ mergedText: '{"merged":true}' }, error); + + await expect(fixture.resolution.resolveJson(fixture.docA, fixture.docB)).resolves.toBe(false); + + expect(fixture.events).toEqual([`delete:${fixture.docA._rev}`, `delete:${fixture.docB._rev}`]); + expect(fixture.storeFile).not.toHaveBeenCalled(); + expect(fixture.log).toHaveBeenCalledWith(error, LOG_LEVEL_VERBOSE, undefined); + fixture.resolution.dispose(); + }); +}); diff --git a/src/features/HiddenFileSync/hiddenFileSyncContext.composition.unit.spec.ts b/src/features/HiddenFileSync/hiddenFileSyncContext.composition.unit.spec.ts new file mode 100644 index 00000000..aa1acd30 --- /dev/null +++ b/src/features/HiddenFileSync/hiddenFileSyncContext.composition.unit.spec.ts @@ -0,0 +1,129 @@ +import { describe, expect, it, vi } from "vitest"; +import { + type DocumentID, + type FilePath, + type FilePathWithPrefix, + type LoadedEntry, + type MetaEntry, + type UXStat, +} from "@vrtmrz/livesync-commonlib/compat/common/types"; +import { ICHeader, ICHeaderEnd } from "@/common/types.ts"; + +vi.mock("@/deps.ts", () => ({})); +vi.mock("./configureHiddenFileSyncMode.ts", () => ({ + configureHiddenFileSyncMode: vi.fn(), +})); + +import { HiddenFileSyncContext } from "./hiddenFileSyncContext.ts"; + +describe("HiddenFileSyncContext operation composition", () => { + it("composes the conflict owner from the current database and path capabilities", async () => { + const path = ".obsidian/app.json" as FilePath; + const prefixedPath = `i:${path}` as FilePathWithPrefix; + const metadata = { + _id: "i:hidden-entry-id" as DocumentID, + _rev: "2-current", + path: prefixedPath, + type: "plain", + datatype: "plain", + ctime: 10, + mtime: 20, + size: 20, + children: [], + eden: {}, + deleted: false, + _conflicts: [], + } as unknown as MetaEntry; + const findEntries = vi.fn(() => ({ + async *[Symbol.asyncIterator]() { + yield metadata; + }, + })); + const getRaw = vi.fn(async () => metadata); + const path2id = vi.fn(async () => metadata._id); + const periodicProcessor = { enable: vi.fn(), disable: vi.fn() }; + const context = new HiddenFileSyncContext({ + createPeriodicProcessor: vi.fn(() => periodicProcessor), + getLocalDatabase: () => ({ findEntries, getRaw }), + path: { path2id }, + log: vi.fn(), + publishActivity: vi.fn(), + closeJsonConflictDialogs: vi.fn(), + hideConfigurationChangeNotice: vi.fn(), + } as never); + + await context.testing.conflictResolution.resolveAll(); + + expect(findEntries).toHaveBeenCalledWith(ICHeader, ICHeaderEnd, { conflicts: true }); + expect(path2id).toHaveBeenCalledWith(prefixedPath, ICHeader); + expect(getRaw).toHaveBeenCalledWith(metadata._id, { conflicts: true }); + context.dispose(); + }); + + it("applies a selected live revision through the narrow repair view", async () => { + const path = ".obsidian/plugins/example/data.json" as FilePath; + const prefixedPath = `i:${path}` as FilePathWithPrefix; + const revision = "2-selected"; + const metadata = { + _id: "hidden-entry-id" as DocumentID, + _rev: revision, + path: prefixedPath, + type: "plain", + datatype: "plain", + ctime: 10, + mtime: 20, + size: 20, + children: [], + eden: {}, + deleted: false, + } as unknown as MetaEntry; + const loaded = { + ...metadata, + data: '{"value":"database"}', + } as LoadedEntry; + const stat = { ctime: 10, mtime: 20, size: 20, type: "file" } as UXStat; + const statHidden = vi.fn<() => Promise>().mockResolvedValueOnce(null).mockResolvedValue(stat); + const writeHiddenFileAuto = vi.fn(async () => true); + const getDBEntryFromMeta = vi.fn(async () => loaded); + const fetchEntryMeta = vi.fn(async () => metadata); + const getConflictedRevs = vi.fn(async () => [] as string[]); + const markChangesAreSame = vi.fn(); + const periodicProcessor = { enable: vi.fn(), disable: vi.fn() }; + const context = new HiddenFileSyncContext({ + createPeriodicProcessor: vi.fn(() => periodicProcessor), + isIgnoredByIgnoreFile: vi.fn(async () => false), + databaseFileAccess: { + fetchEntryMeta, + getConflictedRevs, + }, + getLocalDatabase: () => ({ getDBEntryFromMeta }), + storageAccess: { + statHidden, + isExistsIncludeHidden: vi.fn(async () => false), + ensureDir: vi.fn(async () => true), + writeHiddenFileAuto, + }, + path: { + markChangesAreSame, + unmarkChanges: vi.fn(), + }, + getSettings: () => ({ suppressNotifyHiddenFilesChange: true }), + log: vi.fn(), + publishActivity: vi.fn(), + closeJsonConflictDialogs: vi.fn(), + hideConfigurationChangeNotice: vi.fn(), + } as never); + await expect(context.repair.extractInternalFileRevisionFromDatabase(path, revision, true)).resolves.toBe(true); + + expect(fetchEntryMeta).toHaveBeenCalledWith(prefixedPath, revision, true); + expect(getConflictedRevs).toHaveBeenCalledWith(prefixedPath); + expect(getDBEntryFromMeta).toHaveBeenCalledWith(metadata, false, true); + expect(writeHiddenFileAuto).toHaveBeenCalledWith(path, '{"value":"database"}', { + ctime: metadata.ctime, + mtime: metadata.mtime, + }); + expect(markChangesAreSame).toHaveBeenCalledWith(path, metadata.mtime, stat.mtime); + + context.dispose(); + }); +}); diff --git a/src/features/HiddenFileSync/hiddenFileSyncContext.routing.unit.spec.ts b/src/features/HiddenFileSync/hiddenFileSyncContext.routing.unit.spec.ts index cb727971..32e2e582 100644 --- a/src/features/HiddenFileSync/hiddenFileSyncContext.routing.unit.spec.ts +++ b/src/features/HiddenFileSync/hiddenFileSyncContext.routing.unit.spec.ts @@ -10,58 +10,58 @@ import { HiddenFileSyncContext } from "./hiddenFileSyncContext.ts"; const PATH = ".obsidian/plugins/example/data.json" as FilePath; +function isTargetFile(context: HiddenFileSyncContext, path: FilePath): Promise { + return (context as unknown as { isTargetFile(path: FilePath): Promise }).isTargetFile(path); +} + +function isTargetFileEligible(context: HiddenFileSyncContext, path: FilePath): Promise { + return (context as unknown as { isTargetFileEligible(path: FilePath): Promise }).isTargetFileEligible(path); +} + function createHiddenFileSync( options: { owned?: boolean; ignoredByIgnoreFile?: boolean; patternMatch?: boolean } = {} ) { const ownsLocalFile = vi.fn(() => options.owned ?? true); const isIgnoredByIgnoreFile = vi.fn(async () => options.ignoredByIgnoreFile ?? false); - const isTargetFileInPatterns = vi.fn(() => options.patternMatch ?? true); + const patternTest = vi.fn(() => options.patternMatch ?? true); + const parseRegExpSettings = vi.fn(() => ({ + ignoreFilter: [], + targetFilter: options.patternMatch === undefined ? [] : [{ test: patternTest }], + })); const hiddenFileSync = Object.create(HiddenFileSyncContext.prototype) as HiddenFileSyncContext; Object.assign(hiddenFileSync, { dependencies: { ownsLocalFile, isIgnoredByIgnoreFile }, - isTargetFileInPatterns, + parseRegExpSettings, }); - return { hiddenFileSync, isIgnoredByIgnoreFile, isTargetFileInPatterns, ownsLocalFile }; + return { hiddenFileSync, isIgnoredByIgnoreFile, parseRegExpSettings, ownsLocalFile }; } describe("Hidden File Sync local-path admission", () => { it("checks composition ownership before Hidden File Sync filters", async () => { - const { hiddenFileSync, isTargetFileInPatterns, ownsLocalFile } = createHiddenFileSync({ owned: false }); + const { hiddenFileSync, parseRegExpSettings, ownsLocalFile } = createHiddenFileSync({ owned: false }); - await expect(hiddenFileSync.isTargetFile(PATH)).resolves.toBe(false); + await expect(isTargetFile(hiddenFileSync, PATH)).resolves.toBe(false); expect(ownsLocalFile).toHaveBeenCalledWith(PATH); - expect(isTargetFileInPatterns).not.toHaveBeenCalled(); + expect(parseRegExpSettings).not.toHaveBeenCalled(); }); it("keeps target patterns and ignore-file results as Hidden File Sync eligibility", async () => { const patternExcluded = createHiddenFileSync({ patternMatch: false }); - await expect(patternExcluded.hiddenFileSync.isTargetFile(PATH)).resolves.toBe(false); + await expect(isTargetFile(patternExcluded.hiddenFileSync, PATH)).resolves.toBe(false); expect(patternExcluded.isIgnoredByIgnoreFile).not.toHaveBeenCalled(); const ignoreFileExcluded = createHiddenFileSync({ ignoredByIgnoreFile: true }); - await expect(ignoreFileExcluded.hiddenFileSync.isTargetFile(PATH)).resolves.toBe(false); + await expect(isTargetFile(ignoreFileExcluded.hiddenFileSync, PATH)).resolves.toBe(false); expect(ignoreFileExcluded.isIgnoredByIgnoreFile).toHaveBeenCalledWith(PATH); const admitted = createHiddenFileSync(); - await expect(admitted.hiddenFileSync.isTargetFile(PATH)).resolves.toBe(true); + await expect(isTargetFile(admitted.hiddenFileSync, PATH)).resolves.toBe(true); }); - it("exposes eligibility without consulting the composition owner", async () => { + it("evaluates eligibility without consulting the composition owner", async () => { const { hiddenFileSync, ownsLocalFile } = createHiddenFileSync({ owned: false }); - await expect(hiddenFileSync.isTargetFileEligible(PATH)).resolves.toBe(true); + await expect(isTargetFileEligible(hiddenFileSync, PATH)).resolves.toBe(true); expect(ownsLocalFile).not.toHaveBeenCalled(); }); }); - -describe("compatibility: Hidden File Sync path shape", () => { - const hiddenFileSync = Object.create(HiddenFileSyncContext.prototype) as HiddenFileSyncContext; - - it.each([ - [".obsidian/app.json", true], - [".trash/app.json", false], - ["notes/app.json", false], - ])("recognises %s as a Hidden File Sync path=%s", (path, expected) => { - expect(hiddenFileSync.isHiddenFileSyncHandlingPath(path as FilePath)).toBe(expected); - }); -}); diff --git a/src/features/HiddenFileSync/hiddenFileSyncContext.state.unit.spec.ts b/src/features/HiddenFileSync/hiddenFileSyncContext.state.unit.spec.ts index ec547e3a..62d81dd6 100644 --- a/src/features/HiddenFileSync/hiddenFileSyncContext.state.unit.spec.ts +++ b/src/features/HiddenFileSync/hiddenFileSyncContext.state.unit.spec.ts @@ -1,5 +1,4 @@ import { describe, expect, it, vi } from "vitest"; -import type { FilePath } from "@vrtmrz/livesync-commonlib/compat/common/types"; vi.mock("@/deps.ts", () => ({})); vi.mock("./configureHiddenFileSyncMode.ts", () => ({ @@ -8,12 +7,29 @@ vi.mock("./configureHiddenFileSyncMode.ts", () => ({ import { HiddenFileSyncContext } from "./hiddenFileSyncContext.ts"; -function createContext() { +function getProcessedState(context: HiddenFileSyncContext): unknown { + return (context as unknown as { readonly processedState: unknown }).processedState; +} + +function getPrivate(context: HiddenFileSyncContext, key: string): T { + return (context as unknown as Record)[key]; +} + +function createContext(processedFiles = new Map()) { const periodicProcessor = { enable: vi.fn(), disable: vi.fn() }; const publishActivity = vi.fn(); const hideConfigurationChangeNotice = vi.fn(); + const keyValueDatabase = { + get: vi.fn(async (key: IDBValidKey) => { + if (key == "hidden-file-lastProcessed") return processedFiles; + return new Map(); + }), + }; const context = new HiddenFileSyncContext({ createPeriodicProcessor: vi.fn(() => periodicProcessor), + getKeyValueDatabase: () => keyValueDatabase, + getSettings: () => ({ syncInternalFiles: true }), + log: vi.fn(), publishActivity, closeJsonConflictDialogs: vi.fn(), hideConfigurationChangeNotice, @@ -22,71 +38,41 @@ function createContext() { } describe("HiddenFileSyncContext state ownership", () => { - it("owns queues, caches, concurrency controls, and processors per context instance", () => { + it("owns caches, concurrency controls, processors, and conflict owners per context instance", () => { const first = createContext(); const second = createContext(); - first.context.pendingConflictChecks.add("i:.obsidian/first.json" as never); - first.context.queuedNotificationFiles.add(".obsidian/plugins/first"); - first.context.cacheFileRegExps.set("first", []); + getPrivate>(first.context, "queuedNotificationFiles").add(".obsidian/plugins/first"); + getPrivate>(first.context, "cacheFileRegExps").set("first", []); - expect(second.context.pendingConflictChecks).toEqual(new Set()); - expect(second.context.queuedNotificationFiles).toEqual(new Set()); - expect(second.context.cacheFileRegExps).toEqual(new Map()); - expect(first.context.conflictResolutionProcessor).not.toBe(second.context.conflictResolutionProcessor); - expect(first.context.semaphore).not.toBe(second.context.semaphore); - expect(first.context.periodicInternalFileScanProcessor).toBe(first.periodicProcessor); - expect(second.context.periodicInternalFileScanProcessor).toBe(second.periodicProcessor); + expect(getPrivate>(second.context, "queuedNotificationFiles").size).toBe(0); + expect(getPrivate>(second.context, "cacheFileRegExps")).toEqual(new Map()); + expect(first.context.testing.conflictResolution).not.toBe(second.context.testing.conflictResolution); + expect(getProcessedState(first.context)).not.toBe(getProcessedState(second.context)); + expect(getPrivate(first.context, "changeProcessor")).not.toBe( + getPrivate(second.context, "changeProcessor") + ); + expect(getPrivate(first.context, "periodicInternalFileScanProcessor")).toBe(first.periodicProcessor); + expect(getPrivate(second.context, "periodicInternalFileScanProcessor")).toBe(second.periodicProcessor); first.context.dispose(); second.context.dispose(); }); - it("publishes instance-owned event and processing counts at each transition", async () => { - const { context, publishActivity } = createContext(); - const path = ".obsidian/app.json" as FilePath; - - await context.serializedForEvent(path, async () => { - expect(publishActivity).toHaveBeenLastCalledWith(1, 1); - }); - - expect(publishActivity.mock.calls).toEqual([ - [1, 0], - [1, 1], - [1, 0], - [0, 0], - ]); - context.dispose(); - }); - it.each([ [new Map(), true], [new Map([[".obsidian/app.json", "1-2-3"]]), false], ])( "preserves start-up scan notice selection for the processed-file cache", async (processedFiles, forcedNotice) => { - const { context } = createContext(); - const performStartupScan = vi.fn(async () => undefined); - const keyValueDatabase = { - get: vi.fn(async (key: IDBValidKey) => { - if (key == "hidden-file-lastProcessed") return processedFiles; - return new Map(); - }), - }; - Object.assign(context, { - dependencies: { - createPeriodicProcessor: vi.fn(), - getKeyValueDatabase: () => keyValueDatabase, - getSettings: () => ({ syncInternalFiles: true }), - log: vi.fn(), - }, - performStartupScan, - }); + const { context } = createContext(processedFiles); + const applyOfflineChanges = vi.fn(async () => undefined); + context.applyOfflineChanges = applyOfflineChanges; - await context._everyOnDatabaseInitialized(false); + await context.serviceHandlers.onDatabaseInitialised(false); - expect(performStartupScan).toHaveBeenCalledWith(forcedNotice); - context.conflictResolutionProcessor.terminate(); + expect(applyOfflineChanges).toHaveBeenCalledWith(forcedNotice); + context.dispose(); } ); }); diff --git a/src/features/HiddenFileSync/hiddenFileSyncContext.ts b/src/features/HiddenFileSync/hiddenFileSyncContext.ts index a2e24d14..70978b5d 100644 --- a/src/features/HiddenFileSync/hiddenFileSyncContext.ts +++ b/src/features/HiddenFileSync/hiddenFileSyncContext.ts @@ -6,24 +6,13 @@ import { LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE, - type SavingEntry, type DocumentID, - type UXFileInfo, - type UXStat, - LOG_LEVEL_DEBUG, type MetaEntry, - type UXDataWriteOptions, type ObsidianLiveSyncSettings, type LOG_LEVEL, } from "@vrtmrz/livesync-commonlib/compat/common/types"; import { type InternalFileInfo, ICHeader, ICHeaderEnd } from "@/common/types.ts"; -import { - readAsBlob, - isDocContentSame, - readContent, - createBlob, - type CustomRegExp, -} from "@vrtmrz/livesync-commonlib/compat/common/utils"; +import { type CustomRegExp } from "@vrtmrz/livesync-commonlib/compat/common/utils"; import { compareMTime, isInternalMetadata, @@ -31,28 +20,68 @@ import { cancelTask, scheduleTask, getLogLevel, - autosaveCache, - type MapLike, onlyInNTimes, BASE_IS_NEW, EVEN, - displayRev, } from "@/common/utils.ts"; import { serialized, skipIfDuplicated } from "octagonal-wheels/concurrency/lock"; import { addPrefix, stripAllPrefixes } from "@vrtmrz/livesync-commonlib/compat/string_and_binary/path"; -import { QueueProcessor } from "octagonal-wheels/concurrency/processor"; import { Semaphore } from "octagonal-wheels/concurrency/semaphore"; import { tryGetFilePath } from "@vrtmrz/livesync-commonlib/compat/common/utils.doc"; import { configureHiddenFileSyncMode, type ConfigureHiddenFileSyncResult } from "./configureHiddenFileSyncMode.ts"; import type { OptionalSyncFeatureMode } from "@/features/optionalSyncFeatures.ts"; import { $msg } from "@/common/translation"; import type { LiveSyncLocalDB } from "@vrtmrz/livesync-commonlib/compat/pouchdb/LiveSyncLocalDB"; -import type { StorageAccess } from "@vrtmrz/livesync-commonlib/compat/interfaces/StorageAccess"; import type { DatabaseFileAccess } from "@vrtmrz/livesync-commonlib/compat/interfaces/DatabaseFileAccess"; import type { KeyValueDatabase } from "@vrtmrz/livesync-commonlib/compat/interfaces/KeyValueDatabase"; import type { IPathService } from "@vrtmrz/livesync-commonlib/compat/services/base/IService"; -import type { LogFunction } from "@vrtmrz/livesync-commonlib/compat/services/lib/logUtils"; -import type { HiddenFileSyncCommandView, HiddenFileSyncRepairView } from "./hiddenFileSyncViews.ts"; +import { + createHiddenFileSyncRepairView, + createHiddenFileSyncServiceHandlerView, + createHiddenFileSyncTestingView, + type HiddenFileSyncCommandView, + type HiddenFileSyncRepairView, + type HiddenFileSyncServiceHandlerView, + type HiddenFileSyncTestingRebuild, + type HiddenFileSyncTestingView, +} from "./hiddenFileSyncViews.ts"; +import { isHiddenFileSyncPath, matchesHiddenFileSyncPatterns } from "./hiddenFileSyncPathPolicy.ts"; +import { describeHiddenFileSyncDocument, getHiddenFileSyncComparisonMTime } from "./hiddenFileSyncState.ts"; +import { + collectOptionalFileSyncFiles, + type OptionalFileSyncFileTreeDependencies, +} from "@/features/optionalFileSyncFileTree.ts"; +import { + deleteHiddenFileFromStorage, + ensureHiddenFileDirectory, + readHiddenFileWithInfo, + triggerHiddenFileEvent, + writeHiddenFile, + writeHiddenFileFromDatabase, + type HiddenFileSyncStorageDependencies, +} from "./hiddenFileSyncStorage.ts"; +import { loadHiddenFileSyncBaseEntry, loadLiveHiddenFileSyncRevision } from "./hiddenFileSyncDatabaseLoaders.ts"; +import { + createHiddenFileSyncDatabaseWriteOperations, + type HiddenFileSyncDatabaseWriteOperations, +} from "./hiddenFileSyncDatabaseWriteOperations.ts"; +import { + createHiddenFileSyncDatabaseExtractionOperations, + type HiddenFileSyncDatabaseExtractionOperations, +} from "./hiddenFileSyncDatabaseExtractionOperations.ts"; +import { + createHiddenFileSyncConflictResolution, + type HiddenFileSyncConflictResolution, + type HiddenFileSyncJsonResolution, +} from "./hiddenFileSyncConflictResolution.ts"; +import { + createHiddenFileSyncProcessedState, + type HiddenFileSyncProcessedState, +} from "./hiddenFileSyncProcessedState.ts"; +import { + createHiddenFileSyncChangeProcessor, + type HiddenFileSyncChangeProcessor, +} from "./hiddenFileSyncChangeProcessor.ts"; type SyncDirection = "push" | "pull" | "safe" | "pullForce" | "pushForce"; export type HiddenFileSyncProgress = { @@ -94,17 +123,6 @@ type HiddenFileSyncDatabase = Pick< }; }; -type HiddenFileSyncStorage = Pick< - StorageAccess, - | "ensureDir" - | "isExistsIncludeHidden" - | "readHiddenFileAuto" - | "removeHidden" - | "statHidden" - | "triggerHiddenFile" - | "writeHiddenFileAuto" ->; - type HiddenFileSyncDatabaseFileAccess = Pick< DatabaseFileAccess, "fetchEntryFromMeta" | "fetchEntryMeta" | "getConflictedRevs" | "storeWithBaseRevision" @@ -115,74 +133,206 @@ export type HiddenFileSyncPeriodicProcessor = { disable(): void; }; -export type HiddenFileSyncJsonResolution = { - keepRevision?: string; - mergedText?: string; -}; +export type HiddenFileSyncContextDependencies = OptionalFileSyncFileTreeDependencies & + HiddenFileSyncStorageDependencies & { + getSettings(): HiddenFileSyncSettings; + getLocalDatabase(): HiddenFileSyncDatabase; + getKeyValueDatabase(): KeyValueDatabase; + databaseFileAccess: HiddenFileSyncDatabaseFileAccess; + path: Pick; + createProgress(prefix?: string, level?: LOG_LEVEL): HiddenFileSyncProgress; + createPeriodicProcessor(process: () => Promise): HiddenFileSyncPeriodicProcessor; + isReady(): boolean; + isSuspended(): boolean; + isDatabaseReady(): boolean; + isIgnoredByIgnoreFile(path: string): Promise; + getConfigDir(): string; + getRootPath(): string; + getFileRegExp( + key: + | "syncInternalFileOverwritePatterns" + | "syncInternalFilesIgnorePatterns" + | "syncInternalFilesTargetPatterns" + ): CustomRegExp[]; + applySettings(partial: Partial, saveImmediately?: boolean): Promise; + setSyncInternalFilesEnabled(enabled: boolean): void; + resolveJsonConflict( + path: FilePath, + docs: [LoadedEntry, LoadedEntry], + apply: (resolution: HiddenFileSyncJsonResolution) => Promise + ): Promise; + showConfigurationChangeNotice(updatedFolders: readonly string[]): void; + hideConfigurationChangeNotice(): void; + closeJsonConflictDialogs(): void; + publishActivity(eventCount: number, processingCount: number): void; + ownsLocalFile(path: FilePath): boolean; + }; -export type HiddenFileSyncDirectoryListing = { - files: string[]; - folders: string[]; -}; - -export type HiddenFileSyncContextDependencies = { - getSettings(): HiddenFileSyncSettings; - getLocalDatabase(): HiddenFileSyncDatabase; - getKeyValueDatabase(): KeyValueDatabase; - storageAccess: HiddenFileSyncStorage; - databaseFileAccess: HiddenFileSyncDatabaseFileAccess; - path: Pick; - log: LogFunction; - createProgress(prefix?: string, level?: LOG_LEVEL): HiddenFileSyncProgress; - createPeriodicProcessor(process: () => Promise): HiddenFileSyncPeriodicProcessor; - isReady(): boolean; - isSuspended(): boolean; - isDatabaseReady(): boolean; - isIgnoredByIgnoreFile(path: string): Promise; - getConfigDir(): string; - getRootPath(): string; - listFiles(path: string): Promise; - getFileRegExp( - key: "syncInternalFileOverwritePatterns" | "syncInternalFilesIgnorePatterns" | "syncInternalFilesTargetPatterns" - ): CustomRegExp[]; - applySettings(partial: Partial, saveImmediately?: boolean): Promise; - setSyncInternalFilesEnabled(enabled: boolean): void; - resolveJsonConflict( - path: FilePath, - docs: [LoadedEntry, LoadedEntry], - apply: (resolution: HiddenFileSyncJsonResolution) => Promise - ): Promise; - showConfigurationChangeNotice(updatedFolders: readonly string[]): void; - hideConfigurationChangeNotice(): void; - closeJsonConflictDialogs(): void; - publishActivity(eventCount: number, processingCount: number): void; - ownsLocalFile(path: FilePath): boolean; -}; - -function getComparingMTime( - doc: (MetaEntry | LoadedEntry | false) | UXFileInfo | UXStat | null | undefined, - includeDeleted = false -) { - if (doc === null) return 0; - if (doc === false) return 0; - if (doc === undefined) return 0; - if (!includeDeleted) { - if ("deleted" in doc && doc.deleted) return 0; - if ("_deleted" in doc && doc._deleted) return 0; - } - if ("stat" in doc) return doc.stat?.mtime ?? 0; - return doc.mtime ?? 0; -} - -export class HiddenFileSyncContext implements HiddenFileSyncCommandView, HiddenFileSyncRepairView { +export class HiddenFileSyncContext implements HiddenFileSyncCommandView { private readonly dependencies: HiddenFileSyncContextDependencies; - readonly periodicInternalFileScanProcessor: HiddenFileSyncPeriodicProcessor; - private eventCount = 0; - private processingCount = 0; + private readonly processedState: HiddenFileSyncProcessedState; + private readonly databaseWriteOperations: HiddenFileSyncDatabaseWriteOperations; + private readonly databaseExtractionOperations: HiddenFileSyncDatabaseExtractionOperations; + private readonly conflictResolution: HiddenFileSyncConflictResolution; + private readonly changeProcessor: HiddenFileSyncChangeProcessor; + readonly serviceHandlers: HiddenFileSyncServiceHandlerView; + readonly testing: HiddenFileSyncTestingView; + readonly repair: HiddenFileSyncRepairView; + private readonly periodicInternalFileScanProcessor: HiddenFileSyncPeriodicProcessor; + private rebuildMergingHook: HiddenFileSyncTestingRebuild | undefined; private disposed = false; constructor(dependencies: HiddenFileSyncContextDependencies) { this.dependencies = dependencies; + this.processedState = createHiddenFileSyncProcessedState({ + getKeyValueDatabase: () => this.dependencies.getKeyValueDatabase(), + getLocalDatabase: () => this.dependencies.getLocalDatabase(), + storageAccess: this.dependencies.storageAccess, + path: this.dependencies.path, + log: (message, level, key) => this.dependencies.log(message, level, key), + }); + this.databaseWriteOperations = createHiddenFileSyncDatabaseWriteOperations({ + serialiseFileOperation: async (key, operation) => await serialized(key, operation), + isIgnoredByIgnoreFile: async (path) => await dependencies.isIgnoredByIgnoreFile(path), + readFileWithInfo: async (path) => await readHiddenFileWithInfo(dependencies, path), + loadBaseEntry: async (path) => await loadHiddenFileSyncBaseEntry(dependencies, path, true), + loadBaseMetadata: async (path) => await loadHiddenFileSyncBaseEntry(dependencies, path, false), + loadLiveRevision: async (path, revision) => + await loadLiveHiddenFileSyncRevision(dependencies, path, revision), + fetchEntryFromMeta: async (meta, waitForReady, skipCheck) => + await dependencies.databaseFileAccess.fetchEntryFromMeta(meta, waitForReady, skipCheck), + storeWithBaseRevision: async (file, baseRevision, skipCheck) => + await dependencies.databaseFileAccess.storeWithBaseRevision(file, baseRevision, skipCheck), + putDatabaseEntry: async (entry) => await dependencies.getLocalDatabase().putDBEntry(entry), + putRaw: async (entry) => await dependencies.getLocalDatabase().putRaw(entry), + removeRevision: async (id, revision) => await dependencies.getLocalDatabase().removeRevision(id, revision), + processedState: this.processedState, + now: () => new Date().getTime(), + log: (message, level, key) => dependencies.log(message, level, key), + }); + this.databaseExtractionOperations = createHiddenFileSyncDatabaseExtractionOperations({ + serialiseFileOperation: async (key, operation) => await serialized(key, operation), + isIgnoredByIgnoreFile: async (path) => await dependencies.isIgnoredByIgnoreFile(path), + loadDatabaseMetadata: async (path) => + await dependencies.getLocalDatabase().getDBEntryMeta(path, { conflicts: true }, true), + loadLiveRevision: async (path, revision) => + await loadLiveHiddenFileSyncRevision(dependencies, path, revision), + loadDatabaseEntry: async (entry) => + await dependencies.getLocalDatabase().getDBEntryFromMeta(entry, false, true), + statStorageFile: async (path) => await dependencies.storageAccess.statHidden(path), + writeStorageFile: async (path, entry, force) => + await writeHiddenFileFromDatabase(dependencies, path, entry, force), + deleteStorageFile: async (path) => await deleteHiddenFileFromStorage(dependencies, path), + processedState: this.processedState, + queueNotification: (path) => this.queueNotification(path), + log: (message, level, key) => dependencies.log(message, level, key), + }); + this.conflictResolution = createHiddenFileSyncConflictResolution({ + database: { + scanConflictedEntries: () => + dependencies.getLocalDatabase().findEntries(ICHeader, ICHeaderEnd, { conflicts: true }), + getDocumentId: async (path) => await dependencies.path.path2id(path, ICHeader), + loadCurrentMetadata: async (id) => + await dependencies.getLocalDatabase().getRaw(id, { conflicts: true }), + loadConflictingMetadata: async (id, revision) => + await dependencies.getLocalDatabase().getRaw(id, { rev: revision }), + loadRevisionHistory: async (id) => + await dependencies.getLocalDatabase().getRaw(id, { revs_info: true }), + loadRevisionEntry: async (path, revision) => + await dependencies.getLocalDatabase().getDBEntry(addPrefix(path, ICHeader), { rev: revision }), + mergeJson: async (path, baseRevision, currentRevision, conflictedRevision) => + await dependencies + .getLocalDatabase() + .managers.conflictManager.mergeObject(path, baseRevision, currentRevision, conflictedRevision), + removeRevision: async (id, revision) => + await dependencies.getLocalDatabase().removeRevision(id, revision), + deleteRevision: async (entry) => + await dependencies.getLocalDatabase().deleteDBEntry(dependencies.path.getPath(entry), { + rev: entry._rev, + }), + }, + storage: { + ensureDirectory: async (path) => await ensureHiddenFileDirectory(dependencies, path), + writeFile: async (path, data) => await writeHiddenFile(dependencies, path, data), + triggerEvent: async (path) => await triggerHiddenFileEvent(dependencies, path), + }, + reconciliation: { + storeFile: async (file, forceWrite) => await this.databaseWriteOperations.store(file, forceWrite), + extractFile: async (path) => await this.databaseExtractionOperations.extract(path), + }, + interaction: { + resolveJsonConflict: async (path, docs, apply) => + await dependencies.resolveJsonConflict(path, docs, apply), + }, + shouldOverwrite: (path) => + dependencies.getFileRegExp("syncInternalFileOverwritePatterns").some((pattern) => pattern.test(path)), + log: (message, level, key) => dependencies.log(message, level, key), + }); + this.changeProcessor = createHiddenFileSyncChangeProcessor({ + storageAccess: dependencies.storageAccess, + readFileWithInfo: async (path) => await readHiddenFileWithInfo(dependencies, path), + loadDatabaseMetadata: async (path) => + await dependencies.getLocalDatabase().getDBEntryMeta(path, { conflicts: true }, true), + databaseWriteOperations: this.databaseWriteOperations, + databaseExtractionOperations: this.databaseExtractionOperations, + processedState: this.processedState, + conflictResolution: this.conflictResolution, + log: (message, level, key) => dependencies.log(message, level, key), + publishActivity: (eventCount, processingCount) => dependencies.publishActivity(eventCount, processingCount), + }); + this.repair = createHiddenFileSyncRepairView({ + scanInternalFiles: async () => await this.scanInternalFiles(), + storeInternalFileToDatabase: async (file, forceWrite) => + await this.databaseWriteOperations.store(file, forceWrite), + storeInternalFileToDatabaseWithBaseRevision: async (file, baseRevision, createIfDifferent) => + await this.databaseWriteOperations.storeWithBaseRevision(file, baseRevision, createIfDifferent), + extractInternalFileRevisionFromDatabase: async (storageFilePath, revision, force) => + await this.databaseExtractionOperations.extractRevision(storageFilePath, revision, force), + }); + this.serviceHandlers = createHiddenFileSyncServiceHandlerView({ + processOptionalFileEvent: async (path) => await this.processOptionalFileEvent(path), + processOptionalSyncFiles: async (doc) => await this.processOptionalSyncFiles(doc), + onSettingLoaded: async () => await this.onSettingLoaded(), + realiseSettingSyncMode: async () => await this.realiseSettingSyncMode(), + onResuming: async () => await this.onResuming(), + beforeReplicate: async (showNotice) => await this.beforeReplicate(showNotice), + onDatabaseInitialised: async (showNotice) => await this.onDatabaseInitialised(showNotice), + suspendExtraSync: async () => await this.suspendExtraSync(), + configureOptionalSyncFeature: async (mode) => await this.configureOptionalSyncFeature(mode), + isTargetFileEligible: async (path) => await this.isTargetFileEligible(path), + queueConflict: async (path) => await this.queueConflict(path), + }); + this.testing = createHiddenFileSyncTestingView({ + isManualCommandAvailable: () => this.isManualCommandAvailable(), + scanAllStorageChanges: async (showNotice) => await this.scanAllStorageChanges(showNotice), + scanAllDatabaseChanges: async (showNotice) => await this.scanAllDatabaseChanges(showNotice), + applyOfflineChanges: async (showNotice) => await this.applyOfflineChanges(showNotice), + updateSettingCache: () => this.updateSettingCache(), + initialiseInternalFileSync: async (direction, showMessage, targetFiles) => + await this.initialiseInternalFileSync(direction, showMessage, targetFiles), + conflictResolution: this.conflictResolution.testing, + readFileWithInfo: async (path) => await readHiddenFileWithInfo(dependencies, path), + showConfigurationChangeNotice: (updatedFolders) => { + this.queuedNotificationFiles.clear(); + for (const folder of updatedFolders) { + this.queuedNotificationFiles.add(folder); + } + this.notifyConfigChange(); + }, + interceptRebuildMerging: (interceptor) => { + const previousHook = this.rebuildMergingHook; + const runRebuild = async (showNotice: boolean, targetFiles?: FilePath[] | false) => + await this.rebuildMerging(showNotice, targetFiles); + const hook: HiddenFileSyncTestingRebuild = async (showNotice, targetFiles) => + await interceptor(runRebuild, showNotice, targetFiles); + this.rebuildMergingHook = hook; + return () => { + if (this.rebuildMergingHook === hook) { + this.rebuildMergingHook = previousHook; + } + }; + }, + }); this.periodicInternalFileScanProcessor = dependencies.createPeriodicProcessor( async () => this.isThisModuleEnabled() && this._isDatabaseReady() && (await this.scanAllStorageChanges(false)) @@ -205,14 +355,6 @@ export class HiddenFileSyncContext implements HiddenFileSyncCommandView, HiddenF return this.dependencies.databaseFileAccess; } - private get kvDB() { - return this.dependencies.getKeyValueDatabase(); - } - - private async path2id(filename: FilePathWithPrefix | FilePath, prefix?: string): Promise { - return await this.dependencies.path.path2id(filename, prefix); - } - private getPath(entry: AnyEntry): FilePathWithPrefix { return this.dependencies.path.getPath(entry); } @@ -241,7 +383,7 @@ export class HiddenFileSyncContext implements HiddenFileSyncCommandView, HiddenF return this.dependencies.createProgress(prefix, level); } - isThisModuleEnabled() { + private isThisModuleEnabled() { return this.settings.syncInternalFiles; } @@ -249,34 +391,30 @@ export class HiddenFileSyncContext implements HiddenFileSyncCommandView, HiddenF if (this.disposed) return; this.disposed = true; this.periodicInternalFileScanProcessor?.disable(); - this.conflictResolutionProcessor?.terminate(); - this.pendingConflictChecks.clear(); + this.changeProcessor.dispose(); + this.conflictResolution.dispose(); + this.rebuildMergingHook = undefined; this.queuedNotificationFiles.clear(); this.cacheFileRegExps.clear(); cancelTask(HIDDEN_FILE_NOTIFICATION_TASK); - this.eventCount = 0; - this.processingCount = 0; - this.dependencies.publishActivity(0, 0); this.dependencies.closeJsonConflictDialogs(); this.dependencies.hideConfigurationChangeNotice(); } // The key-value database becomes available before this lifecycle callback. - async _everyOnDatabaseInitialized(showNotice: boolean) { - this._fileInfoLastProcessed = await autosaveCache(this.kvDB, "hidden-file-lastProcessed"); - this._databaseInfoLastProcessed = await autosaveCache(this.kvDB, "hidden-file-lastProcessed-database"); - this._fileInfoLastKnown = await autosaveCache(this.kvDB, "hidden-file-lastKnown"); + private async onDatabaseInitialised(showNotice: boolean) { + await this.processedState.initialise(); if (this.isThisModuleEnabled()) { - if (this._fileInfoLastProcessed.size == 0) { + if (this.processedState.getLastProcessedFileCount() == 0) { this._log(`No cache found. Performing startup scan.`, LOG_LEVEL_VERBOSE); - await this.performStartupScan(true); + await this.applyOfflineChanges(true); } else { - await this.performStartupScan(showNotice); + await this.applyOfflineChanges(showNotice); } } return true; } - async _everyBeforeReplicate(showNotice: boolean) { + private async beforeReplicate(showNotice: boolean) { if ( this.isThisModuleEnabled() && this._isDatabaseReady() && @@ -288,7 +426,7 @@ export class HiddenFileSyncContext implements HiddenFileSyncCommandView, HiddenF return true; } - _everyOnloadAfterLoadSettings(): Promise { + private onSettingLoaded(): Promise { this.updateSettingCache(); return Promise.resolve(true); } @@ -297,7 +435,7 @@ export class HiddenFileSyncContext implements HiddenFileSyncCommandView, HiddenF this.cacheFileRegExps.clear(); } - isReady() { + private isReady() { if (this.disposed) return false; if (!this._isMainReady()) return false; if (this._isMainSuspended()) return false; @@ -309,15 +447,11 @@ export class HiddenFileSyncContext implements HiddenFileSyncCommandView, HiddenF return this.settings.useAdvancedMode && this.isReady() && this._isDatabaseReady(); } - async performStartupScan(showNotice: boolean) { - await this.applyOfflineChanges(showNotice); - } - - async _everyOnResumeProcess(): Promise { + private async onResuming(): Promise { this.periodicInternalFileScanProcessor?.disable(); if (this._isMainSuspended()) return true; if (this.isThisModuleEnabled()) { - await this.performStartupScan(false); + await this.applyOfflineChanges(false); } this.periodicInternalFileScanProcessor.enable( this.isThisModuleEnabled() && this.settings.syncInternalFilesInterval @@ -327,7 +461,7 @@ export class HiddenFileSyncContext implements HiddenFileSyncCommandView, HiddenF return true; } - _everyRealizeSettingSyncMode(): Promise { + private realiseSettingSyncMode(): Promise { this.periodicInternalFileScanProcessor?.disable(); if (this._isMainSuspended()) return Promise.resolve(true); if (!this._isMainReady()) return Promise.resolve(true); @@ -340,22 +474,14 @@ export class HiddenFileSyncContext implements HiddenFileSyncCommandView, HiddenF return Promise.resolve(true); } - async _anyProcessOptionalFileEvent(path: FilePath): Promise { + private async processOptionalFileEvent(path: FilePath): Promise { if (this.isReady()) { return (await this.trackStorageFileModification(path)) || false; } return false; } - _anyGetOptionalConflictCheckMethod(path: FilePathWithPrefix): Promise { - if (isInternalMetadata(path)) { - this.queueConflictCheck(path); - return Promise.resolve(true); - } - return Promise.resolve(false); - } - - async _anyProcessOptionalSyncFiles(doc: LoadedEntry): Promise { + private async processOptionalSyncFiles(doc: LoadedEntry): Promise { if (isInternalMetadata(doc._id)) { if (this.isThisModuleEnabled()) { //system file @@ -379,225 +505,25 @@ export class HiddenFileSyncContext implements HiddenFileSyncCommandView, HiddenF return false; } - async loadFileWithInfo(path: FilePath): Promise { - const stat = await this.storageAccess.statHidden(path); - if (!stat) - return { - name: path.split("/").pop() ?? "", - path, - stat: { - size: 0, - mtime: 0, - ctime: 0, - type: "file", - }, - isInternal: true, - deleted: true, - body: createBlob(new Uint8Array(0)), - }; - const content = await this.storageAccess.readHiddenFileAuto(path); - return { - name: path.split("/").pop() ?? "", - path, - stat, - isInternal: true, - deleted: false, - body: createBlob(content), - }; - } - - _fileInfoLastProcessed!: MapLike; - _fileInfoLastKnown!: MapLike; - _databaseInfoLastProcessed!: MapLike; - - statToKey(stat: UXStat | null) { - return `${stat?.mtime ?? 0}-${stat?.size ?? 0}`; - } - docToKey(doc: LoadedEntry | MetaEntry) { - return `${doc.mtime}-${doc.size}-${doc._rev}-${doc._deleted || doc.deleted || false ? "-0" : "-1"}`; - } - async fileToStatKey(file: FilePath, stat: UXStat | null = null) { - if (!stat) stat = await this.storageAccess.statHidden(file); - return this.statToKey(stat); - } - - updateLastProcessedFile(file: FilePath, keySrc: string | UXStat) { - const key = typeof keySrc == "string" ? keySrc : this.statToKey(keySrc); - const splitted = key.split("-"); - if (splitted[0] != "0") { - this._fileInfoLastKnown.set(file, Number(splitted[0])); - } - this._fileInfoLastProcessed.set(file, key); - } - - async updateLastProcessedAsActualFile(file: FilePath, stat?: UXStat | null) { - if (!stat) stat = await this.storageAccess.statHidden(file); - this._fileInfoLastProcessed.set(file, this.statToKey(stat)); - } - - resetLastProcessedFile(targetFiles: FilePath[] | false) { - if (targetFiles) { - for (const key of targetFiles) { - this._fileInfoLastProcessed.delete(key); - } - } else { - this._log(`Delete all processed mark.`, LOG_LEVEL_VERBOSE); - // THINKING: Should we... - // - delete all `Known file` processed mark? (This is current implementation) - // - delete all `Existing file` processed mark? - // - delete all files inside the config folder of current device mark? - this._fileInfoLastProcessed.clear(); - } - } - - getLastProcessedFileMTime(file: FilePath) { - const key = this._fileInfoLastKnown.get(file); - if (!key) return 0; - return key; - } - - getLastProcessedFileKey(file: FilePath) { - return this._fileInfoLastProcessed.get(file); - } - - getLastProcessedDatabaseKey(file: FilePath) { - return this._databaseInfoLastProcessed.get(file); - } - updateLastProcessedDatabase(file: FilePath, keySrc: string | MetaEntry | LoadedEntry) { - const key = typeof keySrc == "string" ? keySrc : this.docToKey(keySrc); - this._databaseInfoLastProcessed.set(file, key); - } - updateLastProcessed(path: FilePath, db: MetaEntry | LoadedEntry, stat: UXStat) { - this.updateLastProcessedDatabase(path, db); - this.updateLastProcessedFile(path, this.statToKey(stat)); - const dbMTime = getComparingMTime(db); - const storageMTime = getComparingMTime(stat); - if (dbMTime == 0 || storageMTime == 0) { - this.dependencies.path.unmarkChanges(path); - } else { - this.dependencies.path.markChangesAreSame(path, getComparingMTime(db), getComparingMTime(stat)); - } - } - updateLastProcessedDeletion(path: FilePath, db: MetaEntry | LoadedEntry | false) { - this.dependencies.path.unmarkChanges(path); - if (db) this.updateLastProcessedDatabase(path, db); - this.updateLastProcessedFile(path, this.statToKey(null)); - } - async ensureDir(path: FilePath) { - const isExists = await this.storageAccess.isExistsIncludeHidden(path); - if (!isExists) { - await this.storageAccess.ensureDir(path); - } - } - - async writeFile(path: FilePath, data: string | ArrayBuffer, opt?: UXDataWriteOptions): Promise { - await this.storageAccess.writeHiddenFileAuto(path, data, opt); - const stat = await this.storageAccess.statHidden(path); - // this.updateLastProcessedFile(path, this.statToKey(stat)); - return stat; - } - - async __removeFile(path: FilePath): Promise<"OK" | "ALREADY" | false> { - try { - if (!(await this.storageAccess.isExistsIncludeHidden(path))) { - // Already deleted - // this.updateLastProcessedFile(path, this.statToKey(null)); - return "ALREADY"; - } - if (await this.storageAccess.removeHidden(path)) { - // this.updateLastProcessedFile(path, this.statToKey(null)); - return "OK"; - } - } catch (ex) { - this._log(`Failed to remove file:${path}`); - this._log(ex, LOG_LEVEL_VERBOSE); - } - return false; - } - async triggerEvent(path: FilePath) { - try { - await this.storageAccess.triggerHiddenFile(path); - } catch (ex) { - this._log("Failed to call internal API(reconcileInternalFile)", LOG_LEVEL_VERBOSE); - this._log(ex, LOG_LEVEL_VERBOSE); - } - } - - async updateLastProcessedAsActualDatabase(file: FilePath, doc?: MetaEntry | LoadedEntry | null | false) { - const dbPath = addPrefix(file, ICHeader); - if (!doc) doc = await this.localDatabase.getDBEntryMeta(dbPath); - if (!doc) return; - this._databaseInfoLastProcessed.set(file, this.docToKey(doc)); - } - - resetLastProcessedDatabase(targetFiles: FilePath[] | false) { - if (targetFiles) { - for (const key of targetFiles) { - this._databaseInfoLastProcessed.delete(key); - } - } else { - this._log(`Delete all processed mark.`, LOG_LEVEL_VERBOSE); - // THINKING: Should we... - // - delete all `Known file` processed mark? (This is current implementation) - // - delete all `Existing file` processed mark? - // - delete all files inside the config folder of current device mark? - this._databaseInfoLastProcessed.clear(); - } - } - - async adoptCurrentStorageFilesAsProcessed(targetFiles: FilePath[] | false) { + private async adoptCurrentStorageFilesAsProcessed(targetFiles: FilePath[] | false) { const allFiles = await this.scanInternalFileNames(); const files = targetFiles ? allFiles.filter((e) => targetFiles.some((t) => e.indexOf(t) !== -1)) : allFiles; for (const file of files) { - await this.updateLastProcessedAsActualFile(file); + await this.processedState.updateLastProcessedAsActualFile(file); } } - async adoptCurrentDatabaseFilesAsProcessed(targetFiles: FilePath[] | false) { + private async adoptCurrentDatabaseFilesAsProcessed(targetFiles: FilePath[] | false) { const allFiles = await this.getAllDatabaseFiles(); const files = targetFiles ? allFiles.filter((e) => targetFiles.some((t) => e.path.indexOf(t) !== -1)) : allFiles; for (const file of files) { const path = stripAllPrefixes(this.getPath(file)); - await this.updateLastProcessedAsActualDatabase(path, file); + await this.processedState.updateLastProcessedAsActualDatabase(path, file); } } - semaphore = Semaphore(10); - async serializedForEvent(file: FilePath, fn: () => Promise) { - this.eventCount++; - this.publishActivity(); - const rel = await this.semaphore.acquire(); - try { - return await serialized(`hidden-file-event:${file}`, async () => { - this.processingCount++; - this.publishActivity(); - try { - return await fn(); - } finally { - this.processingCount = Math.max(0, this.processingCount - 1); - this.publishActivity(); - } - }); - } finally { - rel(); - this.eventCount = Math.max(0, this.eventCount - 1); - this.publishActivity(); - } - } - - private publishActivity() { - this.dependencies.publishActivity( - this.disposed ? 0 : this.eventCount, - this.disposed ? 0 : this.processingCount - ); - } - - async useStorageFiles(files: FilePath[], showNotice = false, onlyNew = false) { - return await this.trackScannedStorageChanges(files, showNotice, onlyNew, true); - } - - async trackScannedStorageChanges( + private async trackScannedStorageChanges( processFiles: FilePath[], showNotice: boolean = false, onlyNew = false, @@ -629,7 +555,7 @@ export class HiddenFileSyncContext implements HiddenFileSyncCommandView, HiddenF const logLevel = getLogLevel(showNotice); const p = this._progress(`[⚙ Scanning Storage -> DB ]\n`, logLevel); p.log(`Scanning storage files...`); - const knownNames = [...this._fileInfoLastProcessed.keys()] as FilePath[]; + const knownNames = [...this.processedState.getLastProcessedFileKeys()] as FilePath[]; const existNames = await this.scanInternalFileNames(); const files = new Set([...knownNames, ...existNames]); @@ -642,8 +568,8 @@ export class HiddenFileSyncContext implements HiddenFileSyncCommandView, HiddenF const processFiles = nameAndMeta .filter(([path, stat]) => { if (forceWriteAll) return true; - const key = this.getLastProcessedFileKey(path); - const newKey = this.statToKey(stat); + const key = this.processedState.getLastProcessedFileKey(path); + const newKey = this.processedState.storageStateKey(stat); return key != newKey; }) .map(([path, stat]) => path); @@ -663,7 +589,7 @@ Offline Changed files: ${processFiles.length}`; /** * check the file is changed or not, and if changed, process it. */ - async trackStorageFileModification( + private async trackStorageFileModification( path: FilePath, onlyNew = false, forceWrite = false, @@ -676,343 +602,12 @@ Offline Changed files: ${processFiles.length}`; ); return false; } - try { - return await this.serializedForEvent(path, async () => { - let stat = await this.storageAccess.statHidden(path); - // sometimes folder is coming. - if (stat != null && stat.type != "file") { - return false; - } - const key = await this.fileToStatKey(path, stat); - // At here, we need to check to not to respond the same event. - // (a raw event occurs even at the file reading). - // This is only for the events. Not for scanning. Because of the scan is for not to miss any changes. - // Mostly all of before scanning, we should processed the files at the event. - const lastKey = this.getLastProcessedFileKey(path); - if (lastKey == key) { - this._log(`${path} Already processed.`, LOG_LEVEL_DEBUG); - return true; - } - // We should cache the file - const cache = await this.loadFileWithInfo(path); - const cacheMTime = getComparingMTime(cache.stat); - const statMtime = getComparingMTime(stat); - if (cacheMTime != statMtime) { - this._log(`Hidden file:${path} is changed.`, LOG_LEVEL_VERBOSE); - stat = cache.stat; - } - this.updateLastProcessedFile(path, stat!); - const lastIsNotFound = !lastKey || lastKey.endsWith("-0-0"); - const nowIsNotFound = cache.deleted; - const type = lastIsNotFound && nowIsNotFound ? "invalid" : nowIsNotFound ? "delete" : "modified"; - - if (type == "invalid") { - // Maybe the folder is deleted. - return false; - } - - const storageMTimeActual = getComparingMTime(stat); - const storageMTime = - storageMTimeActual == 0 ? this.getLastProcessedFileMTime(path) : storageMTimeActual; - - if (onlyNew) { - // If the file is deleted, and it was not new, we should process it. - const prefixedFileName = addPrefix(path, ICHeader); - const filesOnDB = await this.localDatabase.getDBEntryMeta(prefixedFileName); - const dbMTime = getComparingMTime(filesOnDB, includeDeleted); - const diff = compareMTime(storageMTime, dbMTime); - - if (diff != TARGET_IS_NEW) { - this._log(`Hidden file:${path} is not new.`, LOG_LEVEL_VERBOSE); - if (filesOnDB && stat) { - // OnlyNew not handles the deletion. - this.updateLastProcessed(path, filesOnDB, stat); - } - return true; - } - } - - if (type == "delete") { - this._log(`Deletion detected: ${path}`); - const result = await this.deleteInternalFileOnDatabase(path, forceWrite); - return result; - } else if (type == "modified") { - this._log(`Modification detected:${path}`, LOG_LEVEL_VERBOSE); - const result = await this.storeInternalFileToDatabase(cache, forceWrite); - const resultText = result === undefined ? "Nothing changed" : result ? "Updated" : "Failed"; - this._log(`${resultText}: ${path} ${resultText}`, LOG_LEVEL_VERBOSE); - return result; - } - }); - } catch (ex) { - this._log(`Failed to process hidden file:${path}`); - this._log(ex, LOG_LEVEL_VERBOSE); - } - // Could not be processed. but it was own task. so return true to prevent further processing. - return true; + return await this.changeProcessor.processStorageChange(path, onlyNew, forceWrite, includeDeleted); } - // --> Conflict processing - - // Keep one in-flight conflict check per path so repeated sync events do not close the active merge dialogue. - pendingConflictChecks = new Set(); - - queueConflictCheck(path: FilePathWithPrefix) { - if (this.disposed) return; - if (this.pendingConflictChecks.has(path)) return; - this.pendingConflictChecks.add(path); - this.conflictResolutionProcessor.enqueue(path); - } - - finishConflictCheck(path: FilePathWithPrefix) { - this.pendingConflictChecks.delete(path); - } - - requeueConflictCheck(path: FilePathWithPrefix) { - this.finishConflictCheck(path); - this.queueConflictCheck(path); - } - - async resolveConflictOnInternalFiles() { - // Scan all conflicted internal files - const conflicted = this.localDatabase.findEntries(ICHeader, ICHeaderEnd, { conflicts: true }); - this.conflictResolutionProcessor.suspend(); - try { - for await (const doc of conflicted) { - if (!("_conflicts" in doc)) continue; - if (isInternalMetadata(doc._id)) { - this.queueConflictCheck(doc.path); - } - } - } catch (ex) { - this._log("something went wrong on resolving all conflicted internal files"); - this._log(ex, LOG_LEVEL_VERBOSE); - } - await this.conflictResolutionProcessor.startPipeline().waitForAllProcessed(); - } - - async resolveByNewerEntry( - id: DocumentID, - path: FilePathWithPrefix, - currentDoc: MetaEntry, - currentRev: string, - conflictedRev: string - ) { - const conflictedDoc = await this.localDatabase.getRaw(id, { rev: conflictedRev }); - // determine which revision should been deleted. - // simply check modified time - const mtimeCurrent = getComparingMTime(currentDoc, true); - const mtimeConflicted = getComparingMTime(conflictedDoc, true); - // this._log(`Revisions:${new Date(mtimeA).toLocaleString} and ${new Date(mtimeB).toLocaleString}`); - // console.log(`mtime:${mtimeA} - ${mtimeB}`); - const delRev = mtimeCurrent < mtimeConflicted ? currentRev : conflictedRev; - // delete older one. - await this.localDatabase.removeRevision(id, delRev); - this._log(`Older one has been deleted:${path}`); - const cc = await this.localDatabase.getRaw(id, { conflicts: true }); - if (cc._conflicts?.length === 0) { - await this.extractInternalFileFromDatabase(stripAllPrefixes(path)); - this.finishConflictCheck(path); - } else { - this.requeueConflictCheck(path); - } - // check the file again - } - conflictResolutionProcessor = new QueueProcessor( - async (paths: FilePathWithPrefix[]) => { - const path = paths[0]; - try { - // Retrieve data - const id = await this.path2id(path, ICHeader); - const doc = await this.localDatabase.getRaw(id, { conflicts: true }); - if (doc._conflicts === undefined) { - this.finishConflictCheck(path); - return []; - } - if (doc._conflicts.length == 0) { - this.finishConflictCheck(path); - return []; - } - this._log(`Hidden file conflicted:${path}`); - const conflicts = doc._conflicts.sort((a, b) => Number(a.split("-")[0]) - Number(b.split("-")[0])); - const revA = doc._rev; - const revB = conflicts[0]; - - if (path.endsWith(".json")) { - const conflictedRev = conflicts[0]; - const conflictedRevNo = Number(conflictedRev.split("-")[0]); - //Search - const revFrom = await this.localDatabase.getRaw(id, { revs_info: true }); - const commonBase = - revFrom._revs_info - ?.filter((e) => e.status == "available" && Number(e.rev.split("-")[0]) < conflictedRevNo) - .first()?.rev ?? ""; - const result = await this.localDatabase.managers.conflictManager.mergeObject( - doc.path, - commonBase, - doc._rev, - conflictedRev - ); - if (result) { - this._log(`Object merge:${path}`, LOG_LEVEL_INFO); - const filename = stripAllPrefixes(path); - await this.ensureDir(filename); - const stat = await this.writeFile(filename, result); - if (!stat) { - throw new Error(`conflictResolutionProcessor: Failed to stat file ${filename}`); - } - await this.storeInternalFileToDatabase({ path: filename, ...stat }); - await this.extractInternalFileFromDatabase(filename); - await this.localDatabase.removeRevision(id, revB); - this.requeueConflictCheck(path); - return []; - } else { - this._log(`Object merge is not applicable.`, LOG_LEVEL_VERBOSE); - } - // const pat = this.settings.syncInternalFileOverwritePatterns; - const regExp = this.dependencies.getFileRegExp("syncInternalFileOverwritePatterns"); - if (regExp.some((r) => r.test(stripAllPrefixes(path)))) { - this._log(`Overwrite rule applied for conflicted hidden file: ${path}`, LOG_LEVEL_INFO); - await this.resolveByNewerEntry(id, path, doc, revA, revB); - return []; - } - return [{ path, revA, revB, id, doc }]; - } - // When not JSON file, resolve conflicts by choosing a newer one. - await this.resolveByNewerEntry(id, path, doc, revA, revB); - return []; - } catch (ex) { - this.finishConflictCheck(path); - this._log(`Failed to resolve conflict (Hidden): ${path}`); - this._log(ex, LOG_LEVEL_VERBOSE); - return []; - } - }, - { - suspended: false, - batchSize: 1, - concurrentLimit: 5, - delay: 10, - keepResultUntilDownstreamConnected: true, - yieldThreshold: 10, - pipeTo: new QueueProcessor( - async (results) => { - const { id, doc, path, revA, revB } = results[0]; - const prefixedPath = addPrefix(path, ICHeader); - const docAMerge = await this.localDatabase.getDBEntry(prefixedPath, { rev: revA }); - const docBMerge = await this.localDatabase.getDBEntry(prefixedPath, { rev: revB }); - try { - if (docAMerge != false && docBMerge != false) { - if (await this.showJSONMergeDialogAndMerge(docAMerge, docBMerge)) { - // Again for other conflicted revisions. - this.requeueConflictCheck(path); - } else { - this.finishConflictCheck(path); - } - return; - } else { - // If either revision could not read, force resolving by the newer one. - await this.resolveByNewerEntry(id, path, doc, revA, revB); - } - } catch (ex) { - this.finishConflictCheck(path); - throw ex; - } - }, - { - suspended: false, - batchSize: 1, - concurrentLimit: 1, - delay: 10, - keepResultUntilDownstreamConnected: false, - yieldThreshold: 10, - } - ), - } - ); - - async showJSONMergeDialogAndMerge(docA: LoadedEntry, docB: LoadedEntry): Promise { - this._log("Opening data-merging dialog", LOG_LEVEL_VERBOSE); - const docs: [LoadedEntry, LoadedEntry] = [docA, docB]; - const storageFilePath = stripAllPrefixes(docA.path); - const displayFilename = `${storageFilePath}`; - return await this.dependencies.resolveJsonConflict( - storageFilePath, - docs, - async ({ keepRevision: keep, mergedText: result }) => { - try { - let needFlush = false; - if (!result && !keep) { - this._log(`Skipped merging: ${displayFilename}`); - return false; - } - for (const doc of docs) { - if (doc._rev != keep) { - if (await this.localDatabase.deleteDBEntry(this.getPath(doc), { rev: doc._rev })) { - this._log(`Conflicted revision has been deleted: ${displayFilename}`); - needFlush = true; - } - } - } - if (!keep && result) { - await this.ensureDir(storageFilePath); - const stat = await this.writeFile(storageFilePath, result); - if (!stat) { - throw new Error("Stat failed"); - } - const mtime = getComparingMTime(stat); - await this.storeInternalFileToDatabase( - { path: storageFilePath, mtime, ctime: stat.ctime ?? mtime, size: stat.size ?? 0 }, - true - ); - await this.triggerEvent(storageFilePath); - this._log(`STORAGE <-- DB:${displayFilename}: written (hidden,merged)`); - } - if (needFlush) { - if (await this.extractInternalFileFromDatabase(storageFilePath, false)) { - this._log(`STORAGE --> DB:${displayFilename}: extracted (hidden,merged)`); - } else { - this._log(`STORAGE --> DB:${displayFilename}: extracted (hidden,merged) Failed`); - } - } - return true; - } catch (ex) { - this._log("Could not merge conflicted json"); - this._log(ex, LOG_LEVEL_VERBOSE); - return false; - } - } - ); - } - // <-- Conflict processing - // --> Event Source Handler (Database) - getDocProps(doc: LoadedEntry) { - /* - type DocumentProps = { - id: DocumentID; - rev?: string; - prefixedPath: FilePathWithPrefix; - path: FilePath; - isDeleted: boolean; - revDisplay: string; - shortenedId: string; - shortenedPath: string; - }; - */ - const id = doc._id; - const shortenedId = id.substring(0, 10); - const prefixedPath = this.getPath(doc); - const path = stripAllPrefixes(prefixedPath); - const rev = doc._rev; - const revDisplay = rev ? displayRev(rev) : "0-NOREVS"; - // const prefix = prefixedPath.substring(0, prefixedPath.length - path.length); - const shortenedPath = path.substring(0, 10); - const isDeleted = doc._deleted || doc.deleted || false; - return { id, rev, revDisplay, prefixedPath, path, isDeleted, shortenedId, shortenedPath }; - } - async processReplicationResult(doc: LoadedEntry): Promise { - const info = this.getDocProps(doc); + private async processReplicationResult(doc: LoadedEntry): Promise { + const info = describeHiddenFileSyncDocument(doc, this.getPath(doc)); const path = info.path; const headerLine = `Tracking DB ${info.path} (${info.revDisplay}) :`; const ret = await this.trackDatabaseFileModification(path, headerLine); @@ -1024,12 +619,12 @@ Offline Changed files: ${processFiles.length}`; // --> Database Event Functions - cacheFileRegExps = new Map(); + private readonly cacheFileRegExps = new Map(); /** * Parses the regular expression settings for hidden file synchronization. * @returns An object containing the ignore and target filters. */ - parseRegExpSettings() { + private parseRegExpSettings() { const regExpKey = `${this.settings.syncInternalFilesTargetPatterns}||${this.settings.syncInternalFilesIgnorePatterns}`; let ignoreFilter: CustomRegExp[]; let targetFilter: CustomRegExp[]; @@ -1046,39 +641,8 @@ Offline Changed files: ${processFiles.length}`; return { ignoreFilter, targetFilter }; } - /** - * Checks if the target file path matches the defined patterns. - */ - isTargetFileInPatterns(path: string): boolean { - const { ignoreFilter, targetFilter } = this.parseRegExpSettings(); - - if (ignoreFilter && ignoreFilter.length > 0) { - for (const pattern of ignoreFilter) { - if (pattern.test(path)) { - return false; - } - } - } - if (targetFilter && targetFilter.length > 0) { - for (const pattern of targetFilter) { - if (pattern.test(path)) { - return true; - } - } - // While having target patterns, it effects as an allow-list. - return false; - } - return true; - } - - isHiddenFileSyncHandlingPath(path: FilePath): boolean { - const result = path.startsWith(".") && !path.startsWith(".trash"); - // console.warn(`Assertion: isHiddenFileSyncHandlingPath(${path}) = ${result}`); - return result; - } - - async isTargetFileEligible(path: FilePath): Promise { - const result = this.isTargetFileInPatterns(path) && this.isHiddenFileSyncHandlingPath(path); + private async isTargetFileEligible(path: FilePath): Promise { + const result = matchesHiddenFileSyncPatterns(path, this.parseRegExpSettings()) && isHiddenFileSyncPath(path); // console.warn(`Assertion: isTargetFile(${path}) : ${result ? "✔️" : "❌"}`); if (!result) { return false; @@ -1088,12 +652,12 @@ Offline Changed files: ${processFiles.length}`; return !resultByFile; } - async isTargetFile(path: FilePath): Promise { + private async isTargetFile(path: FilePath): Promise { if (this.dependencies?.ownsLocalFile(path) === false) return false; return await this.isTargetFileEligible(path); } - async trackScannedDatabaseChange( + private async trackScannedDatabaseChange( processFiles: MetaEntry[], showNotice: boolean = false, onlyNew = false, @@ -1144,8 +708,10 @@ Offline Changed files: ${processFiles.length}`; currentDatabaseFiles.map((e) => [stripAllPrefixes(this.getPath(e)), e]) ); const currentDatabaseFileNames = [...Object.keys(allDatabaseMap)] as FilePath[]; - const untrackedLocal = currentStorageFiles.filter((e) => !this._fileInfoLastProcessed.has(e)); - const untrackedDatabase = currentDatabaseFileNames.filter((e) => !this._databaseInfoLastProcessed.has(e)); + const untrackedLocal = currentStorageFiles.filter((e) => !this.processedState.hasLastProcessedFile(e)); + const untrackedDatabase = currentDatabaseFileNames.filter( + (e) => !this.processedState.hasLastProcessedDatabase(e) + ); const bothUntracked = untrackedLocal.filter((e) => untrackedDatabase.indexOf(e) !== -1); p.log("Applying untracked changes..."); const stat = `Tracking statics: @@ -1173,8 +739,8 @@ Common untracked files: ${bothUntracked.length}`; // So, we should skip this. return; } - const fileMTime = getComparingMTime(fileStat); - const dbMTime = getComparingMTime(dbInfo); + const fileMTime = getHiddenFileSyncComparisonMTime(fileStat); + const dbMTime = getHiddenFileSyncComparisonMTime(dbInfo); const diff = compareMTime(fileMTime, dbMTime); if (diff == BASE_IS_NEW) { // Local file is newer than the database file. @@ -1186,7 +752,7 @@ Common untracked files: ${bothUntracked.length}`; await this.trackDatabaseFileModification(file, "[Apply]", true, true, dbInfo); } else if (diff == EVEN) { // Both are same, we may skip this but should update the last processed key. - this.updateLastProcessed(file, dbInfo, fileStat); + this.processedState.updateLastProcessed(file, dbInfo, fileStat); } } finally { rel(); @@ -1210,9 +776,9 @@ Common untracked files: ${bothUntracked.length}`; const databaseFiles = await this.getAllDatabaseFiles(); const files = databaseFiles.filter((e) => { const doc = e; - const key = this.docToKey(doc); + const key = this.processedState.databaseStateKey(doc); const path = stripAllPrefixes(this.getPath(doc)); - const lastKey = this.getLastProcessedDatabaseKey(path); + const lastKey = this.processedState.getLastProcessedDatabaseKey(path); return lastKey != key; }); const logLevel = getLogLevel(showNotice); @@ -1224,7 +790,7 @@ Offline Changed files: ${files.length}`; }); } - async useDatabaseFiles(files: MetaEntry[], showNotice = false, onlyNew = false) { + private async useDatabaseFiles(files: MetaEntry[], showNotice = false, onlyNew = false) { const logLevel = getLogLevel(showNotice); const p = this._progress(`[⚙ Scanning DB -> Storage ]\n`, logLevel); p.log("Scanning database files..."); @@ -1245,7 +811,7 @@ Offline Changed files: ${files.length}`; return true; } - async trackDatabaseFileModification( + private async trackDatabaseFileModification( path: FilePath, headerLine: string, preventDoubleProcess = false, @@ -1253,56 +819,31 @@ Offline Changed files: ${files.length}`; meta: MetaEntry | false = false, includeDeletion = true ): Promise { - return await this.serializedForEvent(path, async () => { - try { - // Fetch the document with conflicts - const prefixedPath = addPrefix(path, ICHeader); - const docMeta = meta - ? meta - : await this.localDatabase.getDBEntryMeta(prefixedPath, { conflicts: true }, true); - if (docMeta === false) { - this._log(`${headerLine}: Failed to read detail of ${path}`); - throw new Error(`Failed to read detail ${path}`); - } - // Check if the file is conflicted, and if so, enqueue to resolve. - // Until the conflict is resolved, the file will not be processed. - if (docMeta._conflicts && docMeta._conflicts.length > 0) { - this.queueConflictCheck(path); - this._log(`${headerLine} Hidden file conflicted, enqueued to resolve`); - return true; - } - // And, extract (or delete) the file to storage. - const extractResult = await this.extractInternalFileFromDatabase( - path, - false, - docMeta, - preventDoubleProcess, - onlyNew, - includeDeletion - ); - if (extractResult) { - this._log(`${headerLine} Hidden file processed`); - } - } catch (ex) { - this._log(`${headerLine} Failed to process hidden file`); - this._log(ex, LOG_LEVEL_VERBOSE); - } - return true; + return await this.changeProcessor.processDatabaseChange(path, headerLine, { + preventDoubleProcess, + onlyNew, + metaEntry: meta, + includeDeletion, }); } + private queueConflict(path: FilePathWithPrefix): Promise { + this.conflictResolution.queue(path); + return Promise.resolve(true); + } + // <-- Database Event Functions // --> Notification for Config Change - queuedNotificationFiles = new Set(); - notifyConfigChange() { + private readonly queuedNotificationFiles = new Set(); + private notifyConfigChange() { const updatedFolders = [...this.queuedNotificationFiles]; this.queuedNotificationFiles.clear(); if (this.disposed) return; this.dependencies.showConfigurationChangeNotice(updatedFolders); } - queueNotification(key: FilePath) { + private queueNotification(key: FilePath) { if (this.disposed) return; if (this.settings.suppressNotifyHiddenFilesChange) { return; @@ -1319,7 +860,7 @@ Offline Changed files: ${files.length}`; // --> Initialization functions - async rebuildMerging(showNotice: boolean, targetFiles: FilePath[] | false = false) { + private async rebuildMerging(showNotice: boolean, targetFiles: FilePath[] | false = false) { const logLevel = getLogLevel(showNotice); const p = this._progress("[⚙ Rebuild by Merge ]\n", logLevel); this._log(`Rebuilding hidden files from the storage and the local database.`, logLevel); @@ -1346,9 +887,9 @@ Offline Changed files: ${files.length}`; for (const file of allFileNames) { eachProgress(); const storageMTime = await this.storageAccess.statHidden(file); - const mtimeStorage = getComparingMTime(storageMTime); + const mtimeStorage = getHiddenFileSyncComparisonMTime(storageMTime); const dbEntry = allDatabaseMap.get(file)!; - const mtimeDB = getComparingMTime(dbEntry); + const mtimeDB = getHiddenFileSyncComparisonMTime(dbEntry); const diff = compareMTime(mtimeStorage, mtimeDB); if (diff == BASE_IS_NEW) { storageToDatabase.push(file); @@ -1362,10 +903,10 @@ Offline Changed files: ${files.length}`; p.once( `Storage to Database: ${storageToDatabase.length} files\n Database to Storage: ${databaseToStorage.length} files` ); - this.resetLastProcessedDatabase(targetFiles); - this.resetLastProcessedFile(targetFiles); + this.processedState.resetLastProcessedDatabase(targetFiles); + this.processedState.resetLastProcessedFile(targetFiles); const processes = [ - this.useStorageFiles(storageToDatabase, showNotice, false), + this.trackScannedStorageChanges(storageToDatabase, showNotice, false, true), this.useDatabaseFiles(databaseToStorage, showNotice, false), ]; p.log("Start processing..."); @@ -1374,7 +915,13 @@ Offline Changed files: ${files.length}`; return [...allFileNames]; } - async rebuildFromStorage(showNotice: boolean, targetFiles: FilePath[] | false = false, onlyNew = false) { + private async runRebuildMerging(showNotice: boolean, targetFiles: FilePath[] | false = false) { + return this.rebuildMergingHook + ? await this.rebuildMergingHook(showNotice, targetFiles) + : await this.rebuildMerging(showNotice, targetFiles); + } + + private async rebuildFromStorage(showNotice: boolean, targetFiles: FilePath[] | false = false, onlyNew = false) { // reset processed file markers const logLevel = getLogLevel(showNotice); this._verbose(`Rebuilding hidden files from the storage.`); @@ -1387,13 +934,13 @@ Offline Changed files: ${files.length}`; : currentFilesAll; p.once(`Storage to Database: ${currentFiles.length} files.`); p.log("Start processing..."); - this.resetLastProcessedFile(targetFiles); - await this.useStorageFiles(currentFiles, showNotice, onlyNew); + this.processedState.resetLastProcessedFile(targetFiles); + await this.trackScannedStorageChanges(currentFiles, showNotice, onlyNew, true); p.done(); return currentFiles; } - async getAllDatabaseFiles() { + private async getAllDatabaseFiles() { const allFiles = ( await this.localDatabase.allDocsRaw({ startkey: ICHeader, endkey: ICHeaderEnd, include_docs: true }) ).rows @@ -1408,7 +955,7 @@ Offline Changed files: ${files.length}`; return files; } - async rebuildFromDatabase(showNotice: boolean, targetFiles: FilePath[] | false = false, onlyNew = false) { + private async rebuildFromDatabase(showNotice: boolean, targetFiles: FilePath[] | false = false, onlyNew = false) { const logLevel = getLogLevel(showNotice); this._verbose(`Rebuilding hidden files from the local database.`); const p = this._progress("[⚙ Rebuild by Database ]\n", logLevel); @@ -1424,7 +971,7 @@ Offline Changed files: ${files.length}`; : allFiles; p.once(`Database to Storage: ${currentFiles.length} files.`); - this.resetLastProcessedDatabase(targetFiles); + this.processedState.resetLastProcessedDatabase(targetFiles); p.log("Start processing..."); await this.useDatabaseFiles(currentFiles, showNotice, onlyNew); p.done(); @@ -1440,8 +987,11 @@ Offline Changed files: ${files.length}`; ) { const logLevel = showMessage ? LOG_LEVEL_NOTICE : LOG_LEVEL_INFO; const p = initialisationProgress ?? this._progress("[⚙ Initialise]\n", logLevel); + // Compatibility question: the legacy preflight was already disabled. + // Enabling it would change initialisation timing and could open a + // conflict dialogue while the feature is being configured. // p.log("Resolving conflicts before starting..."); - // await this.resolveConflictOnInternalFiles(); + // await this.conflictResolution.resolveAll(); p.log("Initialising hidden files sync..."); // The initialisation progress owns the user-visible Notice. Its child // rebuild and scan operations still write ordinary log entries, but @@ -1480,7 +1030,7 @@ Offline Changed files: ${files.length}`; } if (direction == "safe") { p.log(`Started: Database <--> Storage (by modified date)`); - const updatedFiles = await this.rebuildMerging(showChildNotices, targetFiles); + const updatedFiles = await this.runRebuildMerging(showChildNotices, targetFiles); await this.adoptCurrentStorageFilesAsProcessed(updatedFiles); await this.adoptCurrentDatabaseFilesAsProcessed(updatedFiles); // And, scan other changes on the database (i.e. files which are on only other devices) @@ -1492,414 +1042,7 @@ Offline Changed files: ${files.length}`; } // <-- Initialization functions - // --> Storage To Database Functions - - async __loadBaseSaveData(file: FilePath, includeContent = true): Promise { - const prefixedFileName = addPrefix(file, ICHeader); - const id = await this.path2id(prefixedFileName, ICHeader); - try { - const old = includeContent - ? await this.localDatabase.getDBEntry(prefixedFileName, undefined, false, true) - : await this.localDatabase.getDBEntryMeta(prefixedFileName, { conflicts: true }, true); - if (old === false) { - const baseSaveData: LoadedEntry = { - _id: id, - data: [], - path: prefixedFileName, - mtime: 0, - ctime: 0, - datatype: "newnote", - children: [], - size: 0, - deleted: false, - type: "newnote", - eden: {}, - }; - return baseSaveData; - } else { - return old; - } - } catch (ex) { - this._log(`Getting base save data failed`); - this._log(ex, LOG_LEVEL_VERBOSE); - return false; - } - } - - private async getLiveInternalRevision( - prefixedFileName: FilePathWithPrefix, - revision: string - ): Promise { - const [selected, current, conflicts] = await Promise.all([ - this.databaseFileAccess.fetchEntryMeta(prefixedFileName, revision, true), - this.databaseFileAccess.fetchEntryMeta(prefixedFileName, undefined, true), - this.databaseFileAccess.getConflictedRevs(prefixedFileName), - ]); - const liveRevisions = new Set([...(current && current._rev ? [current._rev] : []), ...conflicts]); - if (!selected || selected._rev !== revision || !liveRevisions.has(revision)) { - this._log( - `Could not use hidden-file revision ${revision} of ${stripAllPrefixes(prefixedFileName)}; the selected revision is no longer live`, - LOG_LEVEL_NOTICE - ); - return false; - } - return selected; - } - - async storeInternalFileToDatabase(file: InternalFileInfo | UXFileInfo, forceWrite = false) { - const storeFilePath = stripAllPrefixes(file.path); - const storageFilePath = file.path; - if (await this.dependencies.isIgnoredByIgnoreFile(storageFilePath)) { - return undefined; - } - const prefixedFileName = addPrefix(storeFilePath, ICHeader); - - return await serialized("file-" + prefixedFileName, async () => { - try { - const fileInfo = "stat" in file && "body" in file ? file : await this.loadFileWithInfo(storeFilePath); - if (fileInfo.deleted) { - throw new Error(`Hidden file:${storeFilePath} is deleted. This should not be occurred.`); - } - const baseData = await this.__loadBaseSaveData(storeFilePath, true); - if (baseData === false) throw new Error("Failed to load base data"); - if (baseData._rev && !forceWrite) { - // Not newly created, we should check the content is actually modified. - const isSame = await isDocContentSame(readAsBlob(baseData), fileInfo.body); - if (isSame) { - this.updateLastProcessed(storeFilePath, baseData, fileInfo.stat); - // Not changed. skip. - // TODO: Mark as same? - return undefined; - } - } - const saveData: SavingEntry = { - ...baseData, - data: fileInfo.body, - mtime: fileInfo.stat.mtime, - size: fileInfo.stat.size, - children: [], - deleted: false, - type: baseData.datatype, - }; - const ret = await this.localDatabase.putDBEntry(saveData); - if (ret && ret.ok) { - saveData._rev = ret.rev; - this.updateLastProcessed(storeFilePath, saveData, fileInfo.stat); - } - const success = ret && ret.ok; - this._log(`STORAGE --> DB:${storageFilePath}: (hidden) ${success ? "Done" : "Failed"}`); - return success; - } catch (ex) { - this._log(`STORAGE --> DB:${storageFilePath}: (hidden) Failed`); - this._log(ex, LOG_LEVEL_VERBOSE); - return false; - } - }); - } - - async storeInternalFileToDatabaseWithBaseRevision( - file: InternalFileInfo | UXFileInfo, - baseRevision: string, - createIfDifferent = true - ): Promise { - const storeFilePath = stripAllPrefixes(file.path); - const storageFilePath = file.path; - if (await this.dependencies.isIgnoredByIgnoreFile(storageFilePath)) { - return false; - } - const prefixedFileName = addPrefix(storeFilePath, ICHeader); - - return await serialized("file-" + prefixedFileName, async () => { - try { - const baseData = await this.getLiveInternalRevision(prefixedFileName, baseRevision); - if (baseData === false) { - return false; - } - const fileInfo = "stat" in file && "body" in file ? file : await this.loadFileWithInfo(storeFilePath); - if (fileInfo.deleted) { - throw new Error(`Hidden file:${storeFilePath} is deleted. This should not be occurred.`); - } - if (!baseData.deleted && !baseData._deleted) { - const loadedBase = await this.databaseFileAccess.fetchEntryFromMeta(baseData, true, true); - if (loadedBase && (await isDocContentSame(readAsBlob(loadedBase), fileInfo.body))) { - this.updateLastProcessed(storeFilePath, baseData, fileInfo.stat); - return true; - } - } - if (!createIfDifferent) { - this._log( - `Could not mark hidden file ${storeFilePath} as revision ${baseRevision}; the storage content differs`, - LOG_LEVEL_NOTICE - ); - return false; - } - - const storedRevision = await this.databaseFileAccess.storeWithBaseRevision( - { - ...fileInfo, - path: storeFilePath, - name: fileInfo.name || storeFilePath.split("/").pop() || "", - isInternal: true, - }, - baseRevision, - true - ); - if (storedRevision === false) { - return false; - } - this.updateLastProcessed( - storeFilePath, - { - ...baseData, - _rev: storedRevision, - path: prefixedFileName, - ctime: fileInfo.stat.ctime, - mtime: fileInfo.stat.mtime, - size: fileInfo.stat.size, - deleted: false, - }, - fileInfo.stat - ); - this._log(`STORAGE --> DB:${storageFilePath}: (hidden, selected branch) Done`); - return true; - } catch (ex) { - this._log(`STORAGE --> DB:${storageFilePath}: (hidden, selected branch) Failed`); - this._log(ex, LOG_LEVEL_VERBOSE); - return false; - } - }); - } - - async deleteInternalFileOnDatabase(filenameSrc: FilePath, forceWrite = false) { - const storeFilePath = filenameSrc; - const storageFilePath = filenameSrc; - const displayFileName = filenameSrc; - const prefixedFileName = addPrefix(storeFilePath, ICHeader); - const mtime = new Date().getTime(); - if (await this.dependencies.isIgnoredByIgnoreFile(storageFilePath)) { - return undefined; - } - return await serialized("file-" + prefixedFileName, async () => { - try { - const baseData = await this.__loadBaseSaveData(storeFilePath, false); - if (baseData === false) throw new Error("Failed to load base data during deleting"); - if (baseData._conflicts !== undefined) { - for (const conflictRev of baseData._conflicts) { - await this.localDatabase.removeRevision(baseData._id, conflictRev); - this._log( - `STORAGE -x> DB: ${displayFileName}: (hidden) conflict removed ${baseData._rev} => ${conflictRev}`, - LOG_LEVEL_VERBOSE - ); - } - } - if (baseData.deleted) { - this._log(`STORAGE -x> DB: ${displayFileName}: (hidden) already deleted`, LOG_LEVEL_VERBOSE); - this.updateLastProcessedDeletion(storeFilePath, baseData); - return true; - } - const saveData: LoadedEntry = { - ...baseData, - mtime, - size: 0, - children: [], - deleted: true, - type: baseData.datatype, - }; - const ret = await this.localDatabase.putRaw(saveData); - if (ret && ret.ok) { - this._log(`STORAGE -x> DB: ${displayFileName}: (hidden) Done`); - saveData._rev = ret.rev; - this.updateLastProcessedDeletion(storeFilePath, saveData); - return true; - } else { - this._log(`STORAGE -x> DB: ${displayFileName}: (hidden) Failed`); - return false; - } - } catch (ex) { - this._log(`STORAGE -x> DB: ${displayFileName}: (hidden) Failed`); - this._log(ex, LOG_LEVEL_VERBOSE); - return false; - } - }); - } - - // <-- Storage To Database Functions - - // --> Database To Storage Functions - - async extractInternalFileFromDatabase( - storageFilePath: FilePath, - force = false, - metaEntry?: MetaEntry | LoadedEntry, - preventDoubleProcess = true, - onlyNew = false, - includeDeletion = true, - requiredLiveRevision?: string - ) { - const prefixedFileName = addPrefix(storageFilePath, ICHeader); - if (await this.dependencies.isIgnoredByIgnoreFile(storageFilePath)) { - return undefined; - } - return await serialized("file-" + prefixedFileName, async () => { - try { - // Check conflicted status - const metaOnDB = requiredLiveRevision - ? await this.getLiveInternalRevision(prefixedFileName, requiredLiveRevision) - : metaEntry - ? metaEntry - : await this.localDatabase.getDBEntryMeta(prefixedFileName, { conflicts: true }, true); - if (metaOnDB === false) throw new Error(`File not found on database.:${storageFilePath}`); - // Prevent overwrite for Prevent overwriting while some conflicted revision exists. - if (metaOnDB?._conflicts?.length) { - this._log( - `Hidden file ${storageFilePath} has conflicted revisions, to keep in safe, writing to storage has been prevented`, - LOG_LEVEL_INFO - ); - return false; - } - if (preventDoubleProcess) { - const key = this.docToKey(metaOnDB); - if (this.getLastProcessedDatabaseKey(storageFilePath) == key && !force) { - this._log( - `STORAGE <-- DB: ${storageFilePath}: skipped (hidden, overwrite${force ? ", force" : ""}) (Previously processed)` - ); - return; - } - } - if (onlyNew) { - // Check the file is new or not. - const dbMTime = getComparingMTime(metaOnDB, includeDeletion); // metaOnDB.mtime; - const storageStat = await this.storageAccess.statHidden(storageFilePath); - const storageMTimeActual = storageStat?.mtime ?? 0; - const storageMTime = - storageMTimeActual == 0 ? this.getLastProcessedFileMTime(storageFilePath) : storageMTimeActual; - const diff = compareMTime(storageMTime, dbMTime); - if (diff != TARGET_IS_NEW) { - this._log( - `STORAGE <-- DB: ${storageFilePath}: skipped (hidden, overwrite${force ? ", force" : ""}) (Not new)` - ); - // And this case, we should update the last processed key. - this.updateLastProcessedDatabase(storageFilePath, metaOnDB); - if (storageStat) this.updateLastProcessedFile(storageFilePath, storageStat); - return; - } - } - const deleted = metaOnDB.deleted || metaOnDB._deleted || false; - if (deleted) { - const result = await this.__deleteFile(storageFilePath); - if (result == "OK") { - this.updateLastProcessedDeletion(storageFilePath, metaOnDB); - return true; - } else if (result == "ALREADY") { - this.updateLastProcessedDatabase(storageFilePath, metaOnDB); - return true; - } - return false; - } else { - const fileOnDB = await this.localDatabase.getDBEntryFromMeta(metaOnDB, false, true); - if (fileOnDB === false) { - throw new Error(`Failed to read file from database:${storageFilePath}`); - } - const resultStat = await this.__writeFile(storageFilePath, fileOnDB, force); - if (resultStat) { - this.updateLastProcessed(storageFilePath, metaOnDB, resultStat); - this.queueNotification(storageFilePath); - this._log( - `STORAGE <-- DB: ${storageFilePath}: written (hidden, overwrite${force ? ", force" : ""}) Done` - ); - return true; - } - } - return false; - } catch (ex) { - this._log( - `STORAGE <-- DB: ${storageFilePath}: written (hidden, overwrite${force ? ", force" : ""}) Failed` - ); - this._log(ex, LOG_LEVEL_VERBOSE); - return false; - } - }); - } - - async extractInternalFileRevisionFromDatabase( - storageFilePath: FilePath, - revision: string, - force = false - ): Promise { - return Boolean( - await this.extractInternalFileFromDatabase(storageFilePath, force, undefined, true, false, true, revision) - ); - } - - async __checkIsNeedToWriteFile(storageFilePath: FilePath, content: string | ArrayBuffer): Promise { - try { - const storageContent = await this.storageAccess.readHiddenFileAuto(storageFilePath); - const needWrite = !(await isDocContentSame(storageContent, content)); - return needWrite; - } catch (ex) { - this._log(`Cannot check the content of ${storageFilePath}`); - this._log(ex, LOG_LEVEL_VERBOSE); - return true; - } - } - - async __writeFile(storageFilePath: FilePath, fileOnDB: LoadedEntry, force: boolean): Promise { - try { - const statBefore = await this.storageAccess.statHidden(storageFilePath); - const isExist = statBefore != null; - const writeContent = readContent(fileOnDB); - await this.ensureDir(storageFilePath); - // We have to compare the content, so read it once. - const needWrite = - force || !isExist || (isExist && (await this.__checkIsNeedToWriteFile(storageFilePath, writeContent))); - - if (!needWrite) { - this._log(`STORAGE <-- DB: ${storageFilePath}: skipped (hidden) Not changed`, LOG_LEVEL_DEBUG); - return statBefore; - } - - const writeResultStat = await this.writeFile(storageFilePath, writeContent, { - mtime: fileOnDB.mtime, - ctime: fileOnDB.ctime, - }); - - if (writeResultStat == null) { - this._log( - `STORAGE <-- DB: ${storageFilePath}: written (hidden,new${force ? ", force" : ""}) Failed (writeResult)` - ); - return false; - } - // await this.triggerEvent(storageFilePath); - // markChangesAreSame(storageFilePath, getComparingMTime(writeResultStat), getComparingMTime(fileOnDB)); - this._log(`STORAGE <-- DB: ${storageFilePath}: written (hidden, overwrite${force ? ", force" : ""})`); - return writeResultStat; - } catch (ex) { - this._log( - `STORAGE <-- DB: ${storageFilePath}: written (hidden, overwrite${force ? ", force" : ""}) Failed` - ); - this._log(ex, LOG_LEVEL_VERBOSE); - return false; - } - } - - async __deleteFile(storageFilePath: FilePath): Promise { - const result = await this.__removeFile(storageFilePath); - if (result === false) { - this._log(`STORAGE { + private suspendExtraSync(): Promise { if (this.settings.syncInternalFiles) { this._log( $msg( @@ -1913,12 +1056,12 @@ Offline Changed files: ${files.length}`; } // --> Configuration handling - async _allConfigureOptionalSyncFeature(mode: OptionalSyncFeatureMode) { + private async configureOptionalSyncFeature(mode: OptionalSyncFeatureMode) { await this.configureHiddenFileSync(mode); return true; } - async configureHiddenFileSync(mode: OptionalSyncFeatureMode) { + private async configureHiddenFileSync(mode: OptionalSyncFeatureMode) { let initialisationProgress: HiddenFileSyncProgress | undefined; let result: ConfigureHiddenFileSyncResult; try { @@ -1962,15 +1105,21 @@ Offline Changed files: ${files.length}`; // <-- Configuration handling // --> Local Storage SubFunctions - async scanInternalFileNames() { + private async scanInternalFileNames() { const findRoot = this.dependencies.getRootPath(); - const filenames = await this.getFiles(findRoot, (path) => this.isTargetFile(path)); + const filenames = await collectOptionalFileSyncFiles(this.dependencies, findRoot, { + shouldInclude: (path) => this.isTargetFile(path as FilePath), + onError: (path, error) => { + this._log(`Could not traverse(HiddenSync):${path}`, LOG_LEVEL_INFO); + this._log(error, LOG_LEVEL_VERBOSE); + }, + }); return filenames as FilePath[]; } - async scanInternalFiles(): Promise { + private async scanInternalFiles(): Promise { const fileNames = await this.scanInternalFileNames(); const files = fileNames.map(async (e) => { return { @@ -1997,29 +1146,5 @@ Offline Changed files: ${files.length}`; return result; } - async getFiles(path: string, checkFunction: (path: FilePath) => Promise | boolean) { - let w: HiddenFileSyncDirectoryListing; - try { - w = await this.dependencies.listFiles(path); - } catch (ex) { - this._log(`Could not traverse(HiddenSync):${path}`, LOG_LEVEL_INFO); - this._log(ex, LOG_LEVEL_VERBOSE); - return []; - } - let files = [] as string[]; - for (const file of w.files) { - if (!(await checkFunction(file as FilePath))) { - continue; - } - files.push(file); - } - for (const v of w.folders) { - if (!(await checkFunction(v as FilePath))) { - continue; - } - files = files.concat(await this.getFiles(v, checkFunction)); - } - return files; - } // <-- Local Storage SubFunctions } diff --git a/src/features/HiddenFileSync/hiddenFileSyncContext.unit.spec.ts b/src/features/HiddenFileSync/hiddenFileSyncContext.unit.spec.ts index 99807b43..fb30923d 100644 --- a/src/features/HiddenFileSync/hiddenFileSyncContext.unit.spec.ts +++ b/src/features/HiddenFileSync/hiddenFileSyncContext.unit.spec.ts @@ -1,12 +1,5 @@ import { describe, expect, it, vi } from "vitest"; -import { - type DocumentID, - LOG_LEVEL_NOTICE, - type FilePath, - type FilePathWithPrefix, - type MetaEntry, - type UXFileInfo, -} from "@vrtmrz/livesync-commonlib/compat/common/types"; +import { LOG_LEVEL_NOTICE } from "@vrtmrz/livesync-commonlib/compat/common/types"; vi.mock("@/deps.ts", () => ({})); vi.mock("./configureHiddenFileSyncMode.ts", () => ({ @@ -16,81 +9,27 @@ vi.mock("./configureHiddenFileSyncMode.ts", () => ({ import { HiddenFileSyncContext } from "./hiddenFileSyncContext.ts"; import { configureHiddenFileSyncMode } from "./configureHiddenFileSyncMode.ts"; -function createHiddenRevisionOperation() { - const path = ".obsidian/plugins/example/data.json" as FilePath; - const file = { - path, - name: "data.json", - isInternal: true, - body: new Blob(['{"value":"vault"}']), - stat: { - ctime: 1, - mtime: 2, - size: 17, - type: "file", - }, - } as UXFileInfo; - const selected = { - _id: "i:example" as DocumentID, - _rev: "2-selected", - path: `i:${path}` as FilePathWithPrefix, - ctime: 1, - mtime: 2, - size: 17, - type: "plain", - datatype: "plain", - children: [], - eden: {}, - deleted: false, - } as MetaEntry; - const winner = { - ...selected, - _rev: "3-winner", - } as MetaEntry; - const databaseFileAccess = { - fetchEntryMeta: vi.fn(async (_path: unknown, revision?: string) => - revision === selected._rev ? selected : winner - ), - getConflictedRevs: vi.fn(async () => [selected._rev]), - fetchEntryFromMeta: vi.fn(async () => ({ ...selected, data: '{"value":"database"}' })), - storeWithBaseRevision: vi.fn(async () => "3-vault-child"), - }; - const hiddenFileSync = Object.create(HiddenFileSyncContext.prototype) as HiddenFileSyncContext; - Object.assign(hiddenFileSync, { - dependencies: { - databaseFileAccess, - isIgnoredByIgnoreFile: vi.fn(async () => false), - }, - loadFileWithInfo: vi.fn(async () => file), - updateLastProcessed: vi.fn(), - _log: vi.fn(), - }); - return { - hiddenFileSync, - path, - file, - selected, - winner, - databaseFileAccess, - }; +function callPrivate unknown>(context: HiddenFileSyncContext, key: string): T { + const operation = (context as unknown as Record)[key]; + return operation.bind(context) as T; } describe("HiddenFileSyncContext configuration-change notices", () => { it("releases processors, transient queues, the pattern cache, activity, and the host Notice effect", () => { const periodicInternalFileScanProcessor = { disable: vi.fn() }; - const conflictResolutionProcessor = { terminate: vi.fn() }; - const pendingConflictChecks = new Set(["i:.obsidian/example.json"]); + const conflictResolution = { dispose: vi.fn() }; const queuedNotificationFiles = new Set([".obsidian/plugins/example"]); const cacheFileRegExps = new Map([["patterns", []]]); const publishActivity = vi.fn(); + const changeProcessor = { dispose: vi.fn() }; const hideConfigurationChangeNotice = vi.fn(); const closeJsonConflictDialogs = vi.fn(); const hiddenFileSync = Object.create(HiddenFileSyncContext.prototype) as HiddenFileSyncContext; Object.assign(hiddenFileSync, { dependencies: { publishActivity, hideConfigurationChangeNotice, closeJsonConflictDialogs }, periodicInternalFileScanProcessor, - conflictResolutionProcessor, - pendingConflictChecks, + conflictResolution, + changeProcessor, queuedNotificationFiles, cacheFileRegExps, eventCount: 4, @@ -101,11 +40,10 @@ describe("HiddenFileSyncContext configuration-change notices", () => { hiddenFileSync.dispose(); expect(periodicInternalFileScanProcessor.disable).toHaveBeenCalledOnce(); - expect(conflictResolutionProcessor.terminate).toHaveBeenCalledOnce(); - expect(pendingConflictChecks.size).toBe(0); + expect(conflictResolution.dispose).toHaveBeenCalledOnce(); expect(queuedNotificationFiles.size).toBe(0); expect(cacheFileRegExps.size).toBe(0); - expect(publishActivity).toHaveBeenCalledWith(0, 0); + expect(changeProcessor.dispose).toHaveBeenCalledOnce(); expect(closeJsonConflictDialogs).toHaveBeenCalledOnce(); expect(hideConfigurationChangeNotice).toHaveBeenCalledOnce(); }); @@ -122,7 +60,7 @@ describe("HiddenFileSyncContext configuration-change notices", () => { _isMainSuspended: vi.fn(() => false), }); - expect(hiddenFileSync.isReady()).toBe(false); + expect(callPrivate<() => boolean>(hiddenFileSync, "isReady")()).toBe(false); }); it("settles one batch of changed folders through the host notification effect", () => { @@ -133,14 +71,16 @@ describe("HiddenFileSyncContext configuration-change notices", () => { queuedNotificationFiles: new Set([".obsidian/plugins/alpha", ".obsidian/plugins/beta", ".obsidian"]), }); - hiddenFileSync.notifyConfigChange(); + callPrivate<() => void>(hiddenFileSync, "notifyConfigChange")(); expect(showConfigurationChangeNotice).toHaveBeenCalledWith([ ".obsidian/plugins/alpha", ".obsidian/plugins/beta", ".obsidian", ]); - expect(hiddenFileSync.queuedNotificationFiles.size).toBe(0); + expect((hiddenFileSync as unknown as { queuedNotificationFiles: Set }).queuedNotificationFiles.size).toBe( + 0 + ); }); it("keeps subordinate initialisation phases below Notice level so one progress Notice owns the scan", async () => { @@ -200,7 +140,7 @@ describe("HiddenFileSyncContext configuration-change notices", () => { _log: log, }); - await hiddenFileSync.configureHiddenFileSync("MERGE"); + await callPrivate<(mode: "MERGE") => Promise>(hiddenFileSync, "configureHiddenFileSync")("MERGE"); expect(createProgress).toHaveBeenCalledWith("[⚙ Initialise]\n", LOG_LEVEL_NOTICE); expect(events[0]).toBe("progress:Preparing Hidden File Sync..."); @@ -232,119 +172,10 @@ describe("HiddenFileSyncContext configuration-change notices", () => { _log: vi.fn(), }); - await expect(hiddenFileSync.configureHiddenFileSync("MERGE")).rejects.toBe(error); + await expect( + callPrivate<(mode: "MERGE") => Promise>(hiddenFileSync, "configureHiddenFileSync")("MERGE") + ).rejects.toBe(error); expect(progress.done).toHaveBeenCalledWith("Failed"); }); }); - -describe("HiddenFileSyncContext reconciliation settlement", () => { - it("compatibility: consumes a selected database file even when reading its Metadata fails", async () => { - const error = new Error("metadata unavailable"); - const getDBEntryMeta = vi.fn(async () => { - throw error; - }); - const log = vi.fn(); - const hiddenFileSync = Object.create(HiddenFileSyncContext.prototype) as HiddenFileSyncContext; - Object.assign(hiddenFileSync, { - dependencies: { - getLocalDatabase: () => ({ getDBEntryMeta }), - log, - }, - serializedForEvent: vi.fn(async (_path: FilePath, operation: () => Promise) => await operation()), - }); - - await expect( - hiddenFileSync.trackDatabaseFileModification(".obsidian/app.json" as FilePath, "[Replication]") - ).resolves.toBe(true); - - expect(log).toHaveBeenCalledWith("[Replication] Failed to process hidden file", undefined, undefined); - expect(log).toHaveBeenCalledWith(error, expect.any(Number), undefined); - }); -}); - -describe("HiddenFileSyncContext exact revision repair operations", () => { - it("stores the current hidden Vault file as a child of the selected live revision", async () => { - const { hiddenFileSync, file, selected, databaseFileAccess } = createHiddenRevisionOperation(); - - await expect(hiddenFileSync.storeInternalFileToDatabaseWithBaseRevision(file, selected._rev!)).resolves.toBe( - true - ); - - expect(databaseFileAccess.storeWithBaseRevision).toHaveBeenCalledWith( - expect.objectContaining({ - path: file.path, - body: file.body, - isInternal: true, - }), - selected._rev, - true - ); - expect(hiddenFileSync.updateLastProcessed).toHaveBeenCalledWith( - file.path, - expect.objectContaining({ _rev: "3-vault-child" }), - file.stat - ); - }); - - it("refuses to extend a hidden-file revision which is no longer live", async () => { - const { hiddenFileSync, file, selected, databaseFileAccess } = createHiddenRevisionOperation(); - databaseFileAccess.getConflictedRevs.mockResolvedValue([]); - - await expect(hiddenFileSync.storeInternalFileToDatabaseWithBaseRevision(file, selected._rev!)).resolves.toBe( - false - ); - - expect(databaseFileAccess.storeWithBaseRevision).not.toHaveBeenCalled(); - expect(hiddenFileSync.updateLastProcessed).not.toHaveBeenCalled(); - }); - - it("does not create a hidden-file child when asked only to mark a revision which differs from the Vault", async () => { - const { hiddenFileSync, file, selected, databaseFileAccess } = createHiddenRevisionOperation(); - - await expect( - hiddenFileSync.storeInternalFileToDatabaseWithBaseRevision(file, selected._rev!, false) - ).resolves.toBe(false); - - expect(databaseFileAccess.storeWithBaseRevision).not.toHaveBeenCalled(); - expect(hiddenFileSync.updateLastProcessed).not.toHaveBeenCalled(); - }); - - it("marks a matching hidden-file revision without creating a child", async () => { - const { hiddenFileSync, file, selected, databaseFileAccess } = createHiddenRevisionOperation(); - databaseFileAccess.fetchEntryFromMeta.mockResolvedValue({ - ...selected, - data: '{"value":"vault"}', - }); - - await expect( - hiddenFileSync.storeInternalFileToDatabaseWithBaseRevision(file, selected._rev!, false) - ).resolves.toBe(true); - - expect(databaseFileAccess.storeWithBaseRevision).not.toHaveBeenCalled(); - expect(hiddenFileSync.updateLastProcessed).toHaveBeenCalledWith(file.path, selected, file.stat); - }); - - it("applies the selected live hidden-file revision through the existing extraction path", async () => { - const { hiddenFileSync, path, selected } = createHiddenRevisionOperation(); - const extract = vi.fn(async () => true); - hiddenFileSync.extractInternalFileFromDatabase = extract; - - await expect(hiddenFileSync.extractInternalFileRevisionFromDatabase(path, selected._rev!, true)).resolves.toBe( - true - ); - - expect(extract).toHaveBeenCalledWith(path, true, undefined, true, false, true, selected._rev); - }); - - it("does not apply a hidden-file revision which ceased to be live", async () => { - const { hiddenFileSync, path, selected, databaseFileAccess } = createHiddenRevisionOperation(); - databaseFileAccess.getConflictedRevs.mockResolvedValue([]); - - await expect(hiddenFileSync.extractInternalFileRevisionFromDatabase(path, selected._rev!, true)).resolves.toBe( - false - ); - - expect(databaseFileAccess.fetchEntryFromMeta).not.toHaveBeenCalled(); - }); -}); diff --git a/src/features/HiddenFileSync/hiddenFileSyncDatabaseExtractionOperations.ts b/src/features/HiddenFileSync/hiddenFileSyncDatabaseExtractionOperations.ts new file mode 100644 index 00000000..7ca454d0 --- /dev/null +++ b/src/features/HiddenFileSync/hiddenFileSyncDatabaseExtractionOperations.ts @@ -0,0 +1,223 @@ +import { + LOG_LEVEL_INFO, + LOG_LEVEL_VERBOSE, + type FilePath, + type FilePathWithPrefix, + type LoadedEntry, + type LOG_LEVEL, + type MetaEntry, + type UXStat, +} from "@vrtmrz/livesync-commonlib/compat/common/types"; +import { compareMTime, TARGET_IS_NEW } from "@/common/utils.ts"; +import type { LogFunction } from "@vrtmrz/livesync-commonlib/compat/services/lib/logUtils"; +import { addPrefix } from "@vrtmrz/livesync-commonlib/compat/string_and_binary/path"; + +import { ICHeader } from "@/common/types.ts"; +import { getHiddenFileSyncComparisonMTime } from "./hiddenFileSyncState.ts"; +import type { HiddenFileSyncRemovalResult } from "./hiddenFileSyncStorage.ts"; +import { + serialiseHiddenFileOperation, + type HiddenFileSyncFileSerialisationDependencies, +} from "./hiddenFileSyncFileOperations.ts"; + +export type HiddenFileSyncDatabaseExtractionOptions = Readonly<{ + force?: boolean; + metaEntry?: MetaEntry | LoadedEntry; + preventDoubleProcess?: boolean; + onlyNew?: boolean; + includeDeletion?: boolean; + requiredLiveRevision?: string; +}>; + +type HiddenFileSyncDatabaseExtractionStorageDependencies = { + statStorageFile(path: FilePath): Promise; + writeStorageFile(path: FilePath, entry: LoadedEntry, force: boolean): Promise; + deleteStorageFile(path: FilePath): Promise; +}; + +type HiddenFileSyncDatabaseExtractionReadDependencies = { + loadDatabaseMetadata(path: FilePathWithPrefix): Promise; + loadLiveRevision(path: FilePathWithPrefix, revision: string): Promise; + loadDatabaseEntry(entry: MetaEntry | LoadedEntry): Promise; +}; + +export type HiddenFileSyncDatabaseExtractionProcessedState = { + databaseStateKey(entry: MetaEntry | LoadedEntry): string; + getLastProcessedDatabaseKey(path: FilePath): string | undefined; + getLastProcessedFileMTime(path: FilePath): number; + updateLastProcessedDatabase(path: FilePath, entry: string | MetaEntry | LoadedEntry): void; + updateLastProcessedFile(path: FilePath, storageFile: string | UXStat): void; + updateLastProcessed(path: FilePath, databaseEntry: MetaEntry | LoadedEntry, storageFile: UXStat): void; + updateLastProcessedDeletion(path: FilePath, databaseEntry: MetaEntry | LoadedEntry | false): void; +}; + +export type HiddenFileSyncDatabaseExtractionDependencies = HiddenFileSyncFileSerialisationDependencies & + HiddenFileSyncDatabaseExtractionStorageDependencies & + HiddenFileSyncDatabaseExtractionReadDependencies & { + isIgnoredByIgnoreFile(path: string): Promise; + queueNotification(path: FilePath): void; + processedState: HiddenFileSyncDatabaseExtractionProcessedState; + log: LogFunction; + }; + +export type HiddenFileSyncDatabaseExtractionOperations = { + extract(path: FilePath, options?: HiddenFileSyncDatabaseExtractionOptions): Promise; + extractRevision(path: FilePath, revision: string, force?: boolean): Promise; +}; + +function log( + dependencies: Pick, + message: unknown, + level?: LOG_LEVEL, + key?: string +): void { + dependencies.log(message, level, key); +} + +export async function extractHiddenFileFromDatabase( + dependencies: HiddenFileSyncDatabaseExtractionDependencies, + storageFilePath: FilePath, + options: HiddenFileSyncDatabaseExtractionOptions = {} +): Promise { + const { + force = false, + metaEntry, + preventDoubleProcess = true, + onlyNew = false, + includeDeletion = true, + requiredLiveRevision, + } = options; + const prefixedFileName = addPrefix(storageFilePath, ICHeader); + // Compatibility: admission happens outside the per-file lock, so ignore + // policy errors propagate instead of becoming a false extraction result. + if (await dependencies.isIgnoredByIgnoreFile(storageFilePath)) { + return undefined; + } + return await serialiseHiddenFileOperation(dependencies, prefixedFileName, async () => { + try { + // A caller-supplied Metadata entry is trusted as-is. It can change + // before this lock is acquired; only exact-revision repair performs + // an in-lock liveness check. + const metaOnDatabase = requiredLiveRevision + ? await dependencies.loadLiveRevision(prefixedFileName, requiredLiveRevision) + : metaEntry + ? metaEntry + : await dependencies.loadDatabaseMetadata(prefixedFileName); + // Compatibility: the exact-revision loader validates revision-tree + // membership but normally returns a leaf without `_conflicts`. + // Repair can therefore apply one live branch while ordinary + // reflection remains blocked by conflicted winning Metadata. + if (metaOnDatabase === false) { + throw new Error(`File not found on database.:${storageFilePath}`); + } + if (metaOnDatabase._conflicts?.length) { + log( + dependencies, + `Hidden file ${storageFilePath} has conflicted revisions, to keep in safe, writing to storage has been prevented`, + LOG_LEVEL_INFO + ); + return false; + } + if (preventDoubleProcess) { + const key = dependencies.processedState.databaseStateKey(metaOnDatabase); + if (dependencies.processedState.getLastProcessedDatabaseKey(storageFilePath) == key && !force) { + // Compatibility question: the force suffix is unreachable + // because this branch is entered only when force is false. + log( + dependencies, + `STORAGE <-- DB: ${storageFilePath}: skipped (hidden, overwrite${force ? ", force" : ""}) (Previously processed)` + ); + return undefined; + } + } + if (onlyNew) { + const databaseMTime = getHiddenFileSyncComparisonMTime(metaOnDatabase, includeDeletion); + const storageStat = await dependencies.statStorageFile(storageFilePath); + const storageMTimeActual = storageStat?.mtime ?? 0; + const storageMTime = + storageMTimeActual == 0 + ? dependencies.processedState.getLastProcessedFileMTime(storageFilePath) + : storageMTimeActual; + const difference = compareMTime(storageMTime, databaseMTime); + if (difference != TARGET_IS_NEW) { + log( + dependencies, + `STORAGE <-- DB: ${storageFilePath}: skipped (hidden, overwrite${force ? ", force" : ""}) (Not new)` + ); + // Compatibility: a declined candidate is settled as + // processed, including a deletion excluded by + // includeDeletion. This prevents later scans retrying it. + dependencies.processedState.updateLastProcessedDatabase(storageFilePath, metaOnDatabase); + if (storageStat) dependencies.processedState.updateLastProcessedFile(storageFilePath, storageStat); + return undefined; + } + } + const deleted = metaOnDatabase.deleted || metaOnDatabase._deleted || false; + if (deleted) { + const result = await dependencies.deleteStorageFile(storageFilePath); + if (result == "OK") { + dependencies.processedState.updateLastProcessedDeletion(storageFilePath, metaOnDatabase); + return true; + } + if (result == "ALREADY") { + // Compatibility question: an already absent file updates + // only the database key. It does not record the missing + // storage key or call unmarkChanges through deletion state. + dependencies.processedState.updateLastProcessedDatabase(storageFilePath, metaOnDatabase); + return true; + } + return false; + } + + const fileOnDatabase = await dependencies.loadDatabaseEntry(metaOnDatabase); + if (fileOnDatabase === false) { + throw new Error(`Failed to read file from database:${storageFilePath}`); + } + const resultStat = await dependencies.writeStorageFile(storageFilePath, fileOnDatabase, force); + if (resultStat) { + // Compatibility question: the storage writer also returns the + // existing stat when content is unchanged. That no-op still + // settles state and queues a configuration notification here. + dependencies.processedState.updateLastProcessed(storageFilePath, metaOnDatabase, resultStat); + dependencies.queueNotification(storageFilePath); + log( + dependencies, + `STORAGE <-- DB: ${storageFilePath}: written (hidden, overwrite${force ? ", force" : ""}) Done` + ); + return true; + } + return false; + } catch (error) { + log( + dependencies, + `STORAGE <-- DB: ${storageFilePath}: written (hidden, overwrite${force ? ", force" : ""}) Failed` + ); + log(dependencies, error, LOG_LEVEL_VERBOSE); + return false; + } + }); +} + +export async function extractHiddenFileRevisionFromDatabase( + dependencies: HiddenFileSyncDatabaseExtractionDependencies, + storageFilePath: FilePath, + revision: string, + force = false +): Promise { + return Boolean( + await extractHiddenFileFromDatabase(dependencies, storageFilePath, { + force, + requiredLiveRevision: revision, + }) + ); +} + +export function createHiddenFileSyncDatabaseExtractionOperations( + dependencies: HiddenFileSyncDatabaseExtractionDependencies +): HiddenFileSyncDatabaseExtractionOperations { + return Object.freeze({ + extract: async (path, options) => await extractHiddenFileFromDatabase(dependencies, path, options), + extractRevision: async (path, revision, force) => + await extractHiddenFileRevisionFromDatabase(dependencies, path, revision, force), + }); +} diff --git a/src/features/HiddenFileSync/hiddenFileSyncDatabaseExtractionOperations.unit.spec.ts b/src/features/HiddenFileSync/hiddenFileSyncDatabaseExtractionOperations.unit.spec.ts new file mode 100644 index 00000000..bbb4cb70 --- /dev/null +++ b/src/features/HiddenFileSync/hiddenFileSyncDatabaseExtractionOperations.unit.spec.ts @@ -0,0 +1,366 @@ +import { describe, expect, it, vi } from "vitest"; +import { + LOG_LEVEL_INFO, + LOG_LEVEL_VERBOSE, + type DocumentID, + type FilePath, + type FilePathWithPrefix, + type LoadedEntry, + type MetaEntry, + type UXStat, +} from "@vrtmrz/livesync-commonlib/compat/common/types"; +import type { HiddenFileSyncRemovalResult } from "./hiddenFileSyncStorage.ts"; + +vi.mock("@/deps.ts", () => ({})); + +import { + createHiddenFileSyncDatabaseExtractionOperations, + extractHiddenFileFromDatabase, + extractHiddenFileRevisionFromDatabase, + type HiddenFileSyncDatabaseExtractionDependencies, +} from "./hiddenFileSyncDatabaseExtractionOperations.ts"; +import { toHiddenFileSyncDatabaseStateKey } from "./hiddenFileSyncState.ts"; + +const path = ".obsidian/plugins/example/data.json" as FilePath; +const prefixedPath = `i:${path}` as FilePathWithPrefix; +const id = "hidden-entry-id" as DocumentID; + +function metadata(overrides: Partial = {}): MetaEntry { + return { + _id: id, + _rev: "2-current", + path: prefixedPath, + type: "plain", + datatype: "plain", + ctime: 10, + mtime: 20, + size: 20, + children: [], + eden: {}, + deleted: false, + ...overrides, + } as unknown as MetaEntry; +} + +function loadedEntry(entry = metadata()): LoadedEntry { + return { + ...entry, + data: '{"value":"database"}', + } as LoadedEntry; +} + +function storageStat(mtime = 20): UXStat { + return { + ctime: 11, + mtime, + size: 20, + type: "file", + }; +} + +function createDependencies(entry = metadata()) { + const events: string[] = []; + const serialiseFileOperation = vi.fn(async (_key: string, operation: () => Promise) => { + events.push("lock:start"); + try { + return await operation(); + } finally { + events.push("lock:end"); + } + }); + const log = vi.fn(); + const isIgnoredByIgnoreFile = vi.fn(async () => false); + const loadDatabaseMetadata = vi.fn(async () => entry as MetaEntry | false); + const loadLiveRevision = vi.fn(async (_path: FilePathWithPrefix, revision: string) => + revision === entry._rev ? (entry as MetaEntry) : false + ); + const loadDatabaseEntry = vi.fn(async () => loadedEntry(entry) as LoadedEntry | false); + const statStorageFile = vi.fn(async () => storageStat() as UXStat | null); + const writeStorageFile = vi.fn(async () => { + events.push("storage:write"); + return storageStat(21) as UXStat | false; + }); + const deleteStorageFile = vi.fn(async (): Promise => { + events.push("storage:delete"); + return "OK" as const; + }); + const getLastProcessedDatabaseKey = vi.fn(() => undefined as string | undefined); + const getLastProcessedFileMTime = vi.fn(() => 0); + const updateLastProcessed = vi.fn(() => events.push("state:file")); + const updateLastProcessedDatabase = vi.fn(() => events.push("state:database")); + const updateLastProcessedFile = vi.fn(() => events.push("state:storage")); + const updateLastProcessedDeletion = vi.fn(() => events.push("state:deletion")); + const processedState = { + databaseStateKey: toHiddenFileSyncDatabaseStateKey, + getLastProcessedDatabaseKey, + getLastProcessedFileMTime, + updateLastProcessed, + updateLastProcessedDatabase, + updateLastProcessedFile, + updateLastProcessedDeletion, + }; + const queueNotification = vi.fn(() => events.push("notification")); + const dependencies = { + serialiseFileOperation, + isIgnoredByIgnoreFile, + loadDatabaseMetadata, + loadLiveRevision, + loadDatabaseEntry, + statStorageFile, + writeStorageFile, + deleteStorageFile, + processedState, + queueNotification, + log, + } as HiddenFileSyncDatabaseExtractionDependencies; + return { + deleteStorageFile, + dependencies, + entry, + events, + getLastProcessedDatabaseKey, + getLastProcessedFileMTime, + isIgnoredByIgnoreFile, + loadDatabaseEntry, + loadDatabaseMetadata, + loadLiveRevision, + log, + queueNotification, + serialiseFileOperation, + statStorageFile, + updateLastProcessed, + updateLastProcessedDatabase, + updateLastProcessedDeletion, + updateLastProcessedFile, + writeStorageFile, + }; +} + +describe("hidden-file database-to-storage admission", () => { + it("returns undefined for an ignored path without taking the file lock", async () => { + const fixture = createDependencies(); + fixture.isIgnoredByIgnoreFile.mockResolvedValue(true); + + await expect(extractHiddenFileFromDatabase(fixture.dependencies, path)).resolves.toBeUndefined(); + + expect(fixture.serialiseFileOperation).not.toHaveBeenCalled(); + expect(fixture.loadDatabaseMetadata).not.toHaveBeenCalled(); + }); + + it("propagates ignore-policy errors before taking the guarded path", async () => { + const fixture = createDependencies(); + const error = new Error("ignore policy unavailable"); + fixture.isIgnoredByIgnoreFile.mockRejectedValue(error); + + await expect(extractHiddenFileFromDatabase(fixture.dependencies, path)).rejects.toBe(error); + + expect(fixture.serialiseFileOperation).not.toHaveBeenCalled(); + expect(fixture.log).not.toHaveBeenCalled(); + }); + + it("prevents a conflicted entry from reaching storage", async () => { + const entry = metadata({ _conflicts: ["2-conflict"] }); + const fixture = createDependencies(entry); + + await expect(extractHiddenFileFromDatabase(fixture.dependencies, path, { force: true })).resolves.toBe(false); + + expect(fixture.loadDatabaseEntry).not.toHaveBeenCalled(); + expect(fixture.writeStorageFile).not.toHaveBeenCalled(); + expect(fixture.log).toHaveBeenCalledWith( + `Hidden file ${path} has conflicted revisions, to keep in safe, writing to storage has been prevented`, + LOG_LEVEL_INFO, + undefined + ); + }); +}); + +describe("hidden-file database-to-storage processed-state policy", () => { + it("skips a previously processed revision without settling state again", async () => { + const fixture = createDependencies(); + fixture.getLastProcessedDatabaseKey.mockReturnValue(toHiddenFileSyncDatabaseStateKey(fixture.entry)); + + await expect(extractHiddenFileFromDatabase(fixture.dependencies, path)).resolves.toBeUndefined(); + + expect(fixture.loadDatabaseEntry).not.toHaveBeenCalled(); + expect(fixture.updateLastProcessed).not.toHaveBeenCalled(); + expect(fixture.log).toHaveBeenCalledWith( + `STORAGE <-- DB: ${path}: skipped (hidden, overwrite) (Previously processed)`, + undefined, + undefined + ); + }); + + it("allows force to bypass the previously processed revision", async () => { + const fixture = createDependencies(); + fixture.getLastProcessedDatabaseKey.mockReturnValue(toHiddenFileSyncDatabaseStateKey(fixture.entry)); + + await expect(extractHiddenFileFromDatabase(fixture.dependencies, path, { force: true })).resolves.toBe(true); + + expect(fixture.writeStorageFile).toHaveBeenCalledWith(path, loadedEntry(fixture.entry), true); + }); + + it("settles both sides when onlyNew declines an equally old database entry", async () => { + const fixture = createDependencies(); + + await expect( + extractHiddenFileFromDatabase(fixture.dependencies, path, { + metaEntry: fixture.entry, + preventDoubleProcess: false, + onlyNew: true, + }) + ).resolves.toBeUndefined(); + + expect(fixture.loadDatabaseMetadata).not.toHaveBeenCalled(); + expect(fixture.loadDatabaseEntry).not.toHaveBeenCalled(); + expect(fixture.updateLastProcessedDatabase).toHaveBeenCalledWith(path, fixture.entry); + expect(fixture.updateLastProcessedFile).toHaveBeenCalledWith(path, storageStat()); + expect(fixture.events).toEqual(["lock:start", "state:database", "state:storage", "lock:end"]); + }); + + it("uses the last known mtime when onlyNew sees a zero storage mtime", async () => { + const fixture = createDependencies(metadata({ mtime: 30 })); + fixture.statStorageFile.mockResolvedValue(storageStat(0)); + fixture.getLastProcessedFileMTime.mockReturnValue(40); + + await expect( + extractHiddenFileFromDatabase(fixture.dependencies, path, { onlyNew: true }) + ).resolves.toBeUndefined(); + + expect(fixture.getLastProcessedFileMTime).toHaveBeenCalledWith(path); + expect(fixture.writeStorageFile).not.toHaveBeenCalled(); + }); +}); + +describe("hidden-file database-to-storage application", () => { + it("settles state and queues a notification inside the file lock after a successful write", async () => { + const fixture = createDependencies(); + + await expect(extractHiddenFileFromDatabase(fixture.dependencies, path)).resolves.toBe(true); + + expect(fixture.updateLastProcessed).toHaveBeenCalledWith(path, fixture.entry, storageStat(21)); + expect(fixture.queueNotification).toHaveBeenCalledWith(path); + expect(fixture.serialiseFileOperation).toHaveBeenCalledWith(`file-${prefixedPath}`, expect.any(Function)); + expect(fixture.events).toEqual(["lock:start", "storage:write", "state:file", "notification", "lock:end"]); + }); + + it("settles and notifies when the storage writer reports unchanged content with its existing stat", async () => { + const fixture = createDependencies(); + fixture.writeStorageFile.mockResolvedValue(storageStat()); + + await expect(extractHiddenFileFromDatabase(fixture.dependencies, path)).resolves.toBe(true); + + expect(fixture.updateLastProcessed).toHaveBeenCalledWith(path, fixture.entry, storageStat()); + expect(fixture.queueNotification).toHaveBeenCalledWith(path); + }); + + it("returns false without settlement when the storage writer fails", async () => { + const fixture = createDependencies(); + fixture.writeStorageFile.mockResolvedValue(false); + + await expect(extractHiddenFileFromDatabase(fixture.dependencies, path)).resolves.toBe(false); + + expect(fixture.updateLastProcessed).not.toHaveBeenCalled(); + expect(fixture.queueNotification).not.toHaveBeenCalled(); + }); + + it("records a successful storage deletion through deletion settlement", async () => { + const entry = metadata({ deleted: true }); + const fixture = createDependencies(entry); + + await expect(extractHiddenFileFromDatabase(fixture.dependencies, path)).resolves.toBe(true); + + expect(fixture.updateLastProcessedDeletion).toHaveBeenCalledWith(path, entry); + expect(fixture.updateLastProcessedDatabase).not.toHaveBeenCalled(); + expect(fixture.events).toEqual(["lock:start", "storage:delete", "state:deletion", "lock:end"]); + }); + + it("marks an already absent deleted file as database-processed only", async () => { + const entry = metadata({ deleted: true }); + const fixture = createDependencies(entry); + fixture.deleteStorageFile.mockImplementation(async () => { + fixture.events.push("storage:delete"); + return "ALREADY"; + }); + + await expect(extractHiddenFileFromDatabase(fixture.dependencies, path)).resolves.toBe(true); + + expect(fixture.updateLastProcessedDatabase).toHaveBeenCalledWith(path, entry); + expect(fixture.updateLastProcessedDeletion).not.toHaveBeenCalled(); + expect(fixture.events).toEqual(["lock:start", "storage:delete", "state:database", "lock:end"]); + }); + + it("returns false without settling state when a storage deletion fails", async () => { + const fixture = createDependencies(metadata({ _deleted: true })); + fixture.deleteStorageFile.mockResolvedValue(false); + + await expect(extractHiddenFileFromDatabase(fixture.dependencies, path)).resolves.toBe(false); + + expect(fixture.updateLastProcessedDeletion).not.toHaveBeenCalled(); + expect(fixture.updateLastProcessedDatabase).not.toHaveBeenCalled(); + }); + + it("turns a database content-read failure into false and reports the inherited write diagnostic", async () => { + const fixture = createDependencies(); + const error = new Error("content unavailable"); + fixture.loadDatabaseEntry.mockRejectedValue(error); + + await expect(extractHiddenFileFromDatabase(fixture.dependencies, path, { force: true })).resolves.toBe(false); + + expect(fixture.log).toHaveBeenNthCalledWith( + 1, + `STORAGE <-- DB: ${path}: written (hidden, overwrite, force) Failed`, + undefined, + undefined + ); + expect(fixture.log).toHaveBeenNthCalledWith(2, error, LOG_LEVEL_VERBOSE, undefined); + }); +}); + +describe("selected-revision hidden-file database-to-storage application", () => { + it("applies a selected live revision without reading ordinary Metadata", async () => { + const fixture = createDependencies(); + + await expect( + extractHiddenFileRevisionFromDatabase(fixture.dependencies, path, fixture.entry._rev!, true) + ).resolves.toBe(true); + + expect(fixture.loadLiveRevision).toHaveBeenCalledWith(prefixedPath, fixture.entry._rev); + expect(fixture.loadDatabaseMetadata).not.toHaveBeenCalled(); + expect(fixture.writeStorageFile).toHaveBeenCalledWith(path, loadedEntry(fixture.entry), true); + }); + + it("can apply a selected live branch while ordinary Metadata reports a conflict", async () => { + const selected = metadata({ _rev: "2-selected", _conflicts: undefined }); + const fixture = createDependencies(metadata({ _rev: "3-winner", _conflicts: [selected._rev!] })); + fixture.loadLiveRevision.mockResolvedValue(selected); + fixture.loadDatabaseEntry.mockResolvedValue(loadedEntry(selected)); + + await expect( + extractHiddenFileRevisionFromDatabase(fixture.dependencies, path, selected._rev!, true) + ).resolves.toBe(true); + + expect(fixture.loadDatabaseMetadata).not.toHaveBeenCalled(); + expect(fixture.writeStorageFile).toHaveBeenCalledWith(path, loadedEntry(selected), true); + }); + + it("returns false when the selected revision ceased to be live", async () => { + const fixture = createDependencies(); + fixture.loadLiveRevision.mockResolvedValue(false); + + await expect(extractHiddenFileRevisionFromDatabase(fixture.dependencies, path, "2-stale", true)).resolves.toBe( + false + ); + + expect(fixture.loadDatabaseEntry).not.toHaveBeenCalled(); + expect(fixture.writeStorageFile).not.toHaveBeenCalled(); + }); + + it("exposes frozen operations with the exact-revision Boolean contract", async () => { + const fixture = createDependencies(); + fixture.isIgnoredByIgnoreFile.mockResolvedValue(true); + const operations = createHiddenFileSyncDatabaseExtractionOperations(fixture.dependencies); + + expect(Object.isFrozen(operations)).toBe(true); + await expect(operations.extractRevision(path, fixture.entry._rev!)).resolves.toBe(false); + }); +}); diff --git a/src/features/HiddenFileSync/hiddenFileSyncDatabaseLoaders.ts b/src/features/HiddenFileSync/hiddenFileSyncDatabaseLoaders.ts new file mode 100644 index 00000000..45034e55 --- /dev/null +++ b/src/features/HiddenFileSync/hiddenFileSyncDatabaseLoaders.ts @@ -0,0 +1,111 @@ +import { + LOG_LEVEL_NOTICE, + LOG_LEVEL_VERBOSE, + type FilePath, + type FilePathWithPrefix, + type LoadedEntry, + type LOG_LEVEL, + type MetaEntry, +} from "@vrtmrz/livesync-commonlib/compat/common/types"; +import type { DatabaseFileAccess } from "@vrtmrz/livesync-commonlib/compat/interfaces/DatabaseFileAccess"; +import type { LiveSyncLocalDB } from "@vrtmrz/livesync-commonlib/compat/pouchdb/LiveSyncLocalDB"; +import type { IPathService } from "@vrtmrz/livesync-commonlib/compat/services/base/IService"; +import type { LogFunction } from "@vrtmrz/livesync-commonlib/compat/services/lib/logUtils"; +import { addPrefix, stripAllPrefixes } from "@vrtmrz/livesync-commonlib/compat/string_and_binary/path"; + +import { ICHeader } from "@/common/types.ts"; + +type HiddenFileSyncLogDependency = { + log: LogFunction; +}; + +type HiddenFileSyncDatabaseMethods = { + getLocalDatabase(): Pick; +}; + +type HiddenFileSyncDatabaseFileAccessMethods = { + databaseFileAccess: Pick; +}; + +type HiddenFileSyncPathMethods = { + path: Pick; +}; + +export type HiddenFileSyncBaseEntryLoaderDependencies = HiddenFileSyncDatabaseMethods<"getDBEntry" | "getDBEntryMeta"> & + HiddenFileSyncPathMethods<"path2id"> & + HiddenFileSyncLogDependency; + +export type HiddenFileSyncLiveRevisionLoaderDependencies = HiddenFileSyncDatabaseFileAccessMethods< + "fetchEntryMeta" | "getConflictedRevs" +> & + HiddenFileSyncLogDependency; + +function log(dependencies: HiddenFileSyncLogDependency, message: unknown, level?: LOG_LEVEL, key?: string): void { + dependencies.log(message, level, key); +} + +export async function loadHiddenFileSyncBaseEntry( + dependencies: HiddenFileSyncBaseEntryLoaderDependencies, + file: FilePath, + includeContent = true +): Promise { + const prefixedFileName = addPrefix(file, ICHeader); + // Compatibility question: path-to-ID conversion is performed even when an + // entry already exists, and it sits outside the guarded database lookup. + // Preserve this ordering and error propagation until reviewed separately. + const id = await dependencies.path.path2id(prefixedFileName, ICHeader); + try { + const old = includeContent + ? await dependencies.getLocalDatabase().getDBEntry(prefixedFileName, undefined, false, true) + : await dependencies.getLocalDatabase().getDBEntryMeta(prefixedFileName, { conflicts: true }, true); + if (old !== false) { + return old; + } + // Compatibility question: getDBEntry() also returns false when content + // or Chunks cannot be read. The inherited behaviour treats that exactly + // like absence and synthesises a fresh base entry. + return { + _id: id, + data: [], + path: prefixedFileName, + mtime: 0, + ctime: 0, + datatype: "newnote", + children: [], + size: 0, + deleted: false, + type: "newnote", + eden: {}, + }; + } catch (error) { + log(dependencies, "Getting base save data failed"); + log(dependencies, error, LOG_LEVEL_VERBOSE); + return false; + } +} + +export async function loadLiveHiddenFileSyncRevision( + dependencies: HiddenFileSyncLiveRevisionLoaderDependencies, + prefixedFileName: FilePathWithPrefix, + revision: string +): Promise { + const [selected, current, conflicts] = await Promise.all([ + dependencies.databaseFileAccess.fetchEntryMeta(prefixedFileName, revision, true), + dependencies.databaseFileAccess.fetchEntryMeta(prefixedFileName, undefined, true), + dependencies.databaseFileAccess.getConflictedRevs(prefixedFileName), + ]); + const liveRevisions = new Set([...(current && current._rev ? [current._rev] : []), ...conflicts]); + if (!selected || selected._rev !== revision || !liveRevisions.has(revision)) { + // Compatibility: missing, mismatched, and stale selections share the + // same user-facing diagnostic and false result. + log( + dependencies, + `Could not use hidden-file revision ${revision} of ${stripAllPrefixes(prefixedFileName)}; the selected revision is no longer live`, + LOG_LEVEL_NOTICE + ); + return false; + } + // Compatibility: liveness is revision-tree membership only. A deleted + // Metadata leaf remains selectable while its revision is still live. + return selected; +} diff --git a/src/features/HiddenFileSync/hiddenFileSyncDatabaseLoaders.unit.spec.ts b/src/features/HiddenFileSync/hiddenFileSyncDatabaseLoaders.unit.spec.ts new file mode 100644 index 00000000..6347e5dd --- /dev/null +++ b/src/features/HiddenFileSync/hiddenFileSyncDatabaseLoaders.unit.spec.ts @@ -0,0 +1,202 @@ +import { describe, expect, it, vi } from "vitest"; +import { + LOG_LEVEL_NOTICE, + LOG_LEVEL_VERBOSE, + type DocumentID, + type FilePath, + type FilePathWithPrefix, + type LoadedEntry, + type MetaEntry, +} from "@vrtmrz/livesync-commonlib/compat/common/types"; + +import { loadHiddenFileSyncBaseEntry, loadLiveHiddenFileSyncRevision } from "./hiddenFileSyncDatabaseLoaders.ts"; + +const path = ".obsidian/app.json" as FilePath; +const prefixedPath = `i:${path}` as FilePathWithPrefix; +const id = "hidden-entry-id" as DocumentID; + +function loadedEntry(): LoadedEntry { + return { + _id: id, + _rev: "1-a", + path: prefixedPath, + type: "plain", + datatype: "plain", + data: "content", + ctime: 10, + mtime: 20, + size: 7, + children: [], + eden: {}, + deleted: false, + } as LoadedEntry; +} + +function createBaseEntryDependencies() { + const getDBEntry = vi.fn(); + const getDBEntryMeta = vi.fn(); + const path2id = vi.fn(async () => id); + const log = vi.fn(); + const dependencies = { + getLocalDatabase: () => ({ getDBEntry, getDBEntryMeta }) as never, + path: { path2id } as never, + log, + }; + return { dependencies, getDBEntry, getDBEntryMeta, log, path2id }; +} + +function metaEntry(revision: string): MetaEntry { + return { + ...loadedEntry(), + _rev: revision, + data: undefined, + } as unknown as MetaEntry; +} + +function createLiveRevisionDependencies() { + const selected = metaEntry("2-selected"); + const current = metaEntry("3-current"); + const fetchEntryMeta = vi.fn(async (_path: unknown, revision?: string) => { + if (revision === undefined || revision === current._rev) return current; + if (revision === selected._rev) return selected; + return false; + }); + const getConflictedRevs = vi.fn(async () => [selected._rev!]); + const log = vi.fn(); + const dependencies = { + databaseFileAccess: { fetchEntryMeta, getConflictedRevs } as never, + log, + }; + return { current, dependencies, fetchEntryMeta, getConflictedRevs, log, selected }; +} + +describe("Hidden File Sync base-entry loader", () => { + it("synthesises a new empty base whenever the content lookup reports false", async () => { + const { dependencies, getDBEntry, path2id } = createBaseEntryDependencies(); + getDBEntry.mockResolvedValue(false); + + await expect(loadHiddenFileSyncBaseEntry(dependencies, path)).resolves.toEqual({ + _id: id, + data: [], + path: prefixedPath, + mtime: 0, + ctime: 0, + datatype: "newnote", + children: [], + size: 0, + deleted: false, + type: "newnote", + eden: {}, + }); + expect(path2id).toHaveBeenCalledWith(prefixedPath, "i:"); + expect(getDBEntry).toHaveBeenCalledWith(prefixedPath, undefined, false, true); + }); + + it("returns an existing content entry unchanged", async () => { + const { dependencies, getDBEntry, path2id } = createBaseEntryDependencies(); + const existing = loadedEntry(); + getDBEntry.mockResolvedValue(existing); + + await expect(loadHiddenFileSyncBaseEntry(dependencies, path, true)).resolves.toBe(existing); + expect(path2id).toHaveBeenCalledWith(prefixedPath, "i:"); + }); + + it("uses the conflict-aware metadata lookup when content is not requested", async () => { + const { dependencies, getDBEntry, getDBEntryMeta } = createBaseEntryDependencies(); + const existing = loadedEntry(); + getDBEntryMeta.mockResolvedValue(existing); + + await expect(loadHiddenFileSyncBaseEntry(dependencies, path, false)).resolves.toBe(existing); + expect(getDBEntry).not.toHaveBeenCalled(); + expect(getDBEntryMeta).toHaveBeenCalledWith(prefixedPath, { conflicts: true }, true); + }); + + it("turns a database lookup failure into a logged false result", async () => { + const { dependencies, getDBEntry, log } = createBaseEntryDependencies(); + const error = new Error("database unavailable"); + getDBEntry.mockRejectedValue(error); + + await expect(loadHiddenFileSyncBaseEntry(dependencies, path)).resolves.toBe(false); + expect(log).toHaveBeenNthCalledWith(1, "Getting base save data failed", undefined, undefined); + expect(log).toHaveBeenNthCalledWith(2, error, LOG_LEVEL_VERBOSE, undefined); + }); + + it("propagates path-to-ID failures which occur before the guarded lookup", async () => { + const { dependencies, getDBEntry, log, path2id } = createBaseEntryDependencies(); + const error = new Error("ID conversion failed"); + path2id.mockRejectedValue(error); + + await expect(loadHiddenFileSyncBaseEntry(dependencies, path)).rejects.toBe(error); + expect(getDBEntry).not.toHaveBeenCalled(); + expect(log).not.toHaveBeenCalled(); + }); +}); + +describe("Hidden File Sync live-revision loader", () => { + it("accepts the current winning revision", async () => { + const { current, dependencies, fetchEntryMeta, getConflictedRevs } = createLiveRevisionDependencies(); + + await expect(loadLiveHiddenFileSyncRevision(dependencies, prefixedPath, current._rev!)).resolves.toBe(current); + expect(fetchEntryMeta).toHaveBeenNthCalledWith(1, prefixedPath, current._rev, true); + expect(fetchEntryMeta).toHaveBeenNthCalledWith(2, prefixedPath, undefined, true); + expect(getConflictedRevs).toHaveBeenCalledWith(prefixedPath); + }); + + it("accepts a selected conflict leaf while it remains live", async () => { + const { dependencies, selected } = createLiveRevisionDependencies(); + + await expect(loadLiveHiddenFileSyncRevision(dependencies, prefixedPath, selected._rev!)).resolves.toBe( + selected + ); + }); + + it("accepts a live conflict even when there is no current winner", async () => { + const { dependencies, fetchEntryMeta, selected } = createLiveRevisionDependencies(); + fetchEntryMeta.mockImplementation(async (_path: unknown, revision?: string) => + revision === selected._rev ? selected : false + ); + + await expect(loadLiveHiddenFileSyncRevision(dependencies, prefixedPath, selected._rev!)).resolves.toBe( + selected + ); + }); + + it("accepts deleted metadata while its revision remains live", async () => { + const { dependencies, fetchEntryMeta, selected } = createLiveRevisionDependencies(); + const deleted = { ...selected, deleted: true } as MetaEntry; + fetchEntryMeta.mockImplementation(async (_path: unknown, revision?: string) => + revision === undefined ? metaEntry("3-current") : deleted + ); + + await expect(loadLiveHiddenFileSyncRevision(dependencies, prefixedPath, selected._rev!)).resolves.toBe(deleted); + }); + + it("rejects a selected revision which is no longer current or conflicted", async () => { + const { dependencies, getConflictedRevs, log, selected } = createLiveRevisionDependencies(); + getConflictedRevs.mockResolvedValue([]); + + await expect(loadLiveHiddenFileSyncRevision(dependencies, prefixedPath, selected._rev!)).resolves.toBe(false); + expect(log).toHaveBeenCalledWith( + `Could not use hidden-file revision ${selected._rev} of ${path}; the selected revision is no longer live`, + LOG_LEVEL_NOTICE, + undefined + ); + }); + + it("rejects a lookup result whose revision does not match the selection", async () => { + const { dependencies, fetchEntryMeta, log, selected } = createLiveRevisionDependencies(); + fetchEntryMeta.mockResolvedValue(metaEntry("2-other")); + + await expect(loadLiveHiddenFileSyncRevision(dependencies, prefixedPath, selected._rev!)).resolves.toBe(false); + expect(log).toHaveBeenCalledOnce(); + }); + + it("propagates database-file-access failures", async () => { + const { dependencies, fetchEntryMeta, log, selected } = createLiveRevisionDependencies(); + const error = new Error("revision lookup failed"); + fetchEntryMeta.mockRejectedValue(error); + + await expect(loadLiveHiddenFileSyncRevision(dependencies, prefixedPath, selected._rev!)).rejects.toBe(error); + expect(log).not.toHaveBeenCalled(); + }); +}); diff --git a/src/features/HiddenFileSync/hiddenFileSyncDatabaseWriteOperations.ts b/src/features/HiddenFileSync/hiddenFileSyncDatabaseWriteOperations.ts new file mode 100644 index 00000000..74f99da3 --- /dev/null +++ b/src/features/HiddenFileSync/hiddenFileSyncDatabaseWriteOperations.ts @@ -0,0 +1,337 @@ +import { + LOG_LEVEL_NOTICE, + LOG_LEVEL_VERBOSE, + type FilePath, + type FilePathWithPrefix, + type LoadedEntry, + type LOG_LEVEL, + type MetaEntry, + type SavingEntry, + type UXFileInfo, +} from "@vrtmrz/livesync-commonlib/compat/common/types"; +import { isDocContentSame, readAsBlob } from "@vrtmrz/livesync-commonlib/compat/common/utils"; +import type { LogFunction } from "@vrtmrz/livesync-commonlib/compat/services/lib/logUtils"; +import { addPrefix, stripAllPrefixes } from "@vrtmrz/livesync-commonlib/compat/string_and_binary/path"; + +import type { InternalFileInfo } from "@/common/types.ts"; +import { ICHeader } from "@/common/types.ts"; +import { + serialiseHiddenFileOperation, + type HiddenFileSyncFileSerialisationDependencies, + type HiddenFileSyncFileSerialiser, +} from "./hiddenFileSyncFileOperations.ts"; + +export type { HiddenFileSyncFileSerialiser } from "./hiddenFileSyncFileOperations.ts"; + +type HiddenFileSyncDatabaseWriteResponse = { + readonly ok: boolean; + readonly rev: string; +}; + +export type HiddenFileSyncDatabaseWriteProcessedState = { + updateLastProcessed(path: FilePath, databaseEntry: MetaEntry | LoadedEntry, storageFile: UXFileInfo["stat"]): void; + updateLastProcessedDeletion(path: FilePath, databaseEntry: MetaEntry | LoadedEntry | false): void; +}; + +export type HiddenFileSyncDatabaseWriteDependencies = { + serialiseFileOperation: HiddenFileSyncFileSerialiser; + isIgnoredByIgnoreFile(path: string): Promise; + readFileWithInfo(path: FilePath): Promise; + loadBaseEntry(path: FilePath): Promise; + loadBaseMetadata(path: FilePath): Promise; + loadLiveRevision(path: FilePathWithPrefix, revision: string): Promise; + fetchEntryFromMeta(meta: MetaEntry, waitForReady: boolean, skipCheck: boolean): Promise; + storeWithBaseRevision(file: UXFileInfo, baseRevision: string, skipCheck: boolean): Promise; + putDatabaseEntry(entry: SavingEntry): Promise; + putRaw(entry: LoadedEntry): Promise; + removeRevision(id: LoadedEntry["_id"], revision: string): Promise; + processedState: HiddenFileSyncDatabaseWriteProcessedState; + now(): number; + log: LogFunction; +}; + +type HiddenFileSyncDatabaseWriteCommonDependencies = Pick< + HiddenFileSyncDatabaseWriteDependencies, + "isIgnoredByIgnoreFile" | "log" | "readFileWithInfo" | "serialiseFileOperation" +>; + +export type StoreHiddenFileInDatabaseDependencies = HiddenFileSyncDatabaseWriteCommonDependencies & + Pick; + +export type StoreHiddenFileWithBaseRevisionDependencies = HiddenFileSyncDatabaseWriteCommonDependencies & + Pick< + HiddenFileSyncDatabaseWriteDependencies, + "fetchEntryFromMeta" | "loadLiveRevision" | "storeWithBaseRevision" | "processedState" + >; + +export type DeleteHiddenFileFromDatabaseDependencies = Pick< + HiddenFileSyncDatabaseWriteDependencies, + | "isIgnoredByIgnoreFile" + | "loadBaseMetadata" + | "log" + | "now" + | "putRaw" + | "removeRevision" + | "serialiseFileOperation" + | "processedState" +>; + +export type HiddenFileSyncDatabaseWriteOperations = { + store(file: InternalFileInfo | UXFileInfo, forceWrite?: boolean): Promise; + storeWithBaseRevision( + file: InternalFileInfo | UXFileInfo, + baseRevision: string, + createIfDifferent?: boolean + ): Promise; + delete(path: FilePath, forceWrite?: boolean): Promise; +}; + +type HiddenFileSyncDatabaseWriteGuardDependencies = HiddenFileSyncFileSerialisationDependencies & + Pick; + +function log( + dependencies: Pick, + message: unknown, + level?: LOG_LEVEL, + key?: string +) { + dependencies.log(message, level, key); +} + +async function runGuardedDatabaseWrite( + dependencies: HiddenFileSyncDatabaseWriteGuardDependencies, + prefixedFileName: FilePathWithPrefix, + failureMessage: string, + operation: () => Promise +): Promise { + return await serialiseHiddenFileOperation(dependencies, prefixedFileName, async () => { + try { + return await operation(); + } catch (error) { + log(dependencies, failureMessage); + log(dependencies, error, LOG_LEVEL_VERBOSE); + return false; + } + }); +} + +async function resolvePresentFileInfo( + dependencies: Pick, + file: InternalFileInfo | UXFileInfo, + storeFilePath: FilePath +): Promise { + const fileInfo = "stat" in file && "body" in file ? file : await dependencies.readFileWithInfo(storeFilePath); + if (fileInfo.deleted) { + throw new Error(`Hidden file:${storeFilePath} is deleted. This should not be occurred.`); + } + return fileInfo; +} + +export async function storeHiddenFileInDatabase( + dependencies: StoreHiddenFileInDatabaseDependencies, + file: InternalFileInfo | UXFileInfo, + forceWrite = false +): Promise { + const storeFilePath = stripAllPrefixes(file.path); + const storageFilePath = file.path; + // Compatibility: all three admission checks sit outside the guarded lock, + // so policy errors propagate. The ordinary path reports an ignored file as + // undefined, while the selected-revision path reports false. + if (await dependencies.isIgnoredByIgnoreFile(storageFilePath)) { + return undefined; + } + const prefixedFileName = addPrefix(storeFilePath, ICHeader); + + return await runGuardedDatabaseWrite( + dependencies, + prefixedFileName, + `STORAGE --> DB:${storageFilePath}: (hidden) Failed`, + async () => { + const fileInfo = await resolvePresentFileInfo(dependencies, file, storeFilePath); + const baseData = await dependencies.loadBaseEntry(storeFilePath); + if (baseData === false) throw new Error("Failed to load base data"); + if (baseData._rev && !forceWrite) { + const isSame = await isDocContentSame(readAsBlob(baseData), fileInfo.body); + if (isSame) { + dependencies.processedState.updateLastProcessed(storeFilePath, baseData, fileInfo.stat); + return undefined; + } + } + const saveData: SavingEntry = { + ...baseData, + data: fileInfo.body, + mtime: fileInfo.stat.mtime, + size: fileInfo.stat.size, + children: [], + deleted: false, + type: baseData.datatype, + }; + // Compatibility question: ctime comes from the old database base, + // not from the storage stat. A newly synthesised base therefore + // stores ctime 0. Preserve this until cross-device effects are known. + const ret = await dependencies.putDatabaseEntry(saveData); + if (ret && ret.ok) { + saveData._rev = ret.rev; + dependencies.processedState.updateLastProcessed(storeFilePath, saveData, fileInfo.stat); + } + const success = ret && ret.ok; + log(dependencies, `STORAGE --> DB:${storageFilePath}: (hidden) ${success ? "Done" : "Failed"}`); + return success; + } + ); +} + +export async function storeHiddenFileWithBaseRevision( + dependencies: StoreHiddenFileWithBaseRevisionDependencies, + file: InternalFileInfo | UXFileInfo, + baseRevision: string, + createIfDifferent = true +): Promise { + const storeFilePath = stripAllPrefixes(file.path); + const storageFilePath = file.path; + if (await dependencies.isIgnoredByIgnoreFile(storageFilePath)) { + return false; + } + const prefixedFileName = addPrefix(storeFilePath, ICHeader); + + return await runGuardedDatabaseWrite( + dependencies, + prefixedFileName, + `STORAGE --> DB:${storageFilePath}: (hidden, selected branch) Failed`, + async () => { + // The live check intentionally precedes the storage read. It avoids + // work for a stale selection, but does not make the later write atomic. + const baseData = await dependencies.loadLiveRevision(prefixedFileName, baseRevision); + if (baseData === false) { + return false; + } + const fileInfo = await resolvePresentFileInfo(dependencies, file, storeFilePath); + if (!baseData.deleted && !baseData._deleted) { + const loadedBase = await dependencies.fetchEntryFromMeta(baseData, true, true); + if (loadedBase && (await isDocContentSame(readAsBlob(loadedBase), fileInfo.body))) { + dependencies.processedState.updateLastProcessed(storeFilePath, baseData, fileInfo.stat); + return true; + } + } + if (!createIfDifferent) { + log( + dependencies, + `Could not mark hidden file ${storeFilePath} as revision ${baseRevision}; the storage content differs`, + LOG_LEVEL_NOTICE + ); + return false; + } + + const storedRevision = await dependencies.storeWithBaseRevision( + { + ...fileInfo, + path: storeFilePath, + name: fileInfo.name || storeFilePath.split("/").pop() || "", + isInternal: true, + }, + baseRevision, + true + ); + if (storedRevision === false) { + return false; + } + // Compatibility question: spreading a selected PouchDB tombstone + // retains `_deleted: true` in the processed-state entry even though + // `deleted` is reset. The stored child itself is created separately. + dependencies.processedState.updateLastProcessed( + storeFilePath, + { + ...baseData, + _rev: storedRevision, + path: prefixedFileName, + ctime: fileInfo.stat.ctime, + mtime: fileInfo.stat.mtime, + size: fileInfo.stat.size, + deleted: false, + }, + fileInfo.stat + ); + log(dependencies, `STORAGE --> DB:${storageFilePath}: (hidden, selected branch) Done`); + return true; + } + ); +} + +export async function deleteHiddenFileFromDatabase( + dependencies: DeleteHiddenFileFromDatabaseDependencies, + filenameSrc: FilePath, + forceWrite = false +): Promise { + const storeFilePath = filenameSrc; + const storageFilePath = filenameSrc; + const displayFileName = filenameSrc; + const prefixedFileName = addPrefix(storeFilePath, ICHeader); + // Compatibility question: the timestamp is captured before the ignore + // check and before waiting for the per-file lock. + const mtime = dependencies.now(); + // Compatibility question: forceWrite is part of the inherited call + // contract, but it has never changed deletion behaviour. + void forceWrite; + if (await dependencies.isIgnoredByIgnoreFile(storageFilePath)) { + return undefined; + } + return await runGuardedDatabaseWrite( + dependencies, + prefixedFileName, + `STORAGE -x> DB: ${displayFileName}: (hidden) Failed`, + async () => { + const baseData = await dependencies.loadBaseMetadata(storeFilePath); + if (baseData === false) throw new Error("Failed to load base data during deleting"); + if (baseData._conflicts !== undefined) { + // Compatibility question: these removals are sequential but not + // atomic. Earlier branches remain removed if a later call fails. + for (const conflictRev of baseData._conflicts) { + await dependencies.removeRevision(baseData._id, conflictRev); + log( + dependencies, + `STORAGE -x> DB: ${displayFileName}: (hidden) conflict removed ${baseData._rev} => ${conflictRev}`, + LOG_LEVEL_VERBOSE + ); + } + } + // Compatibility question: only the domain `deleted` marker is + // checked here. `_deleted: true` alone causes another tombstone write. + if (baseData.deleted) { + log(dependencies, `STORAGE -x> DB: ${displayFileName}: (hidden) already deleted`, LOG_LEVEL_VERBOSE); + dependencies.processedState.updateLastProcessedDeletion(storeFilePath, baseData); + return true; + } + const saveData: LoadedEntry = { + ...baseData, + mtime, + size: 0, + children: [], + deleted: true, + type: baseData.datatype, + }; + // A synthesised base has no revision; the inherited behaviour still + // writes it as a tombstone when the requested path was absent. + const ret = await dependencies.putRaw(saveData); + if (ret && ret.ok) { + log(dependencies, `STORAGE -x> DB: ${displayFileName}: (hidden) Done`); + saveData._rev = ret.rev; + dependencies.processedState.updateLastProcessedDeletion(storeFilePath, saveData); + return true; + } else { + log(dependencies, `STORAGE -x> DB: ${displayFileName}: (hidden) Failed`); + return false; + } + } + ); +} + +export function createHiddenFileSyncDatabaseWriteOperations( + dependencies: HiddenFileSyncDatabaseWriteDependencies +): HiddenFileSyncDatabaseWriteOperations { + return Object.freeze({ + store: async (file, forceWrite) => await storeHiddenFileInDatabase(dependencies, file, forceWrite), + storeWithBaseRevision: async (file, baseRevision, createIfDifferent) => + await storeHiddenFileWithBaseRevision(dependencies, file, baseRevision, createIfDifferent), + delete: async (path, forceWrite) => await deleteHiddenFileFromDatabase(dependencies, path, forceWrite), + }); +} diff --git a/src/features/HiddenFileSync/hiddenFileSyncDatabaseWriteOperations.unit.spec.ts b/src/features/HiddenFileSync/hiddenFileSyncDatabaseWriteOperations.unit.spec.ts new file mode 100644 index 00000000..f7b96c4c --- /dev/null +++ b/src/features/HiddenFileSync/hiddenFileSyncDatabaseWriteOperations.unit.spec.ts @@ -0,0 +1,372 @@ +import { describe, expect, it, vi } from "vitest"; +import { + LOG_LEVEL_NOTICE, + LOG_LEVEL_VERBOSE, + type DocumentID, + type FilePath, + type FilePathWithPrefix, + type LoadedEntry, + type MetaEntry, + type UXFileInfo, +} from "@vrtmrz/livesync-commonlib/compat/common/types"; + +import { + deleteHiddenFileFromDatabase, + storeHiddenFileInDatabase, + storeHiddenFileWithBaseRevision, + type HiddenFileSyncDatabaseWriteDependencies, +} from "./hiddenFileSyncDatabaseWriteOperations.ts"; + +const path = ".obsidian/plugins/example/data.json" as FilePath; +const prefixedPath = `i:${path}` as FilePathWithPrefix; +const id = "hidden-entry-id" as DocumentID; + +function fileInfo(content = '{"value":"vault"}'): UXFileInfo { + return { + path, + name: "data.json", + isInternal: true, + body: new Blob([content]), + stat: { + ctime: 41, + mtime: 42, + size: content.length, + type: "file", + }, + deleted: false, + } as UXFileInfo; +} + +function loadedEntry(overrides: Partial = {}): LoadedEntry { + return { + _id: id, + _rev: "2-current", + path: prefixedPath, + type: "plain", + datatype: "plain", + data: '{"value":"database"}', + ctime: 10, + mtime: 20, + size: 20, + children: [], + eden: {}, + deleted: false, + ...overrides, + } as LoadedEntry; +} + +function createDependencies(base = loadedEntry()) { + const events: string[] = []; + const serialiseFileOperation = vi.fn(async (_key: string, operation: () => Promise) => { + events.push("lock:start"); + try { + return await operation(); + } finally { + events.push("lock:end"); + } + }); + const isIgnoredByIgnoreFile = vi.fn(async () => false); + const readFileWithInfo = vi.fn(async () => fileInfo()); + const loadBaseEntry = vi.fn(async () => base as LoadedEntry | false); + const loadBaseMetadata = vi.fn(async () => base as LoadedEntry | false); + const loadLiveRevision = vi.fn(async (_path: FilePathWithPrefix, revision: string) => + revision === base._rev ? (base as MetaEntry) : false + ); + const fetchEntryFromMeta = vi.fn(async () => base as LoadedEntry | false); + const storeWithBaseRevision = vi.fn(async () => "3-selected-child" as string | false); + const putDatabaseEntry = vi.fn(async (_entry: unknown) => ({ ok: true, id, rev: "3-written" })); + const putRaw = vi.fn(async (_entry: LoadedEntry) => ({ ok: true, id, rev: "3-deleted" })); + const removeRevision = vi.fn(async () => true); + const updateLastProcessed = vi.fn(() => events.push("state:file")); + const updateLastProcessedDeletion = vi.fn(() => events.push("state:deletion")); + const processedState = { + updateLastProcessed, + updateLastProcessedDeletion, + }; + const now = vi.fn(() => 1_000); + const log = vi.fn(); + const dependencies = { + serialiseFileOperation, + isIgnoredByIgnoreFile, + readFileWithInfo, + loadBaseEntry, + loadBaseMetadata, + loadLiveRevision, + fetchEntryFromMeta, + storeWithBaseRevision, + putDatabaseEntry, + putRaw, + removeRevision, + processedState, + now, + log, + } as unknown as HiddenFileSyncDatabaseWriteDependencies; + return { + base, + dependencies, + events, + fetchEntryFromMeta, + isIgnoredByIgnoreFile, + loadBaseEntry, + loadBaseMetadata, + loadLiveRevision, + log, + now, + putDatabaseEntry, + putRaw, + readFileWithInfo, + removeRevision, + serialiseFileOperation, + storeWithBaseRevision, + updateLastProcessed, + updateLastProcessedDeletion, + }; +} + +describe("ordinary hidden-file database writes", () => { + it("keeps the synthetic base ctime and settles the new revision inside the file lock", async () => { + const base = loadedEntry({ + _rev: undefined, + ctime: 0, + data: [], + datatype: "newnote", + type: "newnote", + }); + const fixture = createDependencies(base); + const file = fileInfo(); + + await expect(storeHiddenFileInDatabase(fixture.dependencies, file)).resolves.toBe(true); + + expect(fixture.putDatabaseEntry).toHaveBeenCalledWith( + expect.objectContaining({ ctime: 0, mtime: file.stat.mtime, data: file.body }) + ); + expect(fixture.updateLastProcessed).toHaveBeenCalledWith( + path, + expect.objectContaining({ _rev: "3-written", ctime: 0 }), + file.stat + ); + expect(fixture.serialiseFileOperation).toHaveBeenCalledWith(`file-${prefixedPath}`, expect.any(Function)); + expect(fixture.events).toEqual(["lock:start", "state:file", "lock:end"]); + }); + + it("settles matching content without writing or emitting a transfer log", async () => { + const base = loadedEntry({ data: '{"value":"vault"}' }); + const fixture = createDependencies(base); + const file = fileInfo(); + + await expect(storeHiddenFileInDatabase(fixture.dependencies, file)).resolves.toBeUndefined(); + + expect(fixture.putDatabaseEntry).not.toHaveBeenCalled(); + expect(fixture.updateLastProcessed).toHaveBeenCalledWith(path, base, file.stat); + expect(fixture.log).not.toHaveBeenCalled(); + }); + + it("writes matching content when forceWrite is enabled", async () => { + const fixture = createDependencies(loadedEntry({ data: '{"value":"vault"}' })); + + await expect(storeHiddenFileInDatabase(fixture.dependencies, fileInfo(), true)).resolves.toBe(true); + + expect(fixture.putDatabaseEntry).toHaveBeenCalledOnce(); + }); + + it("returns false and reports both messages when a guarded write fails", async () => { + const fixture = createDependencies(); + const error = new Error("storage read failed"); + fixture.readFileWithInfo.mockRejectedValue(error); + + await expect( + storeHiddenFileInDatabase(fixture.dependencies, { + path, + ctime: 1, + mtime: 2, + size: 3, + }) + ).resolves.toBe(false); + + expect(fixture.log).toHaveBeenNthCalledWith(1, `STORAGE --> DB:${path}: (hidden) Failed`, undefined, undefined); + expect(fixture.log).toHaveBeenNthCalledWith(2, error, LOG_LEVEL_VERBOSE, undefined); + }); +}); + +describe("selected-revision hidden-file database writes", () => { + it("stores the Vault content as a child of a selected live revision", async () => { + const fixture = createDependencies(); + const file = fileInfo(); + + await expect(storeHiddenFileWithBaseRevision(fixture.dependencies, file, fixture.base._rev!)).resolves.toBe( + true + ); + + expect(fixture.storeWithBaseRevision).toHaveBeenCalledWith( + expect.objectContaining({ path, body: file.body, isInternal: true }), + fixture.base._rev, + true + ); + expect(fixture.updateLastProcessed).toHaveBeenCalledWith( + path, + expect.objectContaining({ _rev: "3-selected-child" }), + file.stat + ); + }); + + it("validates liveness before reading storage and refuses a stale revision", async () => { + const fixture = createDependencies(); + fixture.loadLiveRevision.mockResolvedValue(false); + + await expect( + storeHiddenFileWithBaseRevision(fixture.dependencies, { path, ctime: 1, mtime: 2, size: 3 }, "2-stale") + ).resolves.toBe(false); + + expect(fixture.readFileWithInfo).not.toHaveBeenCalled(); + expect(fixture.storeWithBaseRevision).not.toHaveBeenCalled(); + expect(fixture.updateLastProcessed).not.toHaveBeenCalled(); + }); + + it("marks matching content without creating a child", async () => { + const base = loadedEntry({ data: '{"value":"vault"}' }); + const fixture = createDependencies(base); + const file = fileInfo(); + + await expect(storeHiddenFileWithBaseRevision(fixture.dependencies, file, base._rev!, false)).resolves.toBe( + true + ); + + expect(fixture.storeWithBaseRevision).not.toHaveBeenCalled(); + expect(fixture.updateLastProcessed).toHaveBeenCalledWith(path, base, file.stat); + }); + + it("reports differing content without creating a child when requested", async () => { + const fixture = createDependencies(); + + await expect( + storeHiddenFileWithBaseRevision(fixture.dependencies, fileInfo(), fixture.base._rev!, false) + ).resolves.toBe(false); + + expect(fixture.storeWithBaseRevision).not.toHaveBeenCalled(); + expect(fixture.updateLastProcessed).not.toHaveBeenCalled(); + expect(fixture.log).toHaveBeenCalledWith( + `Could not mark hidden file ${path} as revision ${fixture.base._rev}; the storage content differs`, + LOG_LEVEL_NOTICE, + undefined + ); + }); + + it("keeps a selected branch's _deleted marker in the processed-state entry", async () => { + const base = loadedEntry({ deleted: true, _deleted: true }); + const fixture = createDependencies(base); + const file = fileInfo(); + + await expect(storeHiddenFileWithBaseRevision(fixture.dependencies, file, base._rev!)).resolves.toBe(true); + + expect(fixture.fetchEntryFromMeta).not.toHaveBeenCalled(); + expect(fixture.updateLastProcessed).toHaveBeenCalledWith( + path, + expect.objectContaining({ _rev: "3-selected-child", deleted: false, _deleted: true }), + file.stat + ); + }); +}); + +describe("hidden-file database deletions", () => { + it("removes conflicts before accepting an already deleted entry", async () => { + const base = loadedEntry({ deleted: true, _conflicts: ["2-conflict"] }); + const fixture = createDependencies(base); + + await expect(deleteHiddenFileFromDatabase(fixture.dependencies, path, true)).resolves.toBe(true); + + expect(fixture.removeRevision).toHaveBeenCalledWith(id, "2-conflict"); + expect(fixture.putRaw).not.toHaveBeenCalled(); + expect(fixture.updateLastProcessedDeletion).toHaveBeenCalledWith(path, base); + }); + + it("writes a deletion when only the PouchDB _deleted marker is present", async () => { + const base = loadedEntry({ deleted: false, _deleted: true }); + const fixture = createDependencies(base); + + await expect(deleteHiddenFileFromDatabase(fixture.dependencies, path)).resolves.toBe(true); + + expect(fixture.putRaw).toHaveBeenCalledWith(expect.objectContaining({ deleted: true, _deleted: true })); + expect(fixture.updateLastProcessedDeletion).toHaveBeenCalledWith( + path, + expect.objectContaining({ _rev: "3-deleted" }) + ); + }); + + it("writes a tombstone for a synthetic base without a revision", async () => { + const base = loadedEntry({ _rev: undefined, data: [], datatype: "newnote", type: "newnote" }); + const fixture = createDependencies(base); + let submitted: LoadedEntry | undefined; + fixture.putRaw.mockImplementation(async (entry: LoadedEntry) => { + submitted = { ...entry }; + return { ok: true, id, rev: "3-deleted" }; + }); + + await expect(deleteHiddenFileFromDatabase(fixture.dependencies, path)).resolves.toBe(true); + + expect(submitted).toEqual(expect.objectContaining({ _rev: undefined, deleted: true, type: "newnote" })); + }); + + it("keeps earlier conflict removals when a later removal fails", async () => { + const base = loadedEntry({ _conflicts: ["2-first", "2-second"] }); + const fixture = createDependencies(base); + const error = new Error("second removal failed"); + fixture.removeRevision.mockResolvedValueOnce(true).mockRejectedValueOnce(error); + + await expect(deleteHiddenFileFromDatabase(fixture.dependencies, path)).resolves.toBe(false); + + expect(fixture.removeRevision.mock.calls).toEqual([ + [id, "2-first"], + [id, "2-second"], + ]); + expect(fixture.putRaw).not.toHaveBeenCalled(); + expect(fixture.updateLastProcessedDeletion).not.toHaveBeenCalled(); + expect(fixture.log).toHaveBeenLastCalledWith(error, LOG_LEVEL_VERBOSE, undefined); + }); + + it("captures the deletion time before evaluating ignore policy", async () => { + const fixture = createDependencies(); + const events: string[] = []; + fixture.now.mockImplementation(() => { + events.push("now"); + return 1_000; + }); + fixture.isIgnoredByIgnoreFile.mockImplementation(async () => { + events.push("ignore"); + return true; + }); + + await expect(deleteHiddenFileFromDatabase(fixture.dependencies, path)).resolves.toBeUndefined(); + + expect(events).toEqual(["now", "ignore"]); + expect(fixture.serialiseFileOperation).not.toHaveBeenCalled(); + }); +}); + +describe("hidden-file database write admission", () => { + it("preserves the different ignored results of the three write paths", async () => { + const fixture = createDependencies(); + fixture.isIgnoredByIgnoreFile.mockResolvedValue(true); + const file = fileInfo(); + + await expect(storeHiddenFileInDatabase(fixture.dependencies, file)).resolves.toBeUndefined(); + await expect(storeHiddenFileWithBaseRevision(fixture.dependencies, file, fixture.base._rev!)).resolves.toBe( + false + ); + await expect(deleteHiddenFileFromDatabase(fixture.dependencies, path)).resolves.toBeUndefined(); + + expect(fixture.loadBaseEntry).not.toHaveBeenCalled(); + expect(fixture.loadBaseMetadata).not.toHaveBeenCalled(); + expect(fixture.loadLiveRevision).not.toHaveBeenCalled(); + }); + + it("propagates ignore-policy failures before entering the guarded lock", async () => { + const fixture = createDependencies(); + const error = new Error("ignore policy unavailable"); + fixture.isIgnoredByIgnoreFile.mockRejectedValue(error); + + await expect(storeHiddenFileInDatabase(fixture.dependencies, fileInfo())).rejects.toBe(error); + + expect(fixture.serialiseFileOperation).not.toHaveBeenCalled(); + expect(fixture.log).not.toHaveBeenCalled(); + }); +}); diff --git a/src/features/HiddenFileSync/hiddenFileSyncFileOperations.ts b/src/features/HiddenFileSync/hiddenFileSyncFileOperations.ts new file mode 100644 index 00000000..3fc0f82f --- /dev/null +++ b/src/features/HiddenFileSync/hiddenFileSyncFileOperations.ts @@ -0,0 +1,18 @@ +import type { FilePathWithPrefix } from "@vrtmrz/livesync-commonlib/compat/common/types"; + +export type HiddenFileSyncFileSerialiser = (key: string, operation: () => Promise) => Promise; + +export type HiddenFileSyncFileSerialisationDependencies = { + serialiseFileOperation: HiddenFileSyncFileSerialiser; +}; + +export async function serialiseHiddenFileOperation( + dependencies: HiddenFileSyncFileSerialisationDependencies, + prefixedFileName: FilePathWithPrefix, + operation: () => Promise +): Promise { + // Compatibility question: this inherited lock uses `file-`, whereas the + // Commonlib database writer uses `file:`. The two writers therefore do not + // mutually exclude one another; changing the key needs concurrency tests. + return await dependencies.serialiseFileOperation(`file-${prefixedFileName}`, operation); +} diff --git a/src/features/HiddenFileSync/hiddenFileSyncPathPolicy.ts b/src/features/HiddenFileSync/hiddenFileSyncPathPolicy.ts new file mode 100644 index 00000000..9e2a3139 --- /dev/null +++ b/src/features/HiddenFileSync/hiddenFileSyncPathPolicy.ts @@ -0,0 +1,22 @@ +import type { CustomRegExp } from "@vrtmrz/livesync-commonlib/compat/common/utils"; + +export type HiddenFileSyncPathFilters = { + ignoreFilter: readonly CustomRegExp[]; + targetFilter: readonly CustomRegExp[]; +}; + +export function isHiddenFileSyncPath(path: string): boolean { + // Compatibility: this prefix check also excludes names such as `.trashcan`. + // Keep the broader exclusion until a separate path-policy decision changes it. + return path.startsWith(".") && !path.startsWith(".trash"); +} + +export function matchesHiddenFileSyncPatterns(path: string, filters: HiddenFileSyncPathFilters): boolean { + if (filters.ignoreFilter.some((pattern) => pattern.test(path))) { + return false; + } + if (filters.targetFilter.length > 0) { + return filters.targetFilter.some((pattern) => pattern.test(path)); + } + return true; +} diff --git a/src/features/HiddenFileSync/hiddenFileSyncPathPolicy.unit.spec.ts b/src/features/HiddenFileSync/hiddenFileSyncPathPolicy.unit.spec.ts new file mode 100644 index 00000000..9d56f6f7 --- /dev/null +++ b/src/features/HiddenFileSync/hiddenFileSyncPathPolicy.unit.spec.ts @@ -0,0 +1,41 @@ +import { describe, expect, it, vi } from "vitest"; +import type { CustomRegExp } from "@vrtmrz/livesync-commonlib/compat/common/utils"; +import { isHiddenFileSyncPath, matchesHiddenFileSyncPatterns } from "./hiddenFileSyncPathPolicy.ts"; + +const pattern = (matches: (path: string) => boolean) => ({ test: vi.fn(matches) }) as unknown as CustomRegExp; + +describe("isHiddenFileSyncPath", () => { + it.each([ + [".obsidian/app.json", true], + [".git/config", true], + [".trash/app.json", false], + [".trashcan/app.json", false], + ["notes/.hidden", false], + ])("classifies %s as a Hidden File Sync path=%s", (path, expected) => { + expect(isHiddenFileSyncPath(path)).toBe(expected); + }); +}); + +describe("matchesHiddenFileSyncPatterns", () => { + it("allows every path when no filters are configured", () => { + expect(matchesHiddenFileSyncPatterns(".obsidian/app.json", { ignoreFilter: [], targetFilter: [] })).toBe(true); + }); + + it("uses target patterns as an allow-list", () => { + const targetFilter = [pattern((path) => path.endsWith(".json"))]; + + expect(matchesHiddenFileSyncPatterns(".obsidian/app.json", { ignoreFilter: [], targetFilter })).toBe(true); + expect(matchesHiddenFileSyncPatterns(".obsidian/theme.css", { ignoreFilter: [], targetFilter })).toBe(false); + }); + + it("gives ignore patterns precedence over target patterns", () => { + const matchesEverything = pattern(() => true); + + expect( + matchesHiddenFileSyncPatterns(".obsidian/app.json", { + ignoreFilter: [matchesEverything], + targetFilter: [matchesEverything], + }) + ).toBe(false); + }); +}); diff --git a/src/features/HiddenFileSync/hiddenFileSyncProcessedState.ts b/src/features/HiddenFileSync/hiddenFileSyncProcessedState.ts new file mode 100644 index 00000000..8131faf2 --- /dev/null +++ b/src/features/HiddenFileSync/hiddenFileSyncProcessedState.ts @@ -0,0 +1,202 @@ +import { + LOG_LEVEL_VERBOSE, + type FilePath, + type LoadedEntry, + type LOG_LEVEL, + type MetaEntry, + type UXStat, +} from "@vrtmrz/livesync-commonlib/compat/common/types"; +import type { KeyValueDatabase } from "@vrtmrz/livesync-commonlib/compat/interfaces/KeyValueDatabase"; +import type { LiveSyncLocalDB } from "@vrtmrz/livesync-commonlib/compat/pouchdb/LiveSyncLocalDB"; +import type { IPathService } from "@vrtmrz/livesync-commonlib/compat/services/base/IService"; +import type { LogFunction } from "@vrtmrz/livesync-commonlib/compat/services/lib/logUtils"; +import { addPrefix } from "@vrtmrz/livesync-commonlib/compat/string_and_binary/path"; + +import { ICHeader } from "@/common/types.ts"; +import { autosaveCache, type MapLike } from "@/common/utils.ts"; +import { + getHiddenFileSyncComparisonMTime, + toHiddenFileSyncDatabaseStateKey, + toHiddenFileSyncStorageStateKey, +} from "./hiddenFileSyncState.ts"; + +type HiddenFileSyncProcessedStateDatabase = Pick; +type HiddenFileSyncProcessedStateStorage = { + statHidden(path: FilePath): Promise; +}; + +export type HiddenFileSyncProcessedStateDependencies = { + getKeyValueDatabase(): KeyValueDatabase; + getLocalDatabase(): HiddenFileSyncProcessedStateDatabase; + storageAccess: HiddenFileSyncProcessedStateStorage; + path: Pick; + log: LogFunction; +}; + +export class HiddenFileSyncProcessedState { + private fileInfoLastProcessed: MapLike = new Map(); + private fileInfoLastKnown: MapLike = new Map(); + private databaseInfoLastProcessed: MapLike = new Map(); + + constructor(private readonly dependencies: HiddenFileSyncProcessedStateDependencies) {} + + private log(message: unknown, level?: LOG_LEVEL, key?: string): void { + this.dependencies.log(message, level, key); + } + + async initialise(): Promise { + // Compatibility question: these reads intentionally remain + // sequential and in this order. + // Compatibility question: autosaveCache has no flush or disposal + // boundary, so a delayed write from an earlier database lifecycle can + // outlive this state owner. Preserve that timing until database + // replacement and unload behaviour have focused coverage. + this.fileInfoLastProcessed = await autosaveCache( + this.dependencies.getKeyValueDatabase(), + "hidden-file-lastProcessed" + ); + this.databaseInfoLastProcessed = await autosaveCache( + this.dependencies.getKeyValueDatabase(), + "hidden-file-lastProcessed-database" + ); + this.fileInfoLastKnown = await autosaveCache(this.dependencies.getKeyValueDatabase(), "hidden-file-lastKnown"); + } + + getLastProcessedFileCount(): number { + return this.fileInfoLastProcessed.size; + } + + getLastProcessedFileKeys(): IterableIterator { + return this.fileInfoLastProcessed.keys(); + } + + hasLastProcessedFile(file: FilePath): boolean { + return this.fileInfoLastProcessed.has(file); + } + + hasLastProcessedDatabase(file: FilePath): boolean { + return this.databaseInfoLastProcessed.has(file); + } + + async fileToStatKey(file: FilePath, stat: UXStat | null = null): Promise { + // Compatibility question: `null` means 'stat not supplied' here + // rather than 'file missing', so a failed earlier stat causes another + // read. Keep that retry until its event-processing effect is + // characterised. + if (!stat) stat = await this.dependencies.storageAccess.statHidden(file); + return this.storageStateKey(stat); + } + + storageStateKey(stat: UXStat | null): string { + return toHiddenFileSyncStorageStateKey(stat); + } + + databaseStateKey(doc: MetaEntry | LoadedEntry): string { + return toHiddenFileSyncDatabaseStateKey(doc); + } + + updateLastProcessedFile(file: FilePath, keySrc: string | UXStat): void { + const key = typeof keySrc == "string" ? keySrc : this.storageStateKey(keySrc); + const splitted = key.split("-"); + if (splitted[0] != "0") { + // Compatibility: a zero storage marker does not replace the last + // known non-zero mtime. Deletion therefore retains that fallback. + this.fileInfoLastKnown.set(file, Number(splitted[0])); + } + this.fileInfoLastProcessed.set(file, key); + } + + async updateLastProcessedAsActualFile(file: FilePath, stat?: UXStat | null): Promise { + if (!stat) stat = await this.dependencies.storageAccess.statHidden(file); + // Compatibility: adoption updates only the processed marker. It does + // not update the last-known non-zero mtime cache. + this.fileInfoLastProcessed.set(file, this.storageStateKey(stat)); + } + + resetLastProcessedFile(targetFiles: FilePath[] | false): void { + if (targetFiles) { + for (const key of targetFiles) { + this.fileInfoLastProcessed.delete(key); + } + } else { + this.log(`Delete all processed mark.`, LOG_LEVEL_VERBOSE); + // THINKING: Should we... + // - delete all `Known file` processed mark? (This is current implementation) + // - delete all `Existing file` processed mark? + // - delete all files inside the config folder of current device mark? + this.fileInfoLastProcessed.clear(); + } + } + + getLastProcessedFileMTime(file: FilePath): number { + const key = this.fileInfoLastKnown.get(file); + if (!key) return 0; + return key; + } + + getLastProcessedFileKey(file: FilePath): string | undefined { + return this.fileInfoLastProcessed.get(file); + } + + getLastProcessedDatabaseKey(file: FilePath): string | undefined { + return this.databaseInfoLastProcessed.get(file); + } + + updateLastProcessedDatabase(file: FilePath, keySrc: string | MetaEntry | LoadedEntry): void { + const key = typeof keySrc == "string" ? keySrc : this.databaseStateKey(keySrc); + this.databaseInfoLastProcessed.set(file, key); + } + + updateLastProcessed(path: FilePath, db: MetaEntry | LoadedEntry, stat: UXStat): void { + this.updateLastProcessedDatabase(path, db); + this.updateLastProcessedFile(path, this.storageStateKey(stat)); + const dbMTime = getHiddenFileSyncComparisonMTime(db); + const storageMTime = getHiddenFileSyncComparisonMTime(stat); + if (dbMTime == 0 || storageMTime == 0) { + this.dependencies.path.unmarkChanges(path); + } else { + this.dependencies.path.markChangesAreSame( + path, + getHiddenFileSyncComparisonMTime(db), + getHiddenFileSyncComparisonMTime(stat) + ); + } + } + + updateLastProcessedDeletion(path: FilePath, db: MetaEntry | LoadedEntry | false): void { + this.dependencies.path.unmarkChanges(path); + if (db) this.updateLastProcessedDatabase(path, db); + this.updateLastProcessedFile(path, this.storageStateKey(null)); + } + + async updateLastProcessedAsActualDatabase( + file: FilePath, + doc?: MetaEntry | LoadedEntry | null | false + ): Promise { + const dbPath = addPrefix(file, ICHeader); + if (!doc) doc = await this.dependencies.getLocalDatabase().getDBEntryMeta(dbPath); + if (!doc) return; + this.databaseInfoLastProcessed.set(file, this.databaseStateKey(doc)); + } + + resetLastProcessedDatabase(targetFiles: FilePath[] | false): void { + if (targetFiles) { + for (const key of targetFiles) { + this.databaseInfoLastProcessed.delete(key); + } + } else { + this.log(`Delete all processed mark.`, LOG_LEVEL_VERBOSE); + // THINKING: Should we... + // - delete all `Known file` processed mark? (This is current implementation) + // - delete all `Existing file` processed mark? + // - delete all files inside the config folder of current device mark? + this.databaseInfoLastProcessed.clear(); + } + } +} + +export function createHiddenFileSyncProcessedState( + dependencies: HiddenFileSyncProcessedStateDependencies +): HiddenFileSyncProcessedState { + return new HiddenFileSyncProcessedState(dependencies); +} diff --git a/src/features/HiddenFileSync/hiddenFileSyncProcessedState.unit.spec.ts b/src/features/HiddenFileSync/hiddenFileSyncProcessedState.unit.spec.ts new file mode 100644 index 00000000..84c58867 --- /dev/null +++ b/src/features/HiddenFileSync/hiddenFileSyncProcessedState.unit.spec.ts @@ -0,0 +1,182 @@ +import { describe, expect, it, vi } from "vitest"; +import { + LOG_LEVEL_VERBOSE, + type FilePath, + type LoadedEntry, + type MetaEntry, + type UXStat, +} from "@vrtmrz/livesync-commonlib/compat/common/types"; +import type { KeyValueDatabase } from "@vrtmrz/livesync-commonlib/compat/interfaces/KeyValueDatabase"; + +vi.mock("@/deps.ts", () => ({})); + +import { + createHiddenFileSyncProcessedState, + type HiddenFileSyncProcessedStateDependencies, +} from "./hiddenFileSyncProcessedState.ts"; +import { toHiddenFileSyncDatabaseStateKey } from "./hiddenFileSyncState.ts"; + +const path = ".obsidian/plugins/example/data.json" as FilePath; + +function stat(mtime: number, size = 20): UXStat { + return { ctime: mtime, mtime, size, type: "file" }; +} + +function metadata(overrides: Partial = {}): MetaEntry { + return { + _id: "hidden-entry-id", + _rev: "2-current", + path: `i:${path}`, + type: "plain", + datatype: "plain", + ctime: 10, + mtime: 20, + size: 20, + children: [], + eden: {}, + deleted: false, + ...overrides, + } as unknown as MetaEntry; +} + +function createState() { + const events: string[] = []; + const caches = new Map>([ + ["hidden-file-lastProcessed", new Map([[path, "40-20"]])], + ["hidden-file-lastProcessed-database", new Map([[path, "20-20-2-current--1"]])], + ["hidden-file-lastKnown", new Map([[path, 40]])], + ]); + let activeReads = 0; + let maximumActiveReads = 0; + const keyValueDatabase = { + get: vi.fn(async (key: IDBValidKey) => { + events.push(`get:${String(key)}`); + activeReads++; + maximumActiveReads = Math.max(maximumActiveReads, activeReads); + await Promise.resolve(); + activeReads--; + return caches.get(String(key)); + }), + set: vi.fn(async () => "ok"), + } as unknown as KeyValueDatabase; + const getDBEntryMeta = vi.fn(async () => false as false | LoadedEntry); + const statHidden = vi.fn(async () => stat(41)); + const markChangesAreSame = vi.fn(() => undefined); + const unmarkChanges = vi.fn(); + const log = vi.fn(); + const dependencies: HiddenFileSyncProcessedStateDependencies = { + getKeyValueDatabase: () => keyValueDatabase, + getLocalDatabase: () => ({ getDBEntryMeta }), + storageAccess: { statHidden }, + path: { markChangesAreSame, unmarkChanges }, + log, + }; + const state = createHiddenFileSyncProcessedState(dependencies); + return { + dependencies, + events, + getDBEntryMeta, + keyValueDatabase, + log, + markChangesAreSame, + maximumActiveReads: () => maximumActiveReads, + state, + statHidden, + unmarkChanges, + }; +} + +describe("Hidden File Sync processed state", () => { + it("loads the three autosave caches sequentially under their existing keys", async () => { + const fixture = createState(); + + await fixture.state.initialise(); + + expect(fixture.events).toEqual([ + "get:hidden-file-lastProcessed", + "get:hidden-file-lastProcessed-database", + "get:hidden-file-lastKnown", + ]); + expect(fixture.maximumActiveReads()).toBe(1); + expect(fixture.state.getLastProcessedFileCount()).toBe(1); + expect(fixture.state.getLastProcessedFileKey(path)).toBe("40-20"); + expect(fixture.state.getLastProcessedDatabaseKey(path)).toBe("20-20-2-current--1"); + expect(fixture.state.getLastProcessedFileMTime(path)).toBe(40); + }); + + it("re-reads a null storage stat and retains the last known mtime on deletion", async () => { + const fixture = createState(); + await fixture.state.initialise(); + + fixture.statHidden.mockResolvedValueOnce(stat(45, 3)); + await expect(fixture.state.fileToStatKey(path, null)).resolves.toBe("45-3"); + expect(fixture.statHidden).toHaveBeenCalledWith(path); + + fixture.state.updateLastProcessedFile(path, stat(45, 3)); + fixture.state.updateLastProcessedDeletion(path, metadata({ mtime: 50, size: 0 })); + + expect(fixture.state.getLastProcessedFileKey(path)).toBe("0-0"); + expect(fixture.state.getLastProcessedFileMTime(path)).toBe(45); + expect(fixture.state.getLastProcessedDatabaseKey(path)).toBe( + toHiddenFileSyncDatabaseStateKey(metadata({ mtime: 50, size: 0 })) + ); + expect(fixture.unmarkChanges).toHaveBeenCalledWith(path); + }); + + it("settles combined state before applying the matching-path marker", async () => { + const fixture = createState(); + await fixture.state.initialise(); + const document = metadata({ mtime: 60, size: 9 }); + const storageStat = stat(61, 9); + + fixture.state.updateLastProcessed(path, document, storageStat); + + expect(fixture.markChangesAreSame).toHaveBeenCalledWith(path, 60, 61); + expect(fixture.unmarkChanges).not.toHaveBeenCalled(); + expect(fixture.state.getLastProcessedDatabaseKey(path)).toBe(toHiddenFileSyncDatabaseStateKey(document)); + expect(fixture.state.getLastProcessedFileKey(path)).toBe("61-9"); + }); + + it("resets each processed side without clearing last-known storage mtimes", async () => { + const fixture = createState(); + await fixture.state.initialise(); + fixture.state.updateLastProcessedFile(path, stat(70, 4)); + fixture.state.updateLastProcessedDatabase(path, "database-key"); + + fixture.state.resetLastProcessedFile([path]); + expect(fixture.state.getLastProcessedFileKey(path)).toBeUndefined(); + expect(fixture.state.getLastProcessedDatabaseKey(path)).toBe("database-key"); + expect(fixture.state.getLastProcessedFileMTime(path)).toBe(70); + + fixture.state.resetLastProcessedDatabase([path]); + expect(fixture.state.getLastProcessedDatabaseKey(path)).toBeUndefined(); + }); + + it("does not settle a database marker for a false or missing database document", async () => { + const fixture = createState(); + await fixture.state.initialise(); + + await fixture.state.updateLastProcessedAsActualDatabase(path, false); + expect(fixture.getDBEntryMeta).toHaveBeenCalledWith(`i:${path}`); + expect(fixture.state.getLastProcessedDatabaseKey(path)).toBe("20-20-2-current--1"); + + fixture.getDBEntryMeta.mockResolvedValueOnce(false); + await fixture.state.updateLastProcessedAsActualDatabase(path); + expect(fixture.state.getLastProcessedDatabaseKey(path)).toBe("20-20-2-current--1"); + }); + + it("logs and clears both sides when a full reset is requested", async () => { + const fixture = createState(); + await fixture.state.initialise(); + + fixture.state.resetLastProcessedFile(false); + fixture.state.resetLastProcessedDatabase(false); + + expect(fixture.log).toHaveBeenNthCalledWith(1, "Delete all processed mark.", LOG_LEVEL_VERBOSE, undefined); + expect(fixture.log).toHaveBeenNthCalledWith(2, "Delete all processed mark.", LOG_LEVEL_VERBOSE, undefined); + expect(fixture.state.getLastProcessedFileCount()).toBe(0); + expect(fixture.state.getLastProcessedFileKey(path)).toBeUndefined(); + expect(fixture.state.getLastProcessedDatabaseKey(path)).toBeUndefined(); + expect(fixture.state.getLastProcessedFileMTime(path)).toBe(40); + }); +}); diff --git a/src/features/HiddenFileSync/hiddenFileSyncState.ts b/src/features/HiddenFileSync/hiddenFileSyncState.ts new file mode 100644 index 00000000..308ec1c0 --- /dev/null +++ b/src/features/HiddenFileSync/hiddenFileSyncState.ts @@ -0,0 +1,48 @@ +import type { + FilePathWithPrefix, + LoadedEntry, + MetaEntry, + UXFileInfo, + UXStat, +} from "@vrtmrz/livesync-commonlib/compat/common/types"; +import { displayRev } from "@vrtmrz/livesync-commonlib/compat/common/utils"; +import { stripAllPrefixes } from "@vrtmrz/livesync-commonlib/compat/string_and_binary/path"; + +export function toHiddenFileSyncStorageStateKey(stat: UXStat | null): string { + return `${stat?.mtime ?? 0}-${stat?.size ?? 0}`; +} + +export function toHiddenFileSyncDatabaseStateKey(doc: LoadedEntry | MetaEntry): string { + // Compatibility: the deletion marker includes its own hyphen, producing `--0` + // or `--1` after the revision. Existing device-local state uses this format. + return `${doc.mtime}-${doc.size}-${doc._rev}-${doc._deleted || doc.deleted || false ? "-0" : "-1"}`; +} + +export function getHiddenFileSyncComparisonMTime( + source: MetaEntry | LoadedEntry | UXFileInfo | UXStat | false | null | undefined, + includeDeleted = false +): number { + if (source === null || source === false || source === undefined) return 0; + if (!includeDeleted) { + if ("deleted" in source && source.deleted) return 0; + if ("_deleted" in source && source._deleted) return 0; + } + if ("stat" in source) return source.stat?.mtime ?? 0; + return source.mtime ?? 0; +} + +export function describeHiddenFileSyncDocument(doc: LoadedEntry, prefixedPath: FilePathWithPrefix) { + const id = doc._id; + const path = stripAllPrefixes(prefixedPath); + const rev = doc._rev; + return { + id, + rev, + revDisplay: rev ? displayRev(rev) : "0-NOREVS", + prefixedPath, + path, + isDeleted: doc._deleted || doc.deleted || false, + shortenedId: id.substring(0, 10), + shortenedPath: path.substring(0, 10), + }; +} diff --git a/src/features/HiddenFileSync/hiddenFileSyncState.unit.spec.ts b/src/features/HiddenFileSync/hiddenFileSyncState.unit.spec.ts new file mode 100644 index 00000000..033c0e38 --- /dev/null +++ b/src/features/HiddenFileSync/hiddenFileSyncState.unit.spec.ts @@ -0,0 +1,70 @@ +import { describe, expect, it } from "vitest"; +import type { FilePathWithPrefix, LoadedEntry, UXStat } from "@vrtmrz/livesync-commonlib/compat/common/types"; +import { + describeHiddenFileSyncDocument, + getHiddenFileSyncComparisonMTime, + toHiddenFileSyncDatabaseStateKey, + toHiddenFileSyncStorageStateKey, +} from "./hiddenFileSyncState.ts"; + +describe("Hidden File Sync state keys", () => { + it("represents a missing storage file with zero values", () => { + expect(toHiddenFileSyncStorageStateKey(null)).toBe("0-0"); + }); + + it("uses storage modification time and size", () => { + expect(toHiddenFileSyncStorageStateKey({ mtime: 123, size: 456 } as UXStat)).toBe("123-456"); + }); + + it.each([ + [false, "123-456-3-example--1"], + [true, "123-456-3-example--0"], + ])("includes database revision and deletion state=%s", (deleted, expected) => { + const doc = { mtime: 123, size: 456, _rev: "3-example", deleted } as LoadedEntry; + + expect(toHiddenFileSyncDatabaseStateKey(doc)).toBe(expected); + }); +}); + +describe("getHiddenFileSyncComparisonMTime", () => { + const absentSources = [null, false, undefined] as const; + + it.each(absentSources)("returns zero for an absent source=%s", (source) => { + expect(getHiddenFileSyncComparisonMTime(source)).toBe(0); + }); + + it("reads a direct stat or a file-info stat", () => { + expect(getHiddenFileSyncComparisonMTime({ mtime: 123 } as UXStat)).toBe(123); + expect(getHiddenFileSyncComparisonMTime({ stat: { mtime: 456 } } as never)).toBe(456); + }); + + it("treats deleted entries as zero unless deletion time is requested", () => { + const deleted = { mtime: 123, deleted: true } as LoadedEntry; + + expect(getHiddenFileSyncComparisonMTime(deleted)).toBe(0); + expect(getHiddenFileSyncComparisonMTime(deleted, true)).toBe(123); + }); +}); + +describe("describeHiddenFileSyncDocument", () => { + it("derives the unprefixed path and diagnostic revision fields", () => { + const doc = { + _id: "0123456789abcdef", + _rev: "3-example", + mtime: 123, + size: 456, + deleted: true, + } as LoadedEntry; + + expect(describeHiddenFileSyncDocument(doc, "i:.obsidian/app.json" as FilePathWithPrefix)).toEqual({ + id: "0123456789abcdef", + rev: "3-example", + revDisplay: "3-exampl", + prefixedPath: "i:.obsidian/app.json", + path: ".obsidian/app.json", + isDeleted: true, + shortenedId: "0123456789", + shortenedPath: ".obsidian/", + }); + }); +}); diff --git a/src/features/HiddenFileSync/hiddenFileSyncStorage.ts b/src/features/HiddenFileSync/hiddenFileSyncStorage.ts new file mode 100644 index 00000000..e68c8442 --- /dev/null +++ b/src/features/HiddenFileSync/hiddenFileSyncStorage.ts @@ -0,0 +1,210 @@ +import { + LOG_LEVEL_DEBUG, + LOG_LEVEL_VERBOSE, + type FilePath, + type LoadedEntry, + type LOG_LEVEL, + type UXDataWriteOptions, + type UXFileInfo, + type UXStat, +} from "@vrtmrz/livesync-commonlib/compat/common/types"; +import type { StorageAccess } from "@vrtmrz/livesync-commonlib/compat/interfaces/StorageAccess"; +import type { LogFunction } from "@vrtmrz/livesync-commonlib/compat/services/lib/logUtils"; +import { + createBlob, + isDocContentSame, + readContent, +} from "@vrtmrz/livesync-commonlib/compat/common/utils"; + +export type HiddenFileSyncStorageAccess = Pick< + StorageAccess, + | "ensureDir" + | "isExistsIncludeHidden" + | "readHiddenFileAuto" + | "removeHidden" + | "statHidden" + | "triggerHiddenFile" + | "writeHiddenFileAuto" +>; + +type HiddenFileSyncStorageMethods = { + storageAccess: Pick; +}; + +type HiddenFileSyncLogDependency = { + log: LogFunction; +}; + +export type HiddenFileSyncStorageDependencies = HiddenFileSyncStorageMethods & + HiddenFileSyncLogDependency; + +export type HiddenFileSyncRemovalResult = "OK" | "ALREADY" | false; + +function log( + dependencies: HiddenFileSyncLogDependency, + message: unknown, + level?: LOG_LEVEL, + key?: string +): void { + dependencies.log(message, level, key); +} + +export async function readHiddenFileWithInfo( + dependencies: HiddenFileSyncStorageMethods<"readHiddenFileAuto" | "statHidden">, + path: FilePath +): Promise { + const stat = await dependencies.storageAccess.statHidden(path); + if (!stat) { + return { + name: path.split("/").pop() ?? "", + path, + stat: { + size: 0, + mtime: 0, + ctime: 0, + type: "file", + }, + isInternal: true, + deleted: true, + body: createBlob(new Uint8Array(0)), + }; + } + const content = await dependencies.storageAccess.readHiddenFileAuto(path); + return { + name: path.split("/").pop() ?? "", + path, + stat, + isInternal: true, + deleted: false, + body: createBlob(content), + }; +} + +export async function ensureHiddenFileDirectory( + dependencies: HiddenFileSyncStorageMethods<"ensureDir" | "isExistsIncludeHidden">, + path: FilePath +): Promise { + if (!(await dependencies.storageAccess.isExistsIncludeHidden(path))) { + // StorageAccess expects the complete target path and ensures its parent. + await dependencies.storageAccess.ensureDir(path); + } +} + +export async function writeHiddenFile( + dependencies: HiddenFileSyncStorageMethods<"statHidden" | "writeHiddenFileAuto">, + path: FilePath, + data: string | ArrayBuffer, + options?: UXDataWriteOptions +): Promise { + // Compatibility: the writer's Boolean result is ignored. The post-write stat + // has historically decided whether the operation produced a usable file. + await dependencies.storageAccess.writeHiddenFileAuto(path, data, options); + return await dependencies.storageAccess.statHidden(path); +} + +export async function removeHiddenFile( + dependencies: HiddenFileSyncStorageMethods<"isExistsIncludeHidden" | "removeHidden"> & + HiddenFileSyncLogDependency, + path: FilePath +): Promise { + try { + if (!(await dependencies.storageAccess.isExistsIncludeHidden(path))) { + return "ALREADY"; + } + if (await dependencies.storageAccess.removeHidden(path)) { + return "OK"; + } + } catch (error) { + log(dependencies, `Failed to remove file:${path}`); + log(dependencies, error, LOG_LEVEL_VERBOSE); + } + return false; +} + +export async function triggerHiddenFileEvent( + dependencies: HiddenFileSyncStorageMethods<"triggerHiddenFile"> & HiddenFileSyncLogDependency, + path: FilePath +): Promise { + try { + await dependencies.storageAccess.triggerHiddenFile(path); + } catch (error) { + log(dependencies, "Failed to call internal API(reconcileInternalFile)", LOG_LEVEL_VERBOSE); + log(dependencies, error, LOG_LEVEL_VERBOSE); + } +} + +export async function isHiddenFileWriteRequired( + dependencies: HiddenFileSyncStorageMethods<"readHiddenFileAuto"> & HiddenFileSyncLogDependency, + path: FilePath, + content: string | ArrayBuffer +): Promise { + try { + const storageContent = await dependencies.storageAccess.readHiddenFileAuto(path); + return !(await isDocContentSame(storageContent, content)); + } catch (error) { + log(dependencies, `Cannot check the content of ${path}`); + log(dependencies, error, LOG_LEVEL_VERBOSE); + // Compatibility: an unreadable current file is treated as requiring a + // write. Changing this policy needs a separate recovery decision. + return true; + } +} + +export async function writeHiddenFileFromDatabase( + dependencies: HiddenFileSyncStorageMethods< + "ensureDir" | "isExistsIncludeHidden" | "readHiddenFileAuto" | "statHidden" | "writeHiddenFileAuto" + > & + HiddenFileSyncLogDependency, + path: FilePath, + fileOnDatabase: LoadedEntry, + force: boolean +): Promise { + try { + const statBefore = await dependencies.storageAccess.statHidden(path); + const isExisting = statBefore != null; + const content = readContent(fileOnDatabase); + await ensureHiddenFileDirectory(dependencies, path); + const writeRequired = + force || !isExisting || (isExisting && (await isHiddenFileWriteRequired(dependencies, path, content))); + + if (!writeRequired) { + log(dependencies, `STORAGE <-- DB: ${path}: skipped (hidden) Not changed`, LOG_LEVEL_DEBUG); + return statBefore; + } + + const statAfter = await writeHiddenFile(dependencies, path, content, { + mtime: fileOnDatabase.mtime, + ctime: fileOnDatabase.ctime, + }); + if (statAfter == null) { + log(dependencies, `STORAGE <-- DB: ${path}: written (hidden,new${force ? ", force" : ""}) Failed (writeResult)`); + return false; + } + log(dependencies, `STORAGE <-- DB: ${path}: written (hidden, overwrite${force ? ", force" : ""})`); + // Compatibility question: ordinary database reflection does not trigger + // a raw storage event here; deletion and manual JSON merging do. Preserve + // this until the event-loop consequences of changing it are characterised. + return statAfter; + } catch (error) { + log(dependencies, `STORAGE <-- DB: ${path}: written (hidden, overwrite${force ? ", force" : ""}) Failed`); + log(dependencies, error, LOG_LEVEL_VERBOSE); + return false; + } +} + +export async function deleteHiddenFileFromStorage( + dependencies: HiddenFileSyncStorageMethods<"isExistsIncludeHidden" | "removeHidden" | "triggerHiddenFile"> & + HiddenFileSyncLogDependency, + path: FilePath +): Promise { + const result = await removeHiddenFile(dependencies, path); + if (result === false) { + log(dependencies, `STORAGE true), + isExistsIncludeHidden: vi.fn(async () => true), + readHiddenFileAuto: vi.fn(async () => "data" as string | ArrayBuffer), + removeHidden: vi.fn(async () => true), + statHidden: vi.fn(async () => stat as UXStat | null), + triggerHiddenFile: vi.fn(async () => undefined), + writeHiddenFileAuto: vi.fn(async () => true), + }; + const log = vi.fn(); + return { dependencies: { storageAccess, log }, log, storageAccess }; +} + +function databaseEntry(content: string, mtime = 30, ctime = 15): LoadedEntry { + return { + path, + type: "plain", + datatype: "plain", + data: content, + mtime, + ctime, + } as LoadedEntry; +} + +describe("Hidden File Sync storage operations", () => { + it("represents a missing hidden file as a deleted empty file", async () => { + const { dependencies, storageAccess } = createStorageDependencies(); + storageAccess.statHidden.mockResolvedValue(null); + + const result = await readHiddenFileWithInfo(dependencies, path); + + expect(result).toMatchObject({ + name: "data.json", + path, + isInternal: true, + deleted: true, + stat: { ctime: 0, mtime: 0, size: 0, type: "file" }, + }); + expect(await result.body.text()).toBe(""); + expect(storageAccess.readHiddenFileAuto).not.toHaveBeenCalled(); + }); + + it("loads an existing hidden file with its storage stat", async () => { + const { dependencies, storageAccess } = createStorageDependencies(); + storageAccess.readHiddenFileAuto.mockResolvedValue("data"); + + const result = await readHiddenFileWithInfo(dependencies, path); + + expect(result).toMatchObject({ name: "data.json", path, isInternal: true, deleted: false, stat }); + expect(await result.body.text()).toBe("data"); + }); + + it("ensures a directory only when the target does not exist", async () => { + const { dependencies, storageAccess } = createStorageDependencies(); + + await ensureHiddenFileDirectory(dependencies, path); + expect(storageAccess.ensureDir).not.toHaveBeenCalled(); + + storageAccess.isExistsIncludeHidden.mockResolvedValue(false); + await ensureHiddenFileDirectory(dependencies, path); + expect(storageAccess.ensureDir).toHaveBeenCalledOnce(); + expect(storageAccess.ensureDir).toHaveBeenCalledWith(path); + }); + + it("writes a hidden file and returns the resulting stat", async () => { + const { dependencies, storageAccess } = createStorageDependencies(); + + await expect(writeHiddenFile(dependencies, path, "data", { mtime: 30, ctime: 15 })).resolves.toBe(stat); + expect(storageAccess.writeHiddenFileAuto).toHaveBeenCalledWith(path, "data", { mtime: 30, ctime: 15 }); + expect(storageAccess.statHidden).toHaveBeenCalledWith(path); + }); + + it("uses the post-write stat even when the storage writer reports false", async () => { + const { dependencies, storageAccess } = createStorageDependencies(); + storageAccess.writeHiddenFileAuto.mockResolvedValue(false); + + await expect(writeHiddenFile(dependencies, path, "data")).resolves.toBe(stat); + expect(storageAccess.statHidden).toHaveBeenCalledAfter(storageAccess.writeHiddenFileAuto); + }); + + it("distinguishes an absent, removed, and unremovable file", async () => { + const { dependencies, storageAccess } = createStorageDependencies(); + + storageAccess.isExistsIncludeHidden.mockResolvedValueOnce(false); + await expect(removeHiddenFile(dependencies, path)).resolves.toBe("ALREADY"); + expect(storageAccess.removeHidden).not.toHaveBeenCalled(); + + storageAccess.isExistsIncludeHidden.mockResolvedValueOnce(true); + storageAccess.removeHidden.mockResolvedValueOnce(true); + await expect(removeHiddenFile(dependencies, path)).resolves.toBe("OK"); + + storageAccess.isExistsIncludeHidden.mockResolvedValueOnce(true); + storageAccess.removeHidden.mockResolvedValueOnce(false); + await expect(removeHiddenFile(dependencies, path)).resolves.toBe(false); + }); + + it("turns a removal error into a logged false result", async () => { + const { dependencies, log, storageAccess } = createStorageDependencies(); + const error = new Error("remove failed"); + storageAccess.isExistsIncludeHidden.mockRejectedValue(error); + + await expect(removeHiddenFile(dependencies, path)).resolves.toBe(false); + expect(log).toHaveBeenNthCalledWith(1, `Failed to remove file:${path}`, undefined, undefined); + expect(log).toHaveBeenNthCalledWith(2, error, LOG_LEVEL_VERBOSE, undefined); + }); + + it("treats a content-read failure as requiring a write", async () => { + const { dependencies, log, storageAccess } = createStorageDependencies(); + const error = new Error("read failed"); + storageAccess.readHiddenFileAuto.mockRejectedValue(error); + + await expect(isHiddenFileWriteRequired(dependencies, path, "data")).resolves.toBe(true); + expect(log).toHaveBeenNthCalledWith(1, `Cannot check the content of ${path}`, undefined, undefined); + expect(log).toHaveBeenNthCalledWith(2, error, LOG_LEVEL_VERBOSE, undefined); + }); + + it("compares binary content without involving the context", async () => { + const { dependencies, storageAccess } = createStorageDependencies(); + storageAccess.readHiddenFileAuto.mockResolvedValue(new Uint8Array([1, 2, 3]).buffer); + + await expect( + isHiddenFileWriteRequired(dependencies, path, new Uint8Array([1, 2, 3]).buffer) + ).resolves.toBe(false); + await expect( + isHiddenFileWriteRequired(dependencies, path, new Uint8Array([1, 2, 4]).buffer) + ).resolves.toBe(true); + }); + + it("skips an unchanged database file and preserves its current stat", async () => { + const { dependencies, storageAccess } = createStorageDependencies(); + storageAccess.readHiddenFileAuto.mockResolvedValue("data"); + + await expect(writeHiddenFileFromDatabase(dependencies, path, databaseEntry("data"), false)).resolves.toBe( + stat + ); + expect(storageAccess.ensureDir).not.toHaveBeenCalled(); + expect(storageAccess.writeHiddenFileAuto).not.toHaveBeenCalled(); + }); + + it("writes changed database content with its original timestamps", async () => { + const { dependencies, storageAccess } = createStorageDependencies(); + storageAccess.readHiddenFileAuto.mockResolvedValue("old"); + + await expect(writeHiddenFileFromDatabase(dependencies, path, databaseEntry("new"), false)).resolves.toBe( + stat + ); + expect(storageAccess.writeHiddenFileAuto).toHaveBeenCalledWith(path, "new", { mtime: 30, ctime: 15 }); + expect(storageAccess.triggerHiddenFile).not.toHaveBeenCalled(); + }); + + it("forces a write without reading existing content", async () => { + const { dependencies, storageAccess } = createStorageDependencies(); + + await expect(writeHiddenFileFromDatabase(dependencies, path, databaseEntry("data"), true)).resolves.toBe(stat); + expect(storageAccess.readHiddenFileAuto).not.toHaveBeenCalled(); + expect(storageAccess.writeHiddenFileAuto).toHaveBeenCalledOnce(); + }); + + it("returns false when a completed write has no resulting stat", async () => { + const { dependencies, storageAccess } = createStorageDependencies(); + storageAccess.statHidden.mockResolvedValue(null); + + await expect(writeHiddenFileFromDatabase(dependencies, path, databaseEntry("data"), false)).resolves.toBe( + false + ); + expect(storageAccess.writeHiddenFileAuto).toHaveBeenCalledOnce(); + }); + + it("triggers a storage event only after an actual deletion", async () => { + const { dependencies, log, storageAccess } = createStorageDependencies(); + + storageAccess.isExistsIncludeHidden.mockResolvedValueOnce(false); + await expect(deleteHiddenFileFromStorage(dependencies, path)).resolves.toBe("ALREADY"); + expect(storageAccess.triggerHiddenFile).not.toHaveBeenCalled(); + log.mockClear(); + + storageAccess.isExistsIncludeHidden.mockResolvedValueOnce(true); + storageAccess.removeHidden.mockResolvedValueOnce(true); + await expect(deleteHiddenFileFromStorage(dependencies, path)).resolves.toBe("OK"); + expect(storageAccess.triggerHiddenFile).toHaveBeenCalledOnce(); + expect(storageAccess.triggerHiddenFile).toHaveBeenCalledWith(path); + expect(storageAccess.triggerHiddenFile).toHaveBeenCalledAfter(storageAccess.removeHidden); + expect(log).toHaveBeenCalledAfter(storageAccess.triggerHiddenFile); + }); + + it("does not trigger a storage event after a failed deletion", async () => { + const { dependencies, storageAccess } = createStorageDependencies(); + storageAccess.removeHidden.mockResolvedValue(false); + + await expect(deleteHiddenFileFromStorage(dependencies, path)).resolves.toBe(false); + expect(storageAccess.triggerHiddenFile).not.toHaveBeenCalled(); + }); + + it("swallows and logs storage-event failures", async () => { + const { dependencies, log, storageAccess } = createStorageDependencies(); + const error = new Error("event failed"); + storageAccess.triggerHiddenFile.mockRejectedValue(error); + + await expect(triggerHiddenFileEvent(dependencies, path)).resolves.toBeUndefined(); + expect(log).toHaveBeenNthCalledWith( + 1, + "Failed to call internal API(reconcileInternalFile)", + LOG_LEVEL_VERBOSE, + undefined + ); + expect(log).toHaveBeenNthCalledWith(2, error, LOG_LEVEL_VERBOSE, undefined); + }); +}); diff --git a/src/features/HiddenFileSync/hiddenFileSyncViews.ts b/src/features/HiddenFileSync/hiddenFileSyncViews.ts index 1fa176f5..019bb1f2 100644 --- a/src/features/HiddenFileSync/hiddenFileSyncViews.ts +++ b/src/features/HiddenFileSync/hiddenFileSyncViews.ts @@ -1,5 +1,12 @@ import type { InternalFileInfo } from "@/common/types.ts"; -import type { FilePath } from "@vrtmrz/livesync-commonlib/compat/common/types"; +import type { + FilePath, + FilePathWithPrefix, + LoadedEntry, + UXFileInfo, +} from "@vrtmrz/livesync-commonlib/compat/common/types"; +import type { OptionalSyncFeatureMode } from "@/features/optionalSyncFeatures.ts"; +import type { HiddenFileSyncConflictTestingView } from "./hiddenFileSyncConflictResolution.ts"; export type HiddenFileSyncInitialisationDirection = "push" | "pull" | "safe" | "pullForce" | "pushForce"; @@ -12,6 +19,91 @@ export interface HiddenFileSyncInitialisationView { ): Promise; } +/** + * Semantic callbacks consumed by the Commonlib service registries. + * + * The context owns the implementations, while the optional-file composition + * adapts this view to Commonlib's aggregation contracts. Callers do not bind + * the context or depend on registry-oriented method names. + */ +export interface HiddenFileSyncServiceHandlerView { + readonly processOptionalFileEvent: (path: FilePath) => Promise; + readonly processOptionalSyncFiles: (doc: LoadedEntry) => Promise; + readonly onSettingLoaded: () => Promise; + readonly realiseSettingSyncMode: () => Promise; + readonly onResuming: () => Promise; + readonly beforeReplicate: (showNotice: boolean) => Promise; + readonly onDatabaseInitialised: (showNotice: boolean) => Promise; + readonly suspendExtraSync: () => Promise; + readonly configureOptionalSyncFeature: (mode: OptionalSyncFeatureMode) => Promise; + readonly isTargetFileEligible: (path: FilePath) => Promise; + readonly queueConflict: (path: FilePathWithPrefix) => Promise; +} + +export function createHiddenFileSyncServiceHandlerView( + operations: HiddenFileSyncServiceHandlerView +): HiddenFileSyncServiceHandlerView { + const view: HiddenFileSyncServiceHandlerView = { + processOptionalFileEvent: async (path) => await operations.processOptionalFileEvent(path), + processOptionalSyncFiles: async (doc) => await operations.processOptionalSyncFiles(doc), + onSettingLoaded: async () => await operations.onSettingLoaded(), + realiseSettingSyncMode: async () => await operations.realiseSettingSyncMode(), + onResuming: async () => await operations.onResuming(), + beforeReplicate: async (showNotice) => await operations.beforeReplicate(showNotice), + onDatabaseInitialised: async (showNotice) => await operations.onDatabaseInitialised(showNotice), + suspendExtraSync: async () => await operations.suspendExtraSync(), + configureOptionalSyncFeature: async (mode) => await operations.configureOptionalSyncFeature(mode), + isTargetFileEligible: async (path) => await operations.isTargetFileEligible(path), + queueConflict: async (path) => await operations.queueConflict(path), + }; + return Object.freeze(view); +} + +export type HiddenFileSyncTestingRebuild = ( + showNotice: boolean, + targetFiles?: FilePath[] | false +) => Promise; + +export type HiddenFileSyncTestingRebuildInterceptor = ( + runRebuild: HiddenFileSyncTestingRebuild, + showNotice: boolean, + targetFiles?: FilePath[] | false +) => Promise; + +/** Operations exposed to the real-Obsidian contract tests. */ +export interface HiddenFileSyncTestingView extends HiddenFileSyncCommandView { + readonly conflictResolution: HiddenFileSyncConflictTestingView; + readFileWithInfo(path: FilePath): Promise; + showConfigurationChangeNotice(updatedFolders: readonly string[]): void; + interceptRebuildMerging(interceptor: HiddenFileSyncTestingRebuildInterceptor): () => void; +} + +export type HiddenFileSyncTestingViewOperations = HiddenFileSyncTestingView; + +/** + * Build the frozen testing seam. Tests can observe behaviour and install a + * scoped timing interceptor, but cannot access mutable context state. + */ +export function createHiddenFileSyncTestingView( + operations: HiddenFileSyncTestingViewOperations +): HiddenFileSyncTestingView { + const view: HiddenFileSyncTestingView = { + isManualCommandAvailable: () => operations.isManualCommandAvailable(), + scanAllStorageChanges: async (showNotice) => await operations.scanAllStorageChanges(showNotice), + scanAllDatabaseChanges: async (showNotice) => await operations.scanAllDatabaseChanges(showNotice), + applyOfflineChanges: async (showNotice) => await operations.applyOfflineChanges(showNotice), + updateSettingCache: () => operations.updateSettingCache(), + initialiseInternalFileSync: async (direction, showMessage, targetFiles) => + await operations.initialiseInternalFileSync(direction, showMessage, targetFiles), + conflictResolution: operations.conflictResolution, + readFileWithInfo: async (path) => await operations.readFileWithInfo(path), + showConfigurationChangeNotice: (updatedFolders) => + operations.showConfigurationChangeNotice(updatedFolders), + interceptRebuildMerging: (interceptor) => operations.interceptRebuildMerging(interceptor), + }; + return Object.freeze(view); +} + /** Exact-revision operations needed by the Hatch repair pane. */ export interface HiddenFileSyncRepairView { scanInternalFiles(): Promise; @@ -28,6 +120,19 @@ export interface HiddenFileSyncRepairView { ): Promise; } +export function createHiddenFileSyncRepairView(operations: HiddenFileSyncRepairView): HiddenFileSyncRepairView { + const view: HiddenFileSyncRepairView = { + scanInternalFiles: async () => await operations.scanInternalFiles(), + storeInternalFileToDatabase: async (file, forceWrite) => + await operations.storeInternalFileToDatabase(file, forceWrite), + storeInternalFileToDatabaseWithBaseRevision: async (file, baseRevision, createIfDifferent) => + await operations.storeInternalFileToDatabaseWithBaseRevision(file, baseRevision, createIfDifferent), + extractInternalFileRevisionFromDatabase: async (storageFilePath, revision, force) => + await operations.extractInternalFileRevisionFromDatabase(storageFilePath, revision, force), + }; + return Object.freeze(view); +} + /** Operations consumed by the host-owned Hidden File Sync commands. */ export interface HiddenFileSyncCommandView extends HiddenFileSyncInitialisationView { isManualCommandAvailable(): boolean; diff --git a/src/features/HiddenFileSync/hiddenFileSyncViews.unit.spec.ts b/src/features/HiddenFileSync/hiddenFileSyncViews.unit.spec.ts new file mode 100644 index 00000000..d0cc02ce --- /dev/null +++ b/src/features/HiddenFileSync/hiddenFileSyncViews.unit.spec.ts @@ -0,0 +1,132 @@ +import { describe, expect, it, vi } from "vitest"; +import type { FilePath } from "@vrtmrz/livesync-commonlib/compat/common/types"; + +import { + createHiddenFileSyncRepairView, + createHiddenFileSyncServiceHandlerView, + createHiddenFileSyncTestingView, + type HiddenFileSyncRepairView, + type HiddenFileSyncServiceHandlerView, + type HiddenFileSyncTestingViewOperations, +} from "./hiddenFileSyncViews.ts"; + +describe("Hidden File Sync repair view", () => { + it("exposes only frozen repair operations and preserves their receiver", async () => { + const path = ".obsidian/app.json" as FilePath; + const file = { path, ctime: 1, mtime: 2, size: 3 }; + const source = { + marker: "source", + scanInternalFiles: vi.fn(async function (this: { marker: string }) { + expect(this.marker).toBe("source"); + return [file]; + }), + storeInternalFileToDatabase: vi.fn(async function (this: { marker: string }) { + expect(this.marker).toBe("source"); + return true; + }), + storeInternalFileToDatabaseWithBaseRevision: vi.fn(async function (this: { marker: string }) { + expect(this.marker).toBe("source"); + return true; + }), + extractInternalFileRevisionFromDatabase: vi.fn(async function (this: { marker: string }) { + expect(this.marker).toBe("source"); + return true; + }), + } as unknown as HiddenFileSyncRepairView; + + const view = createHiddenFileSyncRepairView(source); + + expect(view).not.toBe(source); + expect(Object.isFrozen(view)).toBe(true); + expect(Object.keys(view).sort()).toEqual( + [ + "extractInternalFileRevisionFromDatabase", + "scanInternalFiles", + "storeInternalFileToDatabase", + "storeInternalFileToDatabaseWithBaseRevision", + ].sort() + ); + await expect(view.scanInternalFiles()).resolves.toEqual([file]); + await expect(view.storeInternalFileToDatabase(file)).resolves.toBe(true); + await expect(view.storeInternalFileToDatabaseWithBaseRevision(file, "2-selected", false)).resolves.toBe(true); + await expect(view.extractInternalFileRevisionFromDatabase(path, "2-selected", true)).resolves.toBe(true); + expect(source.storeInternalFileToDatabaseWithBaseRevision).toHaveBeenCalledWith(file, "2-selected", false); + }); +}); + +describe("Hidden File Sync service-handler view", () => { + it("forwards semantic callbacks through a frozen view", async () => { + const operations = { + processOptionalFileEvent: vi.fn(async () => true), + processOptionalSyncFiles: vi.fn(async () => true), + onSettingLoaded: vi.fn(async () => true), + realiseSettingSyncMode: vi.fn(async () => true), + onResuming: vi.fn(async () => true), + beforeReplicate: vi.fn(async () => true), + onDatabaseInitialised: vi.fn(async () => true), + suspendExtraSync: vi.fn(async () => true), + configureOptionalSyncFeature: vi.fn(async () => true), + isTargetFileEligible: vi.fn(async () => true), + queueConflict: vi.fn(async () => true), + } satisfies HiddenFileSyncServiceHandlerView; + + const view = createHiddenFileSyncServiceHandlerView(operations); + + expect(Object.isFrozen(view)).toBe(true); + await view.processOptionalFileEvent(".obsidian/app.json" as FilePath); + await view.processOptionalSyncFiles({} as never); + await view.onSettingLoaded(); + await view.realiseSettingSyncMode(); + await view.onResuming(); + await view.beforeReplicate(true); + await view.onDatabaseInitialised(false); + await view.suspendExtraSync(); + await view.configureOptionalSyncFeature("MERGE"); + await view.isTargetFileEligible(".obsidian/app.json" as FilePath); + await view.queueConflict("i:.obsidian/app.json" as never); + + expect(operations.processOptionalFileEvent).toHaveBeenCalledWith(".obsidian/app.json"); + expect(operations.beforeReplicate).toHaveBeenCalledWith(true); + expect(operations.onDatabaseInitialised).toHaveBeenCalledWith(false); + expect(operations.configureOptionalSyncFeature).toHaveBeenCalledWith("MERGE"); + expect(operations.queueConflict).toHaveBeenCalledWith("i:.obsidian/app.json"); + }); +}); + +describe("Hidden File Sync testing view", () => { + it("keeps test operations focused while retaining a frozen E2E surface", async () => { + const conflictResolution = { + resolveAll: vi.fn(async () => undefined), + resolveJson: vi.fn(async () => true), + pendingPaths: [], + processor: { remaining: 0, totalRemaining: 0, nowProcessing: 0 }, + }; + const restoreRebuild = vi.fn(); + const operations = { + isManualCommandAvailable: vi.fn(() => true), + scanAllStorageChanges: vi.fn(async () => undefined), + scanAllDatabaseChanges: vi.fn(async () => undefined), + applyOfflineChanges: vi.fn(async () => undefined), + updateSettingCache: vi.fn(), + initialiseInternalFileSync: vi.fn(async () => undefined), + conflictResolution, + readFileWithInfo: vi.fn(async () => ({}) as never), + showConfigurationChangeNotice: vi.fn(), + interceptRebuildMerging: vi.fn(() => restoreRebuild), + } satisfies HiddenFileSyncTestingViewOperations; + + const view = createHiddenFileSyncTestingView(operations); + + expect(Object.isFrozen(view)).toBe(true); + await view.scanAllStorageChanges(true); + await view.readFileWithInfo(".obsidian/app.json" as FilePath); + view.showConfigurationChangeNotice([".obsidian"]); + expect(operations.scanAllStorageChanges).toHaveBeenCalledWith(true); + expect(operations.readFileWithInfo).toHaveBeenCalledWith(".obsidian/app.json"); + expect(operations.showConfigurationChangeNotice).toHaveBeenCalledWith([".obsidian"]); + + const interceptor = vi.fn(async () => [] as FilePath[]); + expect(view.interceptRebuildMerging(interceptor)).toBe(restoreRebuild); + expect(operations.interceptRebuildMerging).toHaveBeenCalledWith(interceptor); + }); +}); diff --git a/src/features/optionalFileSyncFileTree.ts b/src/features/optionalFileSyncFileTree.ts new file mode 100644 index 00000000..7843d671 --- /dev/null +++ b/src/features/optionalFileSyncFileTree.ts @@ -0,0 +1,51 @@ +export type OptionalFileSyncDirectoryListing = { + files: readonly string[]; + folders: readonly string[]; +}; + +export type OptionalFileSyncFileTreeDependencies = { + listFiles(path: string): Promise; +}; + +export type OptionalFileSyncFileTreeOptions = { + maxDepth?: number; + shouldInclude?(path: string): boolean | Promise; + onError?(path: string, error: unknown): void; +}; + +export async function collectOptionalFileSyncFiles( + dependencies: OptionalFileSyncFileTreeDependencies, + path: string, + options: OptionalFileSyncFileTreeOptions = {} +): Promise { + if (options.maxDepth !== undefined && options.maxDepth < 0) { + return []; + } + + let listing: OptionalFileSyncDirectoryListing; + try { + listing = await dependencies.listFiles(path); + } catch (error) { + options.onError?.(path, error); + return []; + } + + const files: string[] = []; + for (const file of listing.files) { + if ((await options.shouldInclude?.(file)) ?? true) { + files.push(file); + } + } + for (const folder of listing.folders) { + if (!((await options.shouldInclude?.(folder)) ?? true)) { + continue; + } + files.push( + ...(await collectOptionalFileSyncFiles(dependencies, folder, { + ...options, + maxDepth: options.maxDepth === undefined ? undefined : options.maxDepth - 1, + })) + ); + } + return files; +} diff --git a/src/features/optionalFileSyncFileTree.unit.spec.ts b/src/features/optionalFileSyncFileTree.unit.spec.ts new file mode 100644 index 00000000..075168e7 --- /dev/null +++ b/src/features/optionalFileSyncFileTree.unit.spec.ts @@ -0,0 +1,48 @@ +import { describe, expect, it, vi } from "vitest"; +import { collectOptionalFileSyncFiles } from "./optionalFileSyncFileTree.ts"; + +describe("collectOptionalFileSyncFiles", () => { + it("collects files to the requested directory depth", async () => { + const listings = new Map([ + ["root", { files: ["root/file"], folders: ["root/one"] }], + ["root/one", { files: ["root/one/file"], folders: ["root/one/two"] }], + ["root/one/two", { files: ["root/one/two/file"], folders: ["root/one/two/three"] }], + ["root/one/two/three", { files: ["root/one/two/three/file"], folders: [] }], + ]); + const listFiles = vi.fn(async (path: string) => listings.get(path) ?? { files: [], folders: [] }); + + await expect(collectOptionalFileSyncFiles({ listFiles }, "root", { maxDepth: 2 })).resolves.toEqual([ + "root/file", + "root/one/file", + "root/one/two/file", + ]); + expect(listFiles).not.toHaveBeenCalledWith("root/one/two/three"); + }); + + it("uses the same asynchronous filter for files and directory traversal", async () => { + const listFiles = vi.fn(async (path: string) => + path == "root" + ? { files: ["root/include", "root/skip"], folders: ["root/allowed", "root/blocked"] } + : { files: [`${path}/include`], folders: [] } + ); + const shouldInclude = vi.fn(async (path: string) => !path.includes("skip") && !path.includes("blocked")); + + await expect(collectOptionalFileSyncFiles({ listFiles }, "root", { shouldInclude })).resolves.toEqual([ + "root/include", + "root/allowed/include", + ]); + expect(listFiles).not.toHaveBeenCalledWith("root/blocked"); + }); + + it("reports an unreadable directory and keeps the successful part of the traversal", async () => { + const failure = new Error("unreadable"); + const listFiles = vi.fn(async (path: string) => { + if (path == "root/failing") throw failure; + return { files: ["root/file"], folders: ["root/failing"] }; + }); + const onError = vi.fn(); + + await expect(collectOptionalFileSyncFiles({ listFiles }, "root", { onError })).resolves.toEqual(["root/file"]); + expect(onError).toHaveBeenCalledWith("root/failing", failure); + }); +}); diff --git a/src/serviceFeatures/hiddenFileSyncObsidianAdapter.ts b/src/serviceFeatures/hiddenFileSyncObsidianAdapter.ts index 5bf14a51..b2377faa 100644 --- a/src/serviceFeatures/hiddenFileSyncObsidianAdapter.ts +++ b/src/serviceFeatures/hiddenFileSyncObsidianAdapter.ts @@ -11,9 +11,9 @@ import { getObsidianCommunityPluginManager } from "@/common/obsidianCommunityPlu import { JsonResolveModal } from "@/features/HiddenFileCommon/JsonResolveModal.ts"; import type { HiddenFileSyncContextDependencies, - HiddenFileSyncJsonResolution, HiddenFileSyncProgress, } from "@/features/HiddenFileSync/hiddenFileSyncContext.ts"; +import type { HiddenFileSyncJsonResolution } from "@/features/HiddenFileSync/hiddenFileSyncConflictResolution.ts"; import { MARK_DONE } from "@/modules/features/ModuleLog.ts"; import type { LiveSyncCore } from "@/main.ts"; diff --git a/src/serviceFeatures/optionalFileSyncBoundary.unit.spec.ts b/src/serviceFeatures/optionalFileSyncBoundary.unit.spec.ts index 7c016c61..69b28b05 100644 --- a/src/serviceFeatures/optionalFileSyncBoundary.unit.spec.ts +++ b/src/serviceFeatures/optionalFileSyncBoundary.unit.spec.ts @@ -18,8 +18,14 @@ describe("Optional File Sync ownership boundary", () => { expect(mainSource).not.toContain("optionalFileSync.testing"); expect(featureSource).toContain("new CustomisationSyncContext"); expect(featureSource).toContain("new HiddenFileSyncContext"); + expect(featureSource).toContain("customisationSync.serviceHandlers"); + expect(featureSource).toContain("hiddenFileSync.serviceHandlers"); + expect(featureSource).toContain("customisationSync.testing"); + expect(featureSource).toContain("hiddenFileSync.testing"); expect(customisationSource).not.toContain("onBindFunction("); expect(hiddenSource).not.toContain("onBindFunction("); + expect(customisationSource).not.toMatch(/\b_(?:any|every|all)[A-Z]/); + expect(hiddenSource).not.toMatch(/\b_(?:any|every|all)[A-Z]/); expect(customisationSource).not.toContain("LiveSyncCommands"); expect(hiddenSource).not.toContain("LiveSyncCommands"); expect(customisationSource).not.toContain("extends LiveSyncContext"); diff --git a/src/serviceFeatures/optionalFileSyncRouting.ts b/src/serviceFeatures/optionalFileSyncRouting.ts index c12f60e8..be1aca7a 100644 --- a/src/serviceFeatures/optionalFileSyncRouting.ts +++ b/src/serviceFeatures/optionalFileSyncRouting.ts @@ -15,6 +15,7 @@ import { type CustomisationSyncFileCategory, type CustomisationSyncPathOptions, } from "@/features/ConfigSync/customisationSyncPaths.ts"; +import { isHiddenFileSyncPath } from "@/features/HiddenFileSync/hiddenFileSyncPathPolicy.ts"; export type OptionalFileSyncOwner = "customisation" | "hidden-file" | "none"; @@ -58,10 +59,6 @@ export type CustomisationSyncDocumentOwnershipInput = { pluginSyncExtendedSetting: Readonly>; }; -function isHiddenFileSyncPath(path: string): boolean { - return path.startsWith(".") && !path.startsWith(".trash"); -} - /** Select the sole local writer from persisted feature settings. */ export function selectOptionalFileSyncOwner( input: OptionalFileSyncOwnerSelectionInput diff --git a/src/serviceFeatures/useOptionalFileSync.ts b/src/serviceFeatures/useOptionalFileSync.ts index 99fe96a1..f59fae8e 100644 --- a/src/serviceFeatures/useOptionalFileSync.ts +++ b/src/serviceFeatures/useOptionalFileSync.ts @@ -10,12 +10,16 @@ import { CustomisationSyncContext, type CustomisationSyncContextDependencies, } from "@/features/ConfigSync/customisationSyncContext.ts"; -import type { CustomisationSyncDialogView } from "@/features/ConfigSync/customisationSyncView.ts"; +import type { + CustomisationSyncDialogView, + CustomisationSyncTestingView, +} from "@/features/ConfigSync/customisationSyncView.ts"; import { HiddenFileSyncContext } from "@/features/HiddenFileSync/hiddenFileSyncContext.ts"; import type { HiddenFileSyncCommandView, HiddenFileSyncInitialisationView, HiddenFileSyncRepairView, + HiddenFileSyncTestingView, } from "@/features/HiddenFileSync/hiddenFileSyncViews.ts"; import type { LiveSyncCore } from "@/main.ts"; import { @@ -38,10 +42,10 @@ export interface OptionalFileSyncFeature { readonly hiddenFileSyncCommands: HiddenFileSyncCommandView; readonly hiddenFileSyncInitialisation: HiddenFileSyncInitialisationView; readonly hiddenFileSyncRepair: HiddenFileSyncRepairView; - /** @internal Direct runtime access for the repository's real-Obsidian contract tests. */ + /** @internal Focused operations for the repository's real-Obsidian contract tests. */ readonly testing: { - readonly customisationSync: CustomisationSyncContext; - readonly hiddenFileSync: HiddenFileSyncContext; + readonly customisationSync: CustomisationSyncTestingView; + readonly hiddenFileSync: HiddenFileSyncTestingView; }; } @@ -96,6 +100,9 @@ export function useOptionalFileSync( const hiddenFileSync = createHiddenFileSync({ ownsLocalFile: ownsLocalFile("hidden-file"), }); + const customisationHandlers = customisationSync.serviceHandlers; + const hiddenFileHandlers = hiddenFileSync.serviceHandlers; + const hiddenFileSyncRepair = hiddenFileSync.repair; const { services } = host; const disposers: (() => void)[] = []; @@ -108,7 +115,7 @@ export function useOptionalFileSync( const routeLocalPath = async (path: FilePath) => { const selected = selectOptionalFileSyncOwner(ownerSelectionInput(path)); const hiddenFileEligible = - selected.owner == "hidden-file" ? await hiddenFileSync.isTargetFileEligible(path) : false; + selected.owner == "hidden-file" ? await hiddenFileHandlers.isTargetFileEligible(path) : false; const ready = services.appLifecycle.isReady() && !services.appLifecycle.isSuspended(); return routeOptionalFileSyncPath({ ...ownerSelectionInput(path), @@ -123,86 +130,44 @@ export function useOptionalFileSync( const localPath = normaliseLocalPath(path); const decision = await routeLocalPath(localPath); if (decision.owner == "customisation") { - return await customisationSync._anyProcessOptionalFileEvent(localPath); + return await customisationHandlers.processOptionalFileEvent(localPath); } if (decision.owner == "hidden-file") { - return await hiddenFileSync._anyProcessOptionalFileEvent(localPath); + return await hiddenFileHandlers.processOptionalFileEvent(localPath); } return false; }) ); register( - services.conflict.getOptionalConflictCheckMethod.addHandler(async (path: FilePathWithPrefix) => { + services.conflict.getOptionalConflictCheckMethod.addHandler((path: FilePathWithPrefix) => { if (isPluginMetadata(path) || isCustomisationSyncMetadata(path)) { - return await customisationSync._anyGetOptionalConflictCheckMethod(path); + return Promise.resolve("newer"); } if (isInternalMetadata(path)) { - return await hiddenFileSync._anyGetOptionalConflictCheckMethod(path); + return hiddenFileHandlers.queueConflict(path); } - return false; + return Promise.resolve(false); }) ); + register(services.replication.processVirtualDocument.addHandler(customisationHandlers.processVirtualDocument)); register( - services.replication.processVirtualDocument.addHandler( - customisationSync._anyModuleParsedReplicationResultItem.bind(customisationSync) - ) - ); - register( - services.replication.processOptionalSynchroniseResult.addHandler( - hiddenFileSync._anyProcessOptionalSyncFiles.bind(hiddenFileSync) - ) - ); - register( - services.appLifecycle.onSettingLoaded.addHandler( - hiddenFileSync._everyOnloadAfterLoadSettings.bind(hiddenFileSync) - ) + services.replication.processOptionalSynchroniseResult.addHandler(hiddenFileHandlers.processOptionalSyncFiles) ); + register(services.appLifecycle.onSettingLoaded.addHandler(hiddenFileHandlers.onSettingLoaded)); - register( - services.setting.onRealiseSetting.addHandler( - customisationSync._everyRealizeSettingSyncMode.bind(customisationSync) - ) - ); - register( - services.setting.onRealiseSetting.addHandler(hiddenFileSync._everyRealizeSettingSyncMode.bind(hiddenFileSync)) - ); - register( - services.appLifecycle.onResuming.addHandler(customisationSync._everyOnResumeProcess.bind(customisationSync)) - ); - register(services.appLifecycle.onResuming.addHandler(hiddenFileSync._everyOnResumeProcess.bind(hiddenFileSync))); - register( - services.replication.onBeforeReplicate.addHandler( - customisationSync._everyBeforeReplicate.bind(customisationSync) - ) - ); - register( - services.replication.onBeforeReplicate.addHandler(hiddenFileSync._everyBeforeReplicate.bind(hiddenFileSync)) - ); - register( - services.databaseEvents.onDatabaseInitialised.addHandler( - customisationSync._everyOnDatabaseInitialized.bind(customisationSync) - ) - ); - register( - services.databaseEvents.onDatabaseInitialised.addHandler( - hiddenFileSync._everyOnDatabaseInitialized.bind(hiddenFileSync) - ) - ); - register( - services.setting.suspendExtraSync.addHandler(customisationSync._allSuspendExtraSync.bind(customisationSync)) - ); - register(services.setting.suspendExtraSync.addHandler(hiddenFileSync._allSuspendExtraSync.bind(hiddenFileSync))); - register( - services.setting.enableOptionalFeature.addHandler( - customisationSync._allConfigureOptionalSyncFeature.bind(customisationSync) - ) - ); - register( - services.setting.enableOptionalFeature.addHandler( - hiddenFileSync._allConfigureOptionalSyncFeature.bind(hiddenFileSync) - ) - ); + register(services.setting.onRealiseSetting.addHandler(customisationHandlers.onRealiseSetting)); + register(services.setting.onRealiseSetting.addHandler(hiddenFileHandlers.realiseSettingSyncMode)); + register(services.appLifecycle.onResuming.addHandler(customisationHandlers.onResuming)); + register(services.appLifecycle.onResuming.addHandler(hiddenFileHandlers.onResuming)); + register(services.replication.onBeforeReplicate.addHandler(customisationHandlers.onBeforeReplicate)); + register(services.replication.onBeforeReplicate.addHandler(hiddenFileHandlers.beforeReplicate)); + register(services.databaseEvents.onDatabaseInitialised.addHandler(customisationHandlers.onDatabaseInitialised)); + register(services.databaseEvents.onDatabaseInitialised.addHandler(hiddenFileHandlers.onDatabaseInitialised)); + register(services.setting.suspendExtraSync.addHandler(customisationHandlers.suspendExtraSync)); + register(services.setting.suspendExtraSync.addHandler(hiddenFileHandlers.suspendExtraSync)); + register(services.setting.enableOptionalFeature.addHandler(customisationHandlers.enableOptionalFeature)); + register(services.setting.enableOptionalFeature.addHandler(hiddenFileHandlers.configureOptionalSyncFeature)); register( services.vault.isTargetFileInExtra.addHandler( async (file: string | UXFileInfoStub) => (await routeLocalPath(normaliseLocalPath(file))).owner != "none" @@ -234,7 +199,10 @@ export function useOptionalFileSync( customisationSync, hiddenFileSyncCommands: hiddenFileSync, hiddenFileSyncInitialisation: hiddenFileSync, - hiddenFileSyncRepair: hiddenFileSync, - testing: Object.freeze({ customisationSync, hiddenFileSync }), + hiddenFileSyncRepair, + testing: Object.freeze({ + customisationSync: customisationSync.testing, + hiddenFileSync: hiddenFileSync.testing, + }), }); } diff --git a/src/serviceFeatures/useOptionalFileSync.unit.spec.ts b/src/serviceFeatures/useOptionalFileSync.unit.spec.ts index ee1ef811..5a65ee0b 100644 --- a/src/serviceFeatures/useOptionalFileSync.unit.spec.ts +++ b/src/serviceFeatures/useOptionalFileSync.unit.spec.ts @@ -38,6 +38,15 @@ function handlerRegistry() { }; } +function hiddenFileSyncRepairFixture() { + return Object.freeze({ + scanInternalFiles: vi.fn(async () => []), + storeInternalFileToDatabase: vi.fn(async () => true), + storeInternalFileToDatabaseWithBaseRevision: vi.fn(async () => true), + extractInternalFileRevisionFromDatabase: vi.fn(async () => true), + }); +} + function createFixture() { const processOptionalFileEvent = handlerRegistry(); const getOptionalConflictCheckMethod = handlerRegistry(); @@ -54,31 +63,42 @@ function createFixture() { const onUnload = handlerRegistry(); const calls: string[] = []; + const customisationHandlers = { + processOptionalFileEvent: vi.fn(async () => false), + processVirtualDocument: vi.fn(async () => false), + onRealiseSetting: vi.fn(async () => true), + onResuming: vi.fn(async () => true), + onBeforeReplicate: vi.fn(async () => true), + onDatabaseInitialised: vi.fn(async () => true), + suspendExtraSync: vi.fn(async () => true), + enableOptionalFeature: vi.fn(async () => true), + }; + const customisationTesting = Object.freeze({ kind: "customisation-testing" }); const customisationSync = { dispose: vi.fn(() => calls.push("customisation:unload")), - _anyProcessOptionalFileEvent: vi.fn(async () => false), - _anyGetOptionalConflictCheckMethod: vi.fn(async () => false), - _anyModuleParsedReplicationResultItem: vi.fn(async () => false), - _everyRealizeSettingSyncMode: vi.fn(async () => true), - _everyOnResumeProcess: vi.fn(async () => true), - _everyBeforeReplicate: vi.fn(async () => true), - _everyOnDatabaseInitialized: vi.fn(async () => true), - _allSuspendExtraSync: vi.fn(async () => true), - _allConfigureOptionalSyncFeature: vi.fn(async () => true), + serviceHandlers: Object.freeze(customisationHandlers), + testing: customisationTesting, }; + const hiddenFileHandlers = { + onSettingLoaded: vi.fn(async () => true), + processOptionalFileEvent: vi.fn(async () => false), + queueConflict: vi.fn(async () => true), + processOptionalSyncFiles: vi.fn(async () => false), + realiseSettingSyncMode: vi.fn(async () => true), + onResuming: vi.fn(async () => true), + beforeReplicate: vi.fn(async () => true), + onDatabaseInitialised: vi.fn(async () => true), + suspendExtraSync: vi.fn(async () => true), + configureOptionalSyncFeature: vi.fn(async () => true), + isTargetFileEligible: vi.fn(async () => true), + }; + const hiddenFileSyncRepair = hiddenFileSyncRepairFixture(); + const hiddenFileTesting = Object.freeze({ kind: "hidden-file-testing" }); const hiddenFileSync = { dispose: vi.fn(() => calls.push("hidden:unload")), - _everyOnloadAfterLoadSettings: vi.fn(async () => true), - _anyProcessOptionalFileEvent: vi.fn(async () => false), - _anyGetOptionalConflictCheckMethod: vi.fn(async () => false), - _anyProcessOptionalSyncFiles: vi.fn(async () => false), - _everyRealizeSettingSyncMode: vi.fn(async () => true), - _everyOnResumeProcess: vi.fn(async () => true), - _everyBeforeReplicate: vi.fn(async () => true), - _everyOnDatabaseInitialized: vi.fn(async () => true), - _allSuspendExtraSync: vi.fn(async () => true), - _allConfigureOptionalSyncFeature: vi.fn(async () => true), - isTargetFileEligible: vi.fn(async () => true), + serviceHandlers: Object.freeze(hiddenFileHandlers), + testing: hiddenFileTesting, + repair: hiddenFileSyncRepair, }; const settings = { usePluginSync: true, @@ -121,9 +141,13 @@ function createFixture() { return { calls, + customisationHandlers, customisationSync, + customisationTesting, feature, + hiddenFileHandlers, hiddenFileSync, + hiddenFileTesting, settings, contextDependencies: { customisation: () => customisationDependencies, @@ -148,31 +172,39 @@ function createFixture() { } function createAggregateFixture() { + const customisationHandlers = { + processOptionalFileEvent: vi.fn(async () => false), + processVirtualDocument: vi.fn(async () => false), + onRealiseSetting: vi.fn(async () => true), + onResuming: vi.fn(async () => true), + onBeforeReplicate: vi.fn(async () => true), + onDatabaseInitialised: vi.fn(async () => true), + suspendExtraSync: vi.fn(async () => true), + enableOptionalFeature: vi.fn(async () => true), + }; const customisationSync = { dispose: vi.fn(), - _anyProcessOptionalFileEvent: vi.fn(async () => false), - _anyGetOptionalConflictCheckMethod: vi.fn(async (): Promise => false), - _anyModuleParsedReplicationResultItem: vi.fn(async () => false), - _everyRealizeSettingSyncMode: vi.fn(async () => true), - _everyOnResumeProcess: vi.fn(async () => true), - _everyBeforeReplicate: vi.fn(async () => true), - _everyOnDatabaseInitialized: vi.fn(async () => true), - _allSuspendExtraSync: vi.fn(async () => true), - _allConfigureOptionalSyncFeature: vi.fn(async () => true), + serviceHandlers: Object.freeze(customisationHandlers), + testing: Object.freeze({ kind: "customisation-testing" }), + }; + const hiddenFileHandlers = { + onSettingLoaded: vi.fn(async () => true), + processOptionalFileEvent: vi.fn(async () => false), + queueConflict: vi.fn(async () => true), + processOptionalSyncFiles: vi.fn(async () => false), + realiseSettingSyncMode: vi.fn(async () => true), + onResuming: vi.fn(async () => true), + beforeReplicate: vi.fn(async () => true), + onDatabaseInitialised: vi.fn(async () => true), + suspendExtraSync: vi.fn(async () => true), + configureOptionalSyncFeature: vi.fn(async () => true), + isTargetFileEligible: vi.fn(async () => true), }; const hiddenFileSync = { dispose: vi.fn(), - _everyOnloadAfterLoadSettings: vi.fn(async () => true), - _anyProcessOptionalFileEvent: vi.fn(async () => false), - _anyGetOptionalConflictCheckMethod: vi.fn(async (): Promise => false), - _anyProcessOptionalSyncFiles: vi.fn(async () => false), - _everyRealizeSettingSyncMode: vi.fn(async () => true), - _everyOnResumeProcess: vi.fn(async () => true), - _everyBeforeReplicate: vi.fn(async () => true), - _everyOnDatabaseInitialized: vi.fn(async () => true), - _allSuspendExtraSync: vi.fn(async () => true), - _allConfigureOptionalSyncFeature: vi.fn(async () => true), - isTargetFileEligible: vi.fn(async () => true), + serviceHandlers: Object.freeze(hiddenFileHandlers), + testing: Object.freeze({ kind: "hidden-file-testing" }), + repair: hiddenFileSyncRepairFixture(), }; const settings = { usePluginSync: true, @@ -217,7 +249,7 @@ function createAggregateFixture() { createHiddenFileSync: () => hiddenFileSync as never, }); - return { customisationSync, hiddenFileSync, services, settings }; + return { customisationHandlers, customisationSync, hiddenFileHandlers, hiddenFileSync, services, settings }; } describe("useOptionalFileSync", () => { @@ -239,10 +271,10 @@ describe("useOptionalFileSync", () => { }); it("routes Selective and Automatic paths to exactly one local owner", async () => { - const { customisationSync, hiddenFileSync, registries, settings } = createFixture(); - customisationSync._anyProcessOptionalFileEvent.mockResolvedValue(true); - hiddenFileSync._anyProcessOptionalFileEvent.mockResolvedValue(true); - hiddenFileSync.isTargetFileEligible.mockResolvedValue(false); + const { customisationHandlers, hiddenFileHandlers, registries, settings } = createFixture(); + customisationHandlers.processOptionalFileEvent.mockResolvedValue(true); + hiddenFileHandlers.processOptionalFileEvent.mockResolvedValue(true); + hiddenFileHandlers.isTargetFileEligible.mockResolvedValue(false); await expect(registries.isTargetFileInExtra.handlers[0]!(".obsidian/plugins/example/data.json")).resolves.toBe( true @@ -250,9 +282,9 @@ describe("useOptionalFileSync", () => { await expect( registries.processOptionalFileEvent.handlers[0]!(".obsidian/plugins/example/data.json") ).resolves.toBe(true); - expect(customisationSync._anyProcessOptionalFileEvent).toHaveBeenCalledOnce(); - expect(hiddenFileSync._anyProcessOptionalFileEvent).not.toHaveBeenCalled(); - expect(hiddenFileSync.isTargetFileEligible).not.toHaveBeenCalled(); + expect(customisationHandlers.processOptionalFileEvent).toHaveBeenCalledOnce(); + expect(hiddenFileHandlers.processOptionalFileEvent).not.toHaveBeenCalled(); + expect(hiddenFileHandlers.isTargetFileEligible).not.toHaveBeenCalled(); settings.pluginSyncExtendedSetting = { "PLUGIN_DATA/example": { @@ -261,53 +293,53 @@ describe("useOptionalFileSync", () => { files: [], }, }; - hiddenFileSync.isTargetFileEligible.mockResolvedValue(true); - customisationSync._anyProcessOptionalFileEvent.mockClear(); + hiddenFileHandlers.isTargetFileEligible.mockResolvedValue(true); + customisationHandlers.processOptionalFileEvent.mockClear(); await expect( registries.processOptionalFileEvent.handlers[0]!(".obsidian/plugins/example/data.json") ).resolves.toBe(true); - expect(hiddenFileSync._anyProcessOptionalFileEvent).toHaveBeenCalledOnce(); - expect(customisationSync._anyProcessOptionalFileEvent).not.toHaveBeenCalled(); + expect(hiddenFileHandlers.processOptionalFileEvent).toHaveBeenCalledOnce(); + expect(customisationHandlers.processOptionalFileEvent).not.toHaveBeenCalled(); }); it("does not fall back to the other context when the selected owner skips or fails", async () => { - const { customisationSync, hiddenFileSync, services } = createAggregateFixture(); - customisationSync._anyProcessOptionalFileEvent.mockResolvedValueOnce(false); + const { customisationHandlers, hiddenFileHandlers, services } = createAggregateFixture(); + customisationHandlers.processOptionalFileEvent.mockResolvedValueOnce(false); await expect(services.fileProcessing.processOptionalFileEvent(".obsidian/app.json")).resolves.toBe(false); - expect(hiddenFileSync._anyProcessOptionalFileEvent).not.toHaveBeenCalled(); + expect(hiddenFileHandlers.processOptionalFileEvent).not.toHaveBeenCalled(); - customisationSync._anyProcessOptionalFileEvent.mockRejectedValueOnce(new Error("customisation failed")); + customisationHandlers.processOptionalFileEvent.mockRejectedValueOnce(new Error("customisation failed")); await expect(services.fileProcessing.processOptionalFileEvent(".obsidian/app.json")).resolves.toBe(false); - expect(hiddenFileSync._anyProcessOptionalFileEvent).not.toHaveBeenCalled(); + expect(hiddenFileHandlers.processOptionalFileEvent).not.toHaveBeenCalled(); }); it("dispatches conflict documents by their persisted namespace", async () => { - const { customisationSync, hiddenFileSync, services } = createAggregateFixture(); - customisationSync._anyGetOptionalConflictCheckMethod.mockResolvedValue("newer"); - hiddenFileSync._anyGetOptionalConflictCheckMethod.mockResolvedValue(true); + const { hiddenFileHandlers, services } = createAggregateFixture(); await expect(services.conflict.getOptionalConflictCheckMethod("ix:device/app.json")).resolves.toBe("newer"); - expect(hiddenFileSync._anyGetOptionalConflictCheckMethod).not.toHaveBeenCalled(); + expect(hiddenFileHandlers.queueConflict).not.toHaveBeenCalled(); await expect(services.conflict.getOptionalConflictCheckMethod("i:.obsidian/example.json")).resolves.toBe(true); - expect(customisationSync._anyGetOptionalConflictCheckMethod).toHaveBeenCalledOnce(); + expect(hiddenFileHandlers.queueConflict).toHaveBeenCalledOnce(); + expect(hiddenFileHandlers.queueConflict).toHaveBeenCalledWith("i:.obsidian/example.json"); + await expect(services.conflict.getOptionalConflictCheckMethod("notes/example.md")).resolves.toBe(false); }); it("keeps persisted document acceptance separate from current local ownership", async () => { - const { customisationSync, hiddenFileSync, registries, settings } = createFixture(); + const { customisationHandlers, hiddenFileHandlers, registries, settings } = createFixture(); settings.usePluginSync = false; settings.syncInternalFiles = false; await registries.processVirtualDocument.handlers[0]!({ _id: "ix:device-a/CONFIG/app.json.md" }); await registries.processOptionalSynchroniseResult.handlers[0]!({ _id: "i:.obsidian/app.json" }); - expect(customisationSync._anyModuleParsedReplicationResultItem).toHaveBeenCalledOnce(); - expect(hiddenFileSync._anyProcessOptionalSyncFiles).toHaveBeenCalledOnce(); + expect(customisationHandlers.processVirtualDocument).toHaveBeenCalledOnce(); + expect(hiddenFileHandlers.processOptionalSyncFiles).toHaveBeenCalledOnce(); }); it("routes Ignore mode to neither local context", async () => { - const { customisationSync, hiddenFileSync, registries, settings } = createFixture(); + const { customisationHandlers, hiddenFileHandlers, registries, settings } = createFixture(); settings.pluginSyncExtendedSetting = { "PLUGIN_DATA/example": { key: "PLUGIN_DATA/example", @@ -319,8 +351,8 @@ describe("useOptionalFileSync", () => { await expect( registries.processOptionalFileEvent.handlers[0]!(".obsidian/plugins/example/data.json") ).resolves.toBe(false); - expect(customisationSync._anyProcessOptionalFileEvent).not.toHaveBeenCalled(); - expect(hiddenFileSync._anyProcessOptionalFileEvent).not.toHaveBeenCalled(); + expect(customisationHandlers.processOptionalFileEvent).not.toHaveBeenCalled(); + expect(hiddenFileHandlers.processOptionalFileEvent).not.toHaveBeenCalled(); }); it("injects the same static ownership policy into both scan contexts", () => { @@ -351,11 +383,12 @@ describe("useOptionalFileSync", () => { }); it("preserves bail-first-failure and settled-unload behaviour", async () => { - const { customisationSync, hiddenFileSync, services } = createAggregateFixture(); - customisationSync._everyRealizeSettingSyncMode.mockResolvedValueOnce(false); + const { customisationHandlers, customisationSync, hiddenFileHandlers, hiddenFileSync, services } = + createAggregateFixture(); + customisationHandlers.onRealiseSetting.mockResolvedValueOnce(false); await expect(services.setting.onRealiseSetting()).resolves.toBe(false); - expect(hiddenFileSync._everyRealizeSettingSyncMode).not.toHaveBeenCalled(); + expect(hiddenFileHandlers.realiseSettingSyncMode).not.toHaveBeenCalled(); customisationSync.dispose.mockImplementationOnce(() => { throw new Error("customisation disposal failed"); @@ -380,21 +413,54 @@ describe("useOptionalFileSync", () => { }); it("strips a database prefix before evaluating a Hidden File Sync target", async () => { - const { hiddenFileSync, registries } = createFixture(); + const { hiddenFileHandlers, registries } = createFixture(); await registries.isTargetFileInExtra.handlers[0]!({ path: "i:.obsidian/workspace" }); - expect(hiddenFileSync.isTargetFileEligible).toHaveBeenCalledWith(".obsidian/workspace"); + expect(hiddenFileHandlers.isTargetFileEligible).toHaveBeenCalledWith(".obsidian/workspace"); }); - it("returns focused views without registering either context as an add-on", () => { - const { customisationSync, feature, hiddenFileSync } = createFixture(); + it("returns a concrete repair adapter instead of exposing the broad context to UI", async () => { + const { + customisationSync, + customisationTesting, + feature, + hiddenFileSync, + hiddenFileTesting, + } = createFixture(); expect(feature.customisationSync).toBe(customisationSync); expect(feature.hiddenFileSyncCommands).toBe(hiddenFileSync); expect(feature.hiddenFileSyncInitialisation).toBe(hiddenFileSync); - expect(feature.hiddenFileSyncRepair).toBe(hiddenFileSync); - expect(feature.testing).toEqual({ customisationSync, hiddenFileSync }); + expect(feature.hiddenFileSyncRepair).toBe(hiddenFileSync.repair); + expect(feature.hiddenFileSyncRepair).not.toBe(hiddenFileSync); + expect(Object.isFrozen(feature.hiddenFileSyncRepair)).toBe(true); + expect(Object.keys(feature.hiddenFileSyncRepair).sort()).toEqual( + [ + "extractInternalFileRevisionFromDatabase", + "scanInternalFiles", + "storeInternalFileToDatabase", + "storeInternalFileToDatabaseWithBaseRevision", + ].sort() + ); + const file = { + path: ".obsidian/app.json", + ctime: 1, + mtime: 2, + size: 3, + } as never; + await feature.hiddenFileSyncRepair.storeInternalFileToDatabaseWithBaseRevision(file, "2-selected", false); + expect(hiddenFileSync.repair.storeInternalFileToDatabaseWithBaseRevision).toHaveBeenCalledWith( + file, + "2-selected", + false + ); + expect(feature.testing).toEqual({ + customisationSync: customisationTesting, + hiddenFileSync: hiddenFileTesting, + }); expect(Object.isFrozen(feature.testing)).toBe(true); + expect(feature.testing.customisationSync).not.toBe(customisationSync); + expect(feature.testing.hiddenFileSync).not.toBe(hiddenFileSync); }); }); diff --git a/test/e2e-obsidian/scripts/hidden-file-snippet-sync.ts b/test/e2e-obsidian/scripts/hidden-file-snippet-sync.ts index ceec16db..ba31bf49 100644 --- a/test/e2e-obsidian/scripts/hidden-file-snippet-sync.ts +++ b/test/e2e-obsidian/scripts/hidden-file-snippet-sync.ts @@ -180,7 +180,7 @@ async function resolveHiddenConflicts(cliBinary: string, env: NodeJS.ProcessEnv) "(async()=>{", "const core=app.plugins.plugins['obsidian-livesync'].core;", "const syncContext=app.plugins.plugins['obsidian-livesync'].optionalFileSync.testing.hiddenFileSync;", - "await syncContext.resolveConflictOnInternalFiles();", + "await syncContext.conflictResolution.resolveAll();", "await syncContext.scanAllDatabaseChanges(true);", "return JSON.stringify({ok:true});", "})()", @@ -192,40 +192,46 @@ async function resolveHiddenConflicts(cliBinary: string, env: NodeJS.ProcessEnv) async function autoMergeHiddenJsonConflict(cliBinary: string, env: NodeJS.ProcessEnv, path: string): Promise { await evalObsidianJson( + cliBinary, + [ + "(async()=>{", + `const path=${JSON.stringify(path)};`, + "const syncContext=app.plugins.plugins['obsidian-livesync'].optionalFileSync.testing.hiddenFileSync;", + "await syncContext.conflictResolution.resolveAll();", + "await syncContext.scanAllDatabaseChanges(true);", + "return JSON.stringify({ok:true,path});", + "})()", + ].join(""), + env + ); +} + +async function readHiddenConflictDiagnostics( + cliBinary: string, + env: NodeJS.ProcessEnv, + path: string +): Promise { + return await evalObsidianJson( cliBinary, [ "(async()=>{", `const path=${JSON.stringify(path)};`, "const prefixedPath=`i:${path}`;", "const core=app.plugins.plugins['obsidian-livesync'].core;", - "const syncContext=app.plugins.plugins['obsidian-livesync'].optionalFileSync.testing.hiddenFileSync;", - "let doc=false;", + "const owner=app.plugins.plugins['obsidian-livesync'].optionalFileSync.testing.hiddenFileSync.conflictResolution;", + "const entries=[];", "for await (const entry of core.localDatabase.findEntries('i:','i;',{conflicts:true})){", - " if(entry.path===prefixedPath){ doc=entry; break; }", + " if(entry.path===prefixedPath){", + " entries.push({id:entry._id,path:entry.path,rev:entry._rev,conflicts:entry._conflicts});", + " }", "}", - "if(!doc) throw new Error(`Could not find hidden conflict candidate: ${path}`);", - "if(!doc._conflicts?.length) throw new Error(`Hidden file has no conflicts: ${path}`);", - "const conflicts=doc._conflicts.sort((a,b)=>Number(a.split('-')[0])-Number(b.split('-')[0]));", - "const conflictedRev=conflicts[0];", - "const conflictedRevNo=Number(conflictedRev.split('-')[0]);", - "const revFrom=await core.localDatabase.getRaw(doc._id,{revs_info:true});", - "const commonBase=(revFrom._revs_info||[])", - " .filter((rev)=>rev.status==='available'&&Number(rev.rev.split('-')[0])rev.rev)[0]||'';", - "const result=await core.localDatabase.managers.conflictManager.mergeObject(", - " doc.path, commonBase, doc._rev, conflictedRev", - ");", - "if(!result){", - " throw new Error(`Hidden JSON conflict was not auto-mergeable: ${path}; base=${commonBase}; current=${doc._rev}; conflict=${conflictedRev}`);", - "}", - "await syncContext.ensureDir(path);", - "const stat=await syncContext.writeFile(path,result);", - "if(!stat) throw new Error(`Could not write merged hidden file: ${path}`);", - "await syncContext.storeInternalFileToDatabase({path,mtime:stat.mtime,ctime:stat.ctime,size:stat.size},true);", - "await core.localDatabase.removeRevision(doc._id,conflictedRev);", - "await syncContext.extractInternalFileFromDatabase(path);", - "await syncContext.scanAllDatabaseChanges(true);", - "return JSON.stringify({ok:true,merged:JSON.parse(result)});", + "const modals=Array.from(document.querySelectorAll('.modal-container')).map((element)=>element.textContent?.trim());", + "return JSON.stringify({", + " entries,", + " pendingPaths:Array.from(owner.pendingPaths??[]),", + " processor:{remaining:owner.processor?.remaining,totalRemaining:owner.processor?.totalRemaining,nowProcessing:owner.processor?.nowProcessing},", + " modals,", + "});", "})()", ].join(""), env @@ -250,7 +256,7 @@ async function openHiddenJsonResolveModal(cliBinary: string, env: NodeJS.Process "const docA=await core.localDatabase.getDBEntry(prefixedPath,{rev:doc._rev});", "const docB=await core.localDatabase.getDBEntry(prefixedPath,{rev:conflicts[0]});", "if(docA===false||docB===false) throw new Error(`Could not load conflicted hidden JSON entries: ${path}`);", - "void syncContext.showJSONMergeDialogAndMerge(docA,docB);", + "void syncContext.conflictResolution.resolveJson(docA,docB);", "return JSON.stringify({ok:true});", "})()", ].join(""), @@ -270,11 +276,12 @@ async function storeHiddenFileAsConflict( "(async()=>{", `const path=${JSON.stringify(path)};`, `const baseRev=${JSON.stringify(baseRev)};`, + "const prefixedPath=`i:${path}`;", "const core=app.plugins.plugins['obsidian-livesync'].core;", "const syncContext=app.plugins.plugins['obsidian-livesync'].optionalFileSync.testing.hiddenFileSync;", - "const fileInfo=await syncContext.loadFileWithInfo(path);", + "const fileInfo=await syncContext.readFileWithInfo(path);", "if(fileInfo.deleted) throw new Error(`Hidden file was unexpectedly deleted: ${path}`);", - "const baseData=await syncContext.__loadBaseSaveData(path,true);", + "const baseData=await core.localDatabase.getDBEntry(prefixedPath,undefined,false,true);", "if(baseData===false) throw new Error(`Could not load base save data: ${path}`);", "const saveData={", " ...baseData,", @@ -430,9 +437,21 @@ async function runJsonConflictRoundTrip( await createHiddenJsonConflict(context, session, vaultB, mergeJsonPath, base, left, right); await autoMergeHiddenJsonConflict(context.cliBinary, session.cliEnv, mergeJsonPath); await pushLocalChanges(context.cliBinary, session.cliEnv); - const mergedOnB = await waitForPathContent(vaultB.path, mergeJsonPath, (content) => - hasJsonValues(content, { fromA: true, fromB: true }) - ); + let mergedOnB: string; + try { + mergedOnB = await waitForPathContent(vaultB.path, mergeJsonPath, (content) => + hasJsonValues(content, { fromA: true, fromB: true }) + ); + } catch (error) { + const diagnostics = await readHiddenConflictDiagnostics(context.cliBinary, session.cliEnv, mergeJsonPath).catch( + (diagnosticError: unknown) => ({ + diagnosticError: diagnosticError instanceof Error ? diagnosticError.message : String(diagnosticError), + }) + ); + throw new Error( + `${error instanceof Error ? error.message : String(error)}\nConflict diagnostics: ${JSON.stringify(diagnostics)}` + ); + } await session.app.stop(); session = await startConfiguredSession(context, vaultA); @@ -620,14 +639,11 @@ async function setHiddenFileNoticeFixtures(port: number, itemIds: string[], incl }; obsidianApp.plugins.enabledPlugins.add(pluginId); } - syncContext.queuedNotificationFiles.clear(); - for (const id of nextItemIds) { - syncContext.queuedNotificationFiles.add(`.obsidian/plugins/livesync-e2e-${id}`); - } + const updatedFolders = nextItemIds.map((id) => `.obsidian/plugins/livesync-e2e-${id}`); if (nextIncludeRestart) { - syncContext.queuedNotificationFiles.add(core.services.API.getSystemConfigDir()); + updatedFolders.push(core.services.API.getSystemConfigDir()); } - syncContext.notifyConfigChange(); + syncContext.showConfigurationChangeNotice(updatedFolders); }, { nextItemIds: itemIds, nextIncludeRestart: includeRestart } ); @@ -683,7 +699,6 @@ async function runInitialisationNoticeGrouping(context: RunnerContext, vault: Te const syncContext = plugin.optionalFileSync.testing.hiddenFileSync; const setting = core.services.setting; const originalApplyPartial = setting.applyPartial; - const originalRebuildMerging = syncContext.rebuildMerging; const state = { done: false, reachedPreparation: false, @@ -731,13 +746,15 @@ async function runInitialisationNoticeGrouping(context: RunnerContext, vault: Te return await originalApplyPartial.apply(setting, args); }; - syncContext.rebuildMerging = async (...args: unknown[]) => { - state.reachedInitialisation = true; - await new Promise((resolve) => { - state.releaseInitialisation = resolve; - }); - return await originalRebuildMerging.apply(syncContext, args); - }; + const restoreRebuildMerging = syncContext.interceptRebuildMerging( + async (runRebuild: (...args: unknown[]) => Promise, ...args: unknown[]) => { + state.reachedInitialisation = true; + await new Promise((resolve) => { + state.releaseInitialisation = resolve; + }); + return await runRebuild(...args); + } + ); void core.services.setting .enableOptionalFeature("MERGE") @@ -752,7 +769,7 @@ async function runInitialisationNoticeGrouping(context: RunnerContext, vault: Te ) .finally(() => { setting.applyPartial = originalApplyPartial; - syncContext.rebuildMerging = originalRebuildMerging; + restoreRebuildMerging(); const notices = Array.from(document.querySelectorAll(".notice")); const progressNotices = notices.filter((notice) => notice.textContent?.includes("[⚙")); state.sawStandaloneGatheringNotice ||= notices.some((notice) =>