public/obsidian-livesync
1
0 Fork 0
mirror of https://github.com/vrtmrz/obsidian-livesync.git synced 2026-10-24 18:13:20 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #1282 from vrtmrz/refactor/device-values-commonlib-0.2

Browse Source 
Adopt Commonlib 0.2.0 with device review and synchronisation updates
This commit is contained in:
vorotamoroz authored and GitHub committed 2026-10-24 05:46:03 +09:00
2 parents ce0524d49f 916784c5c7
commit 2468d07f61
99 files changed
+5967 -643
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

devs.md
+13
View File
@@ -166,6 +166,14 @@ Legacy Modules remain grouped by directory:
- `features/` contains optional features; and
- `extras/` contains development and testing tools.
### UI commands and dialogue ownership
Declare shared UI requests with `declareUICommand<Request, Result>` from `@vrtmrz/livesync-commonlib/ui-commands`. The host's `services.UI.commands` registry owns one handler per declaration. A serviceFeature registers the handler, releases it on unload, and owns the associated dialogue lifetime. Callers receive a typed result or an explicit `unavailable` result; handler failures propagate.
`OPEN_JOINED_DEVICES` is shared by Remote Configuration, version Notices, and the command palette. `useJoinedDevicesFeature` registers it once and gives the dialogue narrow refresh, forget, and cancellation callbacks. Each remote operation uses a settings snapshot and releases its finite resource in `finally`. Results from an old remote configuration or a closed dialogue are discarded. A write already sent to the captured remote can still complete after the dialogue closes.
The registry belongs to one UIService instance. Reuse the same declaration object at each call site, and keep active UI cleanup in the feature and dialogue owner. `UIService` disposes the registry when the host's context emits the plug-in-unloaded event.
### Key Architectural Components
- **LiveSyncLocalDB** (`@vrtmrz/livesync-commonlib/compat/pouchdb/LiveSyncLocalDB`): Local PouchDB database wrapper
@@ -204,6 +212,9 @@ The [multiple-device conflict test procedure](test/README.md#multiple-device-con
### File Structure Conventions
- **Class modules**: Start a new or renamed TypeScript or JavaScript filename with an upper-case letter when the file contains a class with the same name.
- **Function modules**: Start the filename with a lower-case letter when the module primarily exports functions, such as `initialFetchFastForward.ts`. Follow the surrounding filename style for the rest of the basename. Corresponding test files use the same basename.
- **Existing files**: Retain established names for unrelated files and separate conventions for components or other file types.
- **Platform-specific code**: Use `.platform.ts` suffix (replaced with `.obsidian.ts` in production builds via esbuild)
- **Development code**: Use `.dev.ts` suffix (replaced with `.prod.ts` in production)
- **Path aliases**: `@/*` maps to `src/*`; Commonlib uses package exports rather than a source alias
@@ -309,6 +320,8 @@ Existing legacy Modules continue to register their handlers in `onBindFunction()
database and Replicator readiness. See
[the declarative settings adapter ADR](docs/adr/2026_08_declarative_settings_adapter.md).
- Use `this.services.setting.saveSettingData()` instead of using plugin methods directly
- Inject `@vrtmrz/value-store` at host composition. `setting.deviceValues` uses exact device-local keys; `setting.config` and the Service Hub `config` share the existing Vault and database-suffix scope. Await reads, writes, and deletions before advancing the workflow. Browser and Obsidian hosts retain their Web Storage keys, and the CLI retains its existing flat JSON string map.
- Seed the settings tab with `await settingTab.loadSettings()` before registration. Its editing buffer remains synchronous for Obsidian rendering; storage failures must not replace a previously loaded value with an empty default.
### Database Operations
docs/glossary.md
+4
View File
@@ -98,6 +98,10 @@ recovery guidance, or diagnostics intended for users.
- **ID recovery code:** A versioned text form of the saved ID key which can be
shown on the current device and imported without deriving a different key.
Treat it as a secret.
- **Joined devices:** The dialogue showing device records held by the active
CouchDB or Object Storage remote, with the current device first. Records
describe participation, not current online activity. **Forget device** removes
an obsolete device record; it does not delete Vault files or revoke credentials.
- **JWT Authentication:** An experimental CouchDB authentication option which
uses a JSON Web Token instead of standard credentials. It requires a private
key or secret, algorithm, expiry duration, subject, and key ID.
docs/settings.md
+38 -5
View File
@@ -234,6 +234,16 @@ Setting key: remoteType
The active connection type. This is automatically projected to the legacy configuration when you activate a connection profile.
### Joined devices
Choose **Review devices** under **Joined devices** to read the device records on the active CouchDB or Object Storage connection. The same **Review devices** action is available from the command palette and version Notices. **Refresh** reads the remote again. This device appears first, including when its record is absent from the remote. Opening or refreshing this dialogue does not create a missing CouchDB database.
The list shows each device ID, recorded platform, plug-in version, app version, and last recorded connection time. Older clients may have incomplete information. These records do not establish whether a device is online. On a locked remote, a device outside the accepted list is shown as **Locked out**.
Before synchronisation starts, and when this dialogue refreshes, Self-hosted LiveSync checks for recorded plug-in versions newer or older than its own. These Notices recommend reviewing and aligning plug-in versions across devices; they are advisory and do not stop synchronisation. The dialogue explains that synchronisation between different versions is generally safe unless the connection is explicitly blocked, but may be less efficient or use more storage. P2P synchronisation may also be unavailable. Unknown development version labels are left unclassified. Automatic checks run at most once per minute when synchronisation starts; they do not poll a continuously running connection.
**Forget device** removes another device's participation records after confirmation. It preserves the remote lock state and every other device. It does not delete Vault files or revoke credentials. A device can register again on an unlocked remote; on a locked remote, the existing recovery process applies. The result is checked after saving to CouchDB; if it cannot be confirmed, refresh the list and try again. CouchDB stores these records together in a local document which does not reject simultaneous updates, so changes from other devices can be overwritten. Pausing synchronisation on the other devices avoids that overlap. Object Storage uses conditional updates; if the server cannot support them, the record is retained.
### 2. Notification
#### Notify when the estimated remote storage size exceeds on start up
@@ -750,11 +760,13 @@ If enabled, the notification of hidden files change will be suppressed.
#### Synchronising files
(RegExp) Empty to sync all files. Set filter as a regular expression to limit synchronising files.
Setting key: syncOnlyRegEx
(RegExp) Limit normal files to matching paths. Leave empty to allow all normal file paths. Hidden File Sync uses its own target patterns.
#### Non-Synchronising files
(RegExp) If this is set, any changes to local and remote files that match this will be skipped.
Setting key: syncIgnoreRegEx
(RegExp) Skip changes to matching normal and hidden files, locally and remotely.
#### Maximum file size
@@ -936,6 +948,18 @@ Setting key: hashCacheMaxAmount
Setting key: customChunkSize
New setup uses 0 for every remote type. With V3 Rabin–Karp, this gives binary
Chunks a 100 KiB maximum, a 25 KiB minimum, and an approximate 25 KiB candidate
gap. The candidate gap is not the mean Chunk length. Text below 4 MiB retains
adaptive smaller boundaries; large text uses the binary thresholds with
UTF-8-safe cuts.
Larger values can reduce document counts for mostly immutable binary files,
but can increase the new content stored after an edit. Existing explicit and
agreed values are retained. Updating the plug-in does not re-chunk the Vault;
editing, forced writes, Chunk recreation, or a full Rebuild can create new
boundaries while existing content remains readable.
#### Chunk Splitter
Setting key: chunkSplitterVersion
@@ -957,7 +981,10 @@ If this enabled, chunks will be split into semantically meaningful segments. Not
#### Fetch chunks on demand
Setting key: readChunksOnline
(ex. Read chunks online) If this option is enabled, LiveSync reads chunks online directly instead of replicating them locally. Increasing Custom chunk size is recommended.
If enabled, this plug-in obtains missing Chunks when they are needed instead
of including all Chunks in ordinary pull replication. New setup retains the
same Chunk-size recommendation of 0. This setting does not re-chunk content
already stored by another device.
#### Batch size of on-demand fetching
@@ -1010,12 +1037,18 @@ The setting remains disabled by default because the measured storage and transfe
#### Batch size
Setting key: batch_size
Number of changes to sync at a time. Defaults to 50. Minimum is 2.
Number of changes in an ordinary CouchDB replication batch. The schema default
is 25, and the minimum is 2. Changing this value alone preserves the
replication checkpoint. Larger counts can reduce round trips, but do not bound
the final request size or memory use.
#### Batch limit
Setting key: batches_limit
Number of batches to process at a time. Defaults to 40. Minimum is 2. This along with batch size controls how many docs are kept in memory at a time.
Number of changes batches which PouchDB can read ahead. The schema default is
25, and the minimum is 2. This does not create parallel body transfers;
PouchDB processes one current replication batch at a time. It does not control
the separate P2P or Object Storage transfer units.
#### Use timeouts instead of heartbeats
docs/tips/hidden-file-sync.md
+4
View File
@@ -32,6 +32,10 @@ Enabling Hidden File Sync requires an initialisation direction:
Prefer a narrow target list. Device-specific workspace state and another plug-in's credentials are usually poor candidates for cross-device synchronisation.
`Normal Files` → `Synchronising files` limits ordinary files only. Hidden File Sync uses its own `Target patterns`. The `Non-Synchronising files` patterns under `Normal Files` also exclude matching hidden files, so review them alongside the hidden-file selectors.
If an earlier version skipped hidden files because of the normal-file allow-list, repeat Hidden File Sync initialisation after updating. A routine scan can skip unchanged files which were already checked. Back up the Vault and choose the intended source direction as described above.
Target patterns also control directory traversal. A pattern must therefore match each parent directory as well as the intended files. For example, the following pattern admits the `.obsidian` parent and only its `snippets` subtree:
```text
package-lock.json
Generated
+14 -4
View File
@@ -23,9 +23,10 @@
"@smithy/types": "^4.14.3",
"@smithy/util-retry": "^4.4.5",
"@vrtmrz/browser-ui-kit": "0.1.0",
"@vrtmrz/livesync-commonlib": "0.1.44",
"@vrtmrz/livesync-commonlib": "0.2.0",
"@vrtmrz/obsidian-plugin-kit": "0.1.4",
"@vrtmrz/ui-interactions": "0.1.2",
"@vrtmrz/value-store": "0.1.0",
"diff-match-patch": "^1.0.5",
"idb": "^8.0.3",
"markdown-it": "^14.3.2",
@@ -4115,9 +4116,9 @@
}
},
"node_modules/@vrtmrz/livesync-commonlib": {
"version": "0.1.44",
"resolved": "https://registry.npmjs.org/@vrtmrz/livesync-commonlib/-/livesync-commonlib-0.1.44.tgz",
"integrity": "sha512-GjXNSLTnWLZzdgrhW3sIOsLqGLP4iMgypPMtcBc2Ju2vMq64TRA1Llfoya1KRIeu6m0G7Xu3ZtkWARTRcHkYYg==",
"version": "0.2.0",
"resolved": "https://registry.npmjs.org/@vrtmrz/livesync-commonlib/-/livesync-commonlib-0.2.0.tgz",
"integrity": "sha512-K24Pkke+FTjZI7DjIV7qkZk+BmC9VebN3da+2cVD9SWOgoGl7OvDyXTHDafbrmmM4vi4ydaydAEh9glO1xHF6A==",
"license": "MIT",
"dependencies": {
"@aws-crypto/sha256-js": "5.2.0",
@@ -4128,6 +4129,7 @@
"@smithy/types": "^4.14.3",
"@smithy/util-retry": "^4.4.5",
"@trystero-p2p/nostr": "0.25.3",
"@vrtmrz/value-store": "0.1.0",
"diff-match-patch": "^1.0.5",
"events": "^3.3.0",
"fflate": "^0.8.2",
@@ -4139,9 +4141,11 @@
"pouchdb-adapter-idb": "^9.0.0",
"pouchdb-adapter-indexeddb": "^9.0.0",
"pouchdb-adapter-memory": "^9.0.0",
"pouchdb-checkpointer": "9.0.0",
"pouchdb-core": "^9.0.0",
"pouchdb-errors": "^9.0.0",
"pouchdb-find": "^9.0.0",
"pouchdb-generate-replication-id": "9.0.0",
"pouchdb-mapreduce": "^9.0.0",
"pouchdb-merge": "^9.0.0",
"pouchdb-replication": "^9.0.0",
@@ -4210,6 +4214,12 @@
"integrity": "sha512-njT2BSFh57imwGDxWXJOcGbZOYFVNGmeXMrS7/RUdBoAeOjWzJB9TxI7WcR1sgQIXKh8TE9hqALU8lue5b8dRw==",
"license": "MIT"
},
"node_modules/@vrtmrz/value-store": {
"version": "0.1.0",
"resolved": "https://registry.npmjs.org/@vrtmrz/value-store/-/value-store-0.1.0.tgz",
"integrity": "sha512-M62f1j0bnJkqpgI5sYxnkXUAOSDEwQbYrLf+w58gCCm4sjIXDmFzDJd2duXabRbktBlACt9P99yjN/9gOAPxrA==",
"license": "MIT"
},
"node_modules/abstract-leveldown": {
"version": "6.2.3",
"resolved": "https://registry.npmjs.org/abstract-leveldown/-/abstract-leveldown-6.2.3.tgz",
package.json
+5 -1
View File
@@ -57,6 +57,8 @@
"test:e2e:obsidian:smoke": "tsx test/e2e-obsidian/scripts/smoke.ts",
"test:e2e:obsidian:onboarding-invitation": "tsx test/e2e-obsidian/scripts/onboarding-invitation.ts",
"test:e2e:obsidian:dialog-mounts": "tsx test/e2e-obsidian/scripts/dialog-mounts.ts",
"test:e2e:obsidian:joined-devices": "tsx test/e2e-obsidian/scripts/joined-devices.ts",
"test:e2e:obsidian:joined-devices-object-storage": "tsx test/e2e-obsidian/scripts/joined-devices.ts --object-storage",
"test:e2e:obsidian:database-diagnostics": "tsx test/e2e-obsidian/scripts/database-diagnostics.ts",
"test:e2e:obsidian:database-diagnostics-couchdb": "tsx test/e2e-obsidian/scripts/database-diagnostics-couchdb.ts",
"test:e2e:obsidian:conflict-dialog-policy": "tsx test/e2e-obsidian/scripts/conflict-dialog-policy.ts",
@@ -92,6 +94,7 @@
"test:e2e:obsidian:two-vault-sync": "tsx test/e2e-obsidian/scripts/two-vault-sync.ts",
"test:e2e:obsidian:security-seed-reconnect": "tsx test/e2e-obsidian/scripts/security-seed-reconnect.ts",
"test:e2e:obsidian:hidden-file-snippet-sync": "tsx test/e2e-obsidian/scripts/hidden-file-snippet-sync.ts",
"test:e2e:obsidian:hidden-file-selectors": "tsx test/e2e-obsidian/scripts/hidden-file-selectors.ts",
"test:e2e:obsidian:customisation-sync": "tsx test/e2e-obsidian/scripts/customisation-sync.ts",
"test:e2e:obsidian:received-change-readiness": "tsx test/e2e-obsidian/scripts/received-change-readiness.ts",
"test:e2e:obsidian:remote-feature-change": "tsx test/e2e-obsidian/scripts/remote-feature-change.ts",
@@ -193,9 +196,10 @@
"@smithy/types": "^4.14.3",
"@smithy/util-retry": "^4.4.5",
"@vrtmrz/browser-ui-kit": "0.1.0",
"@vrtmrz/livesync-commonlib": "0.1.44",
"@vrtmrz/livesync-commonlib": "0.2.0",
"@vrtmrz/obsidian-plugin-kit": "0.1.4",
"@vrtmrz/ui-interactions": "0.1.2",
"@vrtmrz/value-store": "0.1.0",
"diff-match-patch": "^1.0.5",
"idb": "^8.0.3",
"markdown-it": "^14.3.2",
src/apps/browser/LiveSyncBrowserUIService.ts
+2 -2
View File
@@ -1,6 +1,6 @@
import type { ServiceContext } from "@vrtmrz/livesync-commonlib/context";
import type { AppLifecycleService } from "@vrtmrz/livesync-commonlib/compat/services/base/AppLifecycleService";
import type { ConfigService } from "@vrtmrz/livesync-commonlib/compat/services/base/ConfigService";
import type { IConfigService } from "@vrtmrz/livesync-commonlib/compat/services/base/IService";
import type { ControlService } from "@vrtmrz/livesync-commonlib/compat/services/base/ControlService";
import type { ReplicatorService } from "@vrtmrz/livesync-commonlib/compat/services/base/ReplicatorService";
import type { InjectableAPIService } from "@vrtmrz/livesync-commonlib/compat/services/implements/injectable/InjectableAPIService";
@@ -11,7 +11,7 @@ import { BrowserSvelteDialogManager } from "./BrowserSvelteDialogManager";
export interface LiveSyncBrowserUIServiceDependencies<T extends ServiceContext> {
API: InjectableAPIService<T>;
appLifecycle: AppLifecycleService<T>;
config: ConfigService<T>;
config: IConfigService;
control: ControlService<T>;
replicator: ReplicatorService<T>;
}
src/apps/browser/createLiveSyncBrowserServiceHub.ts
+10 -40
View File
@@ -1,11 +1,10 @@
import type { ObsidianLiveSyncSettings } from "@vrtmrz/livesync-commonlib/compat/common/types";
import type { KeyValueDatabaseFactory } from "@vrtmrz/livesync-commonlib/compat/interfaces/KeyValueDatabase";
import { PouchDB } from "@vrtmrz/livesync-commonlib/compat/pouchdb/pouchdb-browser";
import { ConfigService } from "@vrtmrz/livesync-commonlib/compat/services/base/ConfigService";
import { ControlService } from "@vrtmrz/livesync-commonlib/compat/services/base/ControlService";
import { DatabaseService } from "@vrtmrz/livesync-commonlib/compat/services/base/DatabaseService";
import { KeyValueDBService } from "@vrtmrz/livesync-commonlib/compat/services/base/KeyValueDBService";
import type { ISettingService } from "@vrtmrz/livesync-commonlib/compat/services/base/IService";
import type { ValueDeleter, ValueStore } from "@vrtmrz/value-store";
import { ServiceContext } from "@vrtmrz/livesync-commonlib/context";
import { InjectableAppLifecycleService } from "@vrtmrz/livesync-commonlib/compat/services/implements/injectable/InjectableAppLifecycleService";
import { InjectableConflictService } from "@vrtmrz/livesync-commonlib/compat/services/implements/injectable/InjectableConflictService";
@@ -24,10 +23,7 @@ import { InjectableVaultServiceCompat } from "@vrtmrz/livesync-commonlib/compat/
import { setLang, translateLiveSyncMessage } from "@/common/translation";
import { BrowserConfirm } from "./BrowserConfirm";
import { createBrowserKeyValueDatabaseFactory } from "./BrowserKeyValueDatabase";
import {
LiveSyncBrowserAPIService,
type LiveSyncBrowserAPIServiceOptions,
} from "./LiveSyncBrowserAPIService";
import { LiveSyncBrowserAPIService, type LiveSyncBrowserAPIServiceOptions } from "./LiveSyncBrowserAPIService";
import { LiveSyncBrowserUIService } from "./LiveSyncBrowserUIService";
export interface LiveSyncBrowserSettingsPersistence {
@@ -43,6 +39,7 @@ export interface LiveSyncBrowserRestartPolicy {
}
export interface LiveSyncBrowserServiceHubOptions<T extends ServiceContext> {
deviceValues: ValueStore & ValueDeleter;
context?: T;
getSystemVaultName?: () => string;
settings?: LiveSyncBrowserSettingsPersistence;
@@ -51,35 +48,12 @@ export interface LiveSyncBrowserServiceHubOptions<T extends ServiceContext> {
API?: Omit<LiveSyncBrowserAPIServiceOptions, "confirm" | "getSystemVaultName">;
}
class LiveSyncBrowserAppLifecycleService<
T extends ServiceContext,
> extends InjectableAppLifecycleService<T> {}
class LiveSyncBrowserAppLifecycleService<T extends ServiceContext> extends InjectableAppLifecycleService<T> {}
class LiveSyncBrowserDatabaseService<T extends ServiceContext> extends DatabaseService<T> {}
class LiveSyncBrowserKeyValueDBService<T extends ServiceContext> extends KeyValueDBService<T> {}
class LiveSyncBrowserConfigService<T extends ServiceContext> extends ConfigService<T> {
constructor(
context: T,
private readonly setting: ISettingService
) {
super(context);
}
getSmallConfig(key: string): string | null {
return this.setting.getSmallConfig(key);
}
setSmallConfig(key: string, value: string): void {
this.setting.setSmallConfig(key, value);
}
deleteSmallConfig(key: string): void {
this.setting.deleteSmallConfig(key);
}
}
/** LiveSync-owned service composition shared by WebApp and WebPeer. */
export class LiveSyncBrowserServiceHub<T extends ServiceContext> extends InjectableServiceHub<T> {
constructor(options: LiveSyncBrowserServiceHubOptions<T>) {
@@ -98,17 +72,14 @@ export class LiveSyncBrowserServiceHub<T extends ServiceContext> extends Injecta
const setting = new InjectableSettingService(context, {
APIService: API,
onDisplayLanguageChanged: setLang,
deviceValues: options.deviceValues,
});
const settingsPersistence = options.settings;
setting.loadData.setHandler(
settingsPersistence
? () => settingsPersistence.load()
: () => Promise.resolve(undefined)
settingsPersistence ? () => settingsPersistence.load() : () => Promise.resolve(undefined)
);
setting.saveData.setHandler(
settingsPersistence
? (settings) => settingsPersistence.save(settings)
: () => Promise.resolve()
settingsPersistence ? (settings) => settingsPersistence.save(settings) : () => Promise.resolve()
);
const appLifecycle = new LiveSyncBrowserAppLifecycleService(context, {
@@ -142,7 +113,7 @@ export class LiveSyncBrowserServiceHub<T extends ServiceContext> extends Injecta
setting,
API,
});
const config = new LiveSyncBrowserConfigService(context, setting);
const config = setting.config;
const replicator = new InjectableReplicatorService(context, {
settingService: setting,
appLifecycleService: appLifecycle,
@@ -163,8 +134,7 @@ export class LiveSyncBrowserServiceHub<T extends ServiceContext> extends Injecta
databaseService: database,
});
const keyValueDB = new LiveSyncBrowserKeyValueDBService(context, {
openKeyValueDatabase:
options.openKeyValueDatabase ?? createBrowserKeyValueDatabaseFactory(),
openKeyValueDatabase: options.openKeyValueDatabase ?? createBrowserKeyValueDatabaseFactory(),
appLifecycle,
databaseEvents,
vault,
@@ -209,7 +179,7 @@ export class LiveSyncBrowserServiceHub<T extends ServiceContext> extends Injecta
}
export function createLiveSyncBrowserServiceHub<T extends ServiceContext>(
options: LiveSyncBrowserServiceHubOptions<T> = {}
options: LiveSyncBrowserServiceHubOptions<T>
): LiveSyncBrowserServiceHub<T> {
return new LiveSyncBrowserServiceHub(options);
}
src/apps/browser/createLiveSyncBrowserServiceHub.unit.spec.ts
+35 -8
View File
@@ -1,5 +1,6 @@
import { createServiceContext } from "@vrtmrz/livesync-commonlib/context";
import { DEFAULT_SETTINGS } from "@vrtmrz/livesync-commonlib/compat/common/types";
import { MemoryValueStore } from "@vrtmrz/value-store";
import { afterEach, describe, expect, it, vi } from "vitest";
import {
@@ -21,7 +22,8 @@ describe("LiveSync browser service context contract", () => {
const context = createServiceContext({
translate: (key) => `webapp:${key}`,
});
const hub = createLiveSyncBrowserServiceHub({ context });
const deviceValues = new MemoryValueStore();
const hub = createLiveSyncBrowserServiceHub({ context, deviceValues });
expect(observeServiceContext(context, "moduleLocalDatabase.logWaitingForReady")).toEqual({
translation: "webapp:moduleLocalDatabase.logWaitingForReady",
@@ -32,20 +34,17 @@ describe("LiveSync browser service context contract", () => {
expect(SERVICE_CONTEXT_MEMBERS.filter((member) => !composition.servicesUsingExpectedContext[member])).toEqual(
[]
);
expect(hub.setting.deviceValues).toBe(deviceValues);
expect(hub.config).toBe(hub.setting.config);
});
it("binds browser identity, persistence, restart policy, and native Fetch while composing the hub", async () => {
const deviceLocalSettings = new Map<string, string>();
vi.stubGlobal("localStorage", {
getItem: (key: string) => deviceLocalSettings.get(key) ?? null,
setItem: (key: string, value: string) => deviceLocalSettings.set(key, value),
removeItem: (key: string) => deviceLocalSettings.delete(key),
clear: () => deviceLocalSettings.clear(),
});
const deviceValues = new MemoryValueStore();
const savedSettings: (typeof DEFAULT_SETTINGS)[] = [];
const restartCalls: string[] = [];
const nativeFetch = vi.fn(async () => new Response("ok"));
const options: LiveSyncBrowserServiceHubOptions<ReturnType<typeof createServiceContext>> = {
deviceValues,
getSystemVaultName: () => "browser-vault",
settings: {
load: async () => ({
@@ -69,6 +68,8 @@ describe("LiveSync browser service context contract", () => {
const hub = createLiveSyncBrowserServiceHub(options);
expect(hub.setting.deviceValues).toBe(deviceValues);
expect(hub.config).toBe(hub.setting.config);
expect(hub.API.getSystemVaultName()).toBe("browser-vault");
expect(hub.API.getPlatform()).toBe("browser");
const response = await hub.API.nativeFetch("https://example.invalid/");
@@ -86,4 +87,30 @@ describe("LiveSync browser service context contract", () => {
expect(hub.appLifecycle.isReloadingScheduled()).toBe(true);
expect(restartCalls).toEqual(["schedule", "perform", "ask:restart now"]);
});
it("keeps each browser composition bound to its own device value store", async () => {
const firstValues = new MemoryValueStore();
const secondValues = new MemoryValueStore();
const firstHub = createLiveSyncBrowserServiceHub({
deviceValues: firstValues,
getSystemVaultName: () => "first-vault",
});
const secondHub = createLiveSyncBrowserServiceHub({
deviceValues: secondValues,
getSystemVaultName: () => "second-vault",
});
await firstHub.config.setValue("probe", "first");
await secondHub.config.setValue("probe", "second");
expect(firstHub.setting.deviceValues).toBe(firstValues);
expect(secondHub.setting.deviceValues).toBe(secondValues);
expect(firstHub.config).toBe(firstHub.setting.config);
expect(secondHub.config).toBe(secondHub.setting.config);
expect(firstHub.config).not.toBe(secondHub.config);
await expect(firstHub.config.getValue("probe")).resolves.toBe("first");
await expect(secondHub.config.getValue("probe")).resolves.toBe("second");
await expect(firstValues.getValue("first-vault-probe")).resolves.toBe("first");
await expect(secondValues.getValue("second-vault-probe")).resolves.toBe("second");
});
});
src/apps/cli/README.md
+2 -2
View File
@@ -719,8 +719,8 @@ src/apps/cli/
│ └── ServiceFileAccessImpl.ts
├── services/
│ ├── NodeKeyValueDBService.ts
│ ├── NodeServiceHub.ts
│ └── NodeSettingService.ts
│ ├── NodeLocalStorage.ts
│ └── NodeServiceHub.ts
├── test/
│ ├── test-e2e-two-vaults-common.sh
│ ├── test-e2e-two-vaults-matrix.sh
src/apps/cli/main.bootstrap.unit.spec.ts
+21
View File
@@ -88,6 +88,27 @@ describe("CLI database preparation", () => {
await expect(main(standardIo, runner)).rejects.toThrow(`__EXIT__:${exitCode}`);
}
it("loads legacy device values and persists configuration through the service-owned store", async () => {
const localStoragePath = join(directory, ".livesync", "runtime", "local-storage.json");
await mkdir(join(directory, ".livesync", "runtime"), { recursive: true });
await writeFile(localStoragePath, JSON.stringify({ ["__proto__"]: "legacy value" }));
let legacyValue: string | null | undefined;
const runner = vi.fn<CliCommandRunner>(async (_options, { core }) => {
legacyValue = await core.services.setting.deviceValues.getValue("__proto__");
expect(core.services.config).toBe(core.services.setting.config);
await core.services.config.setValue("bootstrap-probe", "config value");
return false;
});
await start("ls", runner);
expect(legacyValue).toBe("legacy value");
const stored = JSON.parse(await readFile(localStoragePath, "utf-8")) as Record<string, string>;
expect(Object.prototype.hasOwnProperty.call(stored, "__proto__")).toBe(true);
expect(stored["__proto__"]).toBe("legacy value");
expect(Object.values(stored)).toContain("config value");
});
it.each([REMOTE_P2P, REMOTE_COUCHDB])(
"restores configured P2P automation before the host connects (%s)",
async (remoteType) => {
src/apps/cli/main.ts
-4
View File
@@ -1,6 +1,5 @@
import { useP2PSettingsPreparation } from "@/serviceFeatures/useP2PSettingsPreparation";
import { NodeServiceContext, NodeServiceHub } from "./services/NodeServiceHub";
import { configureNodeLocalStorage, ensureGlobalNodeLocalStorage } from "./services/NodeLocalStorage";
import { LiveSyncBaseCore, type StartupDatabaseOptions } from "@/LiveSyncBaseCore";
import { initialiseServiceModulesCLI } from "./serviceModules/CLIServiceModules";
import {
@@ -78,7 +77,6 @@ const VAULT_SYNC_MODES: Readonly<Partial<Record<CLICommand, CLIVaultSyncMode>>>
},
};
ensureGlobalNodeLocalStorage();
defaultLoggerEnv.minLogLevel = LOG_LEVEL_DEBUG;
/** Injectable command boundary used by CLI integration probes. */
@@ -400,8 +398,6 @@ export async function main(
const settingsPath = options.settingsPath
? path.resolve(options.settingsPath)
: path.join(databasePath, SETTINGS_FILE);
configureNodeLocalStorage(path.join(databasePath, ".livesync", "runtime", "local-storage.json"));
// Resolve vault path: mirror positional argument takes priority,
// then --vault flag, otherwise fall back to databasePath.
const vaultPath =
src/apps/cli/package.json
+1 -1
View File
@@ -12,7 +12,7 @@
"buildRun": "npm run build && npm run cli --",
"build:docker": "docker build -f Dockerfile -t livesync-cli ../../..",
"check": "tsc -p tsconfig.json",
"test:unit": "cd ../../.. && npx vitest run --config vitest.config.unit.ts src/apps/cli/main.unit.spec.ts src/apps/cli/main.bootstrap.unit.spec.ts src/apps/cli/settingsPersistence.unit.spec.ts src/apps/cli/commands/utils.unit.spec.ts src/apps/cli/commands/runCommand.unit.spec.ts src/apps/cli/commands/daemonCommand.unit.spec.ts src/apps/cli/commands/p2p.unit.spec.ts src/apps/cli/deploy/install.unit.spec.ts src/apps/cli/adapters/NodeFileSystemAdapter.unit.spec.ts",
"test:unit": "cd ../../.. && npx vitest run --config vitest.config.unit.ts src/apps/cli/main.unit.spec.ts src/apps/cli/main.bootstrap.unit.spec.ts src/apps/cli/settingsPersistence.unit.spec.ts src/apps/cli/services/NodeLocalStorage.unit.spec.ts src/apps/cli/commands/utils.unit.spec.ts src/apps/cli/commands/runCommand.unit.spec.ts src/apps/cli/commands/daemonCommand.unit.spec.ts src/apps/cli/commands/p2p.unit.spec.ts src/apps/cli/deploy/install.unit.spec.ts src/apps/cli/adapters/NodeFileSystemAdapter.unit.spec.ts",
"test:e2e:two-vaults": "bash test/test-e2e-two-vaults-with-docker-linux.sh",
"test:e2e:two-vaults:common": "bash test/test-e2e-two-vaults-common.sh",
"test:e2e:two-vaults:matrix": "bash test/test-e2e-two-vaults-matrix.sh",
src/apps/cli/services/NodeLocalStorage.ts
+65 -94
View File
@@ -1,115 +1,86 @@
import { createDeletableValueStore, sanitiseValueStoreError, ValueStoreError } from "@vrtmrz/value-store";
import type { ValueDeleter, ValueStore } from "@vrtmrz/value-store";
import { fs as nodeFs, path as nodePath } from "@vrtmrz/livesync-commonlib/node";
import { compatGlobal } from "@vrtmrz/livesync-commonlib/compat/common/coreEnvFunctions";
type LocalStorageShape = {
getItem(key: string): string | null;
setItem(key: string, value: string): void;
removeItem(key: string): void;
clear(): void;
};
type NodeLocalStorage = ValueStore & ValueDeleter;
class PersistentNodeLocalStorage {
private storagePath: string | undefined;
private localStore: Record<string, string> = {};
class JsonFileValueBackend {
private values: Map<string, string>;
configure(storagePath: string) {
if (this.storagePath === storagePath) {
return;
}
this.storagePath = storagePath;
this.loadFromFile();
constructor(private readonly storagePath: string) {
this.values = this.loadValues();
}
private loadFromFile() {
if (!this.storagePath) {
this.localStore = {};
get(key: string): string | null {
return this.values.get(key) ?? null;
}
set(key: string, value: string): void {
const nextValues = new Map(this.values);
nextValues.set(key, value);
this.writeValues(nextValues);
this.values = nextValues;
}
delete(key: string): void {
if (!this.values.has(key)) {
return;
}
const nextValues = new Map(this.values);
nextValues.delete(key);
this.writeValues(nextValues);
this.values = nextValues;
}
private loadValues(): Map<string, string> {
let contents: string;
try {
const loaded = JSON.parse(nodeFs.readFileSync(this.storagePath, "utf-8")) as Record<string, string>;
this.localStore = { ...loaded };
contents = nodeFs.readFileSync(this.storagePath, "utf-8");
} catch (error) {
if (isMissingFileError(error)) {
return new Map();
}
throw sanitiseValueStoreError(error);
}
let document: unknown;
try {
document = JSON.parse(contents);
} catch {
this.localStore = {};
throw new ValueStoreError("invalid-data");
}
if (document === null || typeof document !== "object" || Array.isArray(document)) {
throw new ValueStoreError("invalid-data");
}
const values = new Map<string, string>();
for (const [key, value] of Object.entries(document)) {
if (typeof value !== "string") {
throw new ValueStoreError("invalid-data");
}
values.set(key, value);
}
return values;
}
private flushToFile() {
if (!this.storagePath) {
return;
}
private writeValues(values: ReadonlyMap<string, string>): void {
nodeFs.mkdirSync(nodePath.dirname(this.storagePath), { recursive: true });
nodeFs.writeFileSync(this.storagePath, JSON.stringify(this.localStore, null, 2), "utf-8");
}
getItem(key: string): string | null {
return this.localStore[key] ?? null;
}
setItem(key: string, value: string) {
this.localStore[key] = value;
this.flushToFile();
}
removeItem(key: string) {
if (!(key in this.localStore)) {
return;
}
delete this.localStore[key];
this.flushToFile();
}
clear() {
this.localStore = {};
this.flushToFile();
nodeFs.writeFileSync(this.storagePath, JSON.stringify(Object.fromEntries(values), null, 2), "utf-8");
}
}
const persistentNodeLocalStorage = new PersistentNodeLocalStorage();
function createNodeLocalStorageShim(): LocalStorageShape {
return {
getItem(key: string) {
return persistentNodeLocalStorage.getItem(key);
},
setItem(key: string, value: string) {
persistentNodeLocalStorage.setItem(key, value);
},
removeItem(key: string) {
persistentNodeLocalStorage.removeItem(key);
},
clear() {
persistentNodeLocalStorage.clear();
},
};
}
export function ensureGlobalNodeLocalStorage() {
const g = compatGlobal as unknown as Record<string, unknown>;
if (
!("localStorage" in g) ||
typeof (g.localStorage as Record<string, unknown> | undefined)?.getItem !== "function"
) {
g.localStorage = createNodeLocalStorageShim();
function isMissingFileError(error: unknown): boolean {
if (typeof error !== "object" || error === null) {
return false;
}
try {
const code = Object.getOwnPropertyDescriptor(error, "code");
return code !== undefined && Object.prototype.hasOwnProperty.call(code, "value") && code.value === "ENOENT";
} catch {
return false;
}
}
export function configureNodeLocalStorage(storagePath: string) {
persistentNodeLocalStorage.configure(storagePath);
ensureGlobalNodeLocalStorage();
}
export function getNodeLocalStorageItem(key: string): string {
return persistentNodeLocalStorage.getItem(key) ?? "";
}
export function setNodeLocalStorageItem(key: string, value: string) {
persistentNodeLocalStorage.setItem(key, value);
}
export function deleteNodeLocalStorageItem(key: string) {
persistentNodeLocalStorage.removeItem(key);
}
export function clearNodeLocalStorage() {
persistentNodeLocalStorage.clear();
export function createNodeLocalStorage(storagePath: string): NodeLocalStorage {
return createDeletableValueStore(new JsonFileValueBackend(storagePath));
}
src/apps/cli/services/NodeLocalStorage.unit.spec.ts
+98 -29
View File
@@ -1,58 +1,127 @@
import { fs, os, path } from "@vrtmrz/livesync-commonlib/node";
import { ValueStoreError } from "@vrtmrz/value-store";
import { afterEach, describe, expect, it } from "vitest";
import {
clearNodeLocalStorage,
configureNodeLocalStorage,
ensureGlobalNodeLocalStorage,
getNodeLocalStorageItem,
setNodeLocalStorageItem,
} from "./NodeLocalStorage";
import { createNodeLocalStorage } from "./NodeLocalStorage";
describe("NodeLocalStorage", () => {
const tempDirs: string[] = [];
afterEach(() => {
clearNodeLocalStorage();
for (const tempDir of tempDirs.splice(0)) {
fs.rmSync(tempDir, { recursive: true, force: true });
}
});
it("persists values to the configured file", () => {
const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), "livesync-node-local-storage-"));
tempDirs.push(tempDir);
const storagePath = path.join(tempDir, "runtime", "local-storage.json");
configureNodeLocalStorage(storagePath);
setNodeLocalStorageItem("checkpoint", "42");
const saved = JSON.parse(fs.readFileSync(storagePath, "utf-8")) as Record<string, string>;
expect(saved.checkpoint).toBe("42");
});
it("reloads persisted values when configured again", () => {
it("loads and preserves legacy flat string data, including empty values and exact keys", async () => {
const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), "livesync-node-local-storage-"));
tempDirs.push(tempDir);
const storagePath = path.join(tempDir, "runtime", "local-storage.json");
const specialKey = "__proto__";
const specialValue = "line one\n雪";
fs.mkdirSync(path.dirname(storagePath), { recursive: true });
fs.writeFileSync(storagePath, JSON.stringify({ persisted: "value" }, null, 2), "utf-8");
fs.writeFileSync(storagePath, JSON.stringify({ [specialKey]: specialValue, empty: "" }), "utf-8");
configureNodeLocalStorage(storagePath);
const store = createNodeLocalStorage(storagePath);
expect(getNodeLocalStorageItem("persisted")).toBe("value");
await expect(store.getValue(specialKey)).resolves.toBe(specialValue);
await expect(store.getValue("empty", "default")).resolves.toBe("");
const saved = JSON.parse(fs.readFileSync(storagePath, "utf-8")) as Record<string, string>;
expect(saved[specialKey]).toBe(specialValue);
expect(saved.empty).toBe("");
});
it("installs a global localStorage shim backed by the same store", () => {
it("persists strings and objects, reloads them, and deletes keys", async () => {
const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), "livesync-node-local-storage-"));
tempDirs.push(tempDir);
const storagePath = path.join(tempDir, "runtime", "local-storage.json");
const store = createNodeLocalStorage(storagePath);
const object = { enabled: false, entries: ["value", null, 0] };
configureNodeLocalStorage(storagePath);
ensureGlobalNodeLocalStorage();
await store.setValue("empty", "");
await store.setValue("text", "stored value");
await store.setObject("object", object);
globalThis.localStorage.setItem("shared", "state");
const reloaded = createNodeLocalStorage(storagePath);
await expect(reloaded.getValue("empty", "default")).resolves.toBe("");
await expect(reloaded.getValue("text")).resolves.toBe("stored value");
await expect(reloaded.getObject("object")).resolves.toEqual(object);
expect(getNodeLocalStorageItem("shared")).toBe("state");
await reloaded.delete("text");
await expect(reloaded.getValue("text")).resolves.toBeNull();
const saved = JSON.parse(fs.readFileSync(storagePath, "utf-8")) as Record<string, string>;
expect(saved.text).toBeUndefined();
expect(saved.object).toBe(JSON.stringify(object));
});
it("treats a missing file as empty and creates its parent on the first write", async () => {
const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), "livesync-node-local-storage-"));
tempDirs.push(tempDir);
const storagePath = path.join(tempDir, "runtime", "local-storage.json");
const store = createNodeLocalStorage(storagePath);
await expect(store.getValue("absent")).resolves.toBeNull();
expect(fs.existsSync(storagePath)).toBe(false);
await store.setValue("created", "value");
expect(fs.existsSync(storagePath)).toBe(true);
});
it("keeps separate store instances isolated by their file paths", async () => {
const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), "livesync-node-local-storage-"));
tempDirs.push(tempDir);
const first = createNodeLocalStorage(path.join(tempDir, "first", "local-storage.json"));
const second = createNodeLocalStorage(path.join(tempDir, "second", "local-storage.json"));
await first.setValue("shared-key", "first");
await second.setValue("shared-key", "second");
await expect(first.getValue("shared-key")).resolves.toBe("first");
await expect(second.getValue("shared-key")).resolves.toBe("second");
});
it("surfaces malformed and unreadable files as sanitised value-store errors", async () => {
const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), "livesync-node-local-storage-"));
tempDirs.push(tempDir);
const malformedPath = path.join(tempDir, "malformed.json");
fs.writeFileSync(malformedPath, "{ malformed", "utf-8");
let malformedError: unknown;
try {
createNodeLocalStorage(malformedPath);
} catch (error) {
malformedError = error;
}
expect(malformedError).toBeInstanceOf(ValueStoreError);
expect(malformedError).toMatchObject({ code: "invalid-data" });
expect(fs.readFileSync(malformedPath, "utf-8")).toBe("{ malformed");
const unreadablePath = path.join(tempDir, "unreadable.json");
fs.mkdirSync(unreadablePath);
expect(() => createNodeLocalStorage(unreadablePath)).toThrowError(ValueStoreError);
});
it("does not change its visible values when a write fails", async () => {
const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), "livesync-node-local-storage-"));
tempDirs.push(tempDir);
const storagePath = path.join(tempDir, "runtime", "local-storage.json");
const store = createNodeLocalStorage(storagePath);
await store.setValue("retained", "before failure");
fs.rmSync(storagePath);
fs.mkdirSync(storagePath);
await expect(store.setValue("new", "write fails")).rejects.toBeInstanceOf(ValueStoreError);
await expect(store.getValue("retained")).resolves.toBe("before failure");
await expect(store.getValue("new")).resolves.toBeNull();
});
it("does not install or modify a global localStorage value", () => {
const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), "livesync-node-local-storage-"));
tempDirs.push(tempDir);
const before = Object.getOwnPropertyDescriptor(globalThis, "localStorage");
createNodeLocalStorage(path.join(tempDir, "runtime", "local-storage.json"));
expect(Object.getOwnPropertyDescriptor(globalThis, "localStorage")).toEqual(before);
});
});
src/apps/cli/services/NodeServiceHub.ts
+8 -11
View File
@@ -1,6 +1,5 @@
import type { AppLifecycleServiceDependencies } from "@vrtmrz/livesync-commonlib/compat/services/base/AppLifecycleService";
import type { ServiceContext } from "@vrtmrz/livesync-commonlib/context";
import { ConfigServiceBrowserCompat } from "@vrtmrz/livesync-commonlib/compat/services/implements/browser/ConfigServiceBrowserCompat";
import type {
ComponentHasResult,
SvelteDialogManager,
@@ -11,6 +10,7 @@ import { InjectableAppLifecycleService } from "@vrtmrz/livesync-commonlib/compat
import { InjectableConflictService } from "@vrtmrz/livesync-commonlib/compat/services/implements/injectable/InjectableConflictService";
import { InjectableDatabaseEventService } from "@vrtmrz/livesync-commonlib/compat/services/implements/injectable/InjectableDatabaseEventService";
import { InjectableFileProcessingService } from "@vrtmrz/livesync-commonlib/compat/services/implements/injectable/InjectableFileProcessingService";
import { InjectableSettingService } from "@vrtmrz/livesync-commonlib/compat/services/implements/injectable/InjectableSettingService";
import { PathServiceCompat } from "@vrtmrz/livesync-commonlib/compat/services/implements/injectable/InjectablePathService";
import { InjectableRemoteService } from "@vrtmrz/livesync-commonlib/compat/services/implements/injectable/InjectableRemoteService";
import { InjectableReplicationService } from "@vrtmrz/livesync-commonlib/compat/services/implements/injectable/InjectableReplicationService";
@@ -21,7 +21,7 @@ import { InjectableVaultServiceCompat } from "@vrtmrz/livesync-commonlib/compat/
import { ControlService } from "@vrtmrz/livesync-commonlib/compat/services/base/ControlService";
import { HeadlessAPIService } from "@vrtmrz/livesync-commonlib/compat/services/implements/headless/HeadlessAPIService";
import { NodeKeyValueDBService } from "./NodeKeyValueDBService";
import { NodeSettingService } from "./NodeSettingService";
import { createNodeLocalStorage } from "./NodeLocalStorage";
import { DatabaseService } from "@vrtmrz/livesync-commonlib/compat/services/base/DatabaseService";
import type { ObsidianLiveSyncSettings } from "@vrtmrz/livesync-commonlib/compat/common/types";
import { path as nodePath } from "@vrtmrz/livesync-commonlib/node";
@@ -96,11 +96,11 @@ export class NodeServiceHub<T extends NodeServiceContext> extends InjectableServ
const conflict = new InjectableConflictService(context);
const fileProcessing = new InjectableFileProcessingService(context);
const setting = new NodeSettingService(
context,
{ APIService: API, onDisplayLanguageChanged: setLang },
localStoragePath
);
const setting = new InjectableSettingService<T>(context, {
APIService: API,
onDisplayLanguageChanged: setLang,
deviceValues: createNodeLocalStorage(localStoragePath),
});
const appLifecycle = new NodeAppLifecycleService<T>(context, {
settingService: setting,
@@ -132,10 +132,7 @@ export class NodeServiceHub<T extends NodeServiceContext> extends InjectableServ
setting,
});
const config = new ConfigServiceBrowserCompat<T>(context, {
settingService: setting,
APIService: API,
});
const config = setting.config;
const replicator = new InjectableReplicatorService(context, {
settingService: setting,
src/apps/cli/services/NodeSettingService.ts
-42
View File
@@ -1,42 +0,0 @@
import { EVENT_SETTING_SAVED } from "@vrtmrz/livesync-commonlib/compat/events/coreEvents";
import { EVENT_REQUEST_RELOAD_SETTING_TAB } from "@/common/events";
import { handlers } from "@vrtmrz/livesync-commonlib/compat/services/lib/HandlerUtils";
import type { ObsidianLiveSyncSettings } from "@vrtmrz/livesync-commonlib/compat/common/types";
import type { ServiceContext } from "@vrtmrz/livesync-commonlib/context";
import { SettingService, type SettingServiceDependencies } from "@vrtmrz/livesync-commonlib/compat/services/base/SettingService";
import {
configureNodeLocalStorage,
deleteNodeLocalStorageItem,
getNodeLocalStorageItem,
setNodeLocalStorageItem,
} from "./NodeLocalStorage";
export class NodeSettingService<T extends ServiceContext> extends SettingService<T> {
constructor(context: T, dependencies: SettingServiceDependencies, storagePath: string) {
super(context, dependencies);
configureNodeLocalStorage(storagePath);
this.onSettingSaved.addHandler((settings) => {
this.context.events.emitEvent(EVENT_SETTING_SAVED, settings);
return Promise.resolve(true);
});
this.onSettingLoaded.addHandler((settings) => {
this.context.events.emitEvent(EVENT_REQUEST_RELOAD_SETTING_TAB);
return Promise.resolve(true);
});
}
protected setItem(key: string, value: string) {
setNodeLocalStorageItem(key, value);
}
protected getItem(key: string): string {
return getNodeLocalStorageItem(key);
}
protected deleteItem(key: string): void {
deleteNodeLocalStorageItem(key);
}
public saveData = handlers<{ saveData: (data: ObsidianLiveSyncSettings) => Promise<void> }>().binder("saveData");
public loadData = handlers<{ loadData: () => Promise<ObsidianLiveSyncSettings | undefined> }>().binder("loadData");
}
src/apps/webapp/WebAppRuntime.ts
+12 -2
View File
@@ -21,6 +21,8 @@ import { useCheckRemoteSize } from "@vrtmrz/livesync-commonlib/compat/serviceFea
import { useRemoteConfiguration } from "@vrtmrz/livesync-commonlib/compat/serviceFeatures/remoteConfig";
import { useP2PReplicatorFeature } from "@vrtmrz/livesync-commonlib/compat/replication/trystero/useP2PReplicatorFeature";
import type { UseP2PReplicatorResult } from "@vrtmrz/livesync-commonlib/compat/replication/trystero/UseP2PReplicatorResult";
import type { ValueStore, ValueDeleter } from "@vrtmrz/value-store";
import { createWebStorageValueStore } from "@vrtmrz/value-store/web-storage";
import { compatGlobal } from "@vrtmrz/livesync-commonlib/compat/common/coreEnvFunctions";
import {
createLiveSyncBrowserServiceHub,
@@ -56,6 +58,7 @@ const DEFAULT_SETTINGS: Partial<ObsidianLiveSyncSettings> = {
export type WebAppRuntimeStatusKind = "info" | "warning" | "error" | "success";
export interface WebAppRuntimeOptions {
deviceValues?: ValueStore & ValueDeleter;
reportStatus?: (kind: WebAppRuntimeStatusKind, message: string) => void;
scheduleReload?: (delayMilliseconds: number) => void;
}
@@ -71,7 +74,10 @@ export class WebAppRuntime {
private paneHost: P2PReplicatorPaneHost | null = null;
private restartScheduled = false;
constructor(rootHandle: FileSystemDirectoryHandle, options: WebAppRuntimeOptions = {}) {
constructor(
rootHandle: FileSystemDirectoryHandle,
private readonly options: WebAppRuntimeOptions = {}
) {
this.rootHandle = rootHandle;
this.reportStatus = options.reportStatus ?? (() => {});
this.scheduleReload =
@@ -89,6 +95,8 @@ export class WebAppRuntime {
private createServiceOptions(): LiveSyncBrowserServiceHubOptions<ServiceContext> {
return {
deviceValues:
this.options.deviceValues ?? createWebStorageValueStore(compatGlobal.localStorage, { prefix: "" }),
getSystemVaultName: () => this.rootHandle.name || "livesync-webapp",
settings: {
save: async (data) => {
@@ -219,7 +227,9 @@ export class WebAppRuntime {
useCheckRemoteSize(core);
useRemoteConfiguration(core);
this.p2p = useP2PReplicatorFeature(core, undefined, undefined, {
prepareP2PSettings: useP2PSettingsPreparation(core.services.API.webCompatFetch.bind(core.services.API)),
prepareP2PSettings: useP2PSettingsPreparation(
core.services.API.webCompatFetch.bind(core.services.API)
),
});
this.paneHost = {
services: core.services,
src/apps/webpeer/src/WebPeerRuntime.ts
+7 -3
View File
@@ -7,6 +7,8 @@ import { P2PLogCollector } from "@vrtmrz/livesync-commonlib/compat/replication/t
import { useP2PReplicatorFeature } from "@vrtmrz/livesync-commonlib/compat/replication/trystero/useP2PReplicatorFeature";
import { ServiceContext, type LiveSyncEventHub } from "@vrtmrz/livesync-commonlib/context";
import type { P2PServiceViews } from "@vrtmrz/livesync-commonlib/p2p";
import type { ValueStore, ValueDeleter } from "@vrtmrz/value-store";
import { createWebStorageValueStore } from "@vrtmrz/value-store/web-storage";
import { unique } from "octagonal-wheels/collection";
import type { SimpleStore } from "octagonal-wheels/databases/SimpleStoreBase";
@@ -20,6 +22,7 @@ import { translateLiveSyncMessage } from "@/common/translation";
import { WEBPEER_STORE_NAME, createWebPeerPersistence } from "./WebPeerPersistence";
export interface WebPeerRuntimeOptions {
deviceValues?: ValueStore & ValueDeleter;
context?: ServiceContext;
store?: SimpleStore<unknown>;
deviceName?: string;
@@ -62,6 +65,7 @@ export class WebPeerRuntime {
this.context = options.context ?? new ServiceContext({ translate: translateLiveSyncMessage });
this.services = createLiveSyncBrowserServiceHub<ServiceContext>({
context: this.context,
deviceValues: options.deviceValues ?? createWebStorageValueStore(compatGlobal.localStorage, { prefix: "" }),
getSystemVaultName: () => options.systemVaultName ?? WEBPEER_STORE_NAME,
settings: persistence.settings,
restart: {
@@ -103,7 +107,7 @@ export class WebPeerRuntime {
await this.services.setting.loadSettings();
const deviceName = this.options.deviceName?.trim();
if (deviceName) {
this.services.config.setSmallConfig(SETTING_KEY_P2P_DEVICE_NAME, deviceName);
await this.services.config.setValue(SETTING_KEY_P2P_DEVICE_NAME, deviceName);
}
const opened = await this.services.database.openDatabase({
replicator: this.services.replicator,
@@ -192,7 +196,7 @@ export class WebPeerRuntime {
);
}
getDeviceName(): string {
return this.services.config.getSmallConfig(SETTING_KEY_P2P_DEVICE_NAME) || this.services.vault.getVaultName();
async getDeviceName(): Promise<string> {
return (await this.services.config.getValue(SETTING_KEY_P2P_DEVICE_NAME)) || this.services.vault.getVaultName();
}
}
src/common/messages/LiveSyncProvisionalMessages.ts
+65
View File
@@ -7,6 +7,71 @@
* remove it from this map in the same change.
*/
export const liveSyncProvisionalEnglishMessages = {
"(RegExp) Limit normal files to matching paths. Leave empty to allow all normal file paths. Hidden File Sync uses its own target patterns.":
"(RegExp) Limit normal files to matching paths. Leave empty to allow all normal file paths. Hidden File Sync uses its own target patterns.",
"(RegExp) Skip changes to matching normal and hidden files, locally and remotely.":
"(RegExp) Skip changes to matching normal and hidden files, locally and remotely.",
"Unknown platform": "Unknown platform",
"This removes only this device's record from the remote.":
"This removes only this device's record from the remote.",
"This remote is unlocked. A device that returns can register again.":
"This remote is unlocked. A device that returns can register again.",
"This remote is locked. Use the existing recovery process for devices that need to join again.":
"This remote is locked. Use the existing recovery process for devices that need to join again.",
"This remote is locked. If the device returns, use the existing recovery process to join it again.":
"This remote is locked. If the device returns, use the existing recovery process to join it again.",
"This remote does not support conditional updates, so the device record could not be removed.":
"This remote does not support conditional updates, so the device record could not be removed.",
"This remote does not currently have a record of joined devices.":
"This remote does not currently have a record of joined devices.",
"This list shows device information recorded by the remote. It does not show whether a device is online now.":
"This list shows device information recorded by the remote. It does not show whether a device is online now.",
"This device has been locked out. Use the existing recovery process to join again.":
"This device has been locked out. Use the existing recovery process to join again.",
"This device": "This device",
"This remote has been cleaned. Use the existing recovery process to join devices again as needed.":
"This remote has been cleaned. Use the existing recovery process to join devices again as needed.",
"The device record was forgotten.": "The device record was forgotten.",
"The device record was already absent.": "The device record was already absent.",
"The device list changed while you were trying to forget this device. Refresh the list before trying again.":
"The device list changed while you were trying to forget this device. Refresh the list before trying again.",
"The device information could not be loaded. Close this dialogue and open Joined devices again if the remote configuration has changed.":
"The device information could not be loaded. Close this dialogue and open Joined devices again if the remote configuration has changed.",
"Review device records for the active CouchDB or Object Storage connection.":
"Review device records for the active CouchDB or Object Storage connection.",
Refresh: "Refresh",
"Plug-in version": "Plug-in version",
Platform: "Platform",
"Older Self-hosted LiveSync versions detected: ${versions}.":
"Older Self-hosted LiveSync versions detected: ${versions}.",
"Newer Self-hosted LiveSync versions detected: ${versions}.":
"Newer Self-hosted LiveSync versions detected: ${versions}.",
"Reviewing and aligning Self-hosted LiveSync versions across devices is recommended.":
"Reviewing and aligning Self-hosted LiveSync versions across devices is recommended.",
Open: "Open",
"Not recorded": "Not recorded",
"Not accepted": "Not accepted",
"No joined devices are recorded on this remote.": "No joined devices are recorded on this remote.",
"Locked out": "Locked out",
"Loading joined devices…": "Loading joined devices…",
"Last recorded": "Last recorded",
"Joined devices could not be updated. Refresh the list and try again. If the remote configuration has changed, close this dialogue and open Joined devices again.":
"Joined devices could not be updated. Refresh the list and try again. If the remote configuration has changed, close this dialogue and open Joined devices again.",
"Joined devices are available after database startup with an active CouchDB or Object Storage connection.":
"Joined devices are available after database startup with an active CouchDB or Object Storage connection.",
"Joined devices": "Joined devices",
"Review devices": "Review devices",
"Synchronisation between different plug-in versions is generally safe unless the connection is explicitly blocked. However, it may be less efficient or use more storage, and P2P synchronisation may be unavailable.":
"Synchronisation between different plug-in versions is generally safe unless the connection is explicitly blocked. However, it may be less efficient or use more storage, and P2P synchronisation may be unavailable.",
"Forgetting…": "Forgetting…",
"Forgetting the record does not delete Vault files or revoke credentials.":
"Forgetting the record does not delete Vault files or revoke credentials.",
"Forget this device record?": "Forget this device record?",
"Forget device": "Forget device",
Close: "Close",
Cancel: "Cancel",
"App version": "App version",
Accepted: "Accepted",
"Preparing follow-up report…": "Preparing follow-up report…",
"Cancelling local search. Checked ${COUNT} entries.": "Cancelling local search. Checked ${COUNT} entries.",
"Search current Vault paths and current local Metadata only. Older history and remote-only files are not searched.":
src/common/replicatorProviders.ts
+13
View File
@@ -28,7 +28,10 @@ import {
getObjectStorageReplicatorConfigurationIdentity,
} from "./replicatorConfigurationIdentity";
import {
createCouchDBJoinedDevicesResourceFactory,
createObjectStorageJoinedDevicesResourceFactory,
createCouchDBConnectionProbeFactory,
createCouchDBInitialFetchFastForwardResourceFactory,
createCouchDBPreferredTweakProbeFactory,
createCouchDBSecuritySeedResourceFactory,
createCouchDBSynchronisationInformationResourceFactory,
@@ -115,6 +118,9 @@ export function createCentralReplicatorProviderDefinitions(
configurationIdentity: getCouchDBReplicatorConfigurationIdentity,
create: () => Promise.resolve(new LiveSyncCouchDBReplicator(host)),
remoteResources: {
[REMOTE_RESOURCE_KINDS.JOINED_DEVICES]: supportedCapability(
createCouchDBJoinedDevicesResourceFactory(host)
),
[REMOTE_RESOURCE_KINDS.CONNECTION]: supportedCapability(createCouchDBConnectionProbeFactory(host)),
[REMOTE_RESOURCE_KINDS.PREFERRED_TWEAK]: supportedCapability(
createCouchDBPreferredTweakProbeFactory(host)
@@ -125,6 +131,9 @@ export function createCentralReplicatorProviderDefinitions(
[REMOTE_RESOURCE_KINDS.SYNCHRONISATION_INFORMATION]: supportedCapability(
createCouchDBSynchronisationInformationResourceFactory(host)
),
[REMOTE_RESOURCE_KINDS.INITIAL_FETCH_FAST_FORWARD]: supportedCapability(
createCouchDBInitialFetchFastForwardResourceFactory(host)
),
},
centralRemoteAdministration: COUCHDB_CENTRAL_REMOTE_ADMINISTRATION_CAPABILITY,
userInitiatedOneShot: supportedCapability(couchDBUserInitiatedOneShot),
@@ -141,6 +150,9 @@ export function createCentralReplicatorProviderDefinitions(
configurationIdentity: getObjectStorageReplicatorConfigurationIdentity,
create: () => Promise.resolve(new LiveSyncJournalReplicator(host)),
remoteResources: {
[REMOTE_RESOURCE_KINDS.JOINED_DEVICES]: supportedCapability(
createObjectStorageJoinedDevicesResourceFactory(host)
),
[REMOTE_RESOURCE_KINDS.CONNECTION]: supportedCapability(
createObjectStorageConnectionProbeFactory(host)
),
@@ -151,6 +163,7 @@ export function createCentralReplicatorProviderDefinitions(
createObjectStorageSecuritySeedResourceFactory(host)
),
[REMOTE_RESOURCE_KINDS.SYNCHRONISATION_INFORMATION]: CAPABILITY_NOT_APPLICABLE,
[REMOTE_RESOURCE_KINDS.INITIAL_FETCH_FAST_FORWARD]: CAPABILITY_NOT_APPLICABLE,
},
centralRemoteAdministration: OBJECT_STORAGE_CENTRAL_REMOTE_ADMINISTRATION_CAPABILITY,
userInitiatedOneShot: supportedCapability(objectStorageUserInitiatedOneShot),
src/common/replicatorProviders.unit.spec.ts
+21 -3
View File
@@ -44,7 +44,16 @@ describe("central Replicator provider definitions", () => {
it("keeps the retained remote-resource catalogue bounded", () => {
expect
.soft(Object.values(REMOTE_RESOURCE_KINDS).sort())
.toEqual(["connection", "preferred-tweak", "security-seed", "synchronisation-information"].sort());
.toEqual(
[
"connection",
"initial-fetch-fast-forward",
"joined-devices",
"preferred-tweak",
"security-seed",
"synchronisation-information",
].sort()
);
});
it("lets Journal prepare its own fresh Security Seed while CouchDB uses central preparation", () => {
@@ -106,18 +115,27 @@ describe("central Replicator provider definitions", () => {
const couchAdministration = definitions.get(REMOTE_COUCHDB)?.centralRemoteAdministration;
const objectAdministration = definitions.get(REMOTE_MINIO)?.centralRemoteAdministration;
expect(Object.keys(couchResources ?? {}).sort()).toEqual(Object.values(REMOTE_RESOURCE_KINDS).sort());
expect(Object.keys(objectResources ?? {}).sort()).toEqual(Object.values(REMOTE_RESOURCE_KINDS).sort());
const resources = Object.values(REMOTE_RESOURCE_KINDS).sort();
expect(Object.keys(couchResources ?? {}).sort()).toEqual(resources);
expect(Object.keys(objectResources ?? {}).sort()).toEqual(resources);
expect(couchResources?.[REMOTE_RESOURCE_KINDS.JOINED_DEVICES]?.kind).toBe(CAPABILITY_SUPPORT_KINDS.SUPPORTED);
expect(objectResources?.[REMOTE_RESOURCE_KINDS.JOINED_DEVICES]?.kind).toBe(CAPABILITY_SUPPORT_KINDS.SUPPORTED);
expect(couchResources?.[REMOTE_RESOURCE_KINDS.CONNECTION].kind).toBe(CAPABILITY_SUPPORT_KINDS.SUPPORTED);
expect(couchResources?.[REMOTE_RESOURCE_KINDS.PREFERRED_TWEAK].kind).toBe(CAPABILITY_SUPPORT_KINDS.SUPPORTED);
expect(couchResources?.[REMOTE_RESOURCE_KINDS.SECURITY_SEED].kind).toBe(CAPABILITY_SUPPORT_KINDS.SUPPORTED);
expect(couchResources?.[REMOTE_RESOURCE_KINDS.SYNCHRONISATION_INFORMATION].kind).toBe(
CAPABILITY_SUPPORT_KINDS.SUPPORTED
);
expect(couchResources?.[REMOTE_RESOURCE_KINDS.INITIAL_FETCH_FAST_FORWARD]?.kind).toBe(
CAPABILITY_SUPPORT_KINDS.SUPPORTED
);
expect(objectResources?.[REMOTE_RESOURCE_KINDS.SECURITY_SEED].kind).toBe(CAPABILITY_SUPPORT_KINDS.SUPPORTED);
expect(objectResources?.[REMOTE_RESOURCE_KINDS.SYNCHRONISATION_INFORMATION].kind).toBe(
CAPABILITY_SUPPORT_KINDS.NOT_APPLICABLE
);
expect(objectResources?.[REMOTE_RESOURCE_KINDS.INITIAL_FETCH_FAST_FORWARD]?.kind).toBe(
CAPABILITY_SUPPORT_KINDS.NOT_APPLICABLE
);
expect(couchAdministration?.kind).toBe(CAPABILITY_SUPPORT_KINDS.SUPPORTED);
expect(objectAdministration?.kind).toBe(CAPABILITY_SUPPORT_KINDS.SUPPORTED);
expect("activeRemoteReads" in definitions.get(REMOTE_COUCHDB)!).toBe(false);
src/common/replicatorResources.unit.spec.ts
+99
View File
@@ -5,6 +5,7 @@ import { createNewVaultSettings } from "@vrtmrz/livesync-commonlib/settings";
const mocks = vi.hoisted(() => ({
logger: vi.fn(),
initialFetchFastForwardResource: undefined as unknown,
couchDB: [] as Array<{
host: unknown;
isMobile: ReturnType<typeof vi.fn>;
@@ -12,6 +13,9 @@ const mocks = vi.hoisted(() => ({
getRemoteStatus: ReturnType<typeof vi.fn>;
getRemotePreferredTweakValues: ReturnType<typeof vi.fn>;
getReplicationPBKDF2Salt: ReturnType<typeof vi.fn>;
getJoinedDevices: ReturnType<typeof vi.fn>;
forgetJoinedDevice: ReturnType<typeof vi.fn>;
createInitialFetchFastForwardResource: ReturnType<typeof vi.fn>;
closeReplication: ReturnType<typeof vi.fn>;
}>,
objectStorage: [] as Array<{
@@ -20,6 +24,8 @@ const mocks = vi.hoisted(() => ({
getRemoteStatus: ReturnType<typeof vi.fn>;
getRemotePreferredTweakValues: ReturnType<typeof vi.fn>;
getReplicationPBKDF2Salt: ReturnType<typeof vi.fn>;
getJoinedDevices: ReturnType<typeof vi.fn>;
forgetJoinedDevice: ReturnType<typeof vi.fn>;
closeReplication: ReturnType<typeof vi.fn>;
}>,
checkSyncInfo: vi.fn(async () => true),
@@ -42,6 +48,9 @@ vi.mock("@vrtmrz/livesync-commonlib/compat/replication/couchdb/LiveSyncReplicato
getRemoteStatus = vi.fn();
getRemotePreferredTweakValues = vi.fn();
getReplicationPBKDF2Salt = vi.fn();
getJoinedDevices = vi.fn();
forgetJoinedDevice = vi.fn();
createInitialFetchFastForwardResource = vi.fn(() => mocks.initialFetchFastForwardResource);
closeReplication = vi.fn();
constructor(host: unknown) {
@@ -58,6 +67,8 @@ vi.mock("@vrtmrz/livesync-commonlib/compat/replication/journal/LiveSyncJournalRe
getRemoteStatus = vi.fn();
getRemotePreferredTweakValues = vi.fn();
getReplicationPBKDF2Salt = vi.fn();
getJoinedDevices = vi.fn();
forgetJoinedDevice = vi.fn();
closeReplication = vi.fn();
constructor(host: unknown) {
@@ -69,10 +80,13 @@ vi.mock("@vrtmrz/livesync-commonlib/compat/replication/journal/LiveSyncJournalRe
import {
createCouchDBConnectionProbeFactory,
createCouchDBJoinedDevicesResourceFactory,
createCouchDBInitialFetchFastForwardResourceFactory,
createCouchDBPreferredTweakProbeFactory,
createCouchDBSecuritySeedResourceFactory,
createCouchDBSynchronisationInformationResourceFactory,
createObjectStorageConnectionProbeFactory,
createObjectStorageJoinedDevicesResourceFactory,
createObjectStoragePreferredTweakProbeFactory,
createObjectStorageSecuritySeedResourceFactory,
} from "./replicatorResources";
@@ -92,6 +106,7 @@ describe("replicator probe factories", () => {
beforeEach(() => {
mocks.couchDB.length = 0;
mocks.objectStorage.length = 0;
mocks.initialFetchFastForwardResource = undefined;
mocks.checkSyncInfo.mockReset().mockResolvedValue(true);
mocks.logger.mockClear();
});
@@ -226,6 +241,65 @@ describe("replicator probe factories", () => {
expect(preferredReplicator.getRemotePreferredTweakValues).toHaveBeenCalledWith(snapshot);
});
it.each([
{
kind: "CouchDB",
remoteType: REMOTE_COUCHDB,
createFactory: createCouchDBJoinedDevicesResourceFactory,
instances: () => mocks.couchDB,
},
{
kind: "Object Storage",
remoteType: REMOTE_MINIO,
createFactory: createObjectStorageJoinedDevicesResourceFactory,
instances: () => mocks.objectStorage,
},
])(
"keeps $kind joined-device operations on an owned settings snapshot",
async ({ remoteType, createFactory, instances }) => {
const host = { name: "host" };
const source = createSettings({ remoteType });
const snapshot = { ...source };
const factory = createFactory(host as never);
const resource = await factory(source);
const nextResource = await factory(source);
const [replicator, nextReplicator] = instances();
expect(instances()).toHaveLength(2);
expect(replicator.host).toBe(host);
expect(replicator.getJoinedDevices).not.toHaveBeenCalled();
expect(replicator.forgetJoinedDevice).not.toHaveBeenCalled();
const devices = {
milestonePresent: true,
locked: false,
cleaned: false,
devices: [{ id: "other", accepted: true }],
};
replicator.getJoinedDevices.mockResolvedValue(devices);
replicator.forgetJoinedDevice.mockResolvedValue("conflict");
source.couchDB_DBNAME = "changed-vault";
source.bucket = "changed-bucket";
await expect(resource.read()).resolves.toBe(devices);
await expect(resource.forget("other")).resolves.toBe("conflict");
expect(replicator.getJoinedDevices).toHaveBeenCalledWith(snapshot);
expect(replicator.getJoinedDevices.mock.calls[0][0]).not.toBe(source);
expect(replicator.forgetJoinedDevice).toHaveBeenCalledWith(snapshot, "other");
const failure = new Error("remote unavailable");
replicator.getJoinedDevices.mockRejectedValue(failure);
await expect(resource.read()).rejects.toBe(failure);
replicator.forgetJoinedDevice.mockRejectedValue(failure);
await expect(resource.forget("other")).rejects.toBe(failure);
const disposal = resource.dispose();
expect(resource.dispose()).toBe(disposal);
await disposal;
expect(replicator.closeReplication).toHaveBeenCalledOnce();
expect(nextReplicator.closeReplication).not.toHaveBeenCalled();
await nextResource.dispose();
expect(nextReplicator.closeReplication).toHaveBeenCalledOnce();
}
);
it("shares one successful asynchronous disposal promise for every probe kind", async () => {
const couchProbe = await createCouchDBPreferredTweakProbeFactory({} as never)(createSettings());
const objectProbe = await createObjectStoragePreferredTweakProbeFactory({} as never)(
@@ -370,4 +444,29 @@ describe("replicator probe factories", () => {
await resource.dispose();
expect(replicator.closeReplication).toHaveBeenCalledOnce();
});
it("creates an Initial Fetch Fast Forward resource from a settings snapshot and leaves ownership to the resource", async () => {
const host = { name: "host" };
const settings = createSettings();
const snapshot = { ...settings };
const prepare = vi.fn(async () => undefined);
const dispose = vi.fn(async () => undefined);
const resource = { prepare, dispose };
const factory = createCouchDBInitialFetchFastForwardResourceFactory(host as never);
mocks.initialFetchFastForwardResource = resource;
const result = await factory(settings);
const replicator = mocks.couchDB[0];
settings.couchDB_URI = "https://changed.example.test";
expect(replicator.createInitialFetchFastForwardResource).toHaveBeenCalledWith(snapshot);
expect(replicator.createInitialFetchFastForwardResource.mock.calls[0][0]).not.toBe(settings);
expect(replicator.host).toBe(host);
expect(result).toBe(resource);
await expect(result.prepare()).resolves.toBeUndefined();
await result.dispose();
expect(prepare).toHaveBeenCalledOnce();
expect(dispose).toHaveBeenCalledOnce();
});
});
src/common/replicatorResources/index.ts
+6
View File
@@ -14,3 +14,9 @@ export {
type SecuritySeedResourceHost,
} from "./securitySeed";
export { createCouchDBSynchronisationInformationResourceFactory } from "./synchronisationInformation";
export {
createCouchDBJoinedDevicesResourceFactory,
createObjectStorageJoinedDevicesResourceFactory,
} from "./joinedDevices";
export { createCouchDBInitialFetchFastForwardResourceFactory } from "./initialFetchFastForward";
src/common/replicatorResources/initialFetchFastForward.ts
+17
View File
@@ -0,0 +1,17 @@
import type { InitialFetchFastForwardResourceFactory } from "@vrtmrz/livesync-commonlib/replication";
import {
LiveSyncCouchDBReplicator,
type LiveSyncCouchDBReplicatorEnv,
} from "@vrtmrz/livesync-commonlib/compat/replication/couchdb/LiveSyncReplicator";
import { snapshotRemoteSettings } from "./shared";
/** Create a private CouchDB Replicator for a fresh Fast Fetch checkpoint handoff. */
export function createCouchDBInitialFetchFastForwardResourceFactory(
host: LiveSyncCouchDBReplicatorEnv
): InitialFetchFastForwardResourceFactory {
return (setting) => {
const snapshot = snapshotRemoteSettings(setting);
const replicator = new LiveSyncCouchDBReplicator(host);
return Promise.resolve(replicator.createInitialFetchFastForwardResource(snapshot));
};
}
src/common/replicatorResources/joinedDevices.ts
+40
View File
@@ -0,0 +1,40 @@
import type { RemoteDBSettings } from "@vrtmrz/livesync-commonlib/compat/common/types";
import type {
JoinedDevicesResource,
JoinedDevicesResourceFactory,
JoinedDevicesSnapshot,
ForgetJoinedDeviceResult,
} from "@vrtmrz/livesync-commonlib/replication";
import {
LiveSyncCouchDBReplicator,
type LiveSyncCouchDBReplicatorEnv,
} from "@vrtmrz/livesync-commonlib/compat/replication/couchdb/LiveSyncReplicator";
import { LiveSyncJournalReplicator } from "@vrtmrz/livesync-commonlib/compat/replication/journal/LiveSyncJournalReplicator";
import { createReplicatorDisposer, snapshotRemoteSettings, type ResourceReplicator } from "./shared";
interface JoinedDevicesReplicator extends ResourceReplicator {
getJoinedDevices(setting: RemoteDBSettings): Promise<JoinedDevicesSnapshot>;
forgetJoinedDevice(setting: RemoteDBSettings, deviceId: string): Promise<ForgetJoinedDeviceResult>;
}
function createResource(replicator: JoinedDevicesReplicator, snapshot: RemoteDBSettings): JoinedDevicesResource {
return {
read: () => replicator.getJoinedDevices(snapshot),
forget: (deviceId) => replicator.forgetJoinedDevice(snapshot, deviceId),
dispose: createReplicatorDisposer(replicator),
};
}
export function createCouchDBJoinedDevicesResourceFactory(
host: LiveSyncCouchDBReplicatorEnv
): JoinedDevicesResourceFactory {
return (setting) =>
Promise.resolve(createResource(new LiveSyncCouchDBReplicator(host), snapshotRemoteSettings(setting)));
}
export function createObjectStorageJoinedDevicesResourceFactory(
host: LiveSyncCouchDBReplicatorEnv
): JoinedDevicesResourceFactory {
return (setting) =>
Promise.resolve(createResource(new LiveSyncJournalReplicator(host), snapshotRemoteSettings(setting)));
}
src/common/turnSettingsPrivacy.unit.spec.ts
+2 -10
View File
@@ -1,3 +1,4 @@
import { MemoryValueStore } from "@vrtmrz/value-store";
import { describe, expect, it } from "vitest";
import {
DEFAULT_SETTINGS,
@@ -18,17 +19,7 @@ import {
} from "./turnSettingsPrivacy";
class MemorySettingService extends SettingService {
readonly items = new Map<string, string>();
saved?: ObsidianLiveSyncSettings;
protected setItem(key: string, value: string) {
this.items.set(key, value);
}
protected getItem(key: string) {
return this.items.get(key) ?? "";
}
protected deleteItem(key: string) {
this.items.delete(key);
}
protected saveData(settings: ObsidianLiveSyncSettings) {
this.saved = structuredClone(settings);
return Promise.resolve();
@@ -70,6 +61,7 @@ describe("managed TURN settings privacy", () => {
const originalURI = ConnectionStringParser.serialize({ type: "p2p", settings: current });
current.remoteConfigurations.managed.uri = originalURI;
const service = new MemorySettingService(new ServiceContext(), {
deviceValues: new MemoryValueStore(),
APIService: {
getSystemVaultName: () => "test-vault",
getAppID: () => "test-app",
src/features/HiddenFileSync/CmdHiddenFileSync.selectors.unit.spec.ts
+127
View File
@@ -0,0 +1,127 @@
import { afterEach, describe, expect, it, vi } from "vitest";
vi.mock("@/deps.ts", () => ({}));
vi.mock("@/features/HiddenFileCommon/JsonResolveModal.ts", () => ({ JsonResolveModal: class JsonResolveModal {} }));
vi.mock("@/modules/features/ModuleLog.ts", () => ({ MARK_DONE: "" }));
vi.mock("./configureHiddenFileSyncMode.ts", () => ({ configureHiddenFileSyncMode: vi.fn() }));
import type { FilePath } from "@vrtmrz/livesync-commonlib/compat/common/types";
import { readContent } from "@vrtmrz/livesync-commonlib/compat/common/utils";
import { createOptionalFileSyncDatabase, optionalFileEntry } from "../../../test/fixtures/optionalFileSyncDatabase";
import { HiddenFileSync } from "./CmdHiddenFileSync";
const hiddenPath = ".obsidian/plugins/example/main.js" as FilePath;
const prefixedPath = `i:${hiddenPath}`;
const body = "module.exports = class ExamplePlugin {};";
const allowList = "\\.(?:md|png|pdf)$";
const fixtureOwners: Awaited<ReturnType<typeof createOptionalFileSyncDatabase>>[] = [];
async function hiddenFiles() {
const fixture = await createOptionalFileSyncDatabase();
fixtureOwners.push(fixture);
fixture.settings.usePluginSync = false;
const hiddenFileSync = Object.create(HiddenFileSync.prototype) as HiddenFileSync;
Object.assign(hiddenFileSync, {
core: {
settings: fixture.settings,
services: {
...fixture.services,
vault: { isIgnoredByIgnoreFile: vi.fn(async () => false) },
},
},
cacheFileRegExps: new Map(),
cacheCustomisationSyncIgnoredFiles: new Map(),
});
return { ...fixture, hiddenFileSync };
}
afterEach(async () => {
vi.restoreAllMocks();
for (const fixture of fixtureOwners.splice(0)) await fixture.dispose();
});
describe("issue #1272: Hidden File Sync with normal-file selectors", () => {
it("stores a selected hidden file while the normal-file allow-list is set", async () => {
const fixture = await hiddenFiles();
fixture.settings.syncOnlyRegEx = allowList as typeof fixture.settings.syncOnlyRegEx;
await expect(fixture.hiddenFileSync.isTargetFile(hiddenPath)).resolves.toBe(true);
await expect(fixture.localDatabase.putDBEntry(optionalFileEntry(prefixedPath, body))).resolves.toMatchObject({
ok: true,
});
const stored = await fixture.localDatabase.getDBEntry(prefixedPath as FilePath);
expect(stored).not.toBe(false);
if (stored) expect(await readContent(stored)).toBe(body);
});
it("reads existing hidden-file content while the normal-file allow-list is set", async () => {
const fixture = await hiddenFiles();
await expect(fixture.localDatabase.putDBEntry(optionalFileEntry(prefixedPath, body))).resolves.toMatchObject({
ok: true,
});
const meta = await fixture.localDatabase.getDBEntryMeta(prefixedPath as FilePath);
if (!meta) throw new Error("The seeded hidden-file Metadata is unavailable");
fixture.settings.syncOnlyRegEx = allowList as typeof fixture.settings.syncOnlyRegEx;
await expect(fixture.hiddenFileSync.isTargetFile(hiddenPath)).resolves.toBe(true);
const entry = await fixture.localDatabase.getDBEntryFromMeta(meta);
expect(entry).not.toBe(false);
if (entry) expect(await readContent(entry)).toBe(body);
});
it("stores and reads the same hidden file when the normal-file allow-list is empty", async () => {
const fixture = await hiddenFiles();
await expect(fixture.localDatabase.putDBEntry(optionalFileEntry(prefixedPath, body))).resolves.toMatchObject({
ok: true,
});
const entry = await fixture.localDatabase.getDBEntry(prefixedPath as FilePath);
expect(entry).not.toBe(false);
if (entry) expect(await readContent(entry)).toBe(body);
});
it("continues to apply the normal-file allow-list to ordinary files", async () => {
const fixture = await hiddenFiles();
fixture.settings.syncOnlyRegEx = allowList as typeof fixture.settings.syncOnlyRegEx;
expect(fixture.localDatabase.isTargetFile("notes/example.md" as FilePath)).toBe(true);
expect(fixture.localDatabase.isTargetFile("notes/example.js" as FilePath)).toBe(false);
expect(fixture.localDatabase.isTargetFile("i:.obsidian/example.md" as FilePath)).toBe(true);
});
it("keeps Customisation Sync documents usable with the same normal-file allow-list", async () => {
const fixture = await hiddenFiles();
fixture.settings.syncOnlyRegEx = allowList as typeof fixture.settings.syncOnlyRegEx;
const path = "ix:device/PLUGIN_MAIN/example%main.js" as FilePath;
await expect(fixture.localDatabase.putDBEntry(optionalFileEntry(path, body))).resolves.toMatchObject({
ok: true,
});
const entry = await fixture.localDatabase.getDBEntry(path);
expect(entry).not.toBe(false);
if (entry) expect(await readContent(entry)).toBe(body);
});
it("keeps Hidden File Sync target and ignore patterns effective", async () => {
const fixture = await hiddenFiles();
fixture.settings.syncInternalFilesTargetPatterns =
"^\\.obsidian/plugins/" as typeof fixture.settings.syncInternalFilesTargetPatterns;
await expect(fixture.hiddenFileSync.isTargetFile(hiddenPath)).resolves.toBe(true);
await expect(fixture.hiddenFileSync.isTargetFile(".obsidian/snippets/example.css" as FilePath)).resolves.toBe(
false
);
fixture.settings.syncInternalFilesIgnorePatterns =
"main\\.js$" as typeof fixture.settings.syncInternalFilesIgnorePatterns;
await expect(fixture.hiddenFileSync.isTargetFile(hiddenPath)).resolves.toBe(false);
});
it("compatibility: applies the normal-file ignore-list to hidden documents", async () => {
const fixture = await hiddenFiles();
fixture.settings.syncOnlyRegEx = allowList as typeof fixture.settings.syncOnlyRegEx;
fixture.settings.syncIgnoreRegEx = "^\\.obsidian/" as typeof fixture.settings.syncIgnoreRegEx;
expect(fixture.localDatabase.isTargetFile(prefixedPath as FilePath)).toBe(false);
});
it("rejects hidden documents when Hidden File Sync is disabled", async () => {
const fixture = await hiddenFiles();
fixture.settings.syncInternalFiles = false;
expect(fixture.localDatabase.isTargetFile(prefixedPath as FilePath)).toBe(false);
});
});
src/features/P2PSync/P2PReplicator/P2PReplicatorPane.svelte
+115 -19
View File
@@ -1,5 +1,6 @@
<script lang="ts">
import { onMount } from "svelte";
import { ValueStoreError } from "@vrtmrz/value-store";
import { AutoAccepting, DEFAULT_SETTINGS, type P2PSyncSetting } from "@vrtmrz/livesync-commonlib/compat/common/types";
import {
AcceptedStatus,
@@ -33,6 +34,14 @@
let settings = $state<P2PSyncSetting>(initialSettings);
let deviceName = $state<string>("");
let deviceNameReady = $state(false);
let dialogStatusReady = $state(false);
type StorageErrorKey = "deviceNameLoad" | "dialogStatusLoad" | "deviceNameWrite" | "dialogStatusWrite";
let storageErrors = $state<Partial<Record<StorageErrorKey, string>>>({});
let mounted = false;
let deviceNameLoadSequence = 0;
let deviceNameWriteQueue: Promise<void> = Promise.resolve();
let dialogStatusWriteQueue: Promise<void> = Promise.resolve();
let eP2PEnabled = $state<boolean>(initialSettings.P2P_Enabled);
let eRelay = $state<string>(initialSettings.P2P_relays);
@@ -68,6 +77,7 @@
);
async function saveAndApply() {
const savedDeviceName = eDeviceName;
// const newSettings = {
// ...currentSettings(),
// P2P_Enabled: eP2PEnabled,
@@ -92,8 +102,39 @@
},
true
);
services.config.setSmallConfig(SETTING_KEY_P2P_DEVICE_NAME, eDeviceName);
deviceName = eDeviceName;
try {
await queueDeviceNameWrite(savedDeviceName);
if (!mounted) return;
deviceName = savedDeviceName;
clearStorageError("deviceNameWrite");
} catch (error) {
setStorageError("deviceNameWrite", error);
}
}
function setStorageError(key: StorageErrorKey, error: unknown) {
if (mounted) {
storageErrors = {
...storageErrors,
[key]: error instanceof Error ? error.message : String(error),
};
}
}
function clearStorageError(key: StorageErrorKey) {
if (!mounted) return;
const nextErrors = { ...storageErrors };
delete nextErrors[key];
storageErrors = nextErrors;
}
function queueDeviceNameWrite(name: string): Promise<void> {
const write = deviceNameWriteQueue.catch(() => undefined).then(async () => {
if (!mounted) return;
await services.config.setValue(SETTING_KEY_P2P_DEVICE_NAME, name);
});
deviceNameWriteQueue = write;
return write;
}
async function revert() {
eP2PEnabled = settings.P2P_Enabled;
@@ -110,10 +151,7 @@
let replicatorInfo = $state<P2PReplicatorStatus | undefined>(undefined);
const applyLoadSettings = (d: P2PSyncSetting, force: boolean) => {
if (force) {
const initDeviceName =
services.config.getSmallConfig(SETTING_KEY_P2P_DEVICE_NAME) ?? services.vault.getVaultName();
deviceName = initDeviceName;
eDeviceName = initDeviceName;
void loadDeviceName(true).catch(() => undefined);
}
const { P2P_relays, P2P_roomID, P2P_passphrase, P2P_AppID, P2P_AutoAccepting } = d;
if (force || !isP2PEnabledModified) eP2PEnabled = d.P2P_Enabled;
@@ -129,6 +167,7 @@
settings = d;
};
onMount(() => {
mounted = true;
const r = events.onEvent("setting-saved", async (d) => {
applyLoadSettings(d, false);
closeServer();
@@ -143,15 +182,52 @@
const r3 = events.onEvent(EVENT_P2P_REPLICATOR_STATUS, (status) => {
replicatorInfo = status;
});
applyLoadSettings(currentSettings(), true);
applyLoadSettings(currentSettings(), false);
void loadInitialPreferences().catch(() => undefined);
host.p2p.diagnostics.requestStatus();
return () => {
mounted = false;
r();
rx();
r2();
r3();
};
});
async function loadInitialPreferences() {
const deviceNameLoad = loadDeviceName(true).catch(() => undefined);
const dialogStatusLoad = readDialogStatus().then(
(dialogStatus) => {
if (!mounted) return;
isNoticeOpened = dialogStatus.notice ?? true;
isSettingOpened = dialogStatus.setting ?? true;
dialogStatusReady = true;
clearStorageError("dialogStatusLoad");
},
(error: unknown) => {
setStorageError("dialogStatusLoad", error);
}
);
await Promise.all([deviceNameLoad, dialogStatusLoad]);
}
async function loadDeviceName(force: boolean) {
const request = ++deviceNameLoadSequence;
try {
const savedDeviceName = await services.config.getValue(SETTING_KEY_P2P_DEVICE_NAME, "");
if (!mounted || request !== deviceNameLoadSequence) return;
const initialDeviceName = savedDeviceName;
if (force || !isDeviceNameModified) {
deviceName = initialDeviceName;
eDeviceName = initialDeviceName;
}
deviceNameReady = true;
clearStorageError("deviceNameLoad");
} catch (error) {
if (mounted && request === deviceNameLoadSequence) setStorageError("deviceNameLoad", error);
throw error;
}
}
let isConnected = $derived.by(() => {
return serverInfo?.isConnected ?? false;
});
@@ -234,26 +310,42 @@
}
const initialDialogStatusKey = `p2p-dialog-status`;
const getDialogStatus = () => {
const readDialogStatus = async (): Promise<{ notice?: boolean; setting?: boolean }> => {
try {
const initialDialogStatus = JSON.parse(services.config.getSmallConfig(initialDialogStatusKey) ?? "{}") as {
notice?: boolean;
setting?: boolean;
const initialDialogStatus = await services.config.getObject(initialDialogStatusKey);
if (typeof initialDialogStatus !== "object" || initialDialogStatus === null || Array.isArray(initialDialogStatus)) {
return {};
}
const status = initialDialogStatus as Record<string, unknown>;
return {
...(typeof status.notice === "boolean" ? { notice: status.notice } : {}),
...(typeof status.setting === "boolean" ? { setting: status.setting } : {}),
};
return initialDialogStatus;
} catch {
return {};
} catch (error) {
if (error instanceof ValueStoreError && error.code === "invalid-data") {
// Invalid optional UI state is recovered with the historical defaults.
return {};
}
throw error;
}
};
const initialDialogStatus = getDialogStatus();
let isNoticeOpened = $state<boolean>(initialDialogStatus.notice ?? true);
let isSettingOpened = $state<boolean>(initialDialogStatus.setting ?? true);
let isNoticeOpened = $state<boolean>(true);
let isSettingOpened = $state<boolean>(true);
$effect(() => {
if (!mounted || !dialogStatusReady) return;
const dialogStatus = {
notice: isNoticeOpened,
setting: isSettingOpened,
};
services.config.setSmallConfig(initialDialogStatusKey, JSON.stringify(dialogStatus));
const write = dialogStatusWriteQueue.catch(() => undefined).then(async () => {
if (!mounted) return;
await services.config.setObject(initialDialogStatusKey, dialogStatus);
});
dialogStatusWriteQueue = write;
void write.then(
() => clearStorageError("dialogStatusWrite"),
(error: unknown) => setStorageError("dialogStatusWrite", error)
);
});
let isObsidian = $derived.by(() => {
return services.API.getPlatform() === "obsidian";
@@ -261,6 +353,9 @@
</script>
<article>
{#each Object.values(storageErrors) as storageError}
<p role="alert">{storageError}</p>
{/each}
<h1>{_msg("Peer to Peer Replicator")}</h1>
<details bind:open={isNoticeOpened}>
<summary>{_msg("P2P.Note.Summary")}</summary>
@@ -345,6 +440,7 @@
type="text"
placeholder="iphone-16"
bind:value={eDeviceName}
disabled={!deviceNameReady}
autocomplete="off"
/>
</label>
@@ -383,7 +479,7 @@
</tr> -->
</tbody>
</table>
<button disabled={!isAnyModified} class="button mod-cta" onclick={saveAndApply}
<button disabled={!isAnyModified || !deviceNameReady} class="button mod-cta" onclick={saveAndApply}
>{_msg("Save and Apply")}</button
>
<button disabled={!isAnyModified} class="button" onclick={revert}>{_msg("Revert changes")}</button>
src/features/ReviewHarness/ReviewHarnessView.ts
+2 -1
View File
@@ -1,4 +1,5 @@
import { ItemView, Notice, Setting, type WorkspaceLeaf } from "@/deps";
import { LOG_LEVEL_NOTICE, Logger } from "octagonal-wheels/common/logger";
import {
REVIEW_HARNESS_SCENARIOS,
type ReviewHarnessScenarioId,
@@ -101,7 +102,7 @@ export class ReviewHarnessView extends ItemView {
true
);
this.addActionButton(actions, "Restart and return", "review-harness-restart", () =>
this.controller.prepareCompatibilityReviewRestart()
this.controller.prepareCompatibilityReviewRestart().catch((error) => Logger(error, LOG_LEVEL_NOTICE))
);
}
}
src/features/ReviewHarness/reviewHarnessController.ts
+19 -8
View File
@@ -23,9 +23,9 @@ export interface ReviewHarnessRuntime {
runVaultRoundTrip(): Promise<ReviewHarnessScenarioResult>;
runIdBenchmark(): Promise<ReviewHarnessScenarioResult>;
runFileContentHashBenchmark(): Promise<ReviewHarnessScenarioResult>;
readContinuation(): string | null;
writeContinuation(value: string): void;
deleteContinuation(): void;
readContinuation(): Promise<string | null>;
writeContinuation(value: string): Promise<void>;
deleteContinuation(): Promise<void>;
restart(): void;
reportError(error: unknown): void;
copyText(value: string): Promise<void>;
@@ -68,6 +68,7 @@ export class ReviewHarnessController {
private current: ReviewHarnessScenarioId | null = null;
private resumedRequestId: string | null = null;
private continuationError: string | null = null;
private continuationConsumption: Promise<void> | undefined;
constructor(private readonly runtime: ReviewHarnessRuntime) {}
@@ -102,10 +103,20 @@ export class ReviewHarnessController {
};
}
consumeContinuation(): void {
const serialised = this.runtime.readContinuation();
consumeContinuation(): Promise<void> {
if (this.continuationConsumption) return this.continuationConsumption;
const consumption = this.consumeContinuationFromStore();
const trackedConsumption = consumption.finally(() => {
if (this.continuationConsumption === trackedConsumption) this.continuationConsumption = undefined;
});
this.continuationConsumption = trackedConsumption;
return trackedConsumption;
}
private async consumeContinuationFromStore(): Promise<void> {
const serialised = await this.runtime.readContinuation();
if (!serialised) return;
this.runtime.deleteContinuation();
await this.runtime.deleteContinuation();
const parsed = parsePendingReviewRun(serialised);
if (!parsed.pendingRun) {
this.continuationError = parsed.error ?? "The Review Harness continuation was invalid.";
@@ -232,7 +243,7 @@ export class ReviewHarnessController {
this.record("scenario-failed", id);
}
prepareCompatibilityReviewRestart(): void {
async prepareCompatibilityReviewRestart(): Promise<void> {
const requestedAt = this.runtime.now().toISOString();
const pending = {
formatVersion: 1,
@@ -241,7 +252,7 @@ export class ReviewHarnessController {
stage: "awaiting-restart",
requestedAt,
} as const;
this.runtime.writeContinuation(JSON.stringify(pending));
await this.runtime.writeContinuation(JSON.stringify(pending));
this.results["compatibility-review"] = {
status: "waiting-for-user",
detail: "Restart requested. The one-shot continuation will be removed before the review resumes.",
src/features/ReviewHarness/reviewHarnessController.unit.spec.ts
+140 -11
View File
@@ -90,14 +90,14 @@ function createRuntime(): ReviewHarnessRuntime & {
detail: "File content hash measurements completed.",
observations: ["SHA-256, 4 KiB: median=1.00 ms; range=1.00–1.00 ms"],
})),
readContinuation() {
async readContinuation() {
return this.continuation;
},
writeContinuation(value) {
async writeContinuation(value) {
this.events.push("write");
this.continuation = value;
},
deleteContinuation() {
async deleteContinuation() {
this.events.push("delete");
this.continuation = null;
},
@@ -251,7 +251,7 @@ describe("ReviewHarnessController", () => {
expect(controller.snapshot().running).toBe(false);
});
it("deletes a one-shot continuation before exposing the resumed guided step", () => {
it("deletes a one-shot continuation before exposing the resumed guided step", async () => {
const runtime = createRuntime();
runtime.continuation = JSON.stringify({
formatVersion: 1,
@@ -262,7 +262,7 @@ describe("ReviewHarnessController", () => {
});
const controller = new ReviewHarnessController(runtime);
controller.consumeContinuation();
await controller.consumeContinuation();
expect(runtime.events).toEqual(["delete"]);
expect(controller.snapshot().results["compatibility-review"]).toMatchObject({
@@ -271,7 +271,7 @@ describe("ReviewHarnessController", () => {
expect(controller.snapshot().resumedRequestId).toBe("compatibility-review-2026-07-18T11:59:00.000Z");
});
it("does not copy rejected continuation values into the report", () => {
it("does not copy rejected continuation values into the report", async () => {
const runtime = createRuntime();
runtime.continuation = JSON.stringify({
formatVersion: 1,
@@ -282,7 +282,7 @@ describe("ReviewHarnessController", () => {
});
const controller = new ReviewHarnessController(runtime);
controller.consumeContinuation();
await controller.consumeContinuation();
expect(controller.snapshot().continuationError).toContain("private-vault");
expect(controller.snapshot().results["compatibility-review"].detail).toBe(
@@ -292,7 +292,7 @@ describe("ReviewHarnessController", () => {
expect(controller.createReport()).not.toContain("secret-note.md");
});
it("does not accept an injected request ID which could enter the report transcript", () => {
it("does not accept an injected request ID which could enter the report transcript", async () => {
const runtime = createRuntime();
runtime.continuation = JSON.stringify({
formatVersion: 1,
@@ -303,18 +303,18 @@ describe("ReviewHarnessController", () => {
});
const controller = new ReviewHarnessController(runtime);
controller.consumeContinuation();
await controller.consumeContinuation();
expect(controller.snapshot().results["compatibility-review"].status).toBe("failed");
expect(controller.createReport()).not.toContain("private-vault");
expect(controller.createReport()).not.toContain("secret-note.md");
});
it("persists the fixed continuation before requesting restart", () => {
it("persists the fixed continuation before requesting restart", async () => {
const runtime = createRuntime();
const controller = new ReviewHarnessController(runtime);
controller.prepareCompatibilityReviewRestart();
await controller.prepareCompatibilityReviewRestart();
expect(runtime.events).toEqual(["write", "restart"]);
expect(JSON.parse(runtime.continuation!)).toMatchObject({
@@ -324,6 +324,135 @@ describe("ReviewHarnessController", () => {
});
});
it("waits for the continuation save before requesting restart", async () => {
const runtime = createRuntime();
let markSaveStarted!: () => void;
const saveStarted = new Promise<void>((resolve) => {
markSaveStarted = resolve;
});
let finishSave!: () => void;
const saving = new Promise<void>((resolve) => {
finishSave = resolve;
});
runtime.writeContinuation = vi.fn(async (value) => {
runtime.events.push("write-started");
markSaveStarted();
await saving;
runtime.continuation = value;
runtime.events.push("write-saved");
});
const controller = new ReviewHarnessController(runtime);
const preparing = controller.prepareCompatibilityReviewRestart();
await saveStarted;
expect(runtime.events).toEqual(["write-started"]);
expect(runtime.restart).not.toHaveBeenCalled();
expect(controller.snapshot().results["compatibility-review"].status).toBe("idle");
finishSave();
await preparing;
expect(runtime.events).toEqual(["write-started", "write-saved", "restart"]);
expect(controller.snapshot().results["compatibility-review"].status).toBe("waiting-for-user");
});
it("keeps the guided review retryable when saving its continuation fails", async () => {
const runtime = createRuntime();
const controller = new ReviewHarnessController(runtime);
await controller.runScenario("compatibility-review");
runtime.writeContinuation = vi.fn(async () => {
runtime.events.push("write-failed");
throw new Error("continuation save failed");
});
await expect(controller.prepareCompatibilityReviewRestart()).rejects.toThrow("continuation save failed");
expect(runtime.events).toEqual(["write-failed"]);
expect(runtime.restart).not.toHaveBeenCalled();
expect(controller.snapshot().results["compatibility-review"].status).toBe("waiting-for-user");
});
it("waits for one continuation deletion before exposing it and coalesces concurrent consumption", async () => {
const runtime = createRuntime();
runtime.continuation = JSON.stringify({
formatVersion: 1,
requestId: "compatibility-review-2026-07-18T11:59:00.000Z",
scenarioId: "compatibility-review",
stage: "awaiting-restart",
requestedAt: "2026-07-18T11:59:00.000Z",
});
let markDeleteStarted!: () => void;
const deleteStarted = new Promise<void>((resolve) => {
markDeleteStarted = resolve;
});
let finishDelete!: () => void;
const deleting = new Promise<void>((resolve) => {
finishDelete = resolve;
});
runtime.readContinuation = vi.fn(async () => {
runtime.events.push("read");
return runtime.continuation;
});
runtime.deleteContinuation = vi.fn(async () => {
runtime.events.push("delete-started");
markDeleteStarted();
await deleting;
runtime.continuation = null;
runtime.events.push("delete-finished");
});
const controller = new ReviewHarnessController(runtime);
const firstConsumption = controller.consumeContinuation();
const duplicateConsumption = controller.consumeContinuation();
await deleteStarted;
expect(duplicateConsumption).toBe(firstConsumption);
expect(runtime.events).toEqual(["read", "delete-started"]);
expect(controller.snapshot().resumedRequestId).toBeNull();
expect(controller.snapshot().results["compatibility-review"].status).toBe("idle");
finishDelete();
await firstConsumption;
expect(runtime.events).toEqual(["read", "delete-started", "delete-finished"]);
expect(runtime.readContinuation).toHaveBeenCalledOnce();
expect(runtime.deleteContinuation).toHaveBeenCalledOnce();
expect(controller.snapshot().resumedRequestId).toBe("compatibility-review-2026-07-18T11:59:00.000Z");
expect(controller.snapshot().results["compatibility-review"].status).toBe("waiting-for-user");
});
it("keeps a failed continuation deletion available for retry", async () => {
const runtime = createRuntime();
runtime.continuation = JSON.stringify({
formatVersion: 1,
requestId: "compatibility-review-2026-07-18T11:59:00.000Z",
scenarioId: "compatibility-review",
stage: "awaiting-restart",
requestedAt: "2026-07-18T11:59:00.000Z",
});
runtime.deleteContinuation = vi
.fn<ReviewHarnessRuntime["deleteContinuation"]>()
.mockRejectedValueOnce(new Error("continuation delete failed"))
.mockImplementation(async () => {
runtime.events.push("delete");
runtime.continuation = null;
});
const controller = new ReviewHarnessController(runtime);
await expect(controller.consumeContinuation()).rejects.toThrow("continuation delete failed");
expect(runtime.continuation).not.toBeNull();
expect(controller.snapshot().resumedRequestId).toBeNull();
expect(controller.snapshot().results["compatibility-review"].status).toBe("idle");
await controller.consumeContinuation();
expect(runtime.continuation).toBeNull();
expect(controller.snapshot().resumedRequestId).toBe("compatibility-review-2026-07-18T11:59:00.000Z");
expect(controller.snapshot().results["compatibility-review"].status).toBe("waiting-for-user");
});
it("passes only after the actual compatibility review clears its pause", async () => {
const runtime = createRuntime();
const controller = new ReviewHarnessController(runtime);
src/main.ts
+2
View File
@@ -51,6 +51,7 @@ import { useInteractiveConflictResolutionFeature } from "./serviceFeatures/inter
import { ConflictResolveModal } from "./modules/features/InteractiveConflictResolving/ConflictResolveModal.ts";
import { useObsidianReplicationRibbonFeature } from "./serviceFeatures/obsidianReplicationRibbon.ts";
import { useStartupLifecycleFeature } from "./serviceFeatures/startupLifecycle";
import { useJoinedDevicesFeature } from "./serviceFeatures/useJoinedDevices";
import { useDatabaseDiagnosticsFeature } from "./serviceFeatures/useDatabaseDiagnostics";
export type LiveSyncCore = LiveSyncBaseCore<ObsidianServiceContext, LiveSyncCommands>;
export default class ObsidianLiveSyncPlugin extends Plugin {
@@ -197,6 +198,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin {
useP2PReplicatorUI(core, core, replicator, createInteractiveP2PReplication(replicator));
useObsidianReplicationRibbonFeature(core);
useDatabaseDiagnosticsFeature(core);
useJoinedDevicesFeature(core, serviceHub.context.noticeGroups);
useRemoteConfiguration(core);
useSetupProtocolFeature(core, setupManager);
src/modules/features/ModuleObsidianSettingTab.ts
+3 -3
View File
@@ -8,13 +8,13 @@ import { openObsidianSettings } from "@/common/obsidianSettings.ts";
export class ModuleObsidianSettingDialogue extends AbstractObsidianModule {
settingTab!: ObsidianLiveSyncSettingTab;
_everyOnloadAfterLoadSettings(): Promise<boolean> {
async _everyOnloadAfterLoadSettings(): Promise<boolean> {
this.settingTab = new ObsidianLiveSyncSettingTab(this.app, this.plugin);
this.settingTab.reloadAllSettings(true);
await this.settingTab.loadSettings();
this.plugin.addSettingTab(this.settingTab);
eventHub.onEvent(EVENT_REQUEST_OPEN_SETTINGS, () => this.openSetting());
return Promise.resolve(true);
return true;
}
openSetting() {
src/modules/features/ModuleObsidianSettingTab.unit.spec.ts
+25 -9
View File
@@ -2,7 +2,7 @@ import { beforeEach, describe, expect, it, vi } from "vitest";
const settingTabState = vi.hoisted(() => ({
callOrder: [] as string[],
reloadAllSettings: vi.fn<(skipUpdate?: boolean) => void>(),
loadSettings: vi.fn<() => Promise<void>>(),
}));
const eventHubState = vi.hoisted(() => ({
@@ -11,9 +11,9 @@ const eventHubState = vi.hoisted(() => ({
vi.mock("./SettingDialogue/ObsidianLiveSyncSettingTab.ts", () => ({
ObsidianLiveSyncSettingTab: class ObsidianLiveSyncSettingTab {
reloadAllSettings(skipUpdate?: boolean) {
settingTabState.callOrder.push(`reload:${String(skipUpdate)}`);
settingTabState.reloadAllSettings(skipUpdate);
async loadSettings() {
await settingTabState.loadSettings();
settingTabState.callOrder.push("settings-loaded");
}
},
}));
@@ -65,7 +65,7 @@ function createModuleHarness() {
describe("ModuleObsidianSettingDialogue startup lifecycle", () => {
beforeEach(() => {
settingTabState.callOrder.length = 0;
settingTabState.reloadAllSettings.mockClear();
settingTabState.loadSettings.mockReset().mockResolvedValue(undefined);
eventHubState.onEvent.mockClear();
});
@@ -79,13 +79,29 @@ describe("ModuleObsidianSettingDialogue startup lifecycle", () => {
});
it("seeds the setting editor without requesting a render before registration", async () => {
const { initialisationHandler, settingsLoadedHandler } = createModuleHarness();
const { initialisationHandler, settingsLoadedHandler, plugin } = createModuleHarness();
let finishRead!: () => void;
settingTabState.loadSettings.mockReturnValueOnce(
new Promise<void>((resolve) => {
finishRead = resolve;
})
);
const handler = settingsLoadedHandler() ?? initialisationHandler();
expect(handler).toBeTypeOf("function");
await handler!();
const loading = handler!();
expect(plugin.addSettingTab).not.toHaveBeenCalled();
finishRead();
await loading;
expect(settingTabState.reloadAllSettings).toHaveBeenCalledWith(true);
expect(settingTabState.callOrder).toEqual(["reload:true", "add-setting-tab"]);
expect(settingTabState.loadSettings).toHaveBeenCalledOnce();
expect(settingTabState.callOrder).toEqual(["settings-loaded", "add-setting-tab"]);
});
it("does not register an editor when reading device values fails", async () => {
const { settingsLoadedHandler, plugin } = createModuleHarness();
const failure = new Error("device storage unavailable");
settingTabState.loadSettings.mockRejectedValueOnce(failure);
await expect(settingsLoadedHandler()!()).rejects.toBe(failure);
expect(plugin.addSettingTab).not.toHaveBeenCalled();
});
});
src/modules/features/SettingDialogue/ObsidianLiveSyncSettingTab.declarative.unit.spec.ts
+8 -6
View File
@@ -1,3 +1,4 @@
import { MemoryValueStore } from "@vrtmrz/value-store";
import { beforeEach, describe, expect, it, vi } from "vitest";
import { DEFAULT_SETTINGS } from "@vrtmrz/livesync-commonlib/compat/common/types";
import type { SettingDefinitionGroup, SettingDefinitionItem, SettingDefinitionPage } from "obsidian";
@@ -158,6 +159,7 @@ function createSettingsTab(options: SettingsTabOptions = {}): ObsidianLiveSyncSe
},
services: {
setting: {
deviceValues: new MemoryValueStore(),
getDeviceAndVaultName: vi.fn(() => ""),
saveSettingData: vi.fn(async () => undefined),
},
@@ -457,7 +459,7 @@ describe("ObsidianLiveSyncSettingTab native page lifecycle", () => {
expect(runtime.pageCleanup).toHaveBeenCalledOnce();
});
it("rebuilds the catalogue when an externally loaded setting changes page visibility", () => {
it("rebuilds the catalogue when an externally loaded setting changes page visibility", async () => {
const tab = createSettingsTab();
const changeLog = findPage(tab, "Change Log");
if (!changeLog.page) {
@@ -466,7 +468,7 @@ describe("ObsidianLiveSyncSettingTab native page lifecycle", () => {
changeLog.page().display();
tab.core.settings.usePowerUserMode = !tab.editingSettings.usePowerUserMode;
tab.requestReload();
await tab.requestReload();
expect(tab.update).toHaveBeenCalledOnce();
});
@@ -480,7 +482,7 @@ describe("ObsidianLiveSyncSettingTab native page lifecycle", () => {
expect(tab.update).toHaveBeenCalledOnce();
});
it("rebuilds translated catalogue names when the display language changes externally", () => {
it("rebuilds translated catalogue names when the display language changes externally", async () => {
const tab = createSettingsTab();
const changeLog = findPage(tab, "Change Log");
if (!changeLog.page) {
@@ -489,12 +491,12 @@ describe("ObsidianLiveSyncSettingTab native page lifecycle", () => {
changeLog.page().display();
tab.core.settings.displayLanguage = "ja";
tab.requestReload();
await tab.requestReload();
expect(tab.update).toHaveBeenCalledOnce();
});
it("rebuilds the catalogue after accepting an external page-visibility setting over a dirty value", () => {
it("rebuilds the catalogue after accepting an external page-visibility setting over a dirty value", async () => {
const tab = createSettingsTab();
const changeLog = findPage(tab, "Change Log");
if (!changeLog.page) {
@@ -505,7 +507,7 @@ describe("ObsidianLiveSyncSettingTab native page lifecycle", () => {
tab.editingSettings.usePowerUserMode = true;
tab.core.settings.usePowerUserMode = true;
tab.requestReload();
await tab.requestReload();
const configureAnchor = vi.mocked(tab.core.confirm.askInPopup).mock.calls[0]?.[2];
expect(configureAnchor).toBeTypeOf("function");
let acceptExternalSetting: (() => void) | undefined;
src/modules/features/SettingDialogue/ObsidianLiveSyncSettingTab.ts
+24 -6
View File
@@ -52,7 +52,6 @@ import {
type PageFunctions,
type UpdateFunction,
} from "./SettingPane.ts";
import { compatGlobal } from "@vrtmrz/livesync-commonlib/compat/common/coreEnvFunctions";
import { JournalSyncCore } from "@vrtmrz/livesync-commonlib/compat/replication/journal/JournalSyncCore";
import { MinioStorageAdapter } from "@vrtmrz/livesync-commonlib/compat/replication/journal/objectstore/MinioStorageAdapter";
import { closeObsidianSettings } from "@/common/obsidianSettings.ts";
@@ -100,6 +99,9 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
}
selectedScreen = "";
private localConfigurationPassphrase = "";
private localSettingsReadVersion = 0;
_editingSettings?: AllSettings;
// Buffered Settings for editing
get editingSettings(): AllSettings {
@@ -162,8 +164,7 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
async saveLocalSetting(key: keyof typeof OnDialogSettingsDefault) {
if (key == "deviceAndVaultName") {
this.services.setting.setDeviceAndVaultName(this.editingSettings?.[key] ?? "");
this.services.setting.saveDeviceAndVaultName();
return await Promise.resolve();
await this.services.setting.saveDeviceAndVaultName();
}
}
/**
@@ -225,6 +226,10 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
appliedConfigurationEncryption.configPassphrase
);
if (appliedConfigurationEncryption.configPassphraseStore === "LOCALSTORAGE") {
this.localSettingsReadVersion++;
this.localConfigurationPassphrase = appliedConfigurationEncryption.configPassphrase;
}
this.copySettingValue(this.initialSettings, appliedConfigurationEncryption, "configPassphraseStore");
this.copySettingValue(this.initialSettings, appliedConfigurationEncryption, "configPassphrase");
}
@@ -285,9 +290,21 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
}
}
private async readLocalSettings(): Promise<boolean> {
const version = ++this.localSettingsReadVersion;
const passphrase = await this.services.setting.deviceValues.getValue("ls-setting-passphrase", "");
if (version !== this.localSettingsReadVersion) return false;
this.localConfigurationPassphrase = passphrase;
return true;
}
async loadSettings(): Promise<void> {
if (await this.readLocalSettings()) this.reloadAllSettings(true);
}
reloadAllLocalSettings() {
const ret = { ...OnDialogSettingsDefault };
ret.configPassphrase = compatGlobal.localStorage.getItem("ls-setting-passphrase") || "";
ret.configPassphrase = this.localConfigurationPassphrase;
ret.preset = "";
ret.deviceAndVaultName = this.services.setting.getDeviceAndVaultName();
return ret;
@@ -402,7 +419,7 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
this.plugin = plugin;
Setting.env = this;
eventHub.onEvent(EVENT_REQUEST_RELOAD_SETTING_TAB, () => {
this.requestReload();
void this.requestReload().catch((error: unknown) => Logger(error, LOG_LEVEL_NOTICE));
});
this.addOnSaved("displayLanguage", () => this.requestCatalogueRefresh());
this.addOnSaved("showStatusOnEditor", () => eventHub.emitEvent(EVENT_ON_UNRESOLVED_ERROR));
@@ -545,7 +562,8 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
);
}
requestReload() {
async requestReload(): Promise<void> {
if (!(await this.readLocalSettings())) return;
const nativeTabIsShown =
this.supportsDeclarativeSettings() && this.containerEl !== undefined && this.containerEl.isShown();
if (this.isShown || nativeTabIsShown) {
src/modules/features/SettingDialogue/ObsidianLiveSyncSettingTab.unit.spec.ts
+56 -9
View File
@@ -1,7 +1,7 @@
import { beforeEach, describe, expect, it, vi } from "vitest";
import { DEFAULT_SETTINGS, LOG_LEVEL_NOTICE, REMOTE_COUCHDB } from "@vrtmrz/livesync-commonlib/compat/common/types";
import { REMOTE_RESOURCE_KINDS } from "@vrtmrz/livesync-commonlib/replication";
import { compatGlobal } from "@vrtmrz/livesync-commonlib/compat/common/coreEnvFunctions";
import { MemoryValueStore } from "@vrtmrz/value-store";
const settingsInitialisationMocks = vi.hoisted(() => ({
applySettingsWithInitialisationChoice: vi.fn(),
@@ -76,8 +76,6 @@ beforeEach(() => {
settingsInitialisationMocks.applySettingsWithInitialisationChoice.mockReset();
loggerMocks.Logger.mockClear();
vi.mocked($msg).mockClear();
vi.mocked(compatGlobal.localStorage.getItem).mockReset().mockReturnValue(null);
vi.mocked(compatGlobal.localStorage.setItem).mockReset();
});
describe("ObsidianLiveSyncSettingTab passphrase verification", () => {
@@ -280,6 +278,7 @@ describe("ObsidianLiveSyncSettingTab pending-setting initialisation", () => {
},
services: {
setting: {
deviceValues: new MemoryValueStore(),
saveSettingData,
getDeviceAndVaultName: vi.fn(() => ""),
suspendAllSync,
@@ -665,6 +664,7 @@ describe("ObsidianLiveSyncSettingTab declarative settings boundary", () => {
},
services: {
setting: {
deviceValues: new MemoryValueStore(),
saveSettingData,
getDeviceAndVaultName: vi.fn(() => ""),
},
@@ -692,6 +692,54 @@ describe("ObsidianLiveSyncSettingTab declarative settings boundary", () => {
return { tab, saveSettingData };
}
it("awaits the stored passphrase before initialising the editing buffer", async () => {
const { tab } = createSettingsTab();
tab.resetEditingSettings();
let resolveRead!: (value: string) => void;
vi.spyOn(tab.services.setting.deviceValues, "getValue").mockReturnValueOnce(
new Promise<string>((resolve) => {
resolveRead = resolve;
})
);
const loading = tab.loadSettings();
expect(tab.initialSettings).toBeUndefined();
resolveRead("stored passphrase");
await loading;
expect(tab.editingSettings.configPassphrase).toBe("stored passphrase");
expect(tab.initialSettings!.configPassphrase).toBe("stored passphrase");
});
it("preserves the loaded settings when device storage cannot be read", async () => {
const { tab } = createSettingsTab();
await tab.services.setting.deviceValues.setValue("ls-setting-passphrase", "stored passphrase");
await tab.loadSettings();
const failure = new Error("device storage unavailable");
vi.spyOn(tab.services.setting.deviceValues, "getValue").mockRejectedValueOnce(failure);
await expect(tab.requestReload()).rejects.toBe(failure);
expect(tab.editingSettings.configPassphrase).toBe("stored passphrase");
expect(tab.initialSettings!.configPassphrase).toBe("stored passphrase");
});
it("does not let a delayed read overwrite a successfully saved passphrase", async () => {
const { tab } = createSettingsTab();
await tab.loadSettings();
let resolveRead!: (value: string) => void;
vi.spyOn(tab.services.setting.deviceValues, "getValue").mockReturnValueOnce(
new Promise<string>((resolve) => {
resolveRead = resolve;
})
);
const loading = tab.loadSettings();
Object.assign(tab.services.setting, { changeConfigurationEncryption: vi.fn(async () => undefined) });
tab.editingSettings.configPassphraseStore = "LOCALSTORAGE";
tab.editingSettings.configPassphrase = "new passphrase";
await tab.saveSettings(["configPassphraseStore", "configPassphrase"]);
resolveRead("old passphrase");
await loading;
tab.reloadAllSettings();
expect(tab.initialSettings!.configPassphrase).toBe("new passphrase");
});
it("loads the imperative fallback without a SettingPage runtime export", () => {
const { tab } = createSettingsTab();
@@ -779,7 +827,7 @@ describe("ObsidianLiveSyncSettingTab declarative settings boundary", () => {
it("saves a custom configuration passphrase edit through the setting service", async () => {
const { tab } = createSettingsTab();
const changeConfigurationEncryption = vi.fn(async () => {
vi.mocked(compatGlobal.localStorage.getItem).mockReturnValue("new custom passphrase");
await tab.services.setting.deviceValues.setValue("ls-setting-passphrase", "new custom passphrase");
});
Object.assign(tab.core.services.setting, { changeConfigurationEncryption });
const store = "LOCALSTORAGE";
@@ -787,13 +835,13 @@ describe("ObsidianLiveSyncSettingTab declarative settings boundary", () => {
tab.initialSettings!.configPassphraseStore = store;
tab.editingSettings.configPassphrase = "new custom passphrase";
tab.initialSettings!.configPassphrase = "old custom passphrase";
vi.mocked(compatGlobal.localStorage.setItem).mockClear();
const setValue = vi.spyOn(tab.services.setting.deviceValues, "setValue");
await tab.saveSettings(["configPassphrase"]);
expect(changeConfigurationEncryption).toHaveBeenCalledOnce();
expect(changeConfigurationEncryption).toHaveBeenCalledWith(store, "new custom passphrase");
expect(compatGlobal.localStorage.setItem).not.toHaveBeenCalled();
expect(setValue).toHaveBeenCalledOnce();
expect(tab.initialSettings!.configPassphrase).toBe("new custom passphrase");
});
@@ -811,7 +859,6 @@ describe("ObsidianLiveSyncSettingTab declarative settings boundary", () => {
tab.initialSettings!.configPassphraseStore = "LOCALSTORAGE";
tab.initialSettings!.configPassphrase = "previous passphrase";
tab.editingSettings.configPassphrase = "passphrase A";
vi.mocked(compatGlobal.localStorage.getItem).mockReturnValue("passphrase A");
const apply = tab.saveSettings(["configPassphrase"]);
expect(changeConfigurationEncryption).toHaveBeenCalledWith("LOCALSTORAGE", "passphrase A");
@@ -840,13 +887,13 @@ describe("ObsidianLiveSyncSettingTab declarative settings boundary", () => {
tab.editingSettings.configPassphraseStore = editedStore;
tab.initialSettings!.configPassphrase = "old custom passphrase";
tab.editingSettings.configPassphrase = "new custom passphrase";
vi.mocked(compatGlobal.localStorage.setItem).mockClear();
const setValue = vi.spyOn(tab.services.setting.deviceValues, "setValue");
await expect(tab.saveSettings(["configPassphraseStore", "configPassphrase"])).rejects.toBe(failure);
expect(changeConfigurationEncryption).toHaveBeenCalledOnce();
expect(changeConfigurationEncryption).toHaveBeenCalledWith(editedStore, "new custom passphrase");
expect(compatGlobal.localStorage.setItem).not.toHaveBeenCalled();
expect(setValue).not.toHaveBeenCalled();
expect(tab.core.settings.configPassphraseStore).toBe(savedStore);
expect(tab.initialSettings!.configPassphraseStore).toBe(savedStore);
expect(tab.initialSettings!.configPassphrase).toBe("old custom passphrase");
src/modules/features/SettingDialogue/PaneRemoteConfig.ts
+17
View File
@@ -7,6 +7,7 @@ import {
type ObsidianLiveSyncSettings,
LOG_LEVEL_VERBOSE,
} from "@vrtmrz/livesync-commonlib/compat/common/types";
import { OPEN_JOINED_DEVICES } from "@/serviceFeatures/joinedDevicesCommand";
import { Menu, type ButtonComponent } from "@/deps.ts";
import { $msg } from "@/common/translation";
import { LiveSyncSetting as Setting } from "./LiveSyncSetting.ts";
@@ -101,6 +102,22 @@ export function paneRemoteConfig(
paneEl: HTMLElement,
{ addPanel }: PageFunctions
): void {
new Setting(paneEl)
.setName($msg("Joined devices"))
.setDesc($msg("Review device records for the active CouchDB or Object Storage connection."))
.addButton((button) =>
button.setButtonText($msg("Review devices")).onClick(async () => {
const result = await this.services.UI.commands.invoke(OPEN_JOINED_DEVICES, undefined);
if (result.status === "unavailable" || result.value === "unavailable") {
this.services.API.addLog(
$msg(
"Joined devices are available after database startup with an active CouchDB or Object Storage connection."
),
LOG_LEVEL_NOTICE
);
}
})
);
{
/* E2EE */
const E2EEInitialProps = {
src/modules/features/SettingDialogue/PaneRemoteConfig.unit.spec.ts
+22 -5
View File
@@ -2,7 +2,7 @@ import { afterEach, describe, expect, it, vi } from "vitest";
const runtime = vi.hoisted(() => ({
buttonClasses: [] as string[],
clickHandlers: [] as Array<() => Promise<void> | void>,
clickHandlers: [] as Array<{ setting: string; run: () => Promise<void> | void }>,
panels: [] as Array<{ destroy: ReturnType<typeof vi.fn> }>,
settingClasses: [] as string[],
}));
@@ -32,7 +32,10 @@ vi.mock("./LiveSyncSetting.ts", () => ({
},
};
setName() {
settingName = "";
setName(name: string) {
this.settingName = name;
return this;
}
@@ -41,6 +44,7 @@ vi.mock("./LiveSyncSetting.ts", () => ({
}
addButton(callback: (button: unknown) => void) {
const setting = this.settingName;
const button = {
buttonEl: {
classList: {
@@ -53,7 +57,7 @@ vi.mock("./LiveSyncSetting.ts", () => ({
return this;
},
onClick(callback: () => Promise<void> | void) {
runtime.clickHandlers.push(callback);
runtime.clickHandlers.push({ setting, run: callback });
return this;
},
setButtonText() {
@@ -107,6 +111,7 @@ vi.mock("./remoteConfigBuffer.ts", () => ({
syncActivatedRemoteSettings: vi.fn(),
}));
import { OPEN_JOINED_DEVICES } from "@/serviceFeatures/joinedDevicesCommand";
import { paneRemoteConfig } from "./PaneRemoteConfig.ts";
function createPanelElement(): HTMLElement {
@@ -124,6 +129,18 @@ afterEach(() => {
});
describe("paneRemoteConfig", () => {
it("opens joined devices through the shared UI command", async () => {
const invoke = vi.fn(async () => ({ status: "handled", value: "closed" }));
const host = {
editingSettings: { remoteConfigurations: {} },
services: { UI: { commands: { invoke } } },
};
const addPanel = vi.fn(() => ({ then: vi.fn() }));
paneRemoteConfig.call(host as never, {} as HTMLElement, { addPanel } as never);
await runtime.clickHandlers.find((handler) => handler.setting === "Joined devices")!.run();
expect(invoke).toHaveBeenCalledWith(OPEN_JOINED_DEVICES, undefined);
});
it("destroys the E2EE info panel when the settings page lifetime unloads", async () => {
const callbacks: Array<() => unknown> = [];
const lifetimeComponent = {
@@ -187,7 +204,7 @@ describe("paneRemoteConfig", () => {
}));
paneRemoteConfig.call(host as never, {} as HTMLElement, { addPanel } as never);
await runtime.clickHandlers[0]();
await runtime.clickHandlers.find((handler) => handler.setting === "Configure E2EE")!.run();
expect(setupManager.onlyE2EEConfiguration).toHaveBeenCalledOnce();
expect(host.editingSettings.encryptInternalMetadata).toBe(true);
@@ -233,7 +250,7 @@ describe("paneRemoteConfig", () => {
}));
paneRemoteConfig.call(host as never, {} as HTMLElement, { addPanel } as never);
await runtime.clickHandlers[0]();
await runtime.clickHandlers.find((handler) => handler.setting === "Configure E2EE")!.run();
expect(host.editingSettings.idDerivationVersion).toBe(1);
expect(host.editingSettings.idDerivationKey).toBe(nextIdKey);
src/modules/features/SettingDialogue/PaneSelector.ts
+2 -4
View File
@@ -13,7 +13,7 @@ export function paneSelector(this: ObsidianLiveSyncSettingTab, paneEl: HTMLEleme
.setName($msg("Synchronising files"))
.setDesc(
$msg(
"(RegExp) Empty to sync all files. Set filter as a regular expression to limit synchronising files."
"(RegExp) Limit normal files to matching paths. Leave empty to allow all normal file paths. Hidden File Sync uses its own target patterns."
)
);
const syncFilesControl = mount(MultipleRegExpControl, {
@@ -32,9 +32,7 @@ export function paneSelector(this: ObsidianLiveSyncSettingTab, paneEl: HTMLEleme
const nonSyncFilesSetting = new Setting(paneEl)
.setName($msg("Non-Synchronising files"))
.setDesc(
$msg("(RegExp) If this is set, any changes to local and remote files that match this will be skipped.")
);
.setDesc($msg("(RegExp) Skip changes to matching normal and hidden files, locally and remotely."));
const nonSyncFilesControl = mount(MultipleRegExpControl, {
target: nonSyncFilesSetting.controlEl,
src/modules/features/SetupManager.unit.spec.ts
+19 -7
View File
@@ -1,3 +1,4 @@
import { MemoryValueStore } from "@vrtmrz/value-store";
import { beforeEach, describe, expect, it, vi } from "vitest";
import {
DEFAULT_SETTINGS,
@@ -37,11 +38,6 @@ import { decodeSettingsFromQRCodeData } from "@vrtmrz/livesync-commonlib/compat/
import { SetupManager, UserMode } from "./SetupManager";
class TestSettingService extends SettingService<ServiceContext> {
protected setItem(_key: string, _value: string): void {}
protected getItem(_key: string): string {
return "";
}
protected deleteItem(_key: string): void {}
protected saveData(_setting: ObsidianLiveSyncSettings): Promise<void> {
return Promise.resolve();
}
@@ -65,6 +61,7 @@ function createLegacyRemoteSetting(): ObsidianLiveSyncSettings {
function createSetupManager() {
const setting = new TestSettingService(new ServiceContext(), {
deviceValues: new MemoryValueStore(),
APIService: {
getSystemVaultName: vi.fn(() => "vault"),
getAppID: vi.fn(() => "app"),
@@ -172,10 +169,12 @@ describe("SetupManager", () => {
const { manager, dialogManager } = createSetupManager();
dialogManager.openWithExplicitCancel.mockResolvedValueOnce("configure-manually");
const configureManually = vi.spyOn(manager, "onConfigureManually").mockResolvedValue(true);
const newVaultSettings = createNewVaultSettings();
await manager.onOnboard(UserMode.NewUser);
expect(configureManually).toHaveBeenCalledWith(createNewVaultSettings(), UserMode.NewUser);
expect(newVaultSettings.customChunkSize).toBe(0);
expect(configureManually).toHaveBeenCalledWith(newVaultSettings, UserMode.NewUser);
});
it("compatibility: normalises imported flat remote settings from a Setup URI before applying", async () => {
@@ -487,8 +486,15 @@ describe("SetupManager", () => {
expect(applySettings).not.toHaveBeenCalled();
});
it("preserves modern profiles, display names, and the active selection from a Setup URI", async () => {
it.each([
{ importedSize: 0, expectedSize: 0 },
{ importedSize: undefined, expectedSize: 60 },
])("preserves modern Setup URI profiles and adopts the supplied chunk size $importedSize", async ({ importedSize, expectedSize }) => {
const { manager, setting, dialogManager } = createSetupManager();
setting.settings = {
...setting.currentSettings(),
customChunkSize: 60,
};
const imported = {
...DEFAULT_SETTINGS,
remoteConfigurations: {
@@ -507,6 +513,8 @@ describe("SetupManager", () => {
},
activeConfigurationId: "archive",
} as ObsidianLiveSyncSettings;
if (importedSize === undefined) delete (imported as Partial<ObsidianLiveSyncSettings>).customChunkSize;
else imported.customChunkSize = importedSize;
dialogManager.openWithExplicitCancel
.mockResolvedValueOnce(imported)
.mockResolvedValueOnce("compatible-existing-user");
@@ -514,6 +522,7 @@ describe("SetupManager", () => {
await manager.onUseSetupURI(UserMode.Unknown, "mock-config://modern-settings");
const current = setting.currentSettings();
expect(current.customChunkSize).toBe(expectedSize);
expect(current.remoteConfigurations).toEqual(imported.remoteConfigurations);
expect(current.activeConfigurationId).toBe("archive");
expect(Object.keys(current.remoteConfigurations).some((id) => id.startsWith("legacy-"))).toBe(false);
@@ -523,6 +532,7 @@ describe("SetupManager", () => {
const { manager, setting, dialogManager } = createSetupManager();
setting.settings = {
...setting.currentSettings(),
customChunkSize: 60,
isConfigured: true,
remoteConfigurations: {
existing: {
@@ -554,6 +564,7 @@ describe("SetupManager", () => {
await manager.onCouchDBManualSetup(UserMode.ExistingUser, setting.currentSettings());
const current = setting.currentSettings();
expect(current.customChunkSize).toBe(60);
expect(current.remoteConfigurations.existing).toBeDefined();
expect(Object.keys(current.remoteConfigurations)).toHaveLength(2);
expect(current.activeConfigurationId).not.toBe("existing");
@@ -662,6 +673,7 @@ describe("SetupManager", () => {
await manager.onP2PManualSetup(UserMode.NewUser, setting.currentSettings());
const current = setting.currentSettings();
expect(current.customChunkSize).toBe(0);
expect(Object.keys(current.remoteConfigurations)).toHaveLength(1);
expect(current.activeConfigurationId).not.toBe("");
expect(current.P2P_ActiveRemoteConfigurationId).toBe(current.activeConfigurationId);
src/modules/features/SetupWizard/dialogs/SetupRemoteP2P.svelte
+37 -7
View File
@@ -52,26 +52,48 @@
const context = getDialogContext();
let error = $state("");
let connectionPathResetNotice = $state(false);
let peerNameReady = $state(false);
let mounted = false;
const hasValidTurnServer = $derived(hasP2PTurnConfiguration(syncSetting));
type Props = GuestDialogProps<SetupRemoteP2PResultType, SetupRemoteP2PInitialData>;
const { setResult, getInitialData }: Props = $props();
let connectionProbe: SetupRemoteP2PInitialData["connectionProbe"] | undefined;
onMount(() => {
mounted = true;
const initialData = getInitialData?.();
connectionProbe = initialData?.connectionProbe;
const initialSettings = initialData?.settings;
if (initialSettings) {
syncSetting = pickP2PSyncSettings(initialSettings);
}
const initialPeerName = (initialSettings?.P2P_DevicePeerName ?? "").trim();
const savedPeerName = initialSettings?.P2P_DevicePeerName;
const initialPeerName = typeof savedPeerName === "string" ? savedPeerName.trim() : "";
if (initialPeerName !== "") {
return;
}
const cachedPeerName = context.services.config.getSmallConfig(SETTING_KEY_P2P_DEVICE_NAME);
if (cachedPeerName) {
syncSetting.P2P_DevicePeerName = cachedPeerName as string;
peerNameReady = true;
return () => {
mounted = false;
};
}
void context.services.config.getValue(SETTING_KEY_P2P_DEVICE_NAME).then(
(cachedPeerName) => {
if (mounted) {
if (typeof cachedPeerName === "string" && cachedPeerName) {
syncSetting.P2P_DevicePeerName = cachedPeerName;
}
peerNameReady = true;
}
},
(loadError: unknown) => {
if (mounted) {
error = loadError instanceof Error ? loadError.message : String(loadError);
peerNameReady = true;
}
}
);
return () => {
mounted = false;
};
});
function generateSetting() {
const connSetting: P2PSyncSetting = {
@@ -193,6 +215,7 @@
}
async function checkAndCommit() {
if (!peerNameReady) return;
error = "";
try {
error = (await checkConnection()) || "";
@@ -207,6 +230,7 @@
}
}
function commit() {
if (!peerNameReady) return;
error = validateManagedTurnSettings(syncSetting) ?? "";
if (error) return;
const setting = pickP2PSyncSettings(generateSetting());
@@ -217,6 +241,7 @@
}
const canProceed = $derived.by(() => {
return (
peerNameReady &&
syncSetting.P2P_relays.trim() !== "" &&
syncSetting.P2P_roomID.trim() !== "" &&
syncSetting.P2P_passphrase.trim() !== "" &&
@@ -291,6 +316,7 @@
autocorrect="off"
autocapitalize="off"
spellcheck="false"
disabled={!peerNameReady}
bind:value={syncSetting.P2P_DevicePeerName}
/>
</InputRow>
@@ -385,7 +411,11 @@
disabled={!canProceed}
commit={() => checkAndCommit()}
/>
<Decision title={translateMessage("Continue anyway")} commit={() => commit()} />
<Decision
title={translateMessage("Continue anyway")}
disabled={!peerNameReady}
commit={() => commit()}
/>
<Decision title={translateMessage("Cancel")} commit={() => cancel()} />
</UserDecisions>
{/if}
src/modules/services/LiveSyncUI/dialogues/JoinedDevicesDialog.svelte
+396
View File
@@ -0,0 +1,396 @@
<script lang="ts">
import { onDestroy, onMount } from "svelte";
import type { GuestDialogProps } from "@/modules/services/LiveSyncUI/svelteDialog";
import DialogHeader from "@/modules/services/LiveSyncUI/components/DialogHeader.svelte";
import InfoNote from "@/modules/services/LiveSyncUI/components/InfoNote.svelte";
import { $msg as translate } from "@/common/translation";
import type {
JoinedDeviceRow,
JoinedDevicesDialogOptions,
JoinedDevicesView,
} from "@/serviceFeatures/joinedDevicesDialogTypes";
const { setResult, getInitialData }: GuestDialogProps<"close", JoinedDevicesDialogOptions> = $props();
let options = $state<JoinedDevicesDialogOptions>();
let view = $state<JoinedDevicesView>();
let isRefreshing = $state(false);
let confirmingId = $state<string>();
let forgettingId = $state<string>();
let statusMessage = $state("");
let errorMessage = $state("");
let disposed = false;
const busy = $derived(isRefreshing || forgettingId !== undefined);
const title = translate("Joined devices");
onMount(() => {
options = getInitialData?.();
if (!options) {
errorMessage = translate(
"The device information could not be loaded. Close this dialogue and open Joined devices again if the remote configuration has changed."
);
return;
}
void refreshView();
});
onDestroy(() => {
disposed = true;
options?.cancel();
});
async function refreshView(afterForget = false) {
const dialogOptions = options;
if (!dialogOptions || disposed || isRefreshing || (forgettingId !== undefined && !afterForget)) return;
isRefreshing = true;
errorMessage = "";
if (!afterForget) {
statusMessage = "";
confirmingId = undefined;
}
try {
const refreshedView = await dialogOptions.refresh();
if (disposed) return;
view = refreshedView;
} catch {
if (!disposed) {
errorMessage = translate(
"The device information could not be loaded. Close this dialogue and open Joined devices again if the remote configuration has changed."
);
}
} finally {
if (!disposed) isRefreshing = false;
}
}
function toggleConfirmation(device: JoinedDeviceRow) {
if (busy || device.self) return;
confirmingId = confirmingId === device.id ? undefined : device.id;
errorMessage = "";
statusMessage = "";
}
async function forgetDevice(device: JoinedDeviceRow) {
const dialogOptions = options;
if (!dialogOptions || disposed || busy || device.self || confirmingId !== device.id) return;
forgettingId = device.id;
confirmingId = undefined;
errorMessage = "";
statusMessage = "";
try {
const result = await dialogOptions.forget(device.id);
if (disposed) return;
if (result === "forgotten") {
statusMessage = translate("The device record was forgotten.");
await refreshView(true);
} else if (result === "not-found") {
statusMessage = translate("The device record was already absent.");
await refreshView(true);
} else if (result === "conflict") {
errorMessage = translate(
"The device list changed while you were trying to forget this device. Refresh the list before trying again."
);
} else {
errorMessage = translate(
"This remote does not support conditional updates, so the device record could not be removed."
);
}
} catch {
if (!disposed) {
errorMessage = translate(
"Joined devices could not be updated. Refresh the list and try again. If the remote configuration has changed, close this dialogue and open Joined devices again."
);
}
} finally {
if (!disposed) forgettingId = undefined;
}
}
function formatLastConnected(timestamp: number | undefined) {
if (timestamp === undefined || !Number.isFinite(timestamp) || timestamp <= 0) return translate("Not recorded");
const date = new Date(timestamp);
if (Number.isNaN(date.getTime())) return translate("Not recorded");
return date.toLocaleString();
}
</script>
<DialogHeader {title} />
<div class="joined-devices-dialog">
<p>
{translate("This list shows device information recorded by the remote. It does not show whether a device is online now.")}
</p>
<InfoNote
message="Synchronisation between different plug-in versions is generally safe unless the connection is explicitly blocked. However, it may be less efficient or use more storage, and P2P synchronisation may be unavailable."
/>
{#if view}
{#if !view.milestonePresent}
<p class="state-message">{translate("This remote does not currently have a record of joined devices.")}</p>
{/if}
{#if view.cleaned}
<p class="state-message">
{translate(
"This remote has been cleaned. Use the existing recovery process to join devices again as needed."
)}
</p>
{/if}
{#if view.locked}
<p class="state-message">
{translate(
"This remote is locked. Use the existing recovery process for devices that need to join again."
)}
</p>
{/if}
{/if}
{#if errorMessage}
<p class="error-message" role="alert">{errorMessage}</p>
{/if}
{#if statusMessage}
<p class="status-message" role="status">{statusMessage}</p>
{/if}
{#if isRefreshing && !view}
<p role="status">{translate("Loading joined devices…")}</p>
{/if}
{#if view}
{#if view.devices.length > 0}
<ul class="device-list">
{#each view.devices as device (device.id)}
<li class="device-card">
<div class="device-heading">
<div class="device-identity">
{#if device.self}
<span class="device-badge">{translate("This device")}</span>
{/if}
<code class="device-id">{device.id}</code>
</div>
<div class="device-badges">
<span class="device-badge">
{device.accepted ? translate("Accepted") : translate("Not accepted")}
</span>
{#if device.lockedOut}
<span class="device-badge device-badge-warning">{translate("Locked out")}</span>
{/if}
</div>
</div>
{#if device.self && device.lockedOut}
<p class="state-message">
{translate(
"This device has been locked out. Use the existing recovery process to join again."
)}
</p>
{/if}
<dl class="device-details">
<div>
<dt>{translate("Platform")}</dt>
<dd>{device.platform || translate("Unknown platform")}</dd>
</div>
<div>
<dt>{translate("Plug-in version")}</dt>
<dd>{device.pluginVersion || translate("Not recorded")}</dd>
</div>
<div>
<dt>{translate("App version")}</dt>
<dd>{device.appVersion || translate("Not recorded")}</dd>
</div>
<div>
<dt>{translate("Last recorded")}</dt>
<dd>{formatLastConnected(device.lastConnected)}</dd>
</div>
</dl>
{#if !device.self}
{#if confirmingId === device.id}
<div class="forget-confirmation">
<p><strong>{translate("Forget this device record?")}</strong></p>
<p>
{translate("This removes only this device's record from the remote.")}
</p>
<p>
{translate(
"Forgetting the record does not delete Vault files or revoke credentials."
)}
</p>
<p>
{view.locked
? translate(
"This remote is locked. If the device returns, use the existing recovery process to join it again."
)
: translate("This remote is unlocked. A device that returns can register again.")}
</p>
<div class="device-actions">
<button disabled={busy} onclick={() => toggleConfirmation(device)}>
{translate("Cancel")}
</button>
<button
class="mod-warning"
disabled={busy}
onclick={() => void forgetDevice(device)}
>
{translate("Forget device")}
</button>
</div>
</div>
{:else}
<div class="device-actions">
<button disabled={busy} onclick={() => toggleConfirmation(device)}>
{forgettingId === device.id
? translate("Forgetting…")
: translate("Forget device")}
</button>
</div>
{/if}
{/if}
</li>
{/each}
</ul>
{:else if !isRefreshing}
<p>{translate("No joined devices are recorded on this remote.")}</p>
{/if}
{/if}
<div class="dialog-controls">
<button disabled={busy} onclick={() => void refreshView()}>{translate("Refresh")}</button>
<button onclick={() => setResult("close")}>{translate("Close")}</button>
</div>
</div>
<style>
.joined-devices-dialog {
display: flex;
flex-direction: column;
gap: var(--size-4-2);
min-width: 0;
max-width: 100%;
}
p {
margin: 0;
overflow-wrap: anywhere;
}
.state-message,
.forget-confirmation {
border-left: 3px solid var(--interactive-accent);
padding: var(--size-4-2);
background: var(--background-secondary);
}
.error-message {
color: var(--text-error);
}
.device-list {
display: grid;
gap: var(--size-4-2);
list-style: none;
margin: 0;
min-width: 0;
padding: 0;
}
.device-card {
display: flex;
flex-direction: column;
gap: var(--size-4-2);
min-width: 0;
overflow-wrap: anywhere;
border: 1px solid var(--background-modifier-border);
border-radius: var(--radius-m);
padding: var(--size-4-3);
}
.device-heading {
display: flex;
flex-wrap: wrap;
align-items: flex-start;
justify-content: space-between;
gap: var(--size-4-2);
min-width: 0;
}
.device-identity,
.device-badges {
display: flex;
flex-wrap: wrap;
align-items: center;
gap: var(--size-4-2);
min-width: 0;
}
.device-id {
display: block;
min-width: 0;
max-width: 100%;
overflow-wrap: anywhere;
white-space: normal;
}
.device-badge {
border-radius: var(--radius-s);
background: var(--background-secondary);
padding: 0 var(--size-4-2);
font-size: var(--font-ui-smaller);
}
.device-badge-warning {
color: var(--text-warning);
}
.device-details {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(min(100%, 13rem), 1fr));
gap: var(--size-4-2) var(--size-4-4);
margin: 0;
min-width: 0;
}
.device-details > div {
min-width: 0;
}
dt {
color: var(--text-muted);
font-size: var(--font-ui-smaller);
}
dd {
margin: 0;
overflow-wrap: anywhere;
}
.forget-confirmation {
display: flex;
flex-direction: column;
gap: var(--size-4-2);
min-width: 0;
}
.device-actions,
.dialog-controls {
display: flex;
flex-wrap: wrap;
justify-content: flex-end;
gap: var(--size-4-2);
}
button {
min-height: 44px;
}
@media (max-width: 480px) {
.device-card {
padding: var(--size-4-2);
}
.device-details {
grid-template-columns: minmax(0, 1fr);
}
}
</style>
src/modules/services/ObsidianServiceHub.ts
+4 -5
View File
@@ -1,4 +1,6 @@
import { InjectableServiceHub } from "@vrtmrz/livesync-commonlib/compat/services/implements/injectable/InjectableServiceHub";
import { compatGlobal } from "@vrtmrz/livesync-commonlib/compat/common/coreEnvFunctions";
import { createWebStorageValueStore } from "@vrtmrz/value-store/web-storage";
import { ObsidianServiceContext } from "@/modules/services/ObsidianServiceContext";
import type { ServiceInstances } from "@vrtmrz/livesync-commonlib/compat/services/ServiceHub";
import type ObsidianLiveSyncPlugin from "@/main";
@@ -11,7 +13,6 @@ import {
ObsidianTweakValueService,
ObsidianTestService,
ObsidianDatabaseEventService,
ObsidianConfigService,
ObsidianKeyValueDBService,
ObsidianControlService,
} from "./ObsidianServices";
@@ -46,6 +47,7 @@ export class ObsidianServiceHub extends InjectableServiceHub<ObsidianServiceCont
const setting = new ObsidianSettingService(context, {
APIService: API,
onDisplayLanguageChanged: setLang,
deviceValues: createWebStorageValueStore(compatGlobal.localStorage, { prefix: "" }),
});
const appLifecycle = new ObsidianAppLifecycleService(context, {
settingService: setting,
@@ -85,10 +87,7 @@ export class ObsidianServiceHub extends InjectableServiceHub<ObsidianServiceCont
databaseEvents: databaseEvents,
vault: vault,
});
const config = new ObsidianConfigService(context, {
settingService: setting,
APIService: API,
});
const config = setting.config;
const replicator = new ObsidianReplicatorService(context, {
settingService: setting,
appLifecycleService: appLifecycle,
src/modules/services/ObsidianServices.ts
+1 -6
View File
@@ -6,7 +6,6 @@ import { InjectableReplicationService } from "@vrtmrz/livesync-commonlib/compat/
import { InjectableReplicatorService } from "@vrtmrz/livesync-commonlib/compat/services/implements/injectable/InjectableReplicatorService";
import { InjectableTestService } from "@vrtmrz/livesync-commonlib/compat/services/implements/injectable/InjectableTestService";
import { InjectableTweakValueService } from "@vrtmrz/livesync-commonlib/compat/services/implements/injectable/InjectableTweakValueService";
import { ConfigServiceBrowserCompat } from "@vrtmrz/livesync-commonlib/compat/services/implements/browser/ConfigServiceBrowserCompat";
import type { ObsidianServiceContext } from "@/modules/services/ObsidianServiceContext";
import { KeyValueDBService } from "@vrtmrz/livesync-commonlib/compat/services/base/KeyValueDBService";
import { ControlService } from "@vrtmrz/livesync-commonlib/compat/services/base/ControlService";
@@ -58,10 +57,7 @@ export class ObsidianReplicatorService extends InjectableReplicatorService<Obsid
super(context, withSleepPreference(dependencies));
}
async runBoundedLocalApplicationActivity<T>(
task: () => T | PromiseLike<T>,
options?: ActivityOptions
): Promise<T> {
async runBoundedLocalApplicationActivity<T>(task: () => T | PromiseLike<T>, options?: ActivityOptions): Promise<T> {
this.boundedLocalApplicationActivityCount.value++;
try {
return this.dependencies.activityRunner
@@ -84,7 +80,6 @@ export class ObsidianConflictService extends InjectableConflictService<ObsidianS
export class ObsidianTweakValueService extends InjectableTweakValueService<ObsidianServiceContext> {}
// InjectableTestService
export class ObsidianTestService extends InjectableTestService<ObsidianServiceContext> {}
export class ObsidianConfigService extends ConfigServiceBrowserCompat<ObsidianServiceContext> {}
export class ObsidianKeyValueDBService extends KeyValueDBService<ObsidianServiceContext> {}
src/modules/services/ObsidianSettingService.ts
-14
View File
@@ -1,4 +1,3 @@
import { compatGlobal } from "@vrtmrz/livesync-commonlib/compat/common/coreEnvFunctions";
import { type ObsidianLiveSyncSettings } from "@vrtmrz/livesync-commonlib/compat/common/types";
import {
EVENT_REQUEST_RELOAD_SETTING_TAB,
@@ -27,19 +26,6 @@ export class ObsidianSettingService<T extends ObsidianServiceContext> extends Se
return Promise.resolve(true);
});
}
protected setItem(key: string, value: string) {
// TODO: Implement nativeLocalStorage.
return compatGlobal.localStorage.setItem(key, value);
}
protected getItem(key: string): string {
// TODO: Implement nativeLocalStorage.
return compatGlobal.localStorage.getItem(key) ?? "";
}
protected deleteItem(key: string): void {
// TODO: Implement nativeLocalStorage.
compatGlobal.localStorage.removeItem(key);
}
protected override async saveData(data: ObsidianLiveSyncSettings): Promise<void> {
return await this.context.liveSyncPlugin.saveData(data);
}
src/modules/services/ObsidianSettingService.unit.spec.ts
+39
View File
@@ -1,6 +1,23 @@
import { describe, expect, it } from "vitest";
import { createServiceContext } from "@vrtmrz/livesync-commonlib/context";
import type { ObsidianLiveSyncSettings } from "@vrtmrz/livesync-commonlib/compat/common/types";
import type { IAPIService } from "@vrtmrz/livesync-commonlib/compat/services/base/IService";
import { MemoryValueStore } from "@vrtmrz/value-store";
import type { ObsidianServiceContext } from "@/modules/services/ObsidianServiceContext";
import { normaliseObsidianSettingsData } from "./ObsidianSettingService.ts";
import { ObsidianSettingService } from "./ObsidianSettingService.ts";
function createSettingService(
vaultName: string,
deviceValues: MemoryValueStore
): ObsidianSettingService<ObsidianServiceContext> {
const context = createServiceContext() as unknown as ObsidianServiceContext;
const APIService = {
addLog: () => undefined,
getSystemVaultName: () => vaultName,
} as unknown as IAPIService;
return new ObsidianSettingService(context, { APIService, deviceValues });
}
describe("normaliseObsidianSettingsData", () => {
it("maps Obsidian's missing data value to Commonlib's new-Vault input", () => {
@@ -12,4 +29,26 @@ describe("normaliseObsidianSettingsData", () => {
expect(normaliseObsidianSettingsData(settings)).toBe(settings);
});
it("keeps injected device values independent from each Vault-scoped configuration store", async () => {
const firstValues = new MemoryValueStore();
const secondValues = new MemoryValueStore();
const firstSetting = createSettingService("first-vault", firstValues);
const secondSetting = createSettingService("second-vault", secondValues);
await firstSetting.deviceValues.setValue("shared-key", "device value");
await firstSetting.config.setValue("shared-key", "first config");
await secondSetting.config.setValue("shared-key", "second config");
expect(firstSetting.deviceValues).toBe(firstValues);
expect(secondSetting.deviceValues).toBe(secondValues);
expect(firstSetting.deviceValues).not.toBe(secondSetting.deviceValues);
expect(firstSetting.config).not.toBe(firstValues);
expect(firstSetting.config).not.toBe(secondSetting.config);
await expect(firstSetting.deviceValues.getValue("shared-key")).resolves.toBe("device value");
await expect(firstSetting.config.getValue("shared-key")).resolves.toBe("first config");
await expect(secondSetting.config.getValue("shared-key")).resolves.toBe("second config");
await expect(firstValues.getValue("first-vault-shared-key")).resolves.toBe("first config");
await expect(secondValues.getValue("second-vault-shared-key")).resolves.toBe("second config");
});
});
src/modules/services/ObsidianUIService.ts
+2 -2
View File
@@ -1,4 +1,4 @@
import type { ConfigService } from "@vrtmrz/livesync-commonlib/compat/services/base/ConfigService";
import type { IConfigService } from "@vrtmrz/livesync-commonlib/compat/services/base/IService";
import type { AppLifecycleService } from "@vrtmrz/livesync-commonlib/compat/services/base/AppLifecycleService";
import type { ReplicatorService } from "@vrtmrz/livesync-commonlib/compat/services/base/ReplicatorService";
import { UIService } from "@vrtmrz/livesync-commonlib/compat/services/implements/base/UIService";
@@ -8,7 +8,7 @@ import DialogToCopy from "@/modules/services/LiveSyncUI/dialogues/DialogueToCopy
import type { IAPIService, IControlService } from "@vrtmrz/livesync-commonlib/compat/services/base/IService";
export type ObsidianUIServiceDependencies<T extends ObsidianServiceContext = ObsidianServiceContext> = {
appLifecycle: AppLifecycleService<T>;
config: ConfigService<T>;
config: IConfigService;
replicator: ReplicatorService<T>;
APIService: IAPIService;
control: IControlService;
src/serviceFeatures/compatibilityReview.ts
+9 -9
View File
@@ -44,17 +44,17 @@ export class CompatibilityReviewController {
return this._initialised;
}
private readAcknowledgedVersion(): string | null {
private async readAcknowledgedVersion(): Promise<string | null> {
const setting = this.core.services.setting;
const currentMarker = setting.getSmallConfig(DATABASE_COMPATIBILITY_VERSION_KEY);
const currentMarker = await this.core.services.config.getValue(DATABASE_COMPATIBILITY_VERSION_KEY, "");
if (currentMarker) return currentMarker;
const legacyKey = legacyDatabaseCompatibilityVersionKey(this.core.services.vault.getVaultName());
const legacyMarker = setting.getDeviceLocalConfig(legacyKey);
const legacyMarker = await setting.deviceValues.getValue(legacyKey, "");
if (!legacyMarker) return null;
setting.setSmallConfig(DATABASE_COMPATIBILITY_VERSION_KEY, legacyMarker);
setting.deleteDeviceLocalConfig(legacyKey);
await this.core.services.config.setValue(DATABASE_COMPATIBILITY_VERSION_KEY, legacyMarker);
await setting.deviceValues.delete(legacyKey);
return legacyMarker;
}
@@ -75,7 +75,7 @@ export class CompatibilityReviewController {
return true;
}
const acknowledgedVersion = this.readAcknowledgedVersion();
const acknowledgedVersion = await this.readAcknowledgedVersion();
const evaluation = evaluateCompatibilityPause({
acknowledgedVersion,
currentVersion: this.currentVersion,
@@ -85,7 +85,7 @@ export class CompatibilityReviewController {
this.pause = evaluation.pause;
if (evaluation.initialiseAcknowledgedVersion) {
setting.setSmallConfig(DATABASE_COMPATIBILITY_VERSION_KEY, `${this.currentVersion}`);
await this.core.services.config.setValue(DATABASE_COMPATIBILITY_VERSION_KEY, `${this.currentVersion}`);
this._initialised = true;
return true;
}
@@ -115,9 +115,9 @@ export class CompatibilityReviewController {
throw error;
}
setting.setSmallConfig(DATABASE_COMPATIBILITY_VERSION_KEY, `${this.currentVersion}`);
await this.core.services.config.setValue(DATABASE_COMPATIBILITY_VERSION_KEY, `${this.currentVersion}`);
const legacyKey = legacyDatabaseCompatibilityVersionKey(this.core.services.vault.getVaultName());
setting.deleteDeviceLocalConfig(legacyKey);
await setting.deviceValues.delete(legacyKey);
this.pause = undefined;
this.ui.clearReminder();
await this.core.services.control.applySettings();
src/serviceFeatures/compatibilityReview.unit.spec.ts
+121 -7
View File
@@ -33,13 +33,32 @@ function createFixture(
} = {}
) {
const local = new Map<string, string>();
const deviceValues = new Map<string, string>();
if (options.marker !== undefined && options.marker !== null) {
local.set(DATABASE_COMPATIBILITY_VERSION_KEY, options.marker);
}
const legacyKey = legacyDatabaseCompatibilityVersionKey("Test Vault");
if (options.legacyMarker !== undefined && options.legacyMarker !== null) {
local.set(legacyKey, options.legacyMarker);
deviceValues.set(legacyKey, options.legacyMarker);
}
const config = {
getValue: vi.fn(async (key: string, defaultValue = "") => local.get(key) ?? defaultValue),
setValue: vi.fn(async (key: string, value: string) => {
local.set(key, value);
}),
delete: vi.fn(async (key: string) => {
local.delete(key);
}),
};
const deviceValueStore = {
getValue: vi.fn(async (key: string, defaultValue = "") => deviceValues.get(key) ?? defaultValue),
setValue: vi.fn(async (key: string, value: string) => {
deviceValues.set(key, value);
}),
delete: vi.fn(async (key: string) => {
deviceValues.delete(key);
}),
};
const settings = {
versionUpFlash: options.versionUpFlash ?? "",
isConfigured: options.isConfigured ?? true,
@@ -49,15 +68,13 @@ function createFixture(
const setting = {
currentSettings: () => settings,
getSettingsMigrationState: () => migrationState(options.migration),
getSmallConfig: (key: string) => local.get(key) ?? "",
setSmallConfig: (key: string, value: string) => local.set(key, value),
getDeviceLocalConfig: (key: string) => local.get(key) ?? null,
deleteDeviceLocalConfig: (key: string) => local.delete(key),
deviceValues: deviceValueStore,
saveSettingData,
};
const core = {
services: {
setting,
config,
vault: { getVaultName: () => "Test Vault" },
control: { applySettings },
},
@@ -69,7 +86,18 @@ function createFixture(
clearReminder: vi.fn(),
};
const controller = new CompatibilityReviewController(core, ui, 12);
return { controller, ui, local, legacyKey, settings, saveSettingData, applySettings };
return {
controller,
ui,
local,
deviceValues,
config,
deviceValueStore,
legacyKey,
settings,
saveSettingData,
applySettings,
};
}
describe("compatibility review controller", () => {
@@ -196,10 +224,96 @@ describe("compatibility review controller", () => {
await fixture.controller.initialise();
expect(fixture.local.get(DATABASE_COMPATIBILITY_VERSION_KEY)).toBe("11");
expect(fixture.local.has(fixture.legacyKey)).toBe(false);
expect(fixture.deviceValues.has(fixture.legacyKey)).toBe(false);
expect(fixture.controller.pendingPause).toBeDefined();
});
it("saves the scoped marker before deleting the legacy device value", async () => {
const fixture = createFixture({ legacyMarker: "11" });
let finishSave!: () => void;
const saving = new Promise<void>((resolve) => {
finishSave = resolve;
});
fixture.config.setValue.mockImplementation(async (key, value) => {
await saving;
fixture.local.set(key, value);
});
const initialising = fixture.controller.initialise();
await Promise.resolve();
await Promise.resolve();
expect(fixture.deviceValues.get(fixture.legacyKey)).toBe("11");
expect(fixture.deviceValueStore.delete).not.toHaveBeenCalled();
expect(fixture.controller.initialised).toBe(false);
finishSave();
await initialising;
expect(fixture.local.get(DATABASE_COMPATIBILITY_VERSION_KEY)).toBe("11");
expect(fixture.deviceValues.has(fixture.legacyKey)).toBe(false);
expect(fixture.controller.initialised).toBe(true);
});
it("does not treat a failed marker read as an absent value", async () => {
const fixture = createFixture({ legacyMarker: "11" });
fixture.config.getValue.mockRejectedValueOnce(new Error("marker read failed"));
await expect(fixture.controller.initialise()).rejects.toThrow("marker read failed");
expect(fixture.deviceValueStore.getValue).not.toHaveBeenCalled();
expect(fixture.deviceValues.get(fixture.legacyKey)).toBe("11");
expect(fixture.controller.initialised).toBe(false);
});
it("keeps the compatibility pause blocked until its async marker write succeeds", async () => {
const fixture = createFixture({ marker: "11" });
vi.mocked(fixture.ui.showSummary).mockResolvedValue("resume");
await fixture.controller.initialise();
let finishSave!: () => void;
const saving = new Promise<void>((resolve) => {
finishSave = resolve;
});
fixture.config.setValue.mockImplementation(async (key, value) => {
await saving;
fixture.local.set(key, value);
});
const reviewing = fixture.controller.openReview();
await Promise.resolve();
await Promise.resolve();
expect(fixture.controller.pendingPause).toBeDefined();
expect(fixture.local.get(DATABASE_COMPATIBILITY_VERSION_KEY)).toBe("11");
expect(fixture.applySettings).not.toHaveBeenCalled();
finishSave();
await reviewing;
expect(fixture.local.get(DATABASE_COMPATIBILITY_VERSION_KEY)).toBe("12");
expect(fixture.controller.pendingPause).toBeUndefined();
expect(fixture.applySettings).toHaveBeenCalledOnce();
});
it("keeps a storage-failed acknowledgement blocked and retryable", async () => {
const fixture = createFixture({ marker: "11" });
vi.mocked(fixture.ui.showSummary).mockResolvedValue("resume");
await fixture.controller.initialise();
fixture.config.setValue.mockRejectedValueOnce(new Error("marker write failed"));
await expect(fixture.controller.openReview()).rejects.toThrow("marker write failed");
expect(fixture.controller.pendingPause).toBeDefined();
expect(fixture.local.get(DATABASE_COMPATIBILITY_VERSION_KEY)).toBe("11");
expect(fixture.applySettings).not.toHaveBeenCalled();
await fixture.controller.openReview();
expect(fixture.controller.pendingPause).toBeUndefined();
expect(fixture.local.get(DATABASE_COMPATIBILITY_VERSION_KEY)).toBe("12");
expect(fixture.applySettings).toHaveBeenCalledOnce();
});
it("restores the runtime gate if persisting an acknowledgement fails", async () => {
const fixture = createFixture({ marker: "11" });
vi.mocked(fixture.ui.showSummary).mockResolvedValue("resume");
src/serviceFeatures/joinedDevices.ts
+64
View File
@@ -0,0 +1,64 @@
import type { JoinedDevice, JoinedDevicesSnapshot } from "@vrtmrz/livesync-commonlib/replication";
import type { JoinedDevicesView } from "./joinedDevicesDialogTypes";
/** Compare recognised SemVer values; unknown development labels have no ordering. */
export function comparePluginVersions(left: string, right: string): number | undefined {
const parse = (value: string) => {
const match =
/^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)(?:-([0-9A-Za-z-]+(?:\.[0-9A-Za-z-]+)*))?(?:\+[0-9A-Za-z-]+(?:\.[0-9A-Za-z-]+)*)?$/.exec(
value
);
if (!match) return undefined;
const prerelease = match[4]?.split(".");
if (prerelease?.some((part) => /^0\d+$/.test(part))) return undefined;
return { core: match.slice(1, 4).map(BigInt), prerelease };
};
const a = parse(left);
const b = parse(right);
if (!a || !b) return undefined;
for (let i = 0; i < 3; i++) {
if (a.core[i] !== b.core[i]) return a.core[i] > b.core[i] ? 1 : -1;
}
if (!a.prerelease || !b.prerelease) return a.prerelease ? -1 : b.prerelease ? 1 : 0;
for (let i = 0; i < Math.max(a.prerelease.length, b.prerelease.length); i++) {
const x = a.prerelease[i];
const y = b.prerelease[i];
if (x === undefined) return -1;
if (y === undefined) return 1;
if (x === y) continue;
const xn = /^\d+$/.test(x);
const yn = /^\d+$/.test(y);
if (xn && yn) return BigInt(x) > BigInt(y) ? 1 : -1;
if (xn !== yn) return xn ? -1 : 1;
return x > y ? 1 : -1;
}
return 0;
}
export function joinedDevicesView(
snapshot: JoinedDevicesSnapshot,
self: Omit<JoinedDevice, "accepted">
): JoinedDevicesView {
const ownRecord = snapshot.devices.find((device) => device.id === self.id);
const ownDevice: JoinedDevice = { ...ownRecord, ...self, accepted: ownRecord?.accepted ?? false };
return {
...snapshot,
devices: [ownDevice, ...snapshot.devices.filter((device) => device.id !== self.id)].map((device) => ({
...device,
self: device.id === self.id,
lockedOut: snapshot.locked && !device.accepted,
})),
};
}
export function differentDeviceVersions(snapshot: JoinedDevicesSnapshot, ownId: string, ownVersion: string) {
const newer = new Set<string>();
const older = new Set<string>();
for (const device of snapshot.devices) {
if (device.id === ownId || !device.pluginVersion) continue;
const comparison = comparePluginVersions(device.pluginVersion, ownVersion);
if (comparison !== undefined && comparison > 0) newer.add(device.pluginVersion);
if (comparison !== undefined && comparison < 0) older.add(device.pluginVersion);
}
return { newer: [...newer].sort(), older: [...older].sort() };
}
src/serviceFeatures/joinedDevices.unit.spec.ts
+69
View File
@@ -0,0 +1,69 @@
import { describe, expect, it } from "vitest";
import { comparePluginVersions, differentDeviceVersions, joinedDevicesView } from "./joinedDevices";
import type { JoinedDevicesSnapshot } from "@vrtmrz/livesync-commonlib/replication";
const empty: JoinedDevicesSnapshot = { milestonePresent: false, locked: false, cleaned: false, devices: [] };
describe("joined device presentation", () => {
it("always shows this device first, including before its remote record exists", () => {
const view = joinedDevicesView(
{ ...empty, devices: [{ id: "another", accepted: true }] },
{ id: "self", platform: "ios", pluginVersion: "1.0.39" }
);
expect(view.devices.map((device) => device.id)).toEqual(["self", "another"]);
expect(view.devices[0]).toMatchObject({ self: true, accepted: false, lockedOut: false, platform: "ios" });
});
it("keeps the remote last-recorded date and acceptance while showing current local versions", () => {
const view = joinedDevicesView(
{
...empty,
milestonePresent: true,
locked: true,
devices: [
{ id: "other", accepted: true },
{ id: "self", accepted: false, lastConnected: 123, pluginVersion: "1.0.0" },
],
},
{ id: "self", pluginVersion: "1.1.0" }
);
expect(view.devices[0]).toMatchObject({
self: true,
lockedOut: true,
lastConnected: 123,
pluginVersion: "1.1.0",
});
expect(view.devices[1].lockedOut).toBe(false);
});
});
describe("version advisories", () => {
it.each([
["1.10.0", "1.9.9", 1],
["1.0.0", "1.0.0-rc.1", 1],
["1.0.0-beta.2", "1.0.0-beta.10", -1],
["1.0.0-1", "1.0.0-alpha", -1],
["1.0.0-alpha", "1.0.0-alpha.1", -1],
["1.0.0+first", "1.0.0+second", 0],
["99999999999999999.0.0", "99999999999999998.0.0", 1],
])("compares %s with %s", (a, b, result) => expect(comparePluginVersions(a, b)).toBe(result));
it.each(["", "development", "1.0", "01.0.0", "1.0.0-01", "v1.0.0"])(
"does not order unknown version %s",
(version) => {
expect(comparePluginVersions(version, "1.0.0")).toBeUndefined();
expect(comparePluginVersions("1.0.0", version)).toBeUndefined();
}
);
it("excludes this device and unknown versions, and deduplicates versions", () => {
const snapshot = {
...empty,
devices: [
{ id: "self", accepted: false, pluginVersion: "9.0.0" },
{ id: "a", accepted: true, pluginVersion: "2.0.0" },
{ id: "b", accepted: false, pluginVersion: "2.0.0" },
{ id: "c", accepted: false, pluginVersion: "0.9.0" },
{ id: "d", accepted: true, pluginVersion: "custom" },
],
};
expect(differentDeviceVersions(snapshot, "self", "1.0.0")).toEqual({ newer: ["2.0.0"], older: ["0.9.0"] });
});
});
src/serviceFeatures/joinedDevicesCommand.ts
+5
View File
@@ -0,0 +1,5 @@
import { declareUICommand } from "@vrtmrz/livesync-commonlib/ui-commands";
export const OPEN_JOINED_DEVICES = declareUICommand<void, "closed" | "already-open" | "unavailable">(
"livesync:joined-devices"
);
src/serviceFeatures/joinedDevicesDialogTypes.ts
+19
View File
@@ -0,0 +1,19 @@
import type { ForgetJoinedDeviceResult, JoinedDevice } from "@vrtmrz/livesync-commonlib/replication";
export interface JoinedDeviceRow extends JoinedDevice {
readonly self: boolean;
readonly lockedOut: boolean;
}
export interface JoinedDevicesView {
readonly milestonePresent: boolean;
readonly locked: boolean;
readonly cleaned: boolean;
readonly devices: readonly JoinedDeviceRow[];
}
export interface JoinedDevicesDialogOptions {
refresh(): Promise<JoinedDevicesView>;
forget(id: string): Promise<ForgetJoinedDeviceResult>;
cancel(): void;
}
src/serviceFeatures/redFlag.simpleFetch.ts
+18 -19
View File
@@ -63,28 +63,27 @@ function buildSimpleFetchResult(stage1: string, stage2?: string) {
return undefined;
}
function rememberSimpleFetchMode(host: NecessaryServices<"setting", never>, stage1: string, stage2?: string) {
host.services.setting.setSmallConfig(SIMPLE_FETCH_MODE_KEY, JSON.stringify({ stage1, stage2 }));
async function rememberSimpleFetchMode(host: NecessaryServices<"setting", never>, stage1: string, stage2?: string) {
await host.services.setting.config.setValue(SIMPLE_FETCH_MODE_KEY, JSON.stringify({ stage1, stage2 }));
}
function getRememberedSimpleFetchMode(host: NecessaryServices<"setting", never>) {
const saved = host.services.setting.getSmallConfig(SIMPLE_FETCH_MODE_KEY);
async function getRememberedSimpleFetchMode(host: NecessaryServices<"setting", never>) {
const saved = await host.services.setting.config.getValue(SIMPLE_FETCH_MODE_KEY, "");
if (!saved) return undefined;
let remembered: ReturnType<typeof buildSimpleFetchResult> = undefined;
try {
const { stage1, stage2 } = JSON.parse(saved) as { stage1?: string; stage2?: string };
if (stage1) {
const remembered = buildSimpleFetchResult(stage1, stage2);
if (remembered) return remembered;
}
if (stage1) remembered = buildSimpleFetchResult(stage1, stage2);
} catch {
// Clear below; the saved choice is optional and can be rebuilt by asking again.
}
host.services.setting.deleteSmallConfig(SIMPLE_FETCH_MODE_KEY);
if (remembered) return remembered;
await host.services.setting.config.delete(SIMPLE_FETCH_MODE_KEY);
return undefined;
}
function clearRememberedSimpleFetchMode(host: NecessaryServices<"setting", never>) {
host.services.setting.deleteSmallConfig(SIMPLE_FETCH_MODE_KEY);
async function clearRememberedSimpleFetchMode(host: NecessaryServices<"setting", never>) {
await host.services.setting.config.delete(SIMPLE_FETCH_MODE_KEY);
}
async function confirmTranslatedChoice(
@@ -104,7 +103,7 @@ async function confirmTranslatedChoice(
export async function askSimpleFetchMode(
host: NecessaryServices<"UI" | "setting", never>
): Promise<{ mode: string; options: Partial<FullScanOptions> } | "cancelled" | "aborted"> {
const remembered = getRememberedSimpleFetchMode(host);
const remembered = await getRememberedSimpleFetchMode(host);
if (remembered) return remembered;
const msg = $msg(
@@ -169,7 +168,7 @@ Firstly, how shall we handle the data retrieved from this remote source?
if (stage2 === STAGE2_ABORT) {
return "aborted";
}
rememberSimpleFetchMode(host, stage1, stage2);
await rememberSimpleFetchMode(host, stage1, stage2);
return buildSimpleFetchResult(stage1, stage2)!;
}
@@ -198,7 +197,7 @@ Firstly, how shall we handle the data retrieved from this remote source?
if (stage2 === STAGE2_ABORT) {
return "aborted";
}
rememberSimpleFetchMode(host, stage1, stage2);
await rememberSimpleFetchMode(host, stage1, stage2);
return buildSimpleFetchResult(stage1, stage2)!;
}
@@ -234,18 +233,18 @@ export async function askAndPerformFastSetupOnScheduledFetchAll(
"Remediation mode is active, so the detailed fetch flow is used instead of Simple Fetch.",
LOG_LEVEL_NOTICE
);
clearRememberedSimpleFetchMode(host);
await clearRememberedSimpleFetchMode(host);
return undefined;
}
const result = await askSimpleFetchMode(host);
if (result === "cancelled") {
log("Fetch cancelled by user.", LOG_LEVEL_NOTICE);
clearRememberedSimpleFetchMode(host);
await clearRememberedSimpleFetchMode(host);
return await cancelScheduledInitialisation(host, cleanupFlag);
}
if (result === "aborted") {
log("Fetch exited by user.", LOG_LEVEL_NOTICE);
clearRememberedSimpleFetchMode(host);
await clearRememberedSimpleFetchMode(host);
host.services.appLifecycle.performRestart();
return false;
}
@@ -256,7 +255,7 @@ export async function askAndPerformFastSetupOnScheduledFetchAll(
const settings = host.services.setting.currentSettings();
if (!(await adjustSettingToRemoteIfNeeded(host, log, { preventFetchingConfig: false }, settings))) {
log("Fetch initialisation cancelled by user.", LOG_LEVEL_NOTICE);
clearRememberedSimpleFetchMode(host);
await clearRememberedSimpleFetchMode(host);
return await cancelScheduledInitialisation(host, cleanupFlag);
}
@@ -295,7 +294,7 @@ export async function askAndPerformFastSetupOnScheduledFetchAll(
}
await host.serviceModules.rebuilder.finishRebuild();
await cleanupFlag();
clearRememberedSimpleFetchMode(host);
await clearRememberedSimpleFetchMode(host);
log("Simple fetch and scan operation completed.", LOG_LEVEL_NOTICE);
return true;
};
src/serviceFeatures/redFlag.simpleFetch.unit.spec.ts
+80 -4
View File
@@ -10,19 +10,26 @@ const KEEP_LOCAL = "Keep local files even if not on remote";
function createHost(savedChoice?: string) {
const smallConfig = new Map<string, string>();
if (savedChoice) smallConfig.set("simple-fetch-mode", savedChoice);
const config = {
getValue: vi.fn(async (key: string, defaultValue = "") => smallConfig.get(key) ?? defaultValue),
setValue: vi.fn(async (key: string, value: string) => {
smallConfig.set(key, value);
}),
delete: vi.fn(async (key: string) => {
smallConfig.delete(key);
}),
};
const confirmWithMessage =
vi.fn<(title: string, message: string, buttons: string[], defaultAction: string) => Promise<string | false>>();
const host = {
services: {
UI: { confirm: { confirmWithMessage } },
setting: {
getSmallConfig: (key: string) => smallConfig.get(key) ?? "",
setSmallConfig: (key: string, value: string) => smallConfig.set(key, value),
deleteSmallConfig: (key: string) => smallConfig.delete(key),
config,
},
},
} as unknown as NecessaryServices<"UI" | "setting", never>;
return { host, confirmWithMessage, smallConfig };
return { host, confirmWithMessage, smallConfig, config };
}
afterEach(() => setLang("def"));
@@ -50,6 +57,75 @@ describe("Fast Setup translated choices", () => {
});
});
it("waits for the saved choice before returning it to the fetch workflow", async () => {
const { host, confirmWithMessage, smallConfig, config } = createHost();
let markSaveStarted!: () => void;
const saveStarted = new Promise<void>((resolve) => {
markSaveStarted = resolve;
});
let finishSave!: () => void;
const save = new Promise<void>((resolve) => {
finishSave = resolve;
});
config.setValue.mockImplementation(async (key, value) => {
markSaveStarted();
await save;
smallConfig.set(key, value);
});
confirmWithMessage.mockImplementation(async (_title, _message, buttons) => buttons[1]);
const asking = askSimpleFetchMode(host);
await saveStarted;
expect(smallConfig.has("simple-fetch-mode")).toBe(false);
expect(confirmWithMessage).toHaveBeenCalledTimes(2);
finishSave();
await expect(asking).resolves.toMatchObject({ mode: "remote-only" });
expect(JSON.parse(smallConfig.get("simple-fetch-mode")!)).toEqual({
stage1: REMOTE_WINS,
stage2: KEEP_LOCAL,
});
});
it("propagates a stored-choice read failure instead of treating it as missing", async () => {
const { host, confirmWithMessage, config } = createHost();
config.getValue.mockRejectedValueOnce(new Error("choice read failed"));
await expect(askSimpleFetchMode(host)).rejects.toThrow("choice read failed");
expect(confirmWithMessage).not.toHaveBeenCalled();
});
it("waits for an invalid optional choice to be deleted before asking again", async () => {
const { host, confirmWithMessage, smallConfig, config } = createHost("not-json");
let markDeleteStarted!: () => void;
const deleteStarted = new Promise<void>((resolve) => {
markDeleteStarted = resolve;
});
let finishDelete!: () => void;
const deletion = new Promise<void>((resolve) => {
finishDelete = resolve;
});
config.delete.mockImplementation(async (key) => {
markDeleteStarted();
await deletion;
smallConfig.delete(key);
});
confirmWithMessage.mockImplementation(async (_title, _message, buttons) => buttons[3]);
const asking = askSimpleFetchMode(host);
await deleteStarted;
expect(confirmWithMessage).not.toHaveBeenCalled();
expect(smallConfig.has("simple-fetch-mode")).toBe(true);
finishDelete();
await expect(asking).resolves.toBe("cancelled");
expect(smallConfig.has("simple-fetch-mode")).toBe(false);
expect(confirmWithMessage).toHaveBeenCalledOnce();
});
it("restores choices saved by the previous English implementation while displaying Spanish", async () => {
const savedChoice = JSON.stringify({ stage1: REMOTE_WINS, stage2: KEEP_LOCAL });
const { host, confirmWithMessage, smallConfig } = createHost(savedChoice);
src/serviceFeatures/redFlag.unit.spec.ts
+12 -10
View File
@@ -107,13 +107,15 @@ const createSettingServiceMock = () => {
}),
suspendAllSync: vi.fn(() => Promise.resolve()),
suspendExtraSync: vi.fn(() => Promise.resolve()),
getSmallConfig: vi.fn((key: string) => smallConfig.get(key) ?? ""),
setSmallConfig: vi.fn((key: string, value: string) => {
smallConfig.set(key, value);
}),
deleteSmallConfig: vi.fn((key: string) => {
smallConfig.delete(key);
}),
config: {
getValue: vi.fn(async (key: string, defaultValue = "") => smallConfig.get(key) ?? defaultValue),
setValue: vi.fn(async (key: string, value: string) => {
smallConfig.set(key, value);
}),
delete: vi.fn(async (key: string) => {
smallConfig.delete(key);
}),
},
};
};
@@ -785,7 +787,7 @@ describe("Red Flag Feature", () => {
);
expect(host.mocks.ui.confirm.confirmWithMessage).not.toHaveBeenCalled();
expect(host.mocks.setting.deleteSmallConfig).toHaveBeenCalledWith("simple-fetch-mode");
expect(host.mocks.setting.config.delete).toHaveBeenCalledWith("simple-fetch-mode");
expect(cleanupFlag).not.toHaveBeenCalled();
});
@@ -830,7 +832,7 @@ describe("Red Flag Feature", () => {
});
expect(host.mocks.rebuilder.finishRebuild).not.toHaveBeenCalled();
expect(cleanupFlag).not.toHaveBeenCalled();
expect(host.mocks.setting.deleteSmallConfig).not.toHaveBeenCalledWith("simple-fetch-mode");
expect(host.mocks.setting.config.delete).not.toHaveBeenCalledWith("simple-fetch-mode");
});
it("re-suspends both reflection directions when finalisation fails after releasing them", async () => {
@@ -892,7 +894,7 @@ describe("Red Flag Feature", () => {
await askAndPerformFastSetupOnScheduledFetchAll(host as any, log, cleanupFlag);
expect(host.mocks.setting.deleteSmallConfig).toHaveBeenCalledWith("simple-fetch-mode");
expect(host.mocks.setting.config.delete).toHaveBeenCalledWith("simple-fetch-mode");
});
it("should return false and cleanup when quick flow is cancelled", async () => {
src/serviceFeatures/useJoinedDevices.ts
+219
View File
@@ -0,0 +1,219 @@
import type { NecessaryServices } from "@vrtmrz/livesync-commonlib/compat/interfaces/ServiceModule";
import {
NODEINFO_DOCID,
REMOTE_COUCHDB,
REMOTE_MINIO,
type EntryNodeInfo,
type RemoteDBSettings,
} from "@vrtmrz/livesync-commonlib/compat/common/types";
import { REMOTE_RESOURCE_KINDS, type JoinedDevicesSnapshot } from "@vrtmrz/livesync-commonlib/replication";
import type { ObsidianNoticeGroups } from "@/modules/services/ObsidianNoticeGroups";
import JoinedDevicesDialog from "@/modules/services/LiveSyncUI/dialogues/JoinedDevicesDialog.svelte";
import {
getCouchDBReplicatorConfigurationIdentity,
getObjectStorageReplicatorConfigurationIdentity,
} from "@/common/replicatorConfigurationIdentity";
import { $msg } from "@/common/translation";
import { OPEN_JOINED_DEVICES } from "./joinedDevicesCommand";
import { differentDeviceVersions, joinedDevicesView } from "./joinedDevices";
import type { JoinedDevicesDialogOptions } from "./joinedDevicesDialogTypes";
type JoinedDevicesHost = NecessaryServices<
"API" | "UI" | "setting" | "database" | "replicator" | "replication" | "appLifecycle",
never
>;
const noticeGroup = "joined-device-versions";
function remoteIdentity(settings: RemoteDBSettings): string | undefined {
if (settings.remoteType === REMOTE_COUCHDB && settings.couchDB_URI && settings.couchDB_DBNAME) {
return getCouchDBReplicatorConfigurationIdentity(settings);
}
if (settings.remoteType === REMOTE_MINIO && settings.endpoint && settings.bucket) {
return getObjectStorageReplicatorConfigurationIdentity(settings);
}
return undefined;
}
/** Own the dialogue, advisory Notices, and their per-session state for one host. */
export function useJoinedDevicesFeature(host: JoinedDevicesHost, notices: ObsidianNoticeGroups): void {
const { services } = host;
let disposed = false;
let active: AbortController | undefined;
let checking = false;
let lastCheck: { identity: string; at: number } | undefined;
const notified = new Map<string, Set<string>>();
const available = () =>
!disposed &&
services.database.isDatabaseReady() &&
remoteIdentity(services.setting.currentSettings()) !== undefined;
const notify = (snapshot: JoinedDevicesSnapshot, ownId: string, identity: string) => {
if (disposed || identity !== remoteIdentity(services.setting.currentSettings())) return;
const version = services.API.getPluginVersion();
const differences = differentDeviceVersions(snapshot, ownId, version);
const seen = notified.get(identity) ?? new Set<string>();
notified.set(identity, seen);
let updated = false;
for (const direction of ["newer", "older"] as const) {
const versions = differences[direction];
if (versions.length === 0) continue;
const key = JSON.stringify([version, direction, versions]);
if (seen.has(key)) continue;
seen.add(key);
notices.setItem(noticeGroup, direction, {
message:
direction === "newer"
? $msg("Newer Self-hosted LiveSync versions detected: ${versions}.", {
versions: versions.join(", "),
})
: $msg("Older Self-hosted LiveSync versions detected: ${versions}.", {
versions: versions.join(", "),
}),
});
updated = true;
}
if (!updated) return;
notices.removeItem(noticeGroup, "review");
notices.setItem(noticeGroup, "review", {
message: $msg("Reviewing and aligning Self-hosted LiveSync versions across devices is recommended."),
action: {
label: $msg("Review devices"),
onSelect: () => {
if (!disposed && identity === remoteIdentity(services.setting.currentSettings())) {
void services.UI.commands.invoke(OPEN_JOINED_DEVICES, undefined);
}
},
},
});
notices.finish(noticeGroup, { durationMs: 15000 });
};
const createOperations = (controller: AbortController): JoinedDevicesDialogOptions => {
const settings = { ...services.setting.currentSettings() };
const identity = remoteIdentity(settings);
const database = services.database.localDatabase;
const assertCurrent = () => {
if (
disposed ||
controller.signal.aborted ||
!identity ||
identity !== remoteIdentity(services.setting.currentSettings()) ||
!services.database.isDatabaseReady() ||
services.database.localDatabase !== database
) {
throw new Error("Joined devices are no longer available for this dialogue");
}
};
const ownId = async () => {
assertCurrent();
const info = await database.localDatabase.get<EntryNodeInfo>(NODEINFO_DOCID);
assertCurrent();
if (typeof info.nodeid !== "string" || !info.nodeid) throw new Error("Device identity is unavailable");
return info.nodeid;
};
return {
refresh: async () => {
const id = await ownId();
const resource = await services.replicator.createRemoteResource(
REMOTE_RESOURCE_KINDS.JOINED_DEVICES,
settings
);
if (!resource) throw new Error("Joined devices are unavailable");
try {
assertCurrent();
const snapshot = await resource.read();
assertCurrent();
notify(snapshot, id, identity!);
return joinedDevicesView(snapshot, {
id,
platform: services.API.getPlatform(),
pluginVersion: services.API.getPluginVersion(),
appVersion: services.API.getAppVersion(),
});
} finally {
await resource.dispose();
}
},
forget: async (id) => {
if (id === (await ownId())) throw new Error("This device cannot be forgotten");
const resource = await services.replicator.createRemoteResource(
REMOTE_RESOURCE_KINDS.JOINED_DEVICES,
settings
);
if (!resource) throw new Error("Joined devices are unavailable");
try {
assertCurrent();
const result = await resource.forget(id);
assertCurrent();
return result;
} finally {
await resource.dispose();
}
},
cancel: () => controller.abort(),
};
};
const unregister = services.UI.commands.register(OPEN_JOINED_DEVICES, async () => {
if (!available()) return "unavailable";
if (active) return "already-open";
const controller = new AbortController();
active = controller;
try {
await services.UI.dialogManager.open<"close", JoinedDevicesDialogOptions>(
JoinedDevicesDialog,
createOperations(controller)
);
} catch {
// The host may close its dialogues during unload.
} finally {
controller.abort();
if (active === controller) active = undefined;
}
return "closed";
});
const checkVersions = () => {
if (!available() || checking) return;
const identity = remoteIdentity(services.setting.currentSettings())!;
if (lastCheck?.identity === identity && Date.now() - lastCheck.at < 60000) return;
lastCheck = { identity, at: Date.now() };
checking = true;
const controller = new AbortController();
void Promise.resolve()
.then(() => createOperations(controller).refresh())
.catch(() => {
// Advisory inspection never changes the outcome of synchronisation.
})
.finally(() => {
controller.abort();
checking = false;
});
};
const stopBeforeReplicate = services.replication.onBeforeReplicate.addHandler(() => {
checkVersions();
return Promise.resolve(true);
});
services.appLifecycle.onInitialise.addHandler(() => {
services.API.addCommand({
id: "joined-devices",
name: $msg("Review devices"),
icon: "monitor-smartphone",
checkCallback: (checking) => {
if (!available()) return false;
if (!checking) void services.UI.commands.invoke(OPEN_JOINED_DEVICES, undefined);
return true;
},
});
return Promise.resolve(true);
});
services.appLifecycle.onUnload.addHandler(() => {
disposed = true;
active?.abort();
unregister();
stopBeforeReplicate();
notices.hide(noticeGroup);
notified.clear();
return Promise.resolve(true);
});
}
src/serviceFeatures/useJoinedDevices.unit.spec.ts
+320
View File
@@ -0,0 +1,320 @@
import { describe, expect, it, vi } from "vitest";
import { UICommandRegistry } from "@vrtmrz/livesync-commonlib/ui-commands";
import { createNewVaultSettings } from "@vrtmrz/livesync-commonlib/settings";
import { REMOTE_COUCHDB } from "@vrtmrz/livesync-commonlib/compat/common/types";
import type { JoinedDevicesSnapshot } from "@vrtmrz/livesync-commonlib/replication";
import type { JoinedDevicesDialogOptions } from "./joinedDevicesDialogTypes";
vi.mock("@/modules/services/LiveSyncUI/dialogues/JoinedDevicesDialog.svelte", () => ({ default: {} }));
vi.mock("@/common/translation", () => ({
$msg: (message: string, values: Record<string, string> = {}) =>
message.replace(/\$\{([^}]+)\}/g, (placeholder, key: string) => values[key] ?? placeholder),
}));
import { useJoinedDevicesFeature } from "./useJoinedDevices";
import { OPEN_JOINED_DEVICES } from "./joinedDevicesCommand";
function fixture() {
const settings = {
...createNewVaultSettings(),
remoteType: REMOTE_COUCHDB,
couchDB_URI: "https://example.test",
couchDB_DBNAME: "vault",
};
const initialise: Array<() => Promise<boolean>> = [];
const unload: Array<() => Promise<boolean>> = [];
const before: Array<() => Promise<boolean>> = [];
const stopBefore = vi.fn();
const commands = new UICommandRegistry();
const read = vi.fn<() => Promise<JoinedDevicesSnapshot>>(async () => ({
milestonePresent: true,
locked: false,
cleaned: false,
devices: [{ id: "other", accepted: true, pluginVersion: "2.0.0" }],
}));
const forget = vi.fn(async (_id: string) => "forgotten" as const);
const dispose = vi.fn(async () => {});
const createResource = vi.fn(async () => ({ read, forget, dispose }));
let close!: () => void;
const dialog = vi.fn(
(_component: unknown, _options: JoinedDevicesDialogOptions) =>
new Promise<void>((resolve) => {
close = resolve;
})
);
const notices = { setItem: vi.fn(), removeItem: vi.fn(), finish: vi.fn(), hide: vi.fn() };
const palette: Array<{ id: string; checkCallback: (checking: boolean) => boolean }> = [];
const database = { localDatabase: { get: vi.fn(async () => ({ nodeid: "self" })) } };
const host = {
services: {
API: {
getPluginVersion: () => "1.0.0",
getAppVersion: () => "1.9.0",
getPlatform: () => "desktop",
addCommand: (command: (typeof palette)[number]) => palette.push(command),
},
UI: { commands, dialogManager: { open: dialog } },
setting: { currentSettings: () => settings },
database: { isDatabaseReady: () => true, localDatabase: database },
replicator: { createRemoteResource: createResource },
replication: {
onBeforeReplicate: {
addHandler: (handler: (typeof before)[number]) => {
before.push(handler);
return stopBefore;
},
},
},
appLifecycle: {
onInitialise: { addHandler: (handler: (typeof initialise)[number]) => initialise.push(handler) },
onUnload: { addHandler: (handler: (typeof unload)[number]) => unload.push(handler) },
},
},
serviceModules: {},
} as unknown as Parameters<typeof useJoinedDevicesFeature>[0];
useJoinedDevicesFeature(host, notices as unknown as Parameters<typeof useJoinedDevicesFeature>[1]);
return {
settings,
initialise,
unload,
before,
stopBefore,
commands,
read,
forget,
dispose,
createResource,
dialog,
notices,
palette,
database,
close: () => close(),
};
}
async function open(data: ReturnType<typeof fixture>) {
const pending = data.commands.invoke(OPEN_JOINED_DEVICES, undefined);
const options = data.dialog.mock.calls[0][1];
return { options, pending };
}
describe("joined devices service feature", () => {
it("opens one dialogue through the command and disposes each completed read", async () => {
const data = fixture();
const { options, pending } = await open(data);
expect(await data.commands.invoke(OPEN_JOINED_DEVICES, undefined)).toEqual({
status: "handled",
value: "already-open",
});
expect((await options.refresh()).devices[0]).toMatchObject({ id: "self", self: true });
expect(data.dispose).toHaveBeenCalledTimes(1);
data.close();
expect(await pending).toEqual({ status: "handled", value: "closed" });
await expect(options.refresh()).rejects.toThrow();
});
it("prevents self removal and fences an old dialogue after a remote switch", async () => {
const data = fixture();
const { options, pending } = await open(data);
await expect(options.forget("self")).rejects.toThrow();
data.settings.couchDB_DBNAME = "new-vault";
await expect(options.forget("other")).rejects.toThrow();
expect(data.forget).not.toHaveBeenCalled();
expect(data.createResource).not.toHaveBeenCalled();
data.close();
await pending;
});
it("discards a stale read, releases the resource, and emits no stale Notice", async () => {
const data = fixture();
let resolve!: (value: JoinedDevicesSnapshot) => void;
data.read.mockReturnValueOnce(
new Promise((done) => {
resolve = done;
})
);
const { options, pending } = await open(data);
const reading = options.refresh();
await vi.waitFor(() => expect(data.read).toHaveBeenCalled());
data.settings.couchDB_DBNAME = "new-vault";
resolve({ milestonePresent: true, locked: false, cleaned: false, devices: [] });
await expect(reading).rejects.toThrow();
expect(data.dispose).toHaveBeenCalledOnce();
expect(data.notices.setItem).not.toHaveBeenCalled();
data.close();
await pending;
});
it.each(["dialogue cancellation", "remote switch", "unload"] as const)(
"does not send a pending forget operation after %s",
async (invalidation) => {
const data = fixture();
const resource = { read: data.read, forget: data.forget, dispose: data.dispose };
let resolve!: (value: typeof resource) => void;
data.createResource.mockReturnValueOnce(
new Promise((done) => {
resolve = done;
})
);
const { options, pending } = await open(data);
const forgetting = options.forget("other");
const rejected = expect(forgetting).rejects.toThrow();
await vi.waitFor(() => expect(data.createResource).toHaveBeenCalledOnce());
if (invalidation === "dialogue cancellation") options.cancel();
if (invalidation === "remote switch") data.settings.couchDB_DBNAME = "new-vault";
if (invalidation === "unload") for (const unload of data.unload) await unload();
resolve(resource);
await rejected;
expect(data.forget).not.toHaveBeenCalled();
expect(data.dispose).toHaveBeenCalledOnce();
data.close();
await pending;
}
);
it("discards a completed forget result after the dialogue closes and releases its resource", async () => {
const data = fixture();
let resolve!: (value: "forgotten") => void;
data.forget.mockReturnValueOnce(
new Promise((done) => {
resolve = done;
})
);
const { options, pending } = await open(data);
const forgetting = options.forget("other");
const rejected = expect(forgetting).rejects.toThrow();
await vi.waitFor(() => expect(data.forget).toHaveBeenCalledWith("other"));
data.close();
await pending;
resolve("forgotten");
await rejected;
expect(data.dispose).toHaveBeenCalledOnce();
});
it("propagates a failed forget operation after releasing its resource", async () => {
const data = fixture();
const failure = new Error("conditional write failed");
data.forget.mockRejectedValueOnce(failure);
const { options, pending } = await open(data);
await expect(options.forget("other")).rejects.toBe(failure);
expect(data.dispose).toHaveBeenCalledOnce();
data.close();
await pending;
});
it("shares one review action across newer and older device versions", async () => {
const data = fixture();
data.read.mockResolvedValue({
milestonePresent: true,
locked: false,
cleaned: false,
devices: [
{ id: "newer-a", accepted: true, pluginVersion: "2.0.0" },
{ id: "newer-b", accepted: true, pluginVersion: "3.0.0" },
{ id: "older-a", accepted: true, pluginVersion: "0.9.0" },
{ id: "older-b", accepted: true, pluginVersion: "0.8.0" },
{ id: "older-duplicate", accepted: true, pluginVersion: "0.8.0" },
],
});
const { options, pending } = await open(data);
await options.refresh();
const items = data.notices.setItem.mock.calls.map(([, , item]) => item);
expect(items.filter((item) => item.action)).toHaveLength(1);
const messages = items.map((item) => item.message).join("\n");
expect(messages).toContain("Self-hosted LiveSync");
expect(messages).toMatch(/Newer.*2\.0\.0, 3\.0\.0/);
expect(messages).toMatch(/Older.*0\.8\.0, 0\.9\.0/);
expect(messages.match(/0\.8\.0/g)).toHaveLength(1);
expect(messages.match(/Reviewing and aligning/g)).toHaveLength(1);
expect(items[items.length - 1]?.action?.label).toBe("Review devices");
expect(data.notices.finish).toHaveBeenCalledExactlyOnceWith("joined-device-versions", {
durationMs: 15000,
});
data.close();
await pending;
});
it("keeps the shared action last when another version direction arrives", async () => {
const data = fixture();
const visible = new Map<string, { message: string; action?: unknown }>();
data.notices.setItem.mockImplementation((_group, key, item) => visible.set(key, item));
data.notices.removeItem.mockImplementation((_group, key) => visible.delete(key));
const { options, pending } = await open(data);
await options.refresh();
expect([...visible.keys()]).toEqual(["newer", "review"]);
data.read.mockResolvedValue({
milestonePresent: true,
locked: false,
cleaned: false,
devices: [
{ id: "other", accepted: true, pluginVersion: "2.0.0" },
{ id: "older", accepted: true, pluginVersion: "0.9.0" },
],
});
await options.refresh();
expect([...visible.keys()]).toEqual(["newer", "older", "review"]);
expect([...visible.values()].filter((item) => item.action)).toHaveLength(1);
visible.clear();
data.read.mockResolvedValue({
milestonePresent: true,
locked: false,
cleaned: false,
devices: [
{ id: "other", accepted: true, pluginVersion: "2.0.0" },
{ id: "older", accepted: true, pluginVersion: "0.8.0" },
],
});
await options.refresh();
expect([...visible.keys()]).toEqual(["older", "review"]);
expect([...visible.values()].filter((item) => item.action)).toHaveLength(1);
data.close();
await pending;
});
it("does not show a review action when no different version is reported", async () => {
const data = fixture();
data.read.mockResolvedValue({
milestonePresent: true,
locked: false,
cleaned: false,
devices: [{ id: "other", accepted: true, pluginVersion: "1.0.0" }],
});
const { options, pending } = await open(data);
await options.refresh();
expect(data.notices.setItem).not.toHaveBeenCalled();
expect(data.notices.removeItem).not.toHaveBeenCalled();
expect(data.notices.finish).not.toHaveBeenCalled();
data.close();
await pending;
});
it("deduplicates advisory Notices and fences their actions after configuration changes", async () => {
const data = fixture();
const { options, pending } = await open(data);
await options.refresh();
await options.refresh();
expect(data.notices.setItem).toHaveBeenCalledTimes(2);
const actions = data.notices.setItem.mock.calls.map(([, , item]) => item.action).filter(Boolean);
expect(actions).toHaveLength(1);
const action = actions[0];
data.close();
await pending;
data.settings.couchDB_DBNAME = "new-vault";
action.onSelect();
expect(data.dialog).toHaveBeenCalledTimes(1);
});
it("keeps inspection failures out of the replication result and unregisters on unload", async () => {
const data = fixture();
data.read.mockRejectedValue(new Error("unavailable"));
expect(await data.before[0]()).toBe(true);
await vi.waitFor(() => expect(data.dispose).toHaveBeenCalledOnce());
expect(await data.before[0]()).toBe(true);
expect(data.read).toHaveBeenCalledTimes(1);
for (const unload of data.unload) await unload();
expect(await data.commands.invoke(OPEN_JOINED_DEVICES, undefined)).toEqual({ status: "unavailable" });
expect(data.stopBefore).toHaveBeenCalledOnce();
expect(data.notices.hide).toHaveBeenCalled();
});
it("uses the same UI command from the command palette", async () => {
const data = fixture();
for (const initialise of data.initialise) await initialise();
expect(data.palette[0].checkCallback(true)).toBe(true);
expect(data.dialog).not.toHaveBeenCalled();
data.palette[0].checkCallback(false);
expect(data.dialog).toHaveBeenCalledOnce();
data.close();
});
});
src/serviceFeatures/useReviewHarness.ts
+7 -7
View File
@@ -69,9 +69,9 @@ export function useReviewHarness(
await createFileContentHashBenchmarkOperations(activeWindow.performance),
activeWindow.performance
),
readContinuation: () => services.setting.getSmallConfig(REVIEW_HARNESS_STATE_KEY),
writeContinuation: (value) => services.setting.setSmallConfig(REVIEW_HARNESS_STATE_KEY, value),
deleteContinuation: () => services.setting.deleteSmallConfig(REVIEW_HARNESS_STATE_KEY),
readContinuation: () => services.config.getValue(REVIEW_HARNESS_STATE_KEY, ""),
writeContinuation: (value) => services.config.setValue(REVIEW_HARNESS_STATE_KEY, value),
deleteContinuation: () => services.config.delete(REVIEW_HARNESS_STATE_KEY),
restart: () => services.appLifecycle.performRestart(),
reportError: (error) => services.API.addLog(error, LOG_LEVEL_NOTICE),
copyText: async (value) => {
@@ -94,14 +94,14 @@ export function useReviewHarness(
let registered = false;
let openAfterLayout = false;
services.appLifecycle.onSettingLoaded.addHandler(() => {
services.appLifecycle.onSettingLoaded.addHandler(async () => {
if (!continuationConsumed) {
await controller.consumeContinuation();
continuationConsumed = true;
controller.consumeContinuation();
}
const snapshot = controller.snapshot();
openAfterLayout = snapshot.resumedRequestId !== null || snapshot.continuationError !== null;
if (!services.setting.currentSettings().enableDebugTools || registered) return Promise.resolve(true);
if (!services.setting.currentSettings().enableDebugTools || registered) return true;
registered = true;
services.API.registerWindow(
@@ -115,7 +115,7 @@ export function useReviewHarness(
void services.API.showWindow(VIEW_TYPE_REVIEW_HARNESS);
},
});
return Promise.resolve(true);
return true;
});
services.appLifecycle.onLayoutReady.addHandler(() => {
src/serviceFeatures/useReviewHarness.unit.spec.ts
+79 -3
View File
@@ -31,6 +31,15 @@ function createFixture(options: { enableDebugTools?: boolean; continuation?: str
const layoutReadyHandlers: Array<() => Promise<boolean>> = [];
const local = new Map<string, string>();
if (options.continuation) local.set(REVIEW_HARNESS_STATE_KEY, options.continuation);
const config = {
getValue: vi.fn(async (key: string, defaultValue = "") => local.get(key) ?? defaultValue),
setValue: vi.fn(async (key: string, value: string) => {
local.set(key, value);
}),
delete: vi.fn(async (key: string) => {
local.delete(key);
}),
};
const settings = {
enableDebugTools: options.enableDebugTools ?? true,
liveSync: false,
@@ -63,10 +72,8 @@ function createFixture(options: { enableDebugTools?: boolean; continuation?: str
requiresSyncReview: true,
reviewReasons: [],
}),
getSmallConfig: (key: string) => local.get(key) ?? "",
setSmallConfig: (key: string, value: string) => local.set(key, value),
deleteSmallConfig: (key: string) => local.delete(key),
},
config,
appLifecycle: {
onSettingLoaded: {
addHandler: vi.fn((handler: () => Promise<boolean>) => settingLoadedHandlers.push(handler)),
@@ -106,6 +113,7 @@ function createFixture(options: { enableDebugTools?: boolean; continuation?: str
api,
settings,
local,
config,
settingLoadedHandlers,
layoutReadyHandlers,
compatibilityReview,
@@ -144,6 +152,74 @@ describe("Review Harness composition", () => {
expect(fixture.api.showWindow).toHaveBeenCalledWith(VIEW_TYPE_REVIEW_HARNESS);
});
it("holds the setting-loaded lifecycle until continuation deletion has completed", async () => {
const requestedAt = "2026-07-18T11:59:00.000Z";
const continuation = JSON.stringify({
formatVersion: 1,
requestId: `compatibility-review-${requestedAt}`,
scenarioId: "compatibility-review",
stage: "awaiting-restart",
requestedAt,
});
const fixture = createFixture({ continuation });
let markDeleteStarted!: () => void;
const deleteStarted = new Promise<void>((resolve) => {
markDeleteStarted = resolve;
});
let finishDelete!: () => void;
const deleting = new Promise<void>((resolve) => {
finishDelete = resolve;
});
fixture.config.delete.mockImplementation(async (key) => {
markDeleteStarted();
await deleting;
fixture.local.delete(key);
});
const loading = fixture.settingLoadedHandlers[0]();
await deleteStarted;
expect(fixture.local.has(REVIEW_HARNESS_STATE_KEY)).toBe(true);
expect(fixture.controller.snapshot().resumedRequestId).toBeNull();
expect(fixture.api.registerWindow).not.toHaveBeenCalled();
finishDelete();
await loading;
expect(fixture.local.has(REVIEW_HARNESS_STATE_KEY)).toBe(false);
expect(fixture.controller.snapshot().resumedRequestId).toBe(`compatibility-review-${requestedAt}`);
expect(fixture.api.registerWindow).toHaveBeenCalledOnce();
});
it("can retry one-shot continuation consumption after a deletion failure", async () => {
const requestedAt = "2026-07-18T11:59:00.000Z";
const continuation = JSON.stringify({
formatVersion: 1,
requestId: `compatibility-review-${requestedAt}`,
scenarioId: "compatibility-review",
stage: "awaiting-restart",
requestedAt,
});
const fixture = createFixture({ continuation });
fixture.config.delete
.mockRejectedValueOnce(new Error("continuation delete failed"))
.mockImplementation(async (key) => {
fixture.local.delete(key);
});
await expect(fixture.settingLoadedHandlers[0]()).rejects.toThrow("continuation delete failed");
expect(fixture.local.has(REVIEW_HARNESS_STATE_KEY)).toBe(true);
expect(fixture.controller.snapshot().resumedRequestId).toBeNull();
expect(fixture.api.registerWindow).not.toHaveBeenCalled();
await fixture.settingLoadedHandlers[0]();
expect(fixture.local.has(REVIEW_HARNESS_STATE_KEY)).toBe(false);
expect(fixture.controller.snapshot().resumedRequestId).toBe(`compatibility-review-${requestedAt}`);
expect(fixture.api.registerWindow).toHaveBeenCalledOnce();
});
it("uses the actual compatibility controller as the guided review boundary", async () => {
const fixture = createFixture();
test/apps/webapp/WebAppRuntime.unit.spec.ts
+9 -8
View File
@@ -1,3 +1,4 @@
import { MemoryValueStore } from "@vrtmrz/value-store";
import { LOG_LEVEL_INFO, type ObsidianLiveSyncSettings } from "@vrtmrz/livesync-commonlib/compat/common/types";
import { beforeEach, describe, expect, it, vi } from "vitest";
import type { LiveSyncBrowserServiceHubOptions } from "@/apps/browser/createLiveSyncBrowserServiceHub";
@@ -168,7 +169,7 @@ describe("WebAppRuntime lifecycle", () => {
it("owns core start, directory scan, status reporting, and shutdown", async () => {
const reportStatus = vi.fn();
const runtime = new WebAppRuntime(createRootHandle(), { reportStatus });
const runtime = new WebAppRuntime(createRootHandle(), { reportStatus, deviceValues: new MemoryValueStore() });
await runtime.start();
@@ -193,7 +194,7 @@ describe("WebAppRuntime lifecycle", () => {
});
it("finalises application readiness after importing local files for optional P2P", async () => {
const runtime = new WebAppRuntime(createRootHandle());
const runtime = new WebAppRuntime(createRootHandle(), { deviceValues: new MemoryValueStore() });
await runtime.start();
vi.clearAllMocks();
runtimeMocks.currentSettings.mockReturnValue(unconfiguredSettings);
@@ -239,7 +240,7 @@ describe("WebAppRuntime lifecycle", () => {
});
it("does not publish readiness after a partial local-file import", async () => {
const runtime = new WebAppRuntime(createRootHandle());
const runtime = new WebAppRuntime(createRootHandle(), { deviceValues: new MemoryValueStore() });
await runtime.start();
vi.clearAllMocks();
runtimeMocks.collectFilesOnStorage.mockResolvedValue({
@@ -269,7 +270,7 @@ describe("WebAppRuntime lifecycle", () => {
});
it("does not publish readiness when database initialisation rejects manual preparation", async () => {
const runtime = new WebAppRuntime(createRootHandle());
const runtime = new WebAppRuntime(createRootHandle(), { deviceValues: new MemoryValueStore() });
await runtime.start();
vi.clearAllMocks();
runtimeMocks.onDatabaseInitialised.mockResolvedValue(false);
@@ -282,7 +283,7 @@ describe("WebAppRuntime lifecycle", () => {
});
it("does not publish readiness when pending file events cannot be committed", async () => {
const runtime = new WebAppRuntime(createRootHandle());
const runtime = new WebAppRuntime(createRootHandle(), { deviceValues: new MemoryValueStore() });
await runtime.start();
vi.clearAllMocks();
runtimeMocks.commitPendingFileEvents.mockResolvedValue(false);
@@ -295,7 +296,7 @@ describe("WebAppRuntime lifecycle", () => {
});
it("does not repeat readiness finalisation after a successful manual scan", async () => {
const runtime = new WebAppRuntime(createRootHandle());
const runtime = new WebAppRuntime(createRootHandle(), { deviceValues: new MemoryValueStore() });
await runtime.start();
vi.clearAllMocks();
@@ -311,7 +312,7 @@ describe("WebAppRuntime lifecycle", () => {
it("rejects a failed core start after cleaning up the partial runtime", async () => {
runtimeMocks.onLoad.mockResolvedValue(false);
const reportStatus = vi.fn();
const runtime = new WebAppRuntime(createRootHandle(), { reportStatus });
const runtime = new WebAppRuntime(createRootHandle(), { reportStatus, deviceValues: new MemoryValueStore() });
await expect(runtime.start()).rejects.toThrow("Failed to initialise LiveSync");
@@ -325,7 +326,7 @@ describe("WebAppRuntime lifecycle", () => {
it("shuts down once before scheduling a host reload", async () => {
const scheduleReload = vi.fn();
const runtime = new WebAppRuntime(createRootHandle(), { scheduleReload });
const runtime = new WebAppRuntime(createRootHandle(), { scheduleReload, deviceValues: new MemoryValueStore() });
await runtime.start();
runtimeMocks.options?.restart?.schedule();
test/apps/webpeer/WebPeerRuntime.unit.spec.ts
+61 -13
View File
@@ -1,10 +1,11 @@
import { MemoryValueStore } from "@vrtmrz/value-store";
import { createServiceContext } from "@vrtmrz/livesync-commonlib/context";
import { DEFAULT_SETTINGS, SETTING_KEY_P2P_DEVICE_NAME } from "@vrtmrz/livesync-commonlib/compat/common/types";
import type { SimpleStore } from "octagonal-wheels/databases/SimpleStoreBase";
import { EVENT_LAYOUT_READY } from "@vrtmrz/livesync-commonlib/compat/events/coreEvents";
import { describe, expect, it, vi } from "vitest";
import { afterEach, describe, expect, it, vi } from "vitest";
import { WebPeerRuntime } from "@/apps/webpeer/src/WebPeerRuntime";
import { WebPeerRuntime, type WebPeerRuntimeOptions } from "@/apps/webpeer/src/WebPeerRuntime";
function createMemoryStore(): SimpleStore<unknown> {
const values = new Map<string, unknown>();
@@ -22,11 +23,31 @@ function createMemoryStore(): SimpleStore<unknown> {
}
describe("WebPeer runtime composition", () => {
const runtimes = new Set<WebPeerRuntime>();
function createRuntime(options: WebPeerRuntimeOptions): WebPeerRuntime {
const runtime = new WebPeerRuntime(options);
runtimes.add(runtime);
return runtime;
}
afterEach(async () => {
try {
for (const runtime of runtimes) {
await runtime.shutdown();
}
} finally {
runtimes.clear();
vi.restoreAllMocks();
}
});
it("preserves one context and live P2P result across the service graph and pane host", () => {
const context = createServiceContext({
translate: (key) => `webpeer:${key}`,
});
const runtime = new WebPeerRuntime({
const runtime = createRuntime({
deviceValues: new MemoryValueStore(),
context,
store: createMemoryStore(),
});
@@ -42,7 +63,8 @@ describe("WebPeer runtime composition", () => {
});
it("loads settings and opens the local database before announcing layout readiness", async () => {
const runtime = new WebPeerRuntime({
const runtime = createRuntime({
deviceValues: new MemoryValueStore(),
store: createMemoryStore(),
});
const loadSettings = vi.spyOn(runtime.services.setting, "loadSettings").mockResolvedValue(undefined);
@@ -67,7 +89,8 @@ describe("WebPeer runtime composition", () => {
it("delegates automatic P2P startup to the resumed lifecycle handler", async () => {
vi.useFakeTimers();
try {
const runtime = new WebPeerRuntime({
const runtime = createRuntime({
deviceValues: new MemoryValueStore(),
store: createMemoryStore(),
});
vi.spyOn(runtime.services.setting, "loadSettings").mockResolvedValue(undefined);
@@ -77,9 +100,7 @@ describe("WebPeer runtime composition", () => {
P2P_Enabled: true,
});
vi.spyOn(runtime.services.database, "openDatabase").mockResolvedValue(true);
const onResumed = vi
.spyOn(runtime.services.appLifecycle, "onResumed")
.mockResolvedValue(true);
const onResumed = vi.spyOn(runtime.services.appLifecycle, "onResumed").mockResolvedValue(true);
const open = vi.spyOn(runtime.p2p.transportLifecycle, "connect").mockResolvedValue(undefined);
await runtime.start();
@@ -93,17 +114,20 @@ describe("WebPeer runtime composition", () => {
});
it("rejects start when the browser-local database cannot be opened", async () => {
const runtime = new WebPeerRuntime({
const runtime = createRuntime({
deviceValues: new MemoryValueStore(),
store: createMemoryStore(),
});
vi.spyOn(runtime.services.setting, "loadSettings").mockResolvedValue(undefined);
vi.spyOn(runtime.services.setting, "currentSettings").mockReturnValue({ ...DEFAULT_SETTINGS });
vi.spyOn(runtime.services.database, "openDatabase").mockResolvedValue(false);
await expect(runtime.start()).rejects.toThrow("WebPeer local database could not be opened");
});
it("isolates a specialised runtime and applies its device name before opening the database", async () => {
const runtime = new WebPeerRuntime({
const runtime = createRuntime({
deviceValues: new MemoryValueStore(),
store: createMemoryStore(),
deviceName: " p2p-check-browser-desktop-abc ",
systemVaultName: "p2p-check-vault",
@@ -114,15 +138,39 @@ describe("WebPeer runtime composition", () => {
P2P_AutoStart: false,
P2P_Enabled: false,
});
const setSmallConfig = vi.spyOn(runtime.services.config, "setSmallConfig").mockImplementation(() => undefined);
let finishWrite!: () => void;
const setValue = vi.spyOn(runtime.services.config, "setValue").mockImplementation(
() =>
new Promise<void>((resolve) => {
finishWrite = resolve;
})
);
const openDatabase = vi.spyOn(runtime.services.database, "openDatabase").mockImplementation(async () => {
expect(setSmallConfig).toHaveBeenCalledWith(SETTING_KEY_P2P_DEVICE_NAME, "p2p-check-browser-desktop-abc");
expect(setValue).toHaveBeenCalledWith(SETTING_KEY_P2P_DEVICE_NAME, "p2p-check-browser-desktop-abc");
return true;
});
await runtime.start();
const starting = runtime.start();
await vi.waitFor(() => expect(setValue).toHaveBeenCalledOnce());
expect(openDatabase).not.toHaveBeenCalled();
finishWrite();
await starting;
expect(runtime.services.API.getSystemVaultName()).toBe("p2p-check-vault");
expect(openDatabase).toHaveBeenCalledOnce();
});
it("reads the current device name after the pane updates the store", async () => {
const runtime = createRuntime({
deviceValues: new MemoryValueStore(),
store: createMemoryStore(),
systemVaultName: "webpeer-vault",
});
vi.spyOn(runtime.services.setting, "currentSettings").mockReturnValue({ ...DEFAULT_SETTINGS });
expect(await runtime.getDeviceName()).toBe("webpeer-vault");
await runtime.services.config.setValue(SETTING_KEY_P2P_DEVICE_NAME, "first-name");
expect(await runtime.getDeviceName()).toBe("first-name");
await runtime.services.config.setValue(SETTING_KEY_P2P_DEVICE_NAME, "renamed-device");
expect(await runtime.getDeviceName()).toBe("renamed-device");
});
});
test/contracts/serviceContext.ts
+2 -3
View File
@@ -17,13 +17,12 @@ export const SERVICE_CONTEXT_MEMBERS = [
"vault",
"test",
"UI",
"config",
"keyValueDB",
"control",
] as const satisfies readonly Exclude<keyof ServiceHub, "context">[];
] as const satisfies readonly Exclude<keyof ServiceHub, "context" | "config">[];
export type ServiceContextMember = (typeof SERVICE_CONTEXT_MEMBERS)[number];
type MissingServiceContextMember = Exclude<Exclude<keyof ServiceHub, "context">, ServiceContextMember>;
type MissingServiceContextMember = Exclude<Exclude<keyof ServiceHub, "context" | "config">, ServiceContextMember>;
const serviceContextMembersAreExhaustive: [MissingServiceContextMember] extends [never] ? true : never = true;
void serviceContextMembersAreExhaustive;
test/e2e-obsidian/README.md
+6 -2
View File
@@ -123,7 +123,9 @@ The underlying `test:e2e:obsidian:<scenario>` scripts remain available for an im
`configuration-encryption` needs no remote service. It applies a custom configuration key through the settings interface, restarts the same isolated Vault, and applies a change to only the custom string. It then copies the encrypted settings into a fresh profile without that key and checks masked input, wrong-input retry, cancellation without a settings write or database initialisation, re-enabling for another attempt, access to the settings tab, and restoration after a further restart. Connection, E2EE, profile, and independent ID values must remain unchanged. These checks establish ordinary Apply and restart behaviour, without simulating process termination between the device-key and settings writes. It also checks changes from custom and ASK modes to Default through the settings interface, and recovery from a wrong saved device key, including cancellation, the validated replacement key, and another restart.
`test:e2e:obsidian:dialog-mounts` starts a temporary real Obsidian session and exercises remote selection and CouchDB settings through `SetupManager`, plus Setup URI entry through the registered command. It verifies the compatibility pause and remote-size review, the distinction between a central data-storage server and P2P signalling, the explicit tested and untested CouchDB save actions, the internal-API warning, the Setup URI controls, automatic adjustment when differences are limited to compatible chunk settings, and both manual configuration-mismatch routes. The same session opens the live log and generated full report, reaches the `Hatch` recovery controls, writes and removes its own persistent log, and runs the missing-chunk recreation and file-verification actions against the empty disposable Vault. It captures representative desktop and mobile dialogues, checks the mobile layout and vertically stacked actions, closes each route through its normal controls, and verifies that each mounted operation settles without an error. It does not apply a remote configuration, contact a remote service, or claim to repair a deliberately damaged database.
`test:e2e:obsidian:dialog-mounts` starts a temporary real Obsidian session and exercises remote selection and CouchDB settings through `SetupManager`, plus Setup URI entry through the registered command. It verifies the compatibility pause and remote-size review, the distinction between a central data-storage server and P2P signalling, the explicit tested and untested CouchDB save actions, the internal-API warning, the Setup URI controls, automatic adjustment when differences are limited to compatible chunk settings, and both manual configuration-mismatch routes. The P2P setup check delays the saved device-name read, requires the name field and both continuation controls to remain disabled, then verifies the saved name and selected connection settings reach the setup workflow. The same session opens the live log and generated full report, reaches the `Hatch` recovery controls, writes and removes its own persistent log, and runs the missing-chunk recreation and file-verification actions against the empty disposable Vault. It captures representative desktop and mobile dialogues, checks the mobile layout and vertically stacked actions, closes each route through its normal controls, and verifies that each mounted operation settles without an error. It does not apply a remote configuration, contact a remote service, or claim to repair a deliberately damaged database.
`test:e2e:obsidian:joined-devices` uses a temporary Vault and isolated CouchDB database to open Joined devices from Remote Configuration, a version Notice, and its registered command. `test:e2e:obsidian:joined-devices-object-storage` runs the same assertions against a unique Object Storage prefix, including the conditional removal supported by the test S3 service. It checks that newer and older Self-hosted LiveSync versions share one Notice with one recommendation and review action, that the current device appears first with current metadata, and that locked-out and other-device records are shown. It confirms that a manual refresh reads an updated remote record and that cancelling the inline confirmation leaves the remote unchanged before forgetting one device. It verifies the raw milestone retains the other device, remote state, and preferred-ID tweaks, then checks mobile scrolling, safe-area bounds, and touch targets. It captures the version Notice, desktop list, forget confirmation, completed removal, and mobile dialogue; unrelated Notices are dismissed before the list captures.
`test:e2e:obsidian:settings-ui` starts with a pending compatibility review and verifies the dedicated pause summary, its detailed explanation, and the explicit resume action in a temporary real Obsidian session. It captures the desktop summary and the iPhone-sized summary and detail dialogues; the mobile checks cover viewport containment, horizontal overflow, safe-area containment, and the close control's touch target. It confirms that the acknowledged internal version advances only after the review is accepted, and checks that the Change Log contains no acknowledgement control. It then enables Advanced mode and persists one numeric Advanced setting. On Obsidian before 1.13, this covers the imperative settings fallback used by the `SettingSpec` proof. On Obsidian 1.13 or later, it verifies the task-oriented landing order, the separate Synchronisation group containing Remote Configuration and Sync Settings, and the General Settings group containing Appearance, Logging, and Extra menus. It reopens and cancels onboarding from Quick Setup, opens all 14 nested settings pages, searches globally for the Advanced control, captures the landing page, search result, Advanced page, and mobile landing page, and verifies that the value remains after the settings dialogue is closed and reopened. The mobile landing check confirms that Remote Configuration is reachable without initial scrolling. Finally, it selects the Synchronisation Settings page and verifies that the deletion panel still exposes the effective 'Keep empty folder' setting without presenting the legacy `trashInsteadDelete` control, whose value no longer changes Obsidian deletion behaviour.
@@ -246,7 +248,9 @@ This proves in real Obsidian the plug-in behaviour shared by supported platforms
`test:e2e:obsidian:document-history-restore` creates a normal note, records a logical deletion while retaining readable chunks, and restores the deleted content through the visible Document History dialogue. It requires the action itself to create and reflect a new non-deleted successor revision, reopens the history at that successor, and captures the file picker, readable deleted revision, restored Vault file, and new successor revision. This scenario owns the ordinary-history restoration boundary; conflict resolution remains with **Inspect conflicts and file/database differences**.
`test:e2e:obsidian:hidden-file-snippet-sync` runs a two-vault hidden file round-trip. It verifies creation and deletion of a real `.obsidian/snippets/*.css` file, automatic JSON conflict merging for a hidden file with the merged result propagated by a second synchronisation, manual JSON Resolve dialogue application through Obsidian's UI, and per-device target patterns where one vault ignores a hidden file that the other vault synchronises. Initial enablement must open one user-visible progress Notice before the enabled setting is saved, then retain that Notice while its nested rebuild and scan phases continue in the ordinary log. The configured fixture starts with a current CouchDB remote profile, so migration from legacy remote settings remains the responsibility of the upgrade scenarios and cannot add unrelated Notices to this check. It also covers [issue #555](https://github.com/vrtmrz/obsidian-livesync/issues/555) by requiring several plug-in and settings changes to share one separate action Notice whose controls remain usable in mobile layouts; a manually dismissed group must not repeat its acknowledged rows when a later change arrives.
`test:e2e:obsidian:hidden-file-snippet-sync` runs a two-vault hidden file round-trip. It verifies creation and deletion of a real `.obsidian/snippets/*.css` file, automatic JSON conflict merging for a hidden file with the merged result propagated by a second synchronisation, manual JSON Resolve dialogue application through Obsidian's UI, and per-device target patterns where one vault ignores a hidden file that the other vault synchronises. That target-pattern case keeps a normal-file allow-list for Markdown and PDF files enabled on both devices, then requires exact hidden JSON content after the receiving device includes it. Initial enablement must open one user-visible progress Notice before the enabled setting is saved, then retain that Notice while its nested rebuild and scan phases continue in the ordinary log. The configured fixture starts with a current CouchDB remote profile, so migration from legacy remote settings remains the responsibility of the upgrade scenarios and cannot add unrelated Notices to this check. It also covers [issue #555](https://github.com/vrtmrz/obsidian-livesync/issues/555) by requiring several plug-in and settings changes to share one separate action Notice whose controls remain usable in mobile layouts; a manually dismissed group must not repeat its acknowledged rows when a later change arrives.
`test:e2e:obsidian:hidden-file-selectors` stores and restores selected JS, CSS, and JSON files with a normal-file allow-list set, and compares both directions with an empty allow-list. It uses one temporary Vault and a local database, and also runs in the complete local suite. Run it through the focused wrapper to build the current bundle first; no remote service is required. Set `E2E_HIDDEN_FILE_SELECTORS_OUTPUT` to choose the JSON report path.
`test:e2e:obsidian:customisation-sync` runs a two-vault Customisation Sync workflow. It scans a real snippet CSS file, config JSON file, and sample plug-in fixture into per-file Customisation Sync data, synchronises the entries through CouchDB, applies them on the second vault, verifies the resulting `.obsidian` files, propagates a snippet update, and verifies deletion of the source-vault snippet sync data without confusing it with the target vault's own applied copy.
test/e2e-obsidian/runner/liveSyncWorkflow.ts
+2 -2
View File
@@ -111,7 +111,7 @@ export async function readE2eCompatibilityMarker(
return await evalObsidianJson<CompatibilityMarkerState>(
cliBinary,
[
"(()=>{",
"(async()=>{",
"const core=app.plugins.plugins['obsidian-livesync'].core;",
"const setting=core.services.setting;",
"const settings=setting.currentSettings();",
@@ -122,7 +122,7 @@ export async function readE2eCompatibilityMarker(
"return JSON.stringify({",
"vaultName,additionalSuffix,expectedStorageKey,",
"rawStorageValue:localStorage.getItem(expectedStorageKey),",
"serviceValue:setting.getSmallConfig(markerKey),",
"serviceValue:await setting.config.getValue(markerKey,''),",
"versionUpFlash:settings.versionUpFlash,",
"});",
"})()",
test/e2e-obsidian/runner/upgradeWorkflow.ts
+2 -2
View File
@@ -288,7 +288,7 @@ export async function readRuntimeUpgradeState(
"localDatabaseDocumentCount:databaseInfo.doc_count,",
"nodeId:nodeInfo?.nodeid??replicator?.nodeid??'',",
"legacyCompatibilityMarker:localStorage.getItem(`obsidian-live-sync-ver${vaultName}`),",
"compatibilityMarker:setting.getSmallConfig('database-compatibility-version')??'',",
"compatibilityMarker:await (setting.config?.getValue?.('database-compatibility-version','')??Promise.resolve(setting.getSmallConfig?.('database-compatibility-version')??'')),",
"compatibilityStorageEntries:Object.fromEntries(Array.from({length:localStorage.length},(_,index)=>localStorage.key(index))",
".filter((key)=>key!==null)",
".filter(key=>key.startsWith('obsidian-live-sync-ver')||key.endsWith('-database-compatibility-version'))",
@@ -335,7 +335,7 @@ export async function readRuntimeSettingsUpgradeState(
"const migrationState=setting.getSettingsMigrationState?.();",
"return JSON.stringify({",
"pluginVersion:app.plugins.manifests['obsidian-livesync']?.version??'unknown',",
"compatibilityMarker:setting.getSmallConfig?.('database-compatibility-version')??'',",
"compatibilityMarker:await (setting.config?.getValue?.('database-compatibility-version','')??Promise.resolve(setting.getSmallConfig?.('database-compatibility-version')??'')),",
"migrationState,",
"settings:{",
"isConfigured:settings.isConfigured,settingVersion:settings.settingVersion,",
test/e2e-obsidian/scripts/couchdb-manual-setup-workflow.ts
+71 -1
View File
@@ -47,6 +47,16 @@ const uiTimeoutMs = Number(process.env.E2E_OBSIDIAN_SETUP_URI_TIMEOUT_MS ?? 3000
const e2eePassphrase = `%${randomBytes(24).toString("base64url")}`;
const notePath = "E2E/manual-couchdb/from-first-device.md";
const noteContent = "# Manual CouchDB setup\n\nThis note was sent by the manually configured first device.\n";
const binaryPath = "E2E/manual-couchdb/content-boundaries.png";
const binaryBlockSize = 32 * 1024;
function binaryFixture(): Uint8Array {
const bytes = new Uint8Array(binaryBlockSize * 8);
for (let index = 0; index < 8; index++) {
bytes[index * binaryBlockSize] = index + 2;
bytes[(index + 1) * binaryBlockSize - 1] = 1;
}
return bytes;
}
const returnNotePath = "E2E/manual-couchdb/from-second-device.md";
const returnNoteContent =
"# Manual CouchDB return journey\n\nThis note returned through a Setup URI generated by the first device.\n";
@@ -395,6 +405,32 @@ async function waitForVaultFile(
throw new Error(`Timed out waiting for ${path}. Last content:\n${lastContent}`);
}
async function waitForBinaryFile(vault: TemporaryVault): Promise<void> {
const expected = Buffer.from(binaryFixture());
const deadline = Date.now() + Number(process.env.E2E_OBSIDIAN_FILE_TIMEOUT_MS ?? 30000);
while (Date.now() < deadline) {
try {
if ((await readFile(join(vault.path, binaryPath))).equals(expected)) return;
} catch (error) {
if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error;
}
await new Promise((resolve) => setTimeout(resolve, 250));
}
throw new Error("The fetched binary did not match its exact original bytes.");
}
async function readChunkInventory(context: RunnerContext, session: ObsidianLiveSyncSession): Promise<string[]> {
return await evalObsidianJson<string[]>(
context.cliBinary,
`(async()=>{
const core=app.plugins.plugins['obsidian-livesync'].core;
const rows=(await core.localDatabase.allDocsRaw({include_docs:true})).rows;
return JSON.stringify(rows.filter(row=>row.doc?.type==='leaf').map(row=>row.id).sort());
})()`,
session.cliEnv
);
}
async function waitForRemoteEntry(context: RunnerContext, entry: { id: string; children: string[] }): Promise<void> {
await waitForCouchDbDocs(context.couchDb, context.dbName, (docs) => {
const ids = new Set(docs.map((doc) => doc._id));
@@ -681,6 +717,8 @@ async function main(): Promise<void> {
process.env.E2E_OBSIDIAN_INDEPENDENT_IDS === "true" ? randomBytes(32).toString("base64url") : undefined;
let firstEntryId: string | undefined;
let returnEntryId: string | undefined;
let binaryRevision: string | undefined;
let chunkInventory: string[] = [];
try {
await assertCouchDbReachable(couchDb);
@@ -712,10 +750,34 @@ async function main(): Promise<void> {
}
await writeNoteViaObsidian(context.cliBinary, session.cliEnv, notePath, noteContent);
const size = await evalObsidianJson<number>(
context.cliBinary,
`JSON.stringify(app.plugins.plugins['obsidian-livesync'].core.settings.customChunkSize)`,
session.cliEnv
);
assertEqual(size, 0, "New manual setup did not retain the common Chunk-size recommendation.");
await evalObsidianJson(
context.cliBinary,
`(async()=>{
const bytes=new Uint8Array(${binaryBlockSize}*8);
for(let index=0;index<8;index++){
bytes[index*${binaryBlockSize}]=index+2;
bytes[(index+1)*${binaryBlockSize}-1]=1;
}
await app.vault.createBinary(${JSON.stringify(binaryPath)},bytes.buffer);
return JSON.stringify(true);
})()`,
session.cliEnv
);
const binaryEntry = await waitForLocalDatabaseEntry(context.cliBinary, session.cliEnv, binaryPath);
assertEqual(binaryEntry.children.length, 8, "V3 size 0 did not use the expected content boundaries.");
binaryRevision = binaryEntry.rev;
const entry = await waitForLocalDatabaseEntry(context.cliBinary, session.cliEnv, notePath);
firstEntryId = entry.id;
await pushLocalChanges(context.cliBinary, session.cliEnv);
await waitForRemoteEntry(context, entry);
await waitForRemoteEntry(context, binaryEntry);
chunkInventory = await readChunkInventory(context, session);
} catch (error) {
await captureFailure(session, "first-device");
throw error;
@@ -729,6 +791,13 @@ async function main(): Promise<void> {
session = await startUnconfiguredSession(context, vaultA);
try {
await assertRestoredE2EEPassphrase(session, context.cliBinary);
const unchanged = await waitForLocalDatabaseEntry(context.cliBinary, session.cliEnv, binaryPath);
assertEqual(unchanged.rev, binaryRevision, "Restart rewrote unchanged binary Metadata.");
assertEqual(
JSON.stringify(await readChunkInventory(context, session)),
JSON.stringify(chunkInventory),
"Restart created Chunks for unchanged Vault files."
);
await scheduleRemoteOverwrite(session.remoteDebuggingPort);
screenshots.push(await confirmRebuild(session.remoteDebuggingPort, e2eeRebuildCaptures));
screenshots.push(
@@ -775,6 +844,7 @@ async function main(): Promise<void> {
await assertPersistedE2EE(vaultB, independentIdSource);
await pushLocalChanges(context.cliBinary, session.cliEnv);
await waitForVaultFile(vaultB, notePath, noteContent);
await waitForBinaryFile(vaultB);
if (independentIdSource) {
assertEqual(
await readDocumentId(context.cliBinary, session.cliEnv, notePath),
@@ -815,7 +885,7 @@ async function main(): Promise<void> {
}
console.log(
`Manual CouchDB onboarding created and tested its database, generated a second-device Setup URI, and completed a bidirectional note round-trip. Screenshots: ${screenshots.join(", ")}`
`Manual CouchDB onboarding retained size 0, used binary content boundaries, preserved unchanged revisions and Chunks across restart, fetched exact binary bytes, and completed a bidirectional note round-trip. Screenshots: ${screenshots.join(", ")}`
);
} finally {
await stopTrackedSessions(context).catch((error: unknown) => {
test/e2e-obsidian/scripts/database-diagnostics.ts
+74 -45
View File
@@ -31,6 +31,18 @@ type DiagnosticPageGlobal = typeof globalThis & {
diagnosticDownload?: { name: string; content: string };
};
async function findVisibleElement(page: Page, selector: string): Promise<Locator> {
const deadline = Date.now() + timeout;
while (Date.now() < deadline) {
for (const candidate of page.context().pages()) {
const element = candidate.locator(selector);
if (await element.isVisible()) return element;
}
await page.waitForTimeout(50);
}
throw new Error(`No Obsidian renderer contains the visible element: ${selector}`);
}
async function openCommand(page: Page, kind: "usage" | "integrity") {
const id = kind === "usage" ? "analyse-database" : "inspect-database-integrity";
const opened = await page.evaluate(
@@ -243,11 +255,14 @@ async function assertDiagnosticFileLookup(
session: ObsidianLiveSyncSession,
mobile = false
): Promise<void> {
const dialoguePage = hatch.page();
const viewport = page.viewportSize();
if (viewport) await dialoguePage.setViewportSize(viewport);
const setting = hatch.locator(".setting-item").filter({
has: page.getByText("Find a file from a report pseudonym", { exact: true }),
has: hatch.page().getByText("Find a file from a report pseudonym", { exact: true }),
});
await setting.getByRole("button", { name: "Find file", exact: true }).click();
const dialogue = page.locator(".diagnostic-file-lookup");
const dialogue = dialoguePage.locator(".diagnostic-file-lookup");
await dialogue.waitFor({ state: "visible", timeout });
const input = dialogue.getByLabel("Report path pseudonym", { exact: true });
await input.fill("file-001");
@@ -259,8 +274,8 @@ async function assertDiagnosticFileLookup(
await candidates.first().waitFor({ timeout });
assert.equal(await candidates.count(), 2, "Distinct Metadata IDs claiming the same path must remain selectable");
if (mobile) {
await assertNoHorizontalOverflow(page, dialogue, { label: "Pseudonym file lookup matches" });
await assertNoHorizontalOverflow(page, dialogue.locator(".lookup-candidates"), {
await assertNoHorizontalOverflow(dialoguePage, dialogue, { label: "Pseudonym file lookup matches" });
await assertNoHorizontalOverflow(dialoguePage, dialogue.locator(".lookup-candidates"), {
label: "Pseudonym file lookup candidate list",
});
assert.ok(
@@ -270,7 +285,7 @@ async function assertDiagnosticFileLookup(
await captureObsidianElement(
session.remoteDebuggingPort,
"database-diagnostics-file-matches-mobile.png",
(page) => page.locator(".modal-container").filter({ has: page.locator(".diagnostic-file-lookup") })
(page) => findVisibleElement(page, ".modal-container:has(.diagnostic-file-lookup)")
);
}
const shadow = candidates.filter({ hasText: shadowId });
@@ -294,28 +309,30 @@ async function assertDiagnosticFileLookup(
for (const privateValue of [fixturePath, shadowId, fixtureContent.trim()]) {
assert.ok(!json.includes(privateValue), "A follow-up report must preserve the original report's pseudonyms");
}
await installDiagnosticExportInterception(page);
await installDiagnosticExportInterception(dialoguePage);
await dialogue.getByRole("button", { name: "Copy report", exact: true }).click();
assert.equal(await page.evaluate(() => (globalThis as DiagnosticPageGlobal).diagnosticClipboard), json);
assert.equal(await dialoguePage.evaluate(() => (globalThis as DiagnosticPageGlobal).diagnosticClipboard), json);
await dialogue.getByRole("button", { name: "Save report", exact: true }).click();
await page.waitForFunction(() => !!(globalThis as DiagnosticPageGlobal).diagnosticDownload, undefined, { timeout });
assert.deepEqual(await page.evaluate(() => (globalThis as DiagnosticPageGlobal).diagnosticDownload), {
await dialoguePage.waitForFunction(() => !!(globalThis as DiagnosticPageGlobal).diagnosticDownload, undefined, {
timeout,
});
assert.deepEqual(await dialoguePage.evaluate(() => (globalThis as DiagnosticPageGlobal).diagnosticDownload), {
name: "diagnostic-file-report.json",
content: json,
});
await assertDesktopDiagnosticResultLayout(page, dialogue, "saving a follow-up report");
await assertDesktopDiagnosticResultLayout(dialoguePage, dialogue, "saving a follow-up report");
if (mobile) {
const modal = page.locator(".modal-container").filter({ has: dialogue });
await assertMobileDialogueLayout(page, modal, "Pseudonym file follow-up report");
await assertNoHorizontalOverflow(page, dialogue, { label: "Pseudonym file follow-up content" });
await assertNoHorizontalOverflow(page, dialogue.locator(".report-token"), {
const modal = dialoguePage.locator(".modal-container").filter({ has: dialogue });
await assertMobileDialogueLayout(dialoguePage, modal, "Pseudonym file follow-up report");
await assertNoHorizontalOverflow(dialoguePage, dialogue, { label: "Pseudonym file follow-up content" });
await assertNoHorizontalOverflow(dialoguePage, dialogue.locator(".report-token"), {
label: "Pseudonym file follow-up token",
});
}
await captureObsidianElement(
session.remoteDebuggingPort,
mobile ? "database-diagnostics-file-follow-up-mobile.png" : "database-diagnostics-file-follow-up.png",
(page) => page.locator(".modal-container").filter({ has: page.locator(".diagnostic-file-lookup") })
(page) => findVisibleElement(page, ".modal-container:has(.diagnostic-file-lookup)")
);
await dialogue.getByRole("radio", { name: "MARKDOWN", exact: true }).check();
const markdown = await dialogue.locator("textarea").inputValue();
@@ -379,10 +396,10 @@ async function assertFollowUpStopsAfterClose(page: Page, hatch: Locator, token:
})()`);
try {
const setting = hatch.locator(".setting-item").filter({
has: page.getByText("Find a file from a report pseudonym", { exact: true }),
has: hatch.page().getByText("Find a file from a report pseudonym", { exact: true }),
});
await setting.getByRole("button", { name: "Find file", exact: true }).click();
const dialogue = page.locator(".diagnostic-file-lookup");
const dialogue = hatch.page().locator(".diagnostic-file-lookup");
await dialogue.getByLabel("Report path pseudonym", { exact: true }).fill(token);
await dialogue.getByRole("button", { name: "Find file", exact: true }).click();
const candidate = dialogue.locator("[data-diagnostic-candidate]").filter({ hasText: shadowId });
@@ -466,7 +483,7 @@ async function main(): Promise<void> {
const activeSession = session;
let usageJSON = "";
let usageFixtureRow: { path: string; id: string; revision: string } | undefined;
await withObsidianPage(session.remoteDebuggingPort, async (page) => {
const desktopScreenshot = await withObsidianPage(session.remoteDebuggingPort, async (page) => {
await page.setViewportSize({ width: 800, height: 600 });
const choice = await openCommand(page, "usage");
await choice.getByRole("radio", { name: "Pseudonyms (recommended)", exact: true }).focus();
@@ -560,13 +577,15 @@ async function main(): Promise<void> {
hatch = await (await openLiveSyncSettings(page, timeout)).openPage("Hatch");
await assertFollowUpStopsAfterClose(page, hatch, usageFixtureRow.path);
await captureObsidianElement(activeSession.remoteDebuggingPort, "database-diagnostics-hatch.png", (page) =>
page.locator(".modal.mod-settings")
findVisibleElement(page, ".modal.mod-settings")
);
const setting = hatch
.locator(".setting-item")
.filter({ has: page.getByText("Inspect database integrity", { exact: true }) });
.filter({ has: hatch.page().getByText("Inspect database integrity", { exact: true }) });
await setting.getByRole("button", { name: "Inspect", exact: true }).click();
const progressDialogue = page.locator(".database-diagnostic");
const progressDialogue = await findVisibleElement(page, ".database-diagnostic");
const progressPage = progressDialogue.page();
await progressPage.setViewportSize({ width: 800, height: 600 });
assert.equal(
await progressDialogue
.getByRole("radio", { name: "Pseudonyms (recommended)", exact: true })
@@ -579,7 +598,7 @@ async function main(): Promise<void> {
.isChecked(),
true
);
await assertDesktopDiagnosticChoiceLayout(page, progressDialogue);
await assertDesktopDiagnosticChoiceLayout(progressPage, progressDialogue);
await installDiagnosticReadGate(cli, activeSession, shadowId, "metadata");
try {
await progressDialogue.getByRole("button", { name: "Begin inspection", exact: true }).click();
@@ -593,7 +612,7 @@ async function main(): Promise<void> {
await captureObsidianElement(
activeSession.remoteDebuggingPort,
"database-diagnostics-progress.png",
(page) => page.locator(".modal-container").filter({ has: page.locator(".database-diagnostic") })
(page) => findVisibleElement(page, ".modal-container:has(.database-diagnostic)")
);
await progressDialogue.getByRole("button", { name: "Cancel inspection", exact: true }).click();
await evalObsidianJson(
@@ -601,7 +620,7 @@ async function main(): Promise<void> {
`(async()=>{globalThis.databaseDiagnosticReadGate?.release();return JSON.stringify(true);})()`,
activeSession.cliEnv
);
await page.locator('[data-diagnostic-phase="result"]').waitFor({ timeout });
await progressPage.locator('[data-diagnostic-phase="result"]').waitFor({ timeout });
await assertFormatChoices(progressDialogue, ["MARKDOWN", "JSON"]);
await chooseFormat(progressDialogue, "json");
assert.equal(JSON.parse(await progressDialogue.locator("textarea").inputValue()).status, "cancelled");
@@ -612,33 +631,41 @@ async function main(): Promise<void> {
}
await setting.getByRole("button", { name: "Inspect", exact: true }).click();
const integrity = page.locator(".database-diagnostic");
const integrity = await findVisibleElement(page, ".database-diagnostic");
const integrityPage = integrity.page();
assert.equal(
await integrity.getByRole("radio", { name: "Pseudonyms (recommended)", exact: true }).isChecked(),
true
);
await integrity.getByRole("button", { name: "Begin inspection", exact: true }).click();
await page.locator('[data-diagnostic-phase="result"]').waitFor({ timeout });
await integrityPage.locator('[data-diagnostic-phase="result"]').waitFor({ timeout });
await assertFormatChoices(integrity, ["MARKDOWN", "JSON"]);
assert.equal(await integrity.getByRole("radio", { name: "MARKDOWN", exact: true }).isChecked(), true);
const markdownReport = await integrity.locator("textarea").inputValue();
const markdownDetailsWithCodeBlock =
/<details>\s*<summary>Database integrity report \(partial\)<\/summary>\s*(`{3,})text\r?\n[\s\S]*?\1\s*<\/details>/;
assert.match(markdownReport, markdownDetailsWithCodeBlock);
await installDiagnosticExportInterception(page);
await installDiagnosticExportInterception(integrityPage);
await integrity.getByRole("button", { name: "Copy report", exact: true }).click();
assert.equal(
await page.evaluate(() => (globalThis as DiagnosticPageGlobal).diagnosticClipboard),
await integrityPage.evaluate(() => (globalThis as DiagnosticPageGlobal).diagnosticClipboard),
markdownReport
);
await integrity.getByRole("button", { name: "Save report", exact: true }).click();
await page.waitForFunction(() => !!(globalThis as DiagnosticPageGlobal).diagnosticDownload, undefined, {
timeout,
});
assert.deepEqual(await page.evaluate(() => (globalThis as DiagnosticPageGlobal).diagnosticDownload), {
name: "database-integrity-partial.md",
content: markdownReport,
});
await integrityPage.waitForFunction(
() => !!(globalThis as DiagnosticPageGlobal).diagnosticDownload,
undefined,
{
timeout,
}
);
assert.deepEqual(
await integrityPage.evaluate(() => (globalThis as DiagnosticPageGlobal).diagnosticDownload),
{
name: "database-integrity-partial.md",
content: markdownReport,
}
);
await chooseFormat(integrity, "json");
const reportText = await integrity.locator("textarea").inputValue();
const integrityReport = JSON.parse(reportText);
@@ -670,15 +697,17 @@ async function main(): Promise<void> {
);
for (const privateValue of [fixturePath, shadowId, fixtureContent.trim()])
assert.ok(!reportText.includes(privateValue));
const screenshot = await captureObsidianElement(
activeSession.remoteDebuggingPort,
"database-diagnostics-desktop.png",
(page) => findVisibleElement(page, ".modal-container:has(.database-diagnostic)")
);
await integrity.getByRole("button", { name: "Close", exact: true }).click();
await integrity.waitFor({ state: "hidden", timeout });
await settingsNavigator.close();
return screenshot;
});
const desktopScreenshot = await captureObsidianElement(
session.remoteDebuggingPort,
"database-diagnostics-desktop.png",
(page) => page.locator(".modal-container").filter({ has: page.locator(".database-diagnostic") })
);
await withObsidianPage(session.remoteDebuggingPort, async (page) => {
await page.locator(".database-diagnostic").getByRole("button", { name: "Close", exact: true }).click();
await page.keyboard.press("Escape");
const localOnly = await openCommand(page, "integrity");
await localOnly.getByRole("checkbox", { name: "Include remote database", exact: true }).uncheck();
await localOnly.getByRole("button", { name: "Begin inspection", exact: true }).click();
@@ -780,7 +809,7 @@ async function main(): Promise<void> {
const mobileScreenshot = await captureObsidianElement(
session.remoteDebuggingPort,
"database-diagnostics-mobile.png",
(page) => page.locator(".modal-container").filter({ has: page.locator(".database-diagnostic") })
(page) => findVisibleElement(page, ".modal-container:has(.database-diagnostic)")
);
await withObsidianPage(session.remoteDebuggingPort, async (page) => {
await page.setViewportSize({ width: 375, height: 667 });
@@ -828,7 +857,7 @@ async function main(): Promise<void> {
const mobileResultScreenshot = await captureObsidianElement(
session.remoteDebuggingPort,
"database-diagnostics-mobile-result.png",
(page) => page.locator(".modal-container").filter({ has: page.locator(".database-diagnostic") })
(page) => findVisibleElement(page, ".modal-container:has(.database-diagnostic)")
);
await withObsidianPage(session.remoteDebuggingPort, async (page) => {
await page.setViewportSize({ width: 375, height: 667 });
@@ -857,7 +886,7 @@ async function main(): Promise<void> {
await captureObsidianElement(
activeSession.remoteDebuggingPort,
"database-diagnostics-mobile-saved.png",
(page) => page.locator(".modal-container").filter({ has: page.locator(".database-diagnostic") })
(page) => findVisibleElement(page, ".modal-container:has(.database-diagnostic)")
);
await dialogue.getByRole("button", { name: "Close", exact: true }).click();
await dialogue.waitFor({ state: "hidden", timeout });
test/e2e-obsidian/scripts/dialog-mounts.ts
+181 -74
View File
@@ -1,7 +1,9 @@
import { readFile } from "node:fs/promises";
import { join } from "node:path";
import type { Page } from "playwright";
import { decodeSettingsFromSetupURI } from "@vrtmrz/livesync-commonlib/setup-uri";
import { decryptString } from "@vrtmrz/livesync-commonlib/compat/encryption/stringEncryption";
import { SETTING_KEY_P2P_DEVICE_NAME } from "@vrtmrz/livesync-commonlib/compat/common/types";
import { $msg } from "../../../src/common/translation.ts";
import { discoverObsidianCli, requireObsidianBinary } from "../runner/environment.ts";
import { createE2eCouchDbPluginData, waitForLiveSyncCoreReady } from "../runner/liveSyncWorkflow.ts";
@@ -13,18 +15,37 @@ import {
captureObsidianPage,
obsidianRemoteDebuggingPort,
openLiveSyncSettings,
withObsidianPage,
waitForVisibleObsidianDialogue,
withObsidianPage as withObsidianSessionPage,
} from "../runner/ui.ts";
import { createTemporaryVault } from "../runner/vault.ts";
const dialogRunStateKey = "__livesyncE2EDialogMount";
const savedP2PDeviceName = "e2e-p2p-saved-device";
const repairRunStateKey = "__livesyncE2ETroubleshootingRepair";
const uiTimeoutMs = Number(process.env.E2E_OBSIDIAN_DIALOG_TIMEOUT_MS ?? 10000);
async function withObsidianPage<T>(port: number, operation: (page: Page) => Promise<T>): Promise<T> {
return await withObsidianSessionPage(port, async (page) => {
for (const candidate of page.context().pages()) {
const hasPlugin = await candidate
.evaluate(
() => (globalThis as ObsidianTestGlobal).app?.plugins?.plugins["obsidian-livesync"] !== undefined
)
.catch(() => false);
if (hasPlugin) return await operation(candidate);
}
throw new Error("The LiveSync Vault renderer is unavailable.");
});
}
type DialogueMode = "desktop" | "mobile";
type DialogueRunState = {
done: boolean;
deviceNameRead?: boolean;
deviceNameReadPending?: boolean;
releaseDeviceNameRead?: () => void;
error?: string;
expected?: unknown;
kind: string;
@@ -46,6 +67,14 @@ type LiveSyncTestPlugin = {
createAllChunks(force: boolean): Promise<unknown>;
};
modules: SetupManagerHandle[];
services: {
setting: {
config: {
getValue(key: string, defaultValue?: string): Promise<string>;
setValue(key: string, value: string): Promise<void>;
};
};
};
settings: Record<string, unknown>;
};
};
@@ -91,38 +120,81 @@ async function openRemoteSelectionDialogue(): Promise<void> {
async function openP2PRemoteSelectionDialogueForInspection(): Promise<void> {
await withObsidianPage(obsidianRemoteDebuggingPort(), async (page) => {
await page.evaluate((stateKey) => {
const plugin = (globalThis as ObsidianTestGlobal).app?.plugins?.plugins["obsidian-livesync"];
if (plugin === undefined) throw new Error("Self-hosted LiveSync is not loaded");
const manager = plugin.core.modules.find((module) => module.constructor.name === "SetupManager");
if (
typeof manager?.onSelectServer !== "function" ||
typeof manager.onConfirmApplySettingsFromWizard !== "function"
) {
throw new Error("Could not find the P2P setup workflow");
}
const originalConfirm = manager.onConfirmApplySettingsFromWizard;
const state: DialogueRunState = { kind: "p2p-compatibility-settings", done: false };
(globalThis as unknown as Record<string, DialogueRunState>)[stateKey] = state;
manager.onConfirmApplySettingsFromWizard = async (settings: unknown) => {
state.expected = settings;
return true;
};
void manager
.onSelectServer(plugin.core.settings, "unknown")
.then(
(result) => {
state.result = result;
state.done = true;
},
(error: unknown) => {
state.error = error instanceof Error ? error.message : String(error);
state.done = true;
}
)
.finally(() => {
manager.onConfirmApplySettingsFromWizard = originalConfirm;
await page.evaluate(
async ({ stateKey, deviceNameKey, deviceName }) => {
const plugin = (globalThis as ObsidianTestGlobal).app?.plugins?.plugins["obsidian-livesync"];
if (plugin === undefined) throw new Error("Self-hosted LiveSync is not loaded");
const initialPeerName = plugin.core.settings.P2P_DevicePeerName;
if (typeof initialPeerName === "string" && initialPeerName.trim() !== "") {
throw new Error("The P2P setup fixture unexpectedly has an initial device name.");
}
const manager = plugin.core.modules.find((module) => module.constructor.name === "SetupManager");
if (
typeof manager?.onSelectServer !== "function" ||
typeof manager.onConfirmApplySettingsFromWizard !== "function"
) {
throw new Error("Could not find the P2P setup workflow");
}
const config = plugin.core.services.setting.config;
await config.setValue(deviceNameKey, deviceName);
const originalGetValue = config.getValue;
const originalConfirm = manager.onConfirmApplySettingsFromWizard;
const state: DialogueRunState = {
kind: "p2p-compatibility-settings",
done: false,
deviceNameRead: false,
deviceNameReadPending: false,
};
(globalThis as unknown as Record<string, DialogueRunState>)[stateKey] = state;
const deviceNameReadGate = new Promise<void>((resolve) => {
state.releaseDeviceNameRead = resolve;
});
config.getValue = async (key, defaultValue) => {
const value = await originalGetValue.call(config, key, defaultValue);
if (key === deviceNameKey) {
state.deviceNameReadPending = true;
await deviceNameReadGate;
state.deviceNameReadPending = false;
if (value === deviceName) state.deviceNameRead = true;
}
return value;
};
manager.onConfirmApplySettingsFromWizard = async (settings: unknown) => {
state.expected = settings;
return true;
};
void manager
.onSelectServer(plugin.core.settings, "unknown")
.then(
(result) => {
state.result = result;
},
(error: unknown) => {
state.error = error instanceof Error ? error.message : String(error);
}
)
.finally(() => {
state.releaseDeviceNameRead?.();
state.releaseDeviceNameRead = undefined;
manager.onConfirmApplySettingsFromWizard = originalConfirm;
config.getValue = originalGetValue;
state.done = true;
});
},
{
stateKey: dialogRunStateKey,
deviceNameKey: SETTING_KEY_P2P_DEVICE_NAME,
deviceName: savedP2PDeviceName,
}
);
});
}
async function releasePendingP2PDeviceNameRead(): Promise<void> {
await withObsidianPage(obsidianRemoteDebuggingPort(), async (page) => {
await page.evaluate((stateKey) => {
const state = (globalThis as unknown as Record<string, DialogueRunState | undefined>)[stateKey];
state?.releaseDeviceNameRead?.();
}, dialogRunStateKey);
});
}
@@ -388,7 +460,7 @@ async function verifyRemoteSelectionDialogue(mode: DialogueMode): Promise<string
if (previousDialogue === null) {
throw new Error("The remote selection dialogue did not expose a close control.");
}
await modal.locator(".modal-close-button").click({ timeout: uiTimeoutMs });
await modal.locator(".modal-close-button, .modal-header-button").last().click({ timeout: uiTimeoutMs });
await page.waitForFunction((element) => !element.isConnected, previousDialogue, { timeout: uiTimeoutMs });
await modal.waitFor({ state: "visible", timeout: uiTimeoutMs });
}
@@ -473,9 +545,11 @@ async function verifyCouchDBSettingsDialogue(mode: DialogueMode): Promise<string
async function verifyP2PCompatibilitySettingsDialogue(): Promise<string> {
await openP2PRemoteSelectionDialogueForInspection();
await withObsidianPage(obsidianRemoteDebuggingPort(), async (page) => {
const remoteSelection = page.locator(".modal-container").filter({
has: page.locator(".modal-title").filter({ hasText: "Choose a synchronisation remote" }),
});
const remoteSelection = await waitForVisibleObsidianDialogue(
page,
"Choose a synchronisation remote",
uiTimeoutMs
);
await remoteSelection
.locator("label")
.filter({ hasText: "Peer-to-Peer (P2P)" })
@@ -491,16 +565,51 @@ async function verifyP2PCompatibilitySettingsDialogue(): Promise<string> {
obsidianRemoteDebuggingPort(),
"setup-p2p-compatibility-dialogue.png",
async (page) => {
const modal = page.locator(".modal-container").filter({
has: page.locator(".modal-title").filter({ hasText: "P2P Configuration" }),
});
const modal = await waitForVisibleObsidianDialogue(page, "P2P Configuration", uiTimeoutMs);
const modalPage = modal.page();
await modal.waitFor({ state: "visible", timeout: uiTimeoutMs });
const peerName = modal.getByLabel("Device Peer ID", { exact: true });
await peerName.waitFor({ state: "visible", timeout: uiTimeoutMs });
await withObsidianPage(obsidianRemoteDebuggingPort(), async (statePage) => {
await statePage.waitForFunction(
(stateKey) =>
(globalThis as unknown as Record<string, DialogueRunState | undefined>)[stateKey]
?.deviceNameReadPending === true,
dialogRunStateKey,
{ timeout: uiTimeoutMs }
);
});
const testAndContinue = modal.getByRole("button", { name: "Test Settings and Continue", exact: true });
const continueAnyway = modal.getByRole("button", { name: "Continue anyway", exact: true });
await testAndContinue.waitFor({ state: "visible", timeout: uiTimeoutMs });
await continueAnyway.waitFor({ state: "visible", timeout: uiTimeoutMs });
if (
!(await peerName.isDisabled()) ||
!(await testAndContinue.isDisabled()) ||
!(await continueAnyway.isDisabled())
) {
throw new Error(
"The P2P device name and setup actions must remain disabled while the saved device name is loading."
);
}
await releasePendingP2PDeviceNameRead();
await modalPage.waitForFunction(
(expectedName) => {
const input = document.querySelector('input[name="p2p-device-peer-id"]');
return input instanceof HTMLInputElement && !input.disabled && input.value === expectedName;
},
savedP2PDeviceName,
{ timeout: uiTimeoutMs }
);
if ((await peerName.inputValue()) !== savedP2PDeviceName) {
throw new Error("The saved P2P device name was not loaded before submitting the setup dialogue.");
}
await modal.locator('input[name="p2p-room-id"]').fill("e2e-p2p-compatibility");
await modal.locator('input[name="p2p-password"]').fill("e2e-passphrase");
await modal.locator('input[name="p2p-device-peer-id"]').fill("e2e-device");
const compatibility = modal.locator("details").filter({
has: page.locator("summary").filter({ hasText: "Connection compatibility" }),
has: modalPage.locator("summary").filter({ hasText: "Connection compatibility" }),
});
await compatibility.waitFor({ state: "visible", timeout: uiTimeoutMs });
if (!(await compatibility.evaluate((element) => (element as HTMLDetailsElement).open))) {
@@ -541,7 +650,7 @@ async function verifyP2PCompatibilitySettingsDialogue(): Promise<string> {
}
const advanced = modal.locator("details").filter({
has: page.locator("summary").filter({ hasText: "Advanced Settings" }),
has: modalPage.locator("summary").filter({ hasText: "Advanced Settings" }),
});
if (!(await advanced.evaluate((element) => (element as HTMLDetailsElement).open))) {
await advanced.locator("summary").click({ timeout: uiTimeoutMs });
@@ -553,20 +662,19 @@ async function verifyP2PCompatibilitySettingsDialogue(): Promise<string> {
if (relayOnlyElement === null || connectionPathElement === null) {
throw new Error("P2P connection-path controls were removed whilst validating TURN settings.");
}
await page.waitForFunction((element) => !(element as HTMLOptionElement).disabled, relayOnlyElement, {
await modalPage.waitForFunction((element) => !(element as HTMLOptionElement).disabled, relayOnlyElement, {
timeout: uiTimeoutMs,
});
await connectionPath.selectOption("relay");
await turnServers.fill("");
await page.waitForFunction(
await modalPage.waitForFunction(
(element) => (element as HTMLSelectElement).value === "automatic",
connectionPathElement,
{ timeout: uiTimeoutMs }
);
const resetNotice = compatibility.locator(".sls-info-note-notice").filter({
hasText:
"TURN relay only requires TURN configuration. Connection path has been restored to Automatic.",
hasText: "TURN relay only requires TURN configuration. Connection path has been restored to Automatic.",
});
await resetNotice.waitFor({ state: "visible", timeout: uiTimeoutMs });
await resetNotice
@@ -575,7 +683,7 @@ async function verifyP2PCompatibilitySettingsDialogue(): Promise<string> {
.waitFor({ state: "visible", timeout: uiTimeoutMs });
await turnServers.fill("turn:turn.example.com:3478");
await page.waitForFunction((element) => !(element as HTMLOptionElement).disabled, relayOnlyElement, {
await modalPage.waitForFunction((element) => !(element as HTMLOptionElement).disabled, relayOnlyElement, {
timeout: uiTimeoutMs,
});
await connectionPath.selectOption("relay");
@@ -583,21 +691,25 @@ async function verifyP2PCompatibilitySettingsDialogue(): Promise<string> {
await advanced.locator("summary").click({ timeout: uiTimeoutMs });
await compatibility.scrollIntoViewIfNeeded();
}
);
).finally(releasePendingP2PDeviceNameRead);
await withObsidianPage(obsidianRemoteDebuggingPort(), async (page) => {
const modal = page.locator(".modal-container").filter({
has: page.locator(".modal-title").filter({ hasText: "P2P Configuration" }),
});
const modal = await waitForVisibleObsidianDialogue(page, "P2P Configuration", uiTimeoutMs);
await modal.getByRole("button", { name: "Continue anyway", exact: true }).click({ timeout: uiTimeoutMs });
await modal.waitFor({ state: "hidden", timeout: uiTimeoutMs });
});
const state = await assertDialogueRunCompleted();
if (state.deviceNameRead !== true) {
throw new Error("The saved P2P device name was not read through setting.config.");
}
if (state.result !== true || typeof state.expected !== "object" || state.expected === null) {
throw new Error("The P2P compatibility settings were not passed to the setup workflow.");
}
const settings = state.expected as Record<string, unknown>;
if (settings.P2P_DevicePeerName !== savedP2PDeviceName) {
throw new Error("The saved P2P device name was not passed through the setup workflow.");
}
if (settings.P2P_maxWirePayloadBytes !== 800 || settings.P2P_connectionPath !== "relay") {
throw new Error(
`The P2P compatibility choices did not round-trip through the setup dialogue: ${JSON.stringify({
@@ -672,15 +784,14 @@ async function verifyGenerateSetupUriDialogue(mode: DialogueMode): Promise<strin
const app = (globalThis as ObsidianTestGlobal).app;
return Boolean(
app?.plugins?.plugins["obsidian-livesync"]?.core?.settings.isConfigured &&
app.commands?.commands?.[commandId]
app.commands?.commands?.[commandId]
);
},
"obsidian-livesync:livesync-copysetupuri",
{ timeout: uiTimeoutMs }
);
return await page.evaluate(
(commandId) =>
(globalThis as ObsidianTestGlobal).app?.commands?.executeCommandById(commandId) === true,
(commandId) => (globalThis as ObsidianTestGlobal).app?.commands?.executeCommandById(commandId) === true,
"obsidian-livesync:livesync-copysetupuri"
);
});
@@ -714,7 +825,8 @@ async function verifyGenerateSetupUriDialogue(mode: DialogueMode): Promise<strin
timeout: uiTimeoutMs,
});
await choice.getByRole("button", { name: "Time-bound", exact: true }).waitFor({
state: "visible", timeout: uiTimeoutMs,
state: "visible",
timeout: uiTimeoutMs,
});
if (mode === "mobile") await assertMobileDialogueLayout(page, choice, "Setup URI availability dialogue");
await choice.getByRole("button", { name: selected, exact: true }).click({ timeout: uiTimeoutMs });
@@ -866,6 +978,7 @@ async function verifyCompatibleAlignmentSettingDefault(): Promise<void> {
if (!(await toggle.evaluate((element) => element.classList.contains("is-enabled")))) {
throw new Error("The automatic compatible-setting policy was displayed as disabled while still undefined.");
}
await settingsNavigator.close();
});
}
@@ -1094,9 +1207,10 @@ async function verifyHatchSurfacesAndSafeActions(): Promise<string> {
);
await withObsidianPage(obsidianRemoteDebuggingPort(), async (page) => {
const liveSyncSettings = page.locator(".sls-setting");
const settingsNavigator = await openLiveSyncSettings(page, uiTimeoutMs);
const liveSyncSettings = await settingsNavigator.openPage("Hatch");
const logSetting = liveSyncSettings.locator(".setting-item").filter({
has: page.getByText("Write logs into the file", { exact: true }),
has: settingsNavigator.page.getByText("Write logs into the file", { exact: true }),
});
await logSetting.locator(".checkbox-container").click({ timeout: uiTimeoutMs });
await page.waitForFunction(
@@ -1130,10 +1244,10 @@ async function verifyHatchSurfacesAndSafeActions(): Promise<string> {
// Saving a toggle refreshes the settings pane. Resolve the visible control again so the
// second action does not target the detached pre-save element.
const refreshedLogToggle = page
.locator(".sls-setting:visible .setting-item:visible")
const refreshedLogToggle = liveSyncSettings
.locator(".setting-item:visible")
.filter({
has: page.getByText("Write logs into the file", { exact: true }),
has: settingsNavigator.page.getByText("Write logs into the file", { exact: true }),
})
.locator(".checkbox-container:visible")
.last();
@@ -1179,13 +1293,9 @@ async function verifyHatchSurfacesAndSafeActions(): Promise<string> {
}
};
}, repairRunStateKey);
await page
.locator(".sls-setting:visible")
.last()
.getByRole("button", { name: "Recreate current chunks", exact: true })
.click({
timeout: uiTimeoutMs,
});
await liveSyncSettings.getByRole("button", { name: "Recreate current chunks", exact: true }).click({
timeout: uiTimeoutMs,
});
await page.waitForFunction(
(stateKey) =>
(globalThis as unknown as Record<string, DialogueRunState | undefined>)[stateKey]?.done === true,
@@ -1200,13 +1310,9 @@ async function verifyHatchSurfacesAndSafeActions(): Promise<string> {
throw new Error(`Recreate missing chunks failed: ${repairState.error}`);
}
await page
.locator(".sls-setting:visible")
.last()
.getByRole("button", { name: "Begin inspection", exact: true })
.click({
timeout: uiTimeoutMs,
});
await liveSyncSettings.getByRole("button", { name: "Begin inspection", exact: true }).click({
timeout: uiTimeoutMs,
});
await page
.locator(".notice")
.filter({ hasText: /^done$/u })
@@ -1214,6 +1320,7 @@ async function verifyHatchSurfacesAndSafeActions(): Promise<string> {
state: "visible",
timeout: uiTimeoutMs,
});
await settingsNavigator.close();
});
return screenshotPath;
test/e2e-obsidian/scripts/hidden-file-selectors.ts
+202
View File
@@ -0,0 +1,202 @@
import { execFileSync } from "node:child_process";
import { createHash } from "node:crypto";
import { mkdir, readFile, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { dirname, join } from "node:path";
import { isDeepStrictEqual } from "node:util";
import { fileURLToPath } from "node:url";
import { evalObsidianJson } from "../runner/cli.ts";
import { discoverObsidianCli, requireObsidianBinary } from "../runner/environment.ts";
import {
createE2eCouchDbPluginData,
createE2eObsidianDeviceLocalState,
waitForLiveSyncCoreReady,
} from "../runner/liveSyncWorkflow.ts";
import { startObsidianLiveSyncSession, type ObsidianLiveSyncSession } from "../runner/session.ts";
import { createTemporaryVault } from "../runner/vault.ts";
const hiddenRoot = ".obsidian/livesync-selector-example";
const hiddenFiles = {
[`${hiddenRoot}/main.js`]: "console.log('hidden fixture');\n",
[`${hiddenRoot}/styles.css`]: ".hidden-fixture { color: blue; }\n",
[`${hiddenRoot}/settings.json`]: '{"fixture":true}\n',
};
const controlRoot = ".obsidian/livesync-selector-control";
const controlFiles = {
[`${controlRoot}/main.js`]: "console.log('control fixture');\n",
[`${controlRoot}/styles.css`]: ".control-fixture { color: red; }\n",
[`${controlRoot}/settings.json`]: '{"control":true}\n',
};
const allowList = "\\.(?:md|png|pdf)$";
const coreCode = "const core=app.plugins.plugins['obsidian-livesync'].core;";
const hiddenCode = `${coreCode}const addOn=core.getAddOn('HiddenFileSync');`;
type Observation = { name: string; expected: unknown; actual: unknown; passed: boolean };
async function writeFixtures(vaultPath: string, files: Record<string, string>) {
for (const [path, content] of Object.entries(files)) {
await mkdir(dirname(join(vaultPath, path)), { recursive: true });
await writeFile(join(vaultPath, path), content);
}
}
async function removeFixtures(vaultPath: string, files: Record<string, string>) {
for (const path of Object.keys(files)) await rm(join(vaultPath, path), { force: true });
}
async function matchingFiles(vaultPath: string, files: Record<string, string>) {
const matches: string[] = [];
for (const [path, content] of Object.entries(files)) {
try {
if ((await readFile(join(vaultPath, path), "utf8")) === content) matches.push(path);
} catch (error) {
if (!(error instanceof Error) || !("code" in error) || error.code !== "ENOENT") throw error;
}
}
return matches.sort();
}
async function main() {
const binary = requireObsidianBinary();
const cli = discoverObsidianCli();
if (!cli.binary) throw new Error("The Obsidian CLI is unavailable");
const cliBinary = cli.binary;
const output =
process.env.E2E_HIDDEN_FILE_SELECTORS_OUTPUT ?? join(tmpdir(), "livesync-hidden-file-selectors.json");
const observations: Observation[] = [];
const report: Record<string, unknown> = {
revision: execFileSync("git", ["rev-parse", "HEAD"], { encoding: "utf8" }).trim(),
sourceDiffSha256: createHash("sha256")
.update(new Uint8Array(execFileSync("git", ["diff", "--binary", "HEAD"])))
.digest("hex"),
scenarioSha256: createHash("sha256")
.update(new Uint8Array(await readFile(fileURLToPath(import.meta.url))))
.digest("hex"),
commonlibEntrySha256: createHash("sha256")
.update(
new Uint8Array(
await readFile(
"node_modules/@vrtmrz/livesync-commonlib/dist/managers/EntryManager/EntryManagerImpls.js"
)
)
)
.digest("hex"),
bundleSha256: createHash("sha256")
.update(new Uint8Array(await readFile("main.js")))
.digest("hex"),
commonlibVersion: JSON.parse(await readFile("node_modules/@vrtmrz/livesync-commonlib/package.json", "utf8"))
.version,
observations,
};
const vault = await createTemporaryVault("livesync-hidden-file-selectors-");
let session: ObsidianLiveSyncSession | undefined;
const record = (name: string, actual: unknown, expected: unknown) => {
const observation = { name, expected, actual, passed: isDeepStrictEqual(actual, expected) };
observations.push(observation);
console.log(JSON.stringify(observation));
};
const evaluate = async <T>(code: string): Promise<T> => {
if (!session) throw new Error("The isolated Obsidian session has not started");
return await evalObsidianJson<T>(cliBinary, `(async()=>{${code}})()`, session.cliEnv, 60_000);
};
try {
session = await startObsidianLiveSyncSession({
binary,
cliBinary,
vault,
localStorageEntries: createE2eObsidianDeviceLocalState(vault.name),
pluginData: createE2eCouchDbPluginData(
{ uri: "http://127.0.0.1:9", username: "", password: "", dbName: "selector-local-only" },
{ periodicReplication: false, syncOnFileOpen: false, useOnlyLocalChunk: true }
),
});
report.pluginVersion = session.readiness.pluginVersion;
await waitForLiveSyncCoreReady(cliBinary, session.cliEnv);
report.runtime = await evaluate(`${coreCode}
core.services.setting.setDeviceAndVaultName('selector-local');
Object.assign(core.settings,{
isConfigured:true,usePluginSync:false,usePluginSyncV2:true,syncInternalFiles:false,
autoSweepPlugins:false,autoSweepPluginsPeriodic:false,watchInternalFileChanges:false,
syncOnlyRegEx:${JSON.stringify(allowList)},syncIgnoreRegEx:''
});
return JSON.stringify({userAgent:navigator.userAgent,databaseReady:core.localDatabase.isReady});
`);
await writeFixtures(vault.path, hiddenFiles);
const uploadCode = (normalAllowList: string, files: Record<string, string>) => `${hiddenCode}
Object.assign(core.settings,{
syncInternalFiles:true,syncOnlyRegEx:${JSON.stringify(normalAllowList)},
syncInternalFilesTargetPatterns:'^\\\\.obsidian(?:$|/livesync-selector-(?:example|control)(?:/|$))',syncInternalFilesIgnorePatterns:''
});
const candidates=await addOn.scanInternalFileNames();
const selected=candidates.filter(path=>${JSON.stringify(Object.keys(files))}.includes(path));
if(selected.length!==3)throw new Error('Hidden-file fixture selection failed: '+JSON.stringify({selected,target:core.settings.syncInternalFilesTargetPatterns}));
for(const path of selected)await addOn.updateLastProcessedAsActualFile(path);
await addOn.rebuildFromStorage(false,${JSON.stringify(Object.keys(files))},false);
const paths=[];
for await(const doc of core.localDatabase.findEntries('i:','i;')){
if(${JSON.stringify(Object.keys(files).map((path) => `i:${path}`))}.includes(doc.path)&&!doc.deleted&&!doc._deleted)paths.push(doc.path.substring(2));
}
return JSON.stringify(paths.sort());
`;
const hiddenUpload = await evaluate<string[]>(uploadCode(allowList, hiddenFiles));
record(
"#1272: a normal allow-list permits selected hidden uploads",
hiddenUpload,
Object.keys(hiddenFiles).sort()
);
await writeFixtures(vault.path, controlFiles);
const uploadControl = await evaluate<string[]>(uploadCode("", controlFiles));
record(
"#1272 control: clearing the normal allow-list permits hidden uploads",
uploadControl,
Object.keys(controlFiles).sort()
);
const pullCode = (normalAllowList: string, files: Record<string, string>) => `${hiddenCode}
core.settings.syncOnlyRegEx=${JSON.stringify(normalAllowList)};
const results=[];
for await(const doc of core.localDatabase.findEntries('i:','i;')){
if(${JSON.stringify(Object.keys(files).map((path) => `i:${path}`))}.includes(doc.path)&&!doc.deleted&&!doc._deleted){
results.push(await addOn.extractInternalFileFromDatabase(doc.path.substring(2),true,doc,false));
}
}
return JSON.stringify(results);
`;
await removeFixtures(vault.path, hiddenFiles);
report.hiddenPullWithAllowList = await evaluate(pullCode(allowList, hiddenFiles));
record(
"#1272: a normal allow-list permits selected hidden downloads",
await matchingFiles(vault.path, hiddenFiles),
Object.keys(hiddenFiles).sort()
);
await removeFixtures(vault.path, controlFiles);
report.hiddenPullWithoutAllowList = await evaluate(pullCode("", controlFiles));
record(
"#1272 control: clearing the normal allow-list permits hidden downloads",
await matchingFiles(vault.path, controlFiles),
Object.keys(controlFiles).sort()
);
} catch (error) {
report.runtimeError = error instanceof Error ? error.stack : String(error);
if (session) report.processOutput = JSON.stringify(session.app.output()).slice(-4_000);
throw error;
} finally {
try {
await writeFile(output, JSON.stringify(report, null, 2) + "\n");
console.log(`Verification report: ${output}`);
} finally {
try {
if (session) await session.app.stop();
} finally {
await vault.dispose();
}
}
}
if (observations.some((observation) => !observation.passed)) process.exitCode = 1;
}
main().catch((error: unknown) => {
console.error(error instanceof Error ? error.stack : error);
process.exitCode = 1;
});
test/e2e-obsidian/scripts/hidden-file-snippet-sync.ts
+33 -16
View File
@@ -69,6 +69,7 @@ type RunnerContext = {
cliBinary: string;
couchDb: CouchDbConfig;
dbName: string;
activeSessions: Set<ObsidianLiveSyncSession>;
};
async function writeVaultFile(vaultPath: string, path: string, content: string): Promise<void> {
@@ -348,6 +349,7 @@ async function startConfiguredSession(
pluginData: createE2eCouchDbPluginData(couchDbSettings, hiddenFileSettings),
localStorageEntries: createE2eObsidianDeviceLocalState(vault.name),
});
context.activeSessions.add(session);
await waitForLiveSyncCoreReady(context.cliBinary, session.cliEnv);
await assertE2eCompatibilityMarker(context.cliBinary, session.cliEnv);
await configureCouchDb(context.cliBinary, session.cliEnv, couchDbSettings, hiddenFileSettings);
@@ -355,6 +357,11 @@ async function startConfiguredSession(
return session;
}
async function stopSession(context: RunnerContext, session: ObsidianLiveSyncSession): Promise<void> {
await session.app.stop();
context.activeSessions.delete(session);
}
async function uploadHiddenFile(
context: RunnerContext,
session: ObsidianLiveSyncSession,
@@ -430,12 +437,12 @@ async function runCreateRoundTrip(
throw new Error("Remote Rebuild did not declare and encrypt internal Metadata for its accepted writer.");
}
console.log("Remote Rebuild declared the feature and preserved encrypted Hidden File Sync Metadata.");
await session.app.stop();
await stopSession(context, session);
session = await startConfiguredSession(context, vaultB);
await pullAndApplyHiddenFiles(context, session);
const received = await waitForPathContent(vaultB.path, snippetPath, (content) => content === snippetContent);
await session.app.stop();
await stopSession(context, session);
assertEqual(received, snippetContent, "Hidden snippet content did not round-trip to the second vault.");
console.log(`Hidden create round-trip copied ${entry.id} to the second vault.`);
@@ -450,12 +457,12 @@ async function runDeleteRoundTrip(
let session = await startConfiguredSession(context, vaultA);
await scanHiddenStorage(context.cliBinary, session.cliEnv);
await pushLocalChanges(context.cliBinary, session.cliEnv);
await session.app.stop();
await stopSession(context, session);
session = await startConfiguredSession(context, vaultB);
await pullAndApplyHiddenFiles(context, session);
await waitForPathDeleted(vaultB.path, snippetPath);
await session.app.stop();
await stopSession(context, session);
console.log("Hidden delete round-trip removed the snippet from the second vault.");
}
@@ -476,14 +483,14 @@ async function runJsonConflictRoundTrip(
const mergedOnB = await waitForPathContent(vaultB.path, mergeJsonPath, (content) =>
hasJsonValues(content, { fromA: true, fromB: true })
);
await session.app.stop();
await stopSession(context, session);
session = await startConfiguredSession(context, vaultA);
await pullAndApplyHiddenFiles(context, session);
const mergedOnA = await waitForPathContent(vaultA.path, mergeJsonPath, (content) =>
hasJsonValues(content, { fromA: true, fromB: true })
);
await session.app.stop();
await stopSession(context, session);
assertEqual(mergedOnA, mergedOnB, "Merged hidden JSON content was not consistent across both vaults.");
console.log("Hidden JSON conflict was automatically merged and round-tripped.");
@@ -503,7 +510,7 @@ async function runJsonManualConflictResolution(context: RunnerContext, vault: Te
const merged = await waitForPathContent(vault.path, manualMergeJsonPath, (content) =>
hasJsonValues(content, { shared: "right", fromA: true, fromB: true })
);
await session.app.stop();
await stopSession(context, session);
const parsed = JSON.parse(merged);
assertEqual(parsed.shared, "right", "Manual JSON conflict resolution did not apply the selected merged result.");
@@ -517,17 +524,19 @@ async function runTargetMismatch(
vaultA: TemporaryVault,
vaultB: TemporaryVault
): Promise<void> {
const selectors = { syncOnlyRegEx: String.raw`\.(md|pdf)$` };
const targetContent = JSON.stringify({ onlyA: true, targetMismatch: true }, null, 4) + "\n";
await writeVaultFile(vaultA.path, targetPath, targetContent);
let session = await startConfiguredSession(context, vaultA);
let session = await startConfiguredSession(context, vaultA, selectors);
try {
await uploadHiddenFile(context, session, targetPath);
} finally {
await session.app.stop();
await stopSession(context, session);
}
session = await startConfiguredSession(context, vaultB, {
...selectors,
syncInternalFilesTargetPatterns: "snippets",
});
try {
@@ -538,10 +547,11 @@ async function runTargetMismatch(
"Hidden file was applied on a device where it was not a target file."
);
} finally {
await session.app.stop();
await stopSession(context, session);
}
session = await startConfiguredSession(context, vaultB, {
...selectors,
syncInternalFilesTargetPatterns: "",
});
let received = "";
@@ -549,11 +559,17 @@ async function runTargetMismatch(
await pullAndApplyHiddenFiles(context, session, { resolveConflicts: false });
received = await waitForPathContent(vaultB.path, targetPath, (content) => content === targetContent);
} finally {
await session.app.stop();
await stopSession(context, session);
}
assertEqual(received, targetContent, "Hidden file was not applied after it became a target file.");
console.log("Hidden target mismatch respected per-device target patterns, then applied after enabling the target.");
assertEqual(
received,
targetContent,
"The normal file allow-list blocked a selected hidden file from the other device."
);
console.log(
"A normal Markdown/PDF allow-list preserved hidden target exclusion and exact remote restoration after enabling the target."
);
}
async function setHiddenFileNoticeFixtures(port: number, itemIds: string[], includeRestart: boolean): Promise<void> {
@@ -828,7 +844,7 @@ async function runInitialisationNoticeGrouping(context: RunnerContext, vault: Te
`Hidden File Sync showed one progress Notice before settings were saved and retained it throughout initialisation. Screenshot: ${screenshotPath}`
);
} finally {
await session.app.stop();
await stopSession(context, session);
}
}
@@ -889,7 +905,7 @@ async function runConfigurationNoticeGrouping(context: RunnerContext, vault: Tem
} finally {
await clearHiddenFileNoticeFixtures(port).catch(() => undefined);
await setObsidianMobileTestMode(port, false, timeoutMs).catch(() => undefined);
await session.app.stop();
await stopSession(context, session);
}
}
@@ -904,7 +920,7 @@ async function main(): Promise<void> {
const dbName = makeUniqueDatabaseName(couchDb.dbPrefix, "hidden-roundtrip");
const vaultA = await createTemporaryVault();
const vaultB = await createTemporaryVault();
const context: RunnerContext = { binary, cliBinary: cli.binary, couchDb, dbName };
const context: RunnerContext = { binary, cliBinary: cli.binary, couchDb, dbName, activeSessions: new Set() };
try {
await assertCouchDbReachable(couchDb);
@@ -923,6 +939,7 @@ async function main(): Promise<void> {
await runInitialisationNoticeGrouping(context, vaultB);
await runConfigurationNoticeGrouping(context, vaultB);
} finally {
for (const session of context.activeSessions) await stopSession(context, session);
await vaultA.dispose();
await vaultB.dispose();
if (process.env.E2E_OBSIDIAN_KEEP_COUCHDB !== "true") {
test/e2e-obsidian/scripts/joined-devices.ts
+588
View File
@@ -0,0 +1,588 @@
import { PutObjectCommand } from "@aws-sdk/client-s3";
import { mkdir } from "node:fs/promises";
import { join } from "node:path";
import { DEVICE_ID_PREFERRED, MILESTONE_DOCID, NODEINFO_DOCID } from "@vrtmrz/livesync-commonlib/compat/common/types";
import { evalObsidianJson } from "../runner/cli.ts";
import {
assertCouchDbReachable,
createCouchDbDatabase,
deleteCouchDbDatabase,
fetchCouchDbDocument,
loadCouchDbConfig,
makeUniqueDatabaseName,
putCouchDbDocument,
type CouchDbDocument,
} from "../runner/couchdb.ts";
import {
createObjectStorageClient,
deleteObjectStoragePrefix,
ensureObjectStorageBucket,
loadObjectStorageConfig,
makeUniqueBucketPrefix,
readObjectStorageJson,
} from "../runner/objectStorage.ts";
import { discoverObsidianCli, requireObsidianBinary } from "../runner/environment.ts";
import {
assertEqual,
assertE2eCompatibilityUnpaused,
configureCouchDb,
configureObjectStorage,
createE2eCouchDbPluginData,
createE2eObjectStoragePluginData,
prepareRemote,
waitForLiveSyncCoreReady,
} from "../runner/liveSyncWorkflow.ts";
import {
assertLocatorHasMinimumTouchTarget,
assertLocatorWithinSafeArea,
assertLocatorWithinViewport,
assertNoHorizontalOverflow,
} from "@vrtmrz/obsidian-test-session";
import { iPhoneSafeArea, setObsidianMobileTestMode } from "../runner/mobileUi.ts";
import { startObsidianLiveSyncSession, type ObsidianLiveSyncSession } from "../runner/session.ts";
import { openLiveSyncSettings, waitForVisibleObsidianDialogue, withObsidianPage } from "../runner/ui.ts";
import { createTemporaryVault } from "../runner/vault.ts";
import type { Locator, Page } from "playwright";
const uiTimeoutMs = Number(process.env.E2E_OBSIDIAN_JOINED_DEVICES_TIMEOUT_MS ?? 10000);
const commandId = "obsidian-livesync:joined-devices";
const targetId = "joined-devices-target";
const retainedId = "joined-devices-retained";
const initialPlatform = "E2E Android tablet";
const refreshedPlatform = "E2E refreshed platform";
const connectedAt = Date.UTC(2026, 0, 12, 9, 30, 0);
type SelfDeviceInfo = {
id: string;
platform: string;
pluginVersion: string;
appVersion: string;
};
type DeviceInfo = {
device_name: string;
vault_name: string;
app_version: string;
plugin_version: string;
platform: string;
progress: string;
last_connected: number;
};
type JoinedDevicesMilestone = CouchDbDocument & {
accepted_nodes: string[];
node_info: Record<string, DeviceInfo>;
locked: boolean;
cleaned: boolean;
node_chunk_info: Record<string, { min: number; max: number; current: number }>;
tweak_values: Record<string, Record<string, unknown>>;
};
type DeviceFixture = {
milestone: JoinedDevicesMilestone;
preferredTweaks: Record<string, unknown>;
};
type JoinedDevicesRemote = {
label: string;
pluginData: Record<string, unknown>;
prepare(): Promise<void>;
configure(cliBinary: string, env: NodeJS.ProcessEnv): Promise<unknown>;
read(): Promise<JoinedDevicesMilestone>;
write(milestone: JoinedDevicesMilestone): Promise<void>;
cleanup(): Promise<void>;
};
async function createRemoteFixture(): Promise<JoinedDevicesRemote> {
const args = process.argv.slice(2);
if (args.some((argument) => argument !== "--object-storage")) {
throw new Error("Usage: joined-devices.ts [--object-storage]");
}
if (args.includes("--object-storage")) {
const config = await loadObjectStorageConfig();
const bucketPrefix = makeUniqueBucketPrefix("joined-devices");
const settings = { ...config, bucketPrefix };
const key = `${bucketPrefix}_00000000-milestone.json`;
return {
label: "Object Storage",
pluginData: createE2eObjectStoragePluginData(settings),
prepare: () => ensureObjectStorageBucket(config),
configure: (cliBinary, env) => configureObjectStorage(cliBinary, env, settings),
read: () => readObjectStorageJson<JoinedDevicesMilestone>(config, key),
write: async (milestone) => {
const client = createObjectStorageClient(config);
try {
await client.send(
new PutObjectCommand({
Bucket: config.bucket,
Key: key,
Body: JSON.stringify(milestone),
ContentType: "application/json",
})
);
} finally {
client.destroy();
}
},
cleanup: () => deleteObjectStoragePrefix(config, bucketPrefix),
};
}
const config = await loadCouchDbConfig();
const dbName = makeUniqueDatabaseName(config.dbPrefix, "joined-devices");
const settings = { uri: config.uri, username: config.username, password: config.password, dbName };
return {
label: "CouchDB",
pluginData: createE2eCouchDbPluginData(settings),
prepare: async () => {
await assertCouchDbReachable(config);
await createCouchDbDatabase(config, dbName);
},
configure: (cliBinary, env) => configureCouchDb(cliBinary, env, settings),
read: async () => {
const milestone = await fetchCouchDbDocument(config, dbName, MILESTONE_DOCID);
assert(Boolean(milestone._rev), "The remote milestone did not have a revision.");
return milestone as JoinedDevicesMilestone;
},
write: async (milestone) => {
await putCouchDbDocument(config, dbName, milestone);
},
cleanup: () => deleteCouchDbDatabase(config, dbName),
};
}
function assert(condition: boolean, message: string): void {
if (!condition) throw new Error(message);
}
function deviceCard(dialogue: Locator, id: string): Locator {
return dialogue.locator(".device-card").filter({ hasText: id });
}
async function assertDetail(card: Locator, label: string, expected: string): Promise<void> {
const rows = card.locator(".device-details > div");
for (let index = 0; index < (await rows.count()); index++) {
const row = rows.nth(index);
if ((await row.locator("dt").innerText()).trim() === label) {
assertEqual((await row.locator("dd").innerText()).trim(), expected, `${label} did not match.`);
return;
}
}
throw new Error(`The device details did not contain '${label}'.`);
}
async function readSelfDeviceInfo(cliBinary: string, env: NodeJS.ProcessEnv): Promise<SelfDeviceInfo> {
return await evalObsidianJson<SelfDeviceInfo>(
cliBinary,
[
"(async()=>{",
"const core=app.plugins.plugins['obsidian-livesync'].core;",
`const nodeInfo=await core.services.database.localDatabase.localDatabase.get(${JSON.stringify(NODEINFO_DOCID)});`,
"return JSON.stringify({",
"id:nodeInfo.nodeid,",
"platform:core.services.API.getPlatform(),",
"pluginVersion:core.services.API.getPluginVersion(),",
"appVersion:core.services.API.getAppVersion(),",
"});",
"})()",
].join(""),
env
);
}
function createFixture(self: SelfDeviceInfo): DeviceFixture {
const preferredTweaks = { customChunkSize: 60, chunkSplitterVersion: "v3-rabin-karp" };
const milestone: JoinedDevicesMilestone = {
_id: MILESTONE_DOCID,
type: "milestoneinfo",
created: Date.now(),
// The current device is deliberately absent: it must still be listed first and identified as locked out.
accepted_nodes: [targetId, retainedId],
node_info: {
[targetId]: {
device_name: "E2E target device",
vault_name: "Joined devices target fixture",
app_version: "1.99.0-test",
plugin_version: "999.0.0",
platform: initialPlatform,
progress: "ready",
last_connected: connectedAt,
},
[retainedId]: {
device_name: "E2E retained device",
vault_name: "Joined devices retained fixture",
app_version: "1.88.0-test",
plugin_version: "0.1.0",
platform: "E2E Linux desktop",
progress: "ready",
last_connected: connectedAt - 86400000,
},
},
locked: true,
cleaned: true,
node_chunk_info: {
[targetId]: { min: 1, max: 3, current: 3 },
[retainedId]: { min: 1, max: 2, current: 2 },
},
tweak_values: {
[targetId]: { customChunkSize: 48 },
[retainedId]: { customChunkSize: 60 },
[DEVICE_ID_PREFERRED]: preferredTweaks,
},
};
assert(self.id !== targetId && self.id !== retainedId, "The local device identity collided with a fixture ID.");
return { milestone, preferredTweaks };
}
async function openFromRemoteSettings(page: Page): Promise<Locator> {
const settings = await openLiveSyncSettings(page, uiTimeoutMs);
const remoteConfiguration = await settings.openPage("Remote Configuration");
await remoteConfiguration
.getByRole("button", { name: "Review devices", exact: true })
.click({ timeout: uiTimeoutMs });
const dialogue = await waitForVisibleObsidianDialogue(page, "Joined devices", uiTimeoutMs);
await dialogue.getByText(targetId, { exact: true }).waitFor({ state: "visible", timeout: uiTimeoutMs });
return dialogue;
}
async function assertInitialDialogue(dialogue: Locator, self: SelfDeviceInfo, page: Page): Promise<void> {
const cards = dialogue.locator(".device-card");
assertEqual(
await cards.count(),
3,
"The joined devices dialogue did not show the self, target, and retained records."
);
const selfCard = cards.nth(0);
assert(Boolean(self.id), "The current device identity was not read from the local NODEINFO_DOCID record.");
assertEqual(
(await selfCard.locator("code.device-id").innerText()).trim(),
self.id,
"The current device was not listed first."
);
await selfCard.getByText("This device", { exact: true }).waitFor({ state: "visible", timeout: uiTimeoutMs });
await selfCard.getByText("Locked out", { exact: true }).waitFor({ state: "visible", timeout: uiTimeoutMs });
await selfCard.getByText("Not accepted", { exact: true }).waitFor({ state: "visible", timeout: uiTimeoutMs });
await selfCard
.getByText("This device has been locked out.", { exact: false })
.waitFor({ state: "visible", timeout: uiTimeoutMs });
await assertDetail(selfCard, "Platform", self.platform);
await assertDetail(selfCard, "Plug-in version", self.pluginVersion);
await assertDetail(selfCard, "App version", self.appVersion);
assertEqual(
await selfCard.getByRole("button", { name: "Forget device", exact: true }).count(),
0,
"The current device exposed a forget control."
);
const targetCard = deviceCard(dialogue, targetId);
await assertDetail(targetCard, "Platform", initialPlatform);
await assertDetail(targetCard, "Plug-in version", "999.0.0");
await assertDetail(targetCard, "App version", "1.99.0-test");
const targetDate = await page.evaluate((timestamp) => new Date(timestamp).toLocaleString(), connectedAt);
await assertDetail(targetCard, "Last recorded", targetDate);
await targetCard.getByText("Accepted", { exact: true }).waitFor({ state: "visible", timeout: uiTimeoutMs });
const retainedCard = deviceCard(dialogue, retainedId);
await assertDetail(retainedCard, "Platform", "E2E Linux desktop");
await assertDetail(retainedCard, "Plug-in version", "0.1.0");
await assertDetail(retainedCard, "App version", "1.88.0-test");
await dialogue
.getByText(
"This list shows device information recorded by the remote. It does not show whether a device is online now.",
{ exact: true }
)
.waitFor({ state: "visible", timeout: uiTimeoutMs });
await dialogue
.locator(".sls-info-note")
.getByText(
"Synchronisation between different plug-in versions is generally safe unless the connection is explicitly blocked. However, it may be less efficient or use more storage, and P2P synchronisation may be unavailable.",
{ exact: true }
)
.waitFor({ state: "visible", timeout: uiTimeoutMs });
await dialogue
.getByText("This remote is locked.", { exact: false })
.waitFor({ state: "visible", timeout: uiTimeoutMs });
await dialogue
.getByText("This remote has been cleaned.", { exact: false })
.waitFor({ state: "visible", timeout: uiTimeoutMs });
}
async function refreshRemotePlatform(page: Page, remote: JoinedDevicesRemote): Promise<void> {
const milestone = await remote.read();
await remote.write({
...milestone,
node_info: {
...milestone.node_info,
[targetId]: { ...milestone.node_info[targetId], platform: refreshedPlatform },
},
});
const dialogue = await waitForVisibleObsidianDialogue(page, "Joined devices", uiTimeoutMs);
await dialogue.getByRole("button", { name: "Refresh", exact: true }).click({ timeout: uiTimeoutMs });
await deviceCard(dialogue, targetId).getByText(refreshedPlatform, { exact: true }).waitFor({
state: "visible",
timeout: uiTimeoutMs,
});
}
async function showForgetConfirmation(dialogue: Locator): Promise<void> {
const targetCard = deviceCard(dialogue, targetId);
await targetCard.getByRole("button", { name: "Forget device", exact: true }).click({ timeout: uiTimeoutMs });
await targetCard.getByText("Forget this device record?", { exact: true }).waitFor({
state: "visible",
timeout: uiTimeoutMs,
});
}
async function assertCancelledForget(dialogue: Locator, remote: JoinedDevicesRemote): Promise<void> {
const before = await remote.read();
await showForgetConfirmation(dialogue);
const confirmation = deviceCard(dialogue, targetId).locator(".forget-confirmation");
await confirmation.getByRole("button", { name: "Cancel", exact: true }).click({ timeout: uiTimeoutMs });
await confirmation.waitFor({ state: "detached", timeout: uiTimeoutMs });
assertEqual(JSON.stringify(await remote.read()), JSON.stringify(before), "Cancelling Forget changed the remote.");
}
async function confirmForgetTarget(dialogue: Locator): Promise<void> {
const targetCard = deviceCard(dialogue, targetId);
await targetCard.getByRole("button", { name: "Forget device", exact: true }).click({ timeout: uiTimeoutMs });
await dialogue.getByText("The device record was forgotten.", { exact: true }).waitFor({
state: "visible",
timeout: uiTimeoutMs,
});
await targetCard.waitFor({ state: "detached", timeout: uiTimeoutMs });
}
async function assertTargetForgotten(remote: JoinedDevicesRemote, fixture: DeviceFixture): Promise<void> {
const milestone = await remote.read();
assertEqual(
JSON.stringify(milestone.accepted_nodes),
JSON.stringify([retainedId]),
"Forgetting changed the wrong accepted device records."
);
assertEqual(
JSON.stringify(Object.keys(milestone.node_info)),
JSON.stringify([retainedId]),
"Forgetting changed the wrong device metadata records."
);
assertEqual(
JSON.stringify(Object.keys(milestone.node_chunk_info)),
JSON.stringify([retainedId]),
"Forgetting changed the wrong chunk metadata records."
);
assertEqual(
JSON.stringify(Object.keys(milestone.tweak_values).sort()),
JSON.stringify([DEVICE_ID_PREFERRED, retainedId].sort()),
"Forgetting changed the wrong tweak metadata records."
);
assertEqual(milestone.locked, true, "Forgetting changed the remote lock state.");
assertEqual(milestone.cleaned, true, "Forgetting changed the remote cleaned state.");
assertEqual(
JSON.stringify(milestone.tweak_values[DEVICE_ID_PREFERRED]),
JSON.stringify(fixture.preferredTweaks),
"Forgetting changed the preferred-ID tweak record."
);
assertEqual(
JSON.stringify(milestone.node_info[retainedId]),
JSON.stringify(fixture.milestone.node_info[retainedId]),
"Forgetting changed the retained device metadata."
);
}
async function assertScrollableMobileDialogue(page: Page, container: Locator): Promise<void> {
const dialogue = container.locator(".modal").last();
const closeButton = container.locator(".modal-close-button, .modal-header-button").last();
await assertLocatorWithinViewport(page, dialogue, { label: "Joined devices dialogue" });
await assertNoHorizontalOverflow(page, dialogue, { label: "Joined devices dialogue" });
await assertLocatorWithinSafeArea(page, dialogue, {
label: "Joined devices dialogue",
safeAreaInsets: iPhoneSafeArea,
});
const buttons = [closeButton, ...(await dialogue.locator("button:visible").all())];
for (const button of buttons) {
// Device records form a scrollable list; every control must be reachable after scrolling.
await button.scrollIntoViewIfNeeded();
const label = (await button.innerText()).trim() || "Close dialogue";
await assertLocatorWithinViewport(page, button, { label });
await assertLocatorWithinSafeArea(page, button, { label, safeAreaInsets: iPhoneSafeArea });
await assertNoHorizontalOverflow(page, button, { label });
await assertLocatorHasMinimumTouchTarget(page, button, { label });
}
await container.locator(".modal-content").evaluate((element) => element.scrollTo(0, 0));
}
async function dismissNotices(page: Page): Promise<void> {
const notices = page.locator(".notice:visible");
for (let remaining = 20; remaining > 0 && (await notices.count()) > 0; remaining--) {
const notice = await notices.first().elementHandle();
if (!notice) continue;
try {
await notice.click({ position: { x: 20, y: 20 }, timeout: uiTimeoutMs });
await notice.waitForElementState("hidden", { timeout: uiTimeoutMs });
} finally {
await notice.dispose();
}
}
assertEqual(await notices.count(), 0, "Notices still obscured the dialogue screenshot.");
}
async function captureJoinedDevices(
port: number,
filename: string,
assertReady: (page: Page) => Promise<Locator>
): Promise<void> {
const outputDirectory = process.env.E2E_OBSIDIAN_DIAGNOSTICS_DIR ?? "/tmp/obsidian-livesync-e2e";
await mkdir(outputDirectory, { recursive: true });
await withObsidianPage(port, async (page) => {
try {
const dialogue = await assertReady(page);
await dialogue.page().screenshot({ path: join(outputDirectory, filename), fullPage: true });
} catch (error) {
const dialogue = await waitForVisibleObsidianDialogue(page, "Joined devices", 500).catch(() => undefined);
const failurePath = join(outputDirectory, filename.replace(/\.png$/u, ".failure.png"));
await (dialogue?.page() ?? page).screenshot({ path: failurePath, fullPage: true }).catch(() => undefined);
console.error(`Joined devices UI failure screenshot: ${failurePath}`);
throw error;
}
});
}
async function main(): Promise<void> {
const binary = requireObsidianBinary();
const cli = discoverObsidianCli();
if (!cli.binary) throw new Error(`Could not find obsidian-cli. Checked paths: ${cli.checked.join(", ")}`);
const remote = await createRemoteFixture();
const vault = await createTemporaryVault();
let session: ObsidianLiveSyncSession | undefined;
try {
await remote.prepare();
session = await startObsidianLiveSyncSession({
binary,
cliBinary: cli.binary,
vault,
startupGraceMs: Number(process.env.E2E_OBSIDIAN_STARTUP_GRACE_MS ?? 1000),
pluginData: remote.pluginData,
});
await waitForLiveSyncCoreReady(cli.binary, session.cliEnv);
await assertE2eCompatibilityUnpaused(cli.binary, session.cliEnv, session.remoteDebuggingPort);
await remote.configure(cli.binary, session.cliEnv);
await waitForLiveSyncCoreReady(cli.binary, session.cliEnv);
await prepareRemote(cli.binary, session.cliEnv);
const self = await readSelfDeviceInfo(cli.binary, session.cliEnv);
const fixture = createFixture(self);
const preparedMilestone = await remote.read();
await remote.write({
...fixture.milestone,
...(preparedMilestone._rev ? { _rev: preparedMilestone._rev } : {}),
});
await captureJoinedDevices(session.remoteDebuggingPort, "joined-devices-notice.png", async (page) => {
const dialogue = await openFromRemoteSettings(page);
const dialoguePage = dialogue.page();
await dialoguePage.setViewportSize({ width: 1280, height: 1100 });
await assertInitialDialogue(dialogue, self, dialoguePage);
const notice = dialoguePage.locator(".notice").filter({
hasText: "Newer Self-hosted LiveSync versions detected: 999.0.0.",
});
await notice.waitFor({ state: "visible", timeout: uiTimeoutMs });
await notice
.getByText("Older Self-hosted LiveSync versions detected: 0.1.0.", { exact: true })
.waitFor({ state: "visible", timeout: uiTimeoutMs });
assertEqual(
await notice
.getByText("Reviewing and aligning Self-hosted LiveSync versions across devices is recommended.", {
exact: true,
})
.count(),
1,
"The version Notice did not contain exactly one recommendation."
);
assertEqual(
await notice.getByRole("button", { name: "Review devices", exact: true }).count(),
1,
"The version Notice did not contain exactly one shared review action."
);
return dialogue;
});
await captureJoinedDevices(session.remoteDebuggingPort, "joined-devices-desktop.png", async (page) => {
const dialogue = await waitForVisibleObsidianDialogue(page, "Joined devices", uiTimeoutMs);
await dialogue.getByRole("button", { name: "Close", exact: true }).click({ timeout: uiTimeoutMs });
await dialogue
.page()
.locator(".notice")
.getByRole("button", { name: "Review devices", exact: true })
.click();
const reopened = await waitForVisibleObsidianDialogue(page, "Joined devices", uiTimeoutMs);
await reopened.getByText(targetId, { exact: true }).waitFor({ state: "visible", timeout: uiTimeoutMs });
await dismissNotices(reopened.page());
return reopened;
});
await captureJoinedDevices(
session.remoteDebuggingPort,
"joined-devices-forget-confirmation.png",
async (page) => {
await refreshRemotePlatform(page, remote);
const dialogue = await waitForVisibleObsidianDialogue(page, "Joined devices", uiTimeoutMs);
await assertCancelledForget(dialogue, remote);
await showForgetConfirmation(dialogue);
await deviceCard(dialogue, targetId).locator(".forget-confirmation").scrollIntoViewIfNeeded();
return dialogue;
}
);
await captureJoinedDevices(session.remoteDebuggingPort, "joined-devices-after-forget.png", async (page) => {
const dialogue = await waitForVisibleObsidianDialogue(page, "Joined devices", uiTimeoutMs);
await confirmForgetTarget(dialogue);
await dialogue.locator(".modal-content").evaluate((element) => element.scrollTo(0, 0));
return dialogue;
});
await withObsidianPage(session.remoteDebuggingPort, async (page) => {
const dialogue = await waitForVisibleObsidianDialogue(page, "Joined devices", uiTimeoutMs);
await dialogue.getByRole("button", { name: "Close", exact: true }).click({ timeout: uiTimeoutMs });
const settings = await openLiveSyncSettings(page, uiTimeoutMs);
await settings.close();
});
await assertTargetForgotten(remote, fixture);
await setObsidianMobileTestMode(session.remoteDebuggingPort, true, uiTimeoutMs);
await waitForLiveSyncCoreReady(cli.binary, session.cliEnv);
await captureJoinedDevices(session.remoteDebuggingPort, "joined-devices-mobile.png", async (page) => {
const opened = await page.evaluate((id) => {
const app = (
window as typeof window & {
app?: { commands?: { executeCommandById?(commandId: string): boolean } };
}
).app;
return app?.commands?.executeCommandById?.(id) === true;
}, commandId);
assert(opened, "The Joined devices command did not open the dialogue.");
const dialogue = await waitForVisibleObsidianDialogue(page, "Joined devices", uiTimeoutMs);
await dialogue.getByText(retainedId, { exact: true }).waitFor({ state: "visible", timeout: uiTimeoutMs });
await assertScrollableMobileDialogue(dialogue.page(), dialogue);
assertEqual(
await dialogue.locator(".device-card").count(),
2,
"The refreshed mobile list did not retain both remaining devices."
);
await dismissNotices(dialogue.page());
return dialogue;
});
console.log(`Joined devices scenario passed against ${remote.label}.`);
} finally {
try {
if (session) await session.app.stop();
} finally {
try {
await vault.dispose();
} finally {
await remote.cleanup();
}
}
}
}
main().catch((error: unknown) => {
console.error(error instanceof Error ? error.stack : error);
process.exit(1);
});
test/e2e-obsidian/scripts/local-suite.ts
+3
View File
@@ -26,6 +26,7 @@ const testSteps: Step[] = [
{ name: "Review Harness", args: ["run", "test:e2e:obsidian:review-harness"] },
{ name: "P2P status pane", args: ["run", "test:e2e:obsidian:p2p-pane"] },
{ name: "vault reflection", args: ["run", "test:e2e:obsidian:vault-reflection"] },
{ name: "joined devices with CouchDB", args: ["run", "test:e2e:obsidian:joined-devices"] },
{ name: "CouchDB upload", args: ["run", "test:e2e:obsidian:couchdb-upload"] },
{
name: "manual CouchDB setup workflow",
@@ -35,6 +36,7 @@ const testSteps: Step[] = [
name: "CLI to real Obsidian synchronisation",
args: ["run", "test:e2e:obsidian:cli-to-obsidian-sync"],
},
{ name: "joined devices with Object Storage", args: ["run", "test:e2e:obsidian:joined-devices-object-storage"] },
{ name: "Object Storage upload", args: ["run", "test:e2e:obsidian:minio-upload"] },
{
name: "Object Storage Setup URI workflow",
@@ -56,6 +58,7 @@ const testSteps: Step[] = [
{ name: "startup scan", args: ["run", "test:e2e:obsidian:startup-scan"] },
{ name: "provisioned Setup URI workflow", args: ["run", "test:e2e:obsidian:setup-uri-workflow"] },
{ name: "two-vault synchronisation", args: ["run", "test:e2e:obsidian:two-vault-sync"] },
{ name: "hidden file selectors", args: ["run", "test:e2e:obsidian:hidden-file-selectors"] },
{ name: "hidden file snippet synchronisation", args: ["run", "test:e2e:obsidian:hidden-file-snippet-sync"] },
{ name: "Customisation Sync", args: ["run", "test:e2e:obsidian:customisation-sync"] },
{ name: "internal Metadata Doctor", args: ["run", "test:e2e:obsidian:internal-metadata-doctor"] },
test/e2e-obsidian/scripts/review-harness.ts
+11 -3
View File
@@ -372,13 +372,21 @@ async function restartAndResumeHarness(): Promise<string> {
await harness
.locator('[data-testid="review-harness-resumed"]')
.waitFor({ state: "visible", timeout: uiTimeoutMs });
const continuationRemoved = await page.evaluate((stateKey) => {
const continuationRemoved = await page.evaluate(async (stateKey) => {
const plugin = (globalThis as ReviewHarnessTestGlobal).app?.plugins?.plugins["obsidian-livesync"];
if (typeof plugin !== "object" || plugin === null || !("core" in plugin)) {
throw new Error("Self-hosted LiveSync is unavailable after restart.");
}
const core = (plugin as { core: { services: { setting: { getSmallConfig(key: string): string } } } }).core;
return core.services.setting.getSmallConfig(stateKey) === "";
const core = (
plugin as {
core: {
services: {
setting: { config: { getValue(key: string, defaultValue: string): Promise<string> } };
};
};
}
).core;
return (await core.services.setting.config.getValue(stateKey, "")) === "";
}, REVIEW_HARNESS_STATE_KEY);
if (!continuationRemoved) throw new Error("The one-shot continuation was not removed before use.");
await assertNoHorizontalOverflow(page, harness, { label: "resumed Review Harness" });
test/e2e-obsidian/scripts/run-focused.ts
+3
View File
@@ -8,6 +8,8 @@ const focusedScenarios = new Set([
"smoke",
"onboarding-invitation",
"dialog-mounts",
"joined-devices",
"joined-devices-object-storage",
"database-diagnostics",
"database-diagnostics-couchdb",
"conflict-dialog-policy",
@@ -39,6 +41,7 @@ const focusedScenarios = new Set([
"two-vault-sync",
"security-seed-reconnect",
"hidden-file-snippet-sync",
"hidden-file-selectors",
"customisation-sync",
"received-change-readiness",
"internal-metadata-doctor",
test/e2e-obsidian/scripts/settings-ui.ts
+17 -9
View File
@@ -58,8 +58,9 @@ type LiveSyncTestPlugin = {
usePowerUserMode: boolean;
useEdgeCaseMode: boolean;
hashCacheMaxCount: number;
customChunkSize: number;
};
getSmallConfig(key: string): string | null;
config: { getValue(key: string, defaultValue: string): Promise<string> };
};
};
};
@@ -484,12 +485,12 @@ async function verifyCompatibilityReview(): Promise<void> {
});
await withObsidianPage(port, async (page) => {
const markerBeforeAcknowledgement = await page.evaluate(() => {
const markerBeforeAcknowledgement = await page.evaluate(async () => {
const plugin = (globalThis as ObsidianTestGlobal).app?.plugins?.plugins["obsidian-livesync"];
if (plugin === undefined) throw new Error("Self-hosted LiveSync is unavailable");
return plugin.core.services.setting.getSmallConfig("database-compatibility-version");
return await plugin.core.services.setting.config.getValue("database-compatibility-version", "");
});
if (markerBeforeAcknowledgement !== null && markerBeforeAcknowledgement !== "") {
if (markerBeforeAcknowledgement !== "") {
throw new Error(
`The database version was marked as acknowledged before review: ${markerBeforeAcknowledgement}`
);
@@ -579,12 +580,12 @@ async function verifyCompatibilityReview(): Promise<void> {
.click();
await summary.waitFor({ state: "hidden", timeout: uiTimeoutMs });
await page.waitForFunction(
(expectedVersion) => {
async (expectedVersion) => {
const plugin = (globalThis as ObsidianTestGlobal).app?.plugins?.plugins["obsidian-livesync"];
if (plugin === undefined) return false;
const setting = plugin.core.services.setting;
return (
setting.getSmallConfig("database-compatibility-version") === expectedVersion &&
(await setting.config.getValue("database-compatibility-version", "")) === expectedVersion &&
setting.currentSettings().versionUpFlash === ""
);
},
@@ -611,10 +612,16 @@ async function verifyConfigDoctorFollowsCompatibilityReview(): Promise<string> {
state: "visible",
timeout: uiTimeoutMs,
});
await doctor.getByText("Enhance chunk size", { exact: true }).waitFor({
state: "visible",
timeout: uiTimeoutMs,
const chunkSize = await page.evaluate(() => {
const plugin = (globalThis as ObsidianTestGlobal).app?.plugins?.plugins["obsidian-livesync"];
return plugin?.core.services.setting.currentSettings().customChunkSize;
});
if (chunkSize !== 0) {
throw new Error("Compatibility review must retain the fixture's zero chunk-size value.");
}
if ((await doctor.getByText("Enhance chunk size", { exact: true }).count()) !== 0) {
throw new Error("Config Doctor must accept the recommended zero chunk-size value.");
}
if ((await doctor.getByText("Data Compression", { exact: true }).count()) !== 0) {
throw new Error("Config Doctor still treats supported Data Compression as a problem.");
}
@@ -978,6 +985,7 @@ function createSettingsPluginData(settingsOnlyRun: boolean): Record<string, unkn
liveSync: false,
versionUpFlash: settingsOnlyRun ? "" : compatibilityReviewMessage,
notifyThresholdOfRemoteStorageSize: 0,
customChunkSize: 0,
syncOnStart: false,
syncOnSave: false,
syncOnEditorSave: false,
test/fixtures/optionalFileSyncDatabase.ts
Vendored
+78
View File
@@ -0,0 +1,78 @@
import { randomUUID } from "node:crypto";
import PouchDB from "pouchdb-core";
import MemoryAdapter from "pouchdb-adapter-memory";
import { vi } from "vitest";
import {
DEFAULT_SETTINGS,
type EntryDoc,
type SavingEntry,
type FilePathWithPrefix,
type DocumentID,
} from "@vrtmrz/livesync-commonlib/compat/common/types";
import { createTextBlob } from "@vrtmrz/livesync-commonlib/compat/common/utils";
import { createLiveSyncEventHub } from "@vrtmrz/livesync-commonlib/context";
import { LiveSyncLocalDB, type LiveSyncLocalDBEnv } from "@vrtmrz/livesync-commonlib/compat/pouchdb/LiveSyncLocalDB";
PouchDB.plugin(MemoryAdapter);
export function optionalFileEntry(path: string, content: string): SavingEntry {
return {
_id: path as DocumentID,
path: path as FilePathWithPrefix,
ctime: 1_000,
mtime: 2_000,
size: content.length,
type: "plain",
datatype: "plain",
data: createTextBlob(content),
children: [],
eden: {},
};
}
export async function createOptionalFileSyncDatabase() {
const name = `optional-file-sync-${randomUUID()}`;
const database = new PouchDB<EntryDoc>(name, { adapter: "memory" });
const settings = {
...DEFAULT_SETTINGS,
syncInternalFiles: true,
useOnlyLocalChunk: true,
usePluginSync: true,
usePluginSyncV2: true,
pluginSyncExtendedSetting: {},
};
const services = {
API: { addLog: vi.fn(), getSystemConfigDir: () => ".obsidian" },
setting: { currentSettings: () => settings, getDeviceAndVaultName: () => "device" },
path: {
path2id: async (path: string) => path,
id2path: (id: string, entry?: { path?: string }) => entry?.path ?? id,
getPath: (entry: { path: FilePathWithPrefix }) => entry.path,
},
context: { events: createLiveSyncEventHub() },
database: { createPouchDBInstance: () => database },
databaseEvents: {
onDatabaseInitialisation: async () => true,
onDatabaseHasReady: async () => true,
onCloseDatabase: async () => true,
onUnloadDatabase: async () => true,
},
replicator: { onCloseActiveReplication: async () => true },
};
const localDatabase = new LiveSyncLocalDB(name, { services } as unknown as LiveSyncLocalDBEnv);
if (!(await localDatabase.initializeDatabase())) {
await database.destroy();
throw new Error("The optional-file test database could not initialise");
}
return {
database,
localDatabase,
services,
settings,
async dispose() {
localDatabase.offRemoteChunkFetchedHandler?.();
await localDatabase.managers.teardownManagers();
await database.destroy();
},
};
}
updates.md
+35
View File
@@ -12,6 +12,41 @@ Earlier releases remain available in the 1.0 release history, the 1.0 preview hi
## Unreleased
### CLI
- Unreadable or malformed device-local state now stops initialisation instead of being treated as empty. Existing valid state files keep their format.
### Hidden File Sync
#### Fixed
- We can synchronise selected hidden files while the normal-file allow-list is set. (#1272)
- Normal-file exclusion patterns still apply to hidden files, and the Selector descriptions now explain their scope.
### Synchronisation
#### Added
- We can choose **Review devices** in Remote Configuration, the command palette, or a version Notice to review participating devices.
- Our device appears first, with its lockout state. Other records show the platform, plug-in version, app version, and last recorded connection time when available.
- An advisory Notice identifies Self-hosted LiveSync and lists newer and older versions separately, with one shared recommendation and **Review devices** action. The dialogue explains possible storage costs and P2P compatibility limits.
- We can forget records belonging to lost or retired devices while preserving the remote lock state and other devices.
- Forgetting a record does not delete Vault files or revoke credentials. A returning device can register again on an unlocked remote.
#### Fixed
- We can reuse unchanged content more reliably after edits with the V3 Rabin–Karp chunk splitter.
- Corrected rolling-window processing lets boundary detection recover after an insertion. Existing content remains readable, and installing this update does not trigger a full re-chunk of our Vaults.
- Chunk-size values 0 and 1 now use content-defined binary boundaries below their maximum lengths, improving reuse after an insertion instead of producing fixed-length pieces.
- Editing or explicitly rewriting a file, recreating its chunks, or rebuilding from our files can produce different chunk boundaries and additional chunks. Older and updated devices can retain different chunk layouts; this affects reuse and storage, not the file contents.
- We can preserve all conflict branches, including deleted branches, during Fast Setup.
- A fresh, complete Fast Fetch can also hand its verified progress to ordinary synchronisation before file watching resumes, reducing the first subsequent identifier scan. Existing resume cursors and On Demand chunk retrieval retain their existing behaviour.
#### Changed
- New setup uses the same Chunk-size recommendation of 0 across remote types, and Setting Doctor no longer recommends increasing every self-hosted V3 configuration to 60.
- Existing explicit and agreed values are retained. Smaller Chunks favour reuse during editing, but increase document counts and can make first uploads and large rebuilds slower. Mostly immutable binary Vaults can still benefit from a larger explicit value.
## 1.0.39
20th October, 2026
utils/chunk-size-benchmark/README.md
+72
View File
@@ -0,0 +1,72 @@
# Chunk-size research harness
These scripts support the October 2026 Chunk-size investigation. The decision,
measurements, and their limits are recorded in Commonlib's
`docs/adr/2026_10_chunking_and_transport_batching.md` and its evidence companion.
They are research tools with experiment-specific assertions, separate from
the maintained unit and integration suites.
The original experiments used Commonlib 0.1.44 and unpublished candidates with
the same version label. `local.mjs` checks that version explicitly. Record the
installed package's source revision and tarball digest when comparing results;
the version label alone does not identify the tested implementation. Scripts
which import internal `dist` paths also depend on that package layout.
## Local measurements
Install the exact Commonlib candidate to assess, then run from the repository
root with Node.js 24:
```sh
NODE_OPTIONS=--max-old-space-size=3072 node utils/chunk-size-benchmark/local.mjs --output .bench/chunk-size --sizes 0,1,4 --repeats 1
node --test utils/chunk-size-benchmark/rabin-window.test.mjs
```
The local run generates a deterministic corpus from repository Markdown,
images, and seeded binary data. It records initial, overwrite, and insertion
stages, Chunk identifiers, reconstructed content checks, and timings. Keep
the generated corpus when comparing revisions: source Markdown may change.
Generated Vaults, logs, and results belong under `.bench/` and are not source
files for the normal test runner.
## Transport measurements
Build the CLI and start the disposable fixtures required for the chosen
transport. These use loopback ports and public benchmark-only credentials.
The scripts create and delete their own test databases and buckets.
GNU `time`, Docker Compose, and a Linux CLI runtime are required.
```sh
NODE_OPTIONS=--max-old-space-size=3072 npm run build --workspace self-hosted-livesync-cli
docker compose -p livesync-chunk-benchmark -f utils/chunk-size-benchmark/compose.yml up -d couchdb object-storage
node utils/chunk-size-benchmark/remote.mjs --output .bench/chunk-size --sizes 0,1,4 --repeats 1 --rtt-ms 0
docker compose -p livesync-chunk-benchmark -f utils/chunk-size-benchmark/compose.yml down --volumes
```
`remote.mjs` accepts `--providers couchdb`, `--providers s3`, or both, plus
`--corpus`, `--source-cli`, and `--target-cli`. Its HTTP proxy adds the requested
RTT across the request and response legs. This models per-request delay rather
than network bandwidth, packet loss, or a production service.
For P2P, start the `relay` fixture instead and run `p2p.mjs` with the same
output, corpus, size, repeat, and CLI options, omitting `--rtt-ms`. Its
DataChannel counters measure application payloads and record the selected
connection path. An archived CLI entry point needs all of its generated
CommonJS chunks beside it. Run transports sequentially and stop the named
Compose project afterwards.
## Historical analysis helpers
- `summarise.mjs`, `report.mjs`, and `verify-results.mjs` consume the original
result layouts. The verifier requires that experiment's exact variants,
configuration counts, and files; it is not a general mixed-version verifier.
- `cloudant-limit.mjs` temporarily changes the disposable CouchDB document
limit and restores it afterwards. `cloudant-json-boundary.mjs` probes final
serialised document sizes against the same fixture. These scripts model
published Cloudant constraints; they do not connect to IBM Cloudant or
certify its acceptance of a Vault.
Some helpers retain historical output-directory and size defaults. Supply
explicit paths and inspect their assertions before reusing them for a new
experiment. The recorded implementation matrix checks file content in both
writer directions; revision-conflict retention has separate Commonlib tests.
utils/chunk-size-benchmark/cloudant-json-boundary.mjs
+108
View File
@@ -0,0 +1,108 @@
import assert from 'node:assert/strict';
import { createHash } from 'node:crypto';
import { promises as fs } from 'node:fs';
import { dirname, join, resolve } from 'node:path';
import { fileURLToPath, pathToFileURL } from 'node:url';
const packageRoot = dirname(fileURLToPath(import.meta.resolve('@vrtmrz/livesync-commonlib/package.json')));
const { splitPiecesRabinKarp } = await import(pathToFileURL(join(packageRoot, 'dist/string_and_binary/chunks.js')).href);
const { getConfiguredFunctionsForEncryption } = await import(pathToFileURL(join(packageRoot, 'dist/pouchdb/encryption.js')).href);
const { compressDoc } = await import(pathToFileURL(join(packageRoot, 'dist/pouchdb/compress.js')).href);
const output = resolve(process.argv[2] || '.bench/rabin-window-fixed/cloudant-json-boundary');
await fs.mkdir(output, { recursive: true });
const uri = 'http://127.0.0.1:15994';
const headers = { Authorization: `Basic ${Buffer.from('bench:benchmark-only-password').toString('base64')}`, 'Content-Type': 'application/json' };
const configPath = '/_node/_local/_config/couchdb/max_document_size';
const previousResponse = await fetch(uri + configPath, { headers });
assert(previousResponse.ok || previousResponse.status === 404);
const previous = previousResponse.ok ? await previousResponse.json() : undefined;
const requestConfigPath = '/_node/_local/_config/chttpd/max_http_request_size';
const previousRequestResponse = await fetch(uri + requestConfigPath, { headers });
assert(previousRequestResponse.ok || previousRequestResponse.status === 404);
const previousRequestLimit = previousRequestResponse.ok ? await previousRequestResponse.json() : undefined;
const database = `chunkbench-json-budget-${Date.now()}`;
const inputs = {
binary: new Blob([new Uint8Array(4 * 1024 * 1024)], { type: 'application/octet-stream' }),
nulText: new Blob(['\0'.repeat(4 * 1024 * 1024)], { type: 'text/plain' }),
escapedText: new Blob(['\\'.repeat(4 * 1024 * 1024)], { type: 'text/plain' }),
};
const cases = [
{ kind: 'binary', customChunkSize: 4, encryption: 'v1', compression: false, expectedAccepted: true },
{ kind: 'binary', customChunkSize: 4, encryption: 'v2', compression: false, expectedAccepted: true },
{ kind: 'binary', customChunkSize: 5, encryption: 'v2', compression: false, expectedAccepted: false },
{ kind: 'nulText', customChunkSize: 0, encryption: 'none', compression: false, expectedAccepted: true },
{ kind: 'nulText', customChunkSize: 4, encryption: 'none', compression: false },
{ kind: 'escapedText', customChunkSize: 4, encryption: 'none', compression: false },
{ kind: 'nulText', customChunkSize: 4, encryption: 'v2', compression: false, expectedAccepted: true },
{ kind: 'nulText', customChunkSize: 4, encryption: 'none', compression: true, expectedAccepted: true },
];
const records = [];
const record = { scope: 'Synthetic leaf-document probes using the installed splitter, compression, encryption, and JSON serialisation, sent to the owned CouchDB fixture with a 1,000,000-byte document limit and 64,000,000-byte request limit. This is not an IBM Cloudant service measurement or a whole-file synchronisation test. CouchDB 3.5.0 estimates document size from parsed strings and does not count JSON escapes; Cloudant internal size accounting has not been verified.', source: 'https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-limits', sizeCalculationSource: 'https://github.com/apache/couchdb/blob/3.5.0/src/couch/src/couch_ejson_size.erl', documentLimitBytes: 1000000, requestLimitBytes: 64000000, previousDocumentLimit: previous ?? null, previousRequestLimit: previousRequestLimit ?? null, cases: records, batches: [] };
try {
assert((await fetch(uri + configPath, { method: 'PUT', headers, body: JSON.stringify('1000000') })).ok);
assert((await fetch(uri + requestConfigPath, { method: 'PUT', headers, body: JSON.stringify('64000000') })).ok);
assert((await fetch(`${uri}/${database}`, { method: 'PUT', headers })).ok);
for (const condition of cases) {
const input = inputs[condition.kind];
const chunks = await splitPiecesRabinKarp(input, 102400 * (condition.customChunkSize + 1), condition.kind !== 'binary', 20);
let largest = '';
for await (const chunk of chunks()) if (chunk.length > largest.length) largest = chunk;
const id = `h:+${createHash('sha256').update(JSON.stringify(condition)).digest('hex')}`;
let doc = { _id: id, type: 'leaf', data: largest };
if (condition.compression) doc = await compressDoc(doc);
if (condition.encryption !== 'none') {
const { incoming, outgoing } = getConfiguredFunctionsForEncryption('public-json-budget-probe-passphrase', false, false, async () => new Uint8Array(16), condition.encryption);
const encrypted = await incoming(doc);
assert.equal(encrypted.e_, true);
const decoded = await outgoing(structuredClone(encrypted));
assert.equal(decoded.data, doc.data);
doc = encrypted;
}
doc = { ...doc, _rev: `1-${'0'.repeat(32)}`, _revisions: { start: 1, ids: ['0'.repeat(32)] } };
const jsonBytes = Buffer.byteLength(JSON.stringify(doc));
const body = JSON.stringify({ docs: [doc], new_edits: false });
const response = await fetch(`${uri}/${database}/_bulk_docs`, { method: 'POST', headers, body });
const result = await response.json();
const accepted = response.ok && Array.isArray(result) && result.every(entry => !entry.error);
const measurement = { ...condition, largestPieceUtf8Bytes: Buffer.byteLength(largest), jsonBytes, jsonBytesUnderLimit: jsonBytes < record.documentLimitBytes, requestBodyBytes: Buffer.byteLength(body), status: response.status, accepted, response: result };
records.push(measurement);
await fs.writeFile(join(output, 'results.json'), JSON.stringify(record, null, 2));
if (condition.expectedAccepted !== undefined) assert.equal(accepted, condition.expectedAccepted, JSON.stringify(measurement));
assert(Buffer.byteLength(body) < record.requestLimitBytes);
console.log(JSON.stringify({ ...measurement, response: undefined }));
}
const batchInput = new Uint8Array(512000 * 25);
for (let index = 0; index < 25; index++) batchInput[(index + 1) * 512000 - 1] = index + 1;
const splitBatch = await splitPiecesRabinKarp(new Blob([batchInput], { type: 'text/plain' }), 512000, true, 20);
const batchPieces = [];
for await (const piece of splitBatch()) batchPieces.push(piece);
assert.equal(batchPieces.length, 25);
assert(batchPieces.every(piece => Buffer.byteLength(piece) === 512000));
assert.equal(new Set(batchPieces).size, 25);
for (const count of [25, 20]) {
const docs = batchPieces.slice(0, count).map((piece, index) => ({
_id: `h:+${createHash('sha256').update(`escaped-batch-${count}-${index}`).digest('hex')}`,
type: 'leaf', data: piece,
_rev: `1-${'0'.repeat(32)}`, _revisions: { start: 1, ids: ['0'.repeat(32)] },
}));
const body = JSON.stringify({ docs, new_edits: false });
const response = await fetch(`${uri}/${database}/_bulk_docs`, { method: 'POST', headers, body });
const result = await response.json();
const accepted = response.ok && Array.isArray(result) && result.every(entry => !entry.error);
const measurement = { count, rawBytesPerLeaf: 512000, requestBodyBytes: Buffer.byteLength(body), largestDocumentJsonBytes: Math.max(...docs.map(doc => Buffer.byteLength(JSON.stringify(doc)))), status: response.status, accepted, response: result };
record.batches.push(measurement);
await fs.writeFile(join(output, 'results.json'), JSON.stringify(record, null, 2));
assert.equal(accepted, count === 20, JSON.stringify(measurement));
if (count === 25) assert.equal(response.status, 413);
console.log(JSON.stringify({ ...measurement, response: undefined }));
}
} finally {
await fetch(`${uri}/${database}`, { method: 'DELETE', headers });
const restored = await fetch(uri + configPath, previous === undefined ? { method: 'DELETE', headers } : { method: 'PUT', headers, body: JSON.stringify(previous) });
record.configurationRestored = restored.ok;
const requestRestored = await fetch(uri + requestConfigPath, previousRequestLimit === undefined ? { method: 'DELETE', headers } : { method: 'PUT', headers, body: JSON.stringify(previousRequestLimit) });
record.requestConfigurationRestored = requestRestored.ok;
await fs.writeFile(join(output, 'results.json'), JSON.stringify(record, null, 2));
assert(record.configurationRestored);
assert(record.requestConfigurationRestored);
}
utils/chunk-size-benchmark/cloudant-limit.mjs
+50
View File
@@ -0,0 +1,50 @@
import assert from 'node:assert/strict';
import { spawn } from 'node:child_process';
import { promises as fs } from 'node:fs';
import { dirname, join, resolve } from 'node:path';
import { fileURLToPath } from 'node:url';
const root = resolve(dirname(fileURLToPath(import.meta.url)), '../..');
const options = new Map();
for (let index = 2; index < process.argv.length; index += 2) {
assert(['--output', '--corpus'].includes(process.argv[index]), `Unknown option: ${process.argv[index]}`);
assert(process.argv[index + 1], `Missing value for ${process.argv[index]}`);
options.set(process.argv[index], process.argv[index + 1]);
}
const output = resolve(options.get('--output') || '.bench/chunk-size-cloudant-model');
const corpusPath = resolve(options.get('--corpus') || '.bench/chunk-size/corpus/manifest.json');
const configUri = 'http://127.0.0.1:15994/_node/_local/_config/couchdb/max_document_size';
const headers = { Authorization: `Basic ${Buffer.from('bench:benchmark-only-password').toString('base64')}`, 'Content-Type': 'application/json' };
const previousResponse = await fetch(configUri, { headers });
assert(previousResponse.ok || previousResponse.status === 404);
const previous = previousResponse.ok ? await previousResponse.json() : undefined;
const manifest = JSON.parse(await fs.readFile(corpusPath, 'utf8'));
manifest.stages = manifest.stages.filter((stage) => stage.name === 'initial');
for (const stage of manifest.stages) stage.directory = resolve(dirname(corpusPath), stage.directory);
await fs.mkdir(output, { recursive: true });
const initialManifest = join(output, 'initial-only-manifest.json');
await fs.writeFile(initialManifest, JSON.stringify(manifest, null, 2));
const record = { scope: 'CouchDB with a 1,000,000-byte document limit, modelling the published Cloudant document-size constraint. The IBM Cloudant service itself is not measured.', source: 'https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-limits', limitBytes: 1000000, publishedRequestLimitBytes: 64000000, variant: process.env.LIVESYNC_BENCH_VARIANT || 'unlabelled', previousConfiguredValue: previous ?? null };
try {
const configured = await fetch(configUri, { method: 'PUT', headers, body: JSON.stringify(String(record.limitBytes)) });
assert(configured.ok, `Could not set fixture document limit: ${configured.status}`);
const child = spawn(process.execPath, [join(root, 'utils/chunk-size-benchmark/remote.mjs'), '--output', output, '--corpus', initialManifest, '--providers', 'couchdb', '--sizes', '4,10', '--repeats', '1'], {
cwd: root, stdio: ['ignore', 'pipe', 'pipe'], env: { ...process.env, NODE_OPTIONS: '--max-old-space-size=3072' },
});
let log = ''; child.stdout.on('data', (data) => { log += data; process.stdout.write(data); }); child.stderr.on('data', (data) => { log += data; });
record.exitCode = await new Promise((accept, reject) => { child.once('error', reject); child.once('close', accept); });
await fs.writeFile(join(output, 'run.log'), log);
const measured = JSON.parse(await fs.readFile(join(output, 'remote/results-rtt0.json'), 'utf8'));
record.cases = measured.results.map((entry) => ({ name: entry.name, customChunkSize: entry.customChunkSize, complete: entry.complete, error: entry.error, stages: entry.stages.map((stage) => ({ name: stage.name, largestDocumentJsonBytes: stage.remote.largestDocumentJsonBytes, maxUploadRequestBodyBytes: stage.upload.http.maxRequestBodyBytes, verifiedFiles: stage.verifiedFiles })) }));
assert(record.cases.some((entry) => entry.customChunkSize === 4 && entry.complete), 'The 500 KiB candidate must complete under the modelled limit');
assert(record.cases.some((entry) => entry.customChunkSize === 10 && !entry.complete), 'The 1,100 KiB candidate must hit the modelled document limit');
const rejectionLog = await fs.readFile(join(output, 'remote/couchdb-c10-r1/logs/initial-upload.txt'), 'utf8');
record.sizeRejectionObserved = /413|document_too_large|document.*too large|too_large|request entity too large/iu.test(rejectionLog);
assert(record.sizeRejectionObserved, 'The rejection log must identify the document-size limit');
} finally {
const restored = await fetch(configUri, previous === undefined ? { method: 'DELETE', headers } : { method: 'PUT', headers, body: JSON.stringify(previous) });
record.configurationRestored = restored.ok;
await fs.writeFile(join(output, 'constraint-result.json'), JSON.stringify(record, null, 2));
assert(record.configurationRestored, 'Fixture document limit restoration failed');
}
console.log(JSON.stringify(record));
utils/chunk-size-benchmark/compose.yml
+30
View File
@@ -0,0 +1,30 @@
services:
relay:
image: ghcr.io/hoytech/strfry@sha256:36f1886d185a88ca57c66ebe52e6e9e8428dac2486eea0a5d50ff934f18b60c3
command: ["--config", "/etc/strfry.conf", "relay"]
volumes:
- ../../test/fixtures/p2p-relay/strfry.conf:/etc/strfry.conf:ro
tmpfs:
- /app/strfry-db:rw,size=128m,mode=1777
ports:
- "127.0.0.1:14094:7777"
mem_limit: 256m
couchdb:
image: couchdb:3.5.0@sha256:1c0a30a54535377bc9ae23f42efb1ca4e92abe796f78ec98b768ac1d5874cc1b
environment:
COUCHDB_USER: bench
COUCHDB_PASSWORD: benchmark-only-password
COUCHDB_SINGLE_NODE: "true"
ports:
- "127.0.0.1:15994:5984"
mem_limit: 768m
object-storage:
image: rustfs/rustfs:1.0.0-rc.6@sha256:97171b3d72cd47dc81000f92ea84de25608bfc35a94c965501afaeb5d99f6035
environment:
RUSTFS_ACCESS_KEY: bench-access-key
RUSTFS_SECRET_KEY: benchmark-only-secret-key
RUSTFS_CONSOLE_ENABLE: "false"
command: ["/data"]
ports:
- "127.0.0.1:19094:9000"
mem_limit: 1g
utils/chunk-size-benchmark/corpus.mjs
+310
View File
@@ -0,0 +1,310 @@
import { createHash } from "node:crypto";
import { copyFile, mkdir, readFile, readdir, stat, utimes, writeFile } from "node:fs/promises";
import path from "node:path";
import { fileURLToPath } from "node:url";
const MODULE_DIR = path.dirname(fileURLToPath(import.meta.url));
const DEFAULT_REPO_ROOT = path.resolve(MODULE_DIR, "../..");
const SEED = 0x4c53594e;
const KIB = 1024;
const MIB = 1024 * KIB;
const BINARY_BYTES = 8 * MIB;
const SMALL_MARKDOWN_BYTES = 512 * KIB;
const LARGE_MARKDOWN_BYTES = 5 * MIB;
const BINARY_EDIT_BYTES = 4 * KIB;
const INITIAL_MTIME = new Date("2020-01-01T00:00:00.000Z");
const STAGE_MTIMES = {
overwrite: new Date("2020-01-02T00:00:00.000Z"),
insert: new Date("2020-01-03T00:00:00.000Z"),
};
const MARKDOWN_FILES = [
"README.md",
"devs.md",
"docs/settings.md",
"docs/troubleshooting.md",
"docs/quick_setup.md",
"docs/datastructure.md",
"docs/p2p.md",
"docs/tips/fast-setup.md",
"docs/design_docs/intention_of_chunks.md",
"docs/specs_data_compression.md",
];
const SNIPPET_FILES = [
"docs/terms.md",
"docs/glossary.md",
"docs/quick_setup_ja.md",
"docs/design_docs/unicode_path_identity.md",
"docs/settings.md",
];
const CORPUS_PATHS = {
markdownSmall: "benchmark/generated-small.md",
markdownLarge: "benchmark/generated-large.md",
binary: "benchmark/random-8m.bin",
cloudantImage: "assets/cloudant-setup.png",
setupImage: "assets/quick-setup-first-setup-uri.png",
};
function sha256(bytes) {
return createHash("sha256").update(bytes).digest("hex");
}
function createPrng(seed) {
let state = seed >>> 0;
return () => {
state ^= state << 13;
state ^= state >>> 17;
state ^= state << 5;
return state >>> 0;
};
}
function makeRandomBytes(length, seed) {
const next = createPrng(seed);
const output = Buffer.allocUnsafe(length);
for (let offset = 0; offset < length; offset += 4) {
const value = next();
output[offset] = value & 0xff;
if (offset + 1 < length) output[offset + 1] = (value >>> 8) & 0xff;
if (offset + 2 < length) output[offset + 2] = (value >>> 16) & 0xff;
if (offset + 3 < length) output[offset + 3] = (value >>> 24) & 0xff;
}
return output;
}
function extractParagraphs(source) {
const withoutFrontmatter = source.replace(/^---\r?\n[\s\S]*?\r?\n---\r?\n/u, "");
return withoutFrontmatter
.replace(/\r\n/gu, "\n")
.split(/\n{2,}/u)
.map((paragraph) => paragraph.trim())
.filter((paragraph) => {
if (paragraph.length < 80) return false;
if (/^(?:#{1,6}\s|!\[|\|)/u.test(paragraph)) return false;
return true;
});
}
async function readSnippetParagraphs(repoRoot) {
const excerpts = [];
for (const file of SNIPPET_FILES) {
const source = await readFile(path.join(repoRoot, file), "utf8");
for (const paragraph of extractParagraphs(source)) {
const excerpt = Array.from(paragraph).slice(0, 1000).join("");
excerpts.push({ source: file, text: excerpt });
}
}
if (excerpts.length === 0) throw new Error("No repository Markdown snippets were available.");
return excerpts;
}
function buildGeneratedMarkdown({ targetBytes, title, variant, excerpts }) {
const lines = [
`# ${title}`,
"",
"A deterministic UTF-8 fixture contains varied passages from the project documentation.",
"",
];
let byteLength = Buffer.byteLength(lines.join("\n"), "utf8");
const profiles = ["replication", "metadata", "recovery", "integrity", "configuration", "history", "incremental", "transport"];
const unicodeLabels = ["résumé", "café", "日本語", "中文", "Δοκιμή", "ångström", "naïve", "雪", "λ", "mañana"];
let record = 0;
while (byteLength < targetBytes) {
const excerpt = excerpts[(record * 7 + variant.length) % excerpts.length];
const profile = profiles[(record * 3 + variant.length) % profiles.length];
const unicode = unicodeLabels[(record * 5 + variant.length) % unicodeLabels.length];
const tag = ((SEED ^ (record * 0x9e3779b1)) >>> 0).toString(16).padStart(8, "0");
const paragraph = [
`## Record ${String(record + 1).padStart(6, "0")} · ${profile} · ${variant}`,
"",
excerpt.text,
"",
`Fixture label ${tag} uses ${unicode}; source excerpt: ${excerpt.source}; revision ${record % 97}.`,
"",
].join("\n");
lines.push(paragraph);
byteLength += Buffer.byteLength(paragraph, "utf8");
record += 1;
}
return lines.join("\n");
}
function insertNearStart(markdown, paragraph) {
const firstGap = markdown.indexOf("\n\n");
const introGap = markdown.indexOf("\n\n", firstGap + 2);
if (firstGap < 0 || introGap < 0) throw new Error("Generated Markdown fixture has no introductory paragraph.");
const insertionPoint = introGap + 2;
return `${markdown.slice(0, insertionPoint)}${paragraph}\n\n${markdown.slice(insertionPoint)}`;
}
async function writeFileWithMtime(directory, relativePath, contents, mtime) {
const outputPath = path.join(directory, relativePath);
await mkdir(path.dirname(outputPath), { recursive: true });
await writeFile(outputPath, contents);
await utimes(outputPath, mtime, mtime);
}
async function copyStage(sourceDirectory, targetDirectory, relativePaths) {
for (const relativePath of relativePaths) {
const sourcePath = path.join(sourceDirectory, relativePath);
const targetPath = path.join(targetDirectory, relativePath);
await mkdir(path.dirname(targetPath), { recursive: true });
const sourceStat = await stat(sourcePath);
await copyFile(sourcePath, targetPath);
await utimes(targetPath, sourceStat.atime, sourceStat.mtime);
}
}
async function setMtime(directory, relativePath, mtime) {
const outputPath = path.join(directory, relativePath);
await utimes(outputPath, mtime, mtime);
}
async function listFiles(directory) {
const files = [];
async function visit(currentDirectory) {
const entries = await readdir(currentDirectory, { withFileTypes: true });
entries.sort((left, right) => left.name.localeCompare(right.name));
for (const entry of entries) {
const fullPath = path.join(currentDirectory, entry.name);
if (entry.isDirectory()) {
await visit(fullPath);
} else if (entry.isFile()) {
files.push(path.relative(directory, fullPath).split(path.sep).join("/"));
}
}
}
await visit(directory);
return files.sort();
}
async function describeStage(stageName, stageDirectory, paths) {
const files = [];
for (const relativePath of paths) {
const bytes = await readFile(path.join(stageDirectory, relativePath));
files.push({ path: relativePath, bytes: bytes.byteLength, sha256: sha256(bytes) });
}
return { name: stageName, directory: stageName, files };
}
export async function createCorpus(outDir, repoRoot = DEFAULT_REPO_ROOT) {
const outputDirectory = path.resolve(outDir);
const sourceRoot = path.resolve(repoRoot);
const initialDirectory = path.join(outputDirectory, "initial");
const overwriteDirectory = path.join(outputDirectory, "overwrite");
const insertDirectory = path.join(outputDirectory, "insert");
await mkdir(outputDirectory, { recursive: true });
const initialContents = new Map();
for (const relativePath of MARKDOWN_FILES) {
initialContents.set(relativePath, await readFile(path.join(sourceRoot, relativePath)));
}
const excerpts = await readSnippetParagraphs(sourceRoot);
initialContents.set(
CORPUS_PATHS.markdownSmall,
Buffer.from(
buildGeneratedMarkdown({
targetBytes: SMALL_MARKDOWN_BYTES,
title: "Generated Markdown · 512 KiB",
variant: "small",
excerpts,
}),
"utf8",
),
);
initialContents.set(
CORPUS_PATHS.markdownLarge,
Buffer.from(
buildGeneratedMarkdown({
targetBytes: LARGE_MARKDOWN_BYTES,
title: "Generated Markdown · 5 MiB",
variant: "large",
excerpts,
}),
"utf8",
),
);
initialContents.set(CORPUS_PATHS.binary, makeRandomBytes(BINARY_BYTES, SEED));
initialContents.set(CORPUS_PATHS.cloudantImage, await readFile(path.join(sourceRoot, "instruction_images/cloudant_1.png")));
initialContents.set(CORPUS_PATHS.setupImage, await readFile(path.join(sourceRoot, "images/quick-setup/guide-quick-setup-first-setup-uri.png")));
const relativePaths = [...initialContents.keys()].sort();
for (const [relativePath, contents] of initialContents) {
await writeFileWithMtime(initialDirectory, relativePath, contents, INITIAL_MTIME);
}
await copyStage(initialDirectory, overwriteDirectory, relativePaths);
const smallMarkdownPath = path.join(overwriteDirectory, CORPUS_PATHS.markdownSmall);
const smallMarkdownBefore = await readFile(smallMarkdownPath, "utf8");
const originalIntro = "A deterministic UTF-8 fixture contains varied passages from the project documentation.";
const changedIntro = "A deterministic UTF-8 fixture records a short content update using varied project passages.";
if (!smallMarkdownBefore.includes(originalIntro)) throw new Error("Could not locate the generated Markdown paragraph to overwrite.");
await writeFile(smallMarkdownPath, smallMarkdownBefore.replace(originalIntro, changedIntro));
const binaryPath = path.join(overwriteDirectory, CORPUS_PATHS.binary);
const initialBinary = await readFile(binaryPath);
const binaryOverwriteOffset = Math.floor(initialBinary.byteLength * 0.25);
const overwrittenBinary = Buffer.from(initialBinary);
makeRandomBytes(BINARY_EDIT_BYTES, (SEED ^ 0x00a55a5a) >>> 0).copy(overwrittenBinary, binaryOverwriteOffset);
await writeFile(binaryPath, overwrittenBinary);
const alternatePng = await readFile(path.join(sourceRoot, "instruction_images/cloudant_2.png"));
await writeFile(path.join(overwriteDirectory, CORPUS_PATHS.cloudantImage), alternatePng);
for (const relativePath of [CORPUS_PATHS.markdownSmall, CORPUS_PATHS.binary, CORPUS_PATHS.cloudantImage]) {
await setMtime(overwriteDirectory, relativePath, STAGE_MTIMES.overwrite);
}
await copyStage(overwriteDirectory, insertDirectory, relativePaths);
const insertedBinaryPath = path.join(insertDirectory, CORPUS_PATHS.binary);
const overwrittenBinaryForInsert = await readFile(insertedBinaryPath);
const binaryInsertOffset = Math.floor(overwrittenBinaryForInsert.byteLength * 0.25);
const insertedBinary = Buffer.concat([
overwrittenBinaryForInsert.subarray(0, binaryInsertOffset),
makeRandomBytes(BINARY_EDIT_BYTES, (SEED ^ 0x005aa5a5) >>> 0),
overwrittenBinaryForInsert.subarray(binaryInsertOffset),
]);
await writeFile(insertedBinaryPath, insertedBinary);
const insertedParagraphs = [
[CORPUS_PATHS.markdownSmall, "The insert stage adds record 0047, profile delta-small, while retaining Unicode labels such as 日本語 and café."],
[CORPUS_PATHS.markdownLarge, "The insert stage adds record 0913, profile delta-large, while retaining Unicode labels such as 中文 and Δοκιμή."],
];
for (const [relativePath, paragraph] of insertedParagraphs) {
const markdownPath = path.join(insertDirectory, relativePath);
const markdown = await readFile(markdownPath, "utf8");
await writeFile(markdownPath, insertNearStart(markdown, paragraph));
}
for (const relativePath of [CORPUS_PATHS.binary, ...insertedParagraphs.map(([relativePath]) => relativePath)]) {
await setMtime(insertDirectory, relativePath, STAGE_MTIMES.insert);
}
const actualPaths = await listFiles(initialDirectory);
for (const directory of [overwriteDirectory, insertDirectory]) {
const stagePaths = await listFiles(directory);
if (JSON.stringify(stagePaths) !== JSON.stringify(actualPaths)) {
throw new Error(`Corpus stage ${path.basename(directory)} has different relative paths.`);
}
}
const sourceForPath = new Map(MARKDOWN_FILES.map((file) => [file, `repo:${file}`]));
sourceForPath.set(CORPUS_PATHS.markdownSmall, "generated:utf8-markdown-512KiB-from-repository-snippets");
sourceForPath.set(CORPUS_PATHS.markdownLarge, "generated:utf8-markdown-5MiB-from-repository-snippets");
sourceForPath.set(CORPUS_PATHS.binary, `generated:seeded-high-entropy-binary-seed-${SEED.toString(16)}`);
sourceForPath.set(CORPUS_PATHS.cloudantImage, "repo:instruction_images/cloudant_1.png");
sourceForPath.set(CORPUS_PATHS.setupImage, "repo:images/quick-setup/guide-quick-setup-first-setup-uri.png");
const files = actualPaths.map((relativePath) => {
const kind = relativePath.endsWith(".md") ? "markdown" : "binary";
return { path: relativePath, kind, source: sourceForPath.get(relativePath) };
});
const stages = [
await describeStage("initial", initialDirectory, actualPaths),
await describeStage("overwrite", overwriteDirectory, actualPaths),
await describeStage("insert", insertDirectory, actualPaths),
];
const manifest = { seed: SEED, files, stages };
await writeFile(path.join(outputDirectory, "manifest.json"), `${JSON.stringify(manifest, null, 2)}\n`);
return manifest;
}
utils/chunk-size-benchmark/datachannel-metrics.cjs
+23
View File
@@ -0,0 +1,23 @@
const { writeFileSync } = require('node:fs');
const { RTCDataChannel } = require('werift');
const output = process.env.LIVESYNC_BENCH_DATACHANNEL_JSON;
if (!output) throw new Error('LIVESYNC_BENCH_DATACHANNEL_JSON is required');
const channels = new Set();
const originalSend = RTCDataChannel.prototype.send;
RTCDataChannel.prototype.send = function (...args) {
channels.add(this);
return originalSend.apply(this, args);
};
process.once('exit', () => {
const rows = [...channels].map((channel) => ({ label: channel.label, payloadBytesSent: channel.bytesSent, messagesSent: channel.messagesSent }));
const cpu = process.cpuUsage();
writeFileSync(output, JSON.stringify({
source: 'werift 0.24.4 RTCDataChannel counters; observed channels through a benchmark-only preload',
scope: 'Application DataChannel payload; includes application retransmissions, excludes SCTP/DTLS/UDP/IP overhead, transport retransmissions, and relay signalling',
payloadBytesSent: rows.reduce((sum, row) => sum + row.payloadBytesSent, 0),
messagesSent: rows.reduce((sum, row) => sum + row.messagesSent, 0),
processUsage: { userCpuMs: cpu.user / 1000, systemCpuMs: cpu.system / 1000, maxResidentSetKiB: process.resourceUsage().maxRSS },
channels: rows,
}, null, 2));
});
utils/chunk-size-benchmark/local.mjs
+596
View File
@@ -0,0 +1,596 @@
import { createHash } from "node:crypto";
import { readFile, mkdir, writeFile } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { fileURLToPath, pathToFileURL } from "node:url";
import { HashManager } from "@vrtmrz/livesync-commonlib/hashing";
import { createCorpus } from "./corpus.mjs";
const MODULE_DIR = path.dirname(fileURLToPath(import.meta.url));
const PROJECT_ROOT = path.resolve(MODULE_DIR, "../..");
const DEFAULT_SIZES = [0, 1, 4, 10, 20, 40, 60];
const DEFAULT_REPEATS = 3;
const ID_DERIVATION_KEY = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef";
const LEGACY_PASSPHRASE = "public-local-chunk-size-benchmark-fixture";
const CHUNK_MANAGER_WITHOUT_CACHE = {
getCachedChunk: () => undefined,
getChunkIDFromCache: () => false,
};
const COMMONLIB_PACKAGE_JSON = fileURLToPath(import.meta.resolve("@vrtmrz/livesync-commonlib/package.json"));
const COMMONLIB_ROOT = path.dirname(COMMONLIB_PACKAGE_JSON);
const COMMONLIB_PACKAGE = JSON.parse(await readFile(COMMONLIB_PACKAGE_JSON, "utf8"));
const { ContentSplitterRabinKarp } = await import(
pathToFileURL(path.join(COMMONLIB_ROOT, "dist/ContentSplitter/ContentSplitterRabinKarp.js")).href
);
const { prepareChunk } = await import(
pathToFileURL(path.join(COMMONLIB_ROOT, "dist/managers/EntryManager/EntryManagerImpls.js")).href
);
function sha256(bytes) {
return createHash("sha256").update(bytes).digest("hex");
}
function parseArguments(args) {
const parsed = { output: undefined, sizes: [...DEFAULT_SIZES], repeats: DEFAULT_REPEATS, help: false };
for (let index = 0; index < args.length; index += 1) {
const argument = args[index];
if (argument === "--help" || argument === "-h") {
parsed.help = true;
continue;
}
const value = args[index + 1];
if (!value || value.startsWith("--")) throw new Error(`Missing value for ${argument}.`);
if (argument === "--output") {
parsed.output = value;
} else if (argument === "--sizes") {
parsed.sizes = value.split(",").map((item) => {
const trimmed = item.trim();
if (!/^\d+$/u.test(trimmed)) throw new Error(`Invalid customChunkSize value: ${item}`);
const size = Number(trimmed);
if (!Number.isSafeInteger(size) || size > 10000) throw new Error(`customChunkSize is out of range: ${item}`);
return size;
});
if (parsed.sizes.length === 0 || new Set(parsed.sizes).size !== parsed.sizes.length) {
throw new Error("--sizes must contain one or more distinct integers.");
}
} else if (argument === "--repeats") {
if (!/^\d+$/u.test(value)) throw new Error(`Invalid repeat count: ${value}`);
parsed.repeats = Number(value);
if (!Number.isSafeInteger(parsed.repeats) || parsed.repeats < 1 || parsed.repeats > 100) {
throw new Error("--repeats must be an integer from 1 to 100.");
}
} else {
throw new Error(`Unknown argument: ${argument}`);
}
index += 1;
}
if (!parsed.help && !parsed.output) throw new Error("--output <dir> is required.");
return parsed;
}
function usage() {
return "Usage: node utils/chunk-size-benchmark/local.mjs --output <dir> [--sizes 0,1,4,10,20,40,60] [--repeats 3]";
}
function settingsFor(customChunkSize, mode) {
return {
chunkSplitterVersion: "v3-rabin-karp",
customChunkSize,
minimumChunkSize: 20,
disableWorkerForGeneratingChunks: true,
processSmallFilesInUIThread: false,
encrypt: true,
E2EEAlgorithm: "v2",
hashAlg: "xxhash64",
passphrase: mode === "legacy" ? LEGACY_PASSPHRASE : "",
idDerivationVersion: mode === "primary" ? 1 : 0,
idDerivationKey: mode === "primary" ? ID_DERIVATION_KEY : "",
enableCompression: false,
useEden: false,
};
}
async function createCondition(customChunkSize) {
const primarySettings = settingsFor(customChunkSize, "primary");
const legacySettings = settingsFor(customChunkSize, "legacy");
const primarySettingService = { currentSettings: () => primarySettings };
const legacySettingService = { currentSettings: () => legacySettings };
const splitter = new ContentSplitterRabinKarp({ settingService: primarySettingService });
const primaryHashManager = new HashManager({ settingService: primarySettingService });
const legacyHashManager = new HashManager({ settingService: legacySettingService });
await Promise.all([splitter.initialised, primaryHashManager.initialise(), legacyHashManager.initialise()]);
return { customChunkSize, splitter, primaryHashManager, legacyHashManager };
}
async function warmCondition(condition, file) {
await splitAndHashFile(condition, file, new Map(), new Map());
}
async function loadCorpus(manifest, corpusDirectory) {
const filesByPath = new Map(manifest.files.map((file) => [file.path, file]));
const loadedStages = [];
for (const stage of manifest.stages) {
const stageDirectory = path.join(corpusDirectory, stage.directory);
const loadedFiles = [];
for (const expected of stage.files) {
const metadata = filesByPath.get(expected.path);
if (!metadata) throw new Error(`Corpus manifest has no file metadata for ${expected.path}.`);
const data = await readFile(path.join(stageDirectory, expected.path));
if (data.byteLength !== expected.bytes || sha256(data) !== expected.sha256) {
throw new Error(`Corpus file does not match its manifest: ${stage.name}/${expected.path}`);
}
loadedFiles.push({ ...metadata, expectedSha256: expected.sha256, data });
}
loadedStages.push({ name: stage.name, files: loadedFiles });
}
return loadedStages;
}
function decodeChunk(kind, piece) {
return kind === "markdown" ? Buffer.from(piece, "utf8") : Buffer.from(piece, "base64");
}
function assertSameContentForId(idsToPieces, id, piece, label) {
const existing = idsToPieces.get(id);
if (existing !== undefined && existing !== piece) {
throw new Error(`${label} collision: chunk ID ${id} was produced for different content.`);
}
if (existing === undefined) idsToPieces.set(id, piece);
}
async function splitAndHashFile(condition, file, primaryIdContents, legacyIdContents) {
const blobType = file.kind === "markdown" ? "text/plain" : "application/octet-stream";
const blob = new Blob([file.data], { type: blobType });
const entry = { path: file.path, data: blob };
const primaryChunks = [];
const primaryCpuStart = process.cpuUsage();
const primaryStart = process.hrtime.bigint();
const generator = await condition.splitter.splitContent(entry);
for await (const piece of generator) {
const prepared = await prepareChunk(
{ hashManager: condition.primaryHashManager, chunkManager: CHUNK_MANAGER_WITHOUT_CACHE },
piece,
);
assertSameContentForId(primaryIdContents, prepared.id, piece, "Primary keyed hash");
primaryChunks.push({ piece, id: prepared.id });
}
const primaryElapsedMs = Number(process.hrtime.bigint() - primaryStart) / 1e6;
const primaryCpu = process.cpuUsage(primaryCpuStart);
const primaryCpuMs = (primaryCpu.user + primaryCpu.system) / 1000;
const legacyCpuStart = process.cpuUsage();
const legacyStart = process.hrtime.bigint();
for (const chunk of primaryChunks) {
const prepared = await prepareChunk(
{ hashManager: condition.legacyHashManager, chunkManager: CHUNK_MANAGER_WITHOUT_CACHE },
chunk.piece,
);
assertSameContentForId(legacyIdContents, prepared.id, chunk.piece, "Legacy encrypted xxhash64");
chunk.legacyId = prepared.id;
}
const legacyIdElapsedMs = Number(process.hrtime.bigint() - legacyStart) / 1e6;
const legacyCpu = process.cpuUsage(legacyCpuStart);
const legacyIdCpuMs = (legacyCpu.user + legacyCpu.system) / 1000;
const decoded = primaryChunks.map((chunk) => decodeChunk(file.kind, chunk.piece));
const reconstructed = Buffer.concat(decoded);
if (!reconstructed.equals(file.data) || sha256(reconstructed) !== file.expectedSha256) {
throw new Error(`Splitter output did not reconstruct ${file.path} in the ${file.kind} encoding.`);
}
const chunks = primaryChunks.map((chunk, index) => {
const raw = decoded[index];
return {
id: chunk.id,
legacyId: chunk.legacyId,
rawBytes: raw.byteLength,
storedUtf8Bytes: Buffer.byteLength(chunk.piece, "utf8"),
dataSha256: sha256(raw),
idUtf8Bytes: Buffer.byteLength(chunk.id, "utf8"),
legacyIdUtf8Bytes: Buffer.byteLength(chunk.legacyId, "utf8"),
piece: chunk.piece,
};
});
const fingerprintInput = chunks.map(({ id, legacyId, rawBytes, dataSha256 }) => [id, legacyId, rawBytes, dataSha256]);
const chunkFingerprintSha256 = sha256(Buffer.from(JSON.stringify(fingerprintInput), "utf8"));
return {
chunks,
primaryElapsedMs,
primaryCpuMs,
legacyIdElapsedMs,
legacyIdCpuMs,
chunkFingerprintSha256,
};
}
function percentile(sortedValues, fraction) {
if (sortedValues.length === 0) return 0;
return sortedValues[Math.max(0, Math.ceil(sortedValues.length * fraction) - 1)];
}
function summarizeRawChunkBytes(values) {
const sorted = [...values].sort((left, right) => left - right);
return {
count: sorted.length,
p50: percentile(sorted, 0.5),
p90: percentile(sorted, 0.9),
p95: percentile(sorted, 0.95),
p99: percentile(sorted, 0.99),
max: sorted.at(-1) ?? 0,
};
}
function summariseIdLengths(chunks, key) {
const lengths = chunks.map((chunk) => Buffer.byteLength(chunk[key], "utf8")).sort((left, right) => left - right);
const uniqueLengths = new Map();
for (const chunk of chunks) uniqueLengths.set(chunk[key], Buffer.byteLength(chunk[key], "utf8"));
const uniqueValues = [...uniqueLengths.values()].sort((left, right) => left - right);
return {
occurrences: lengths.length,
totalUtf8Bytes: lengths.reduce((sum, value) => sum + value, 0),
minLengthBytes: lengths[0] ?? 0,
medianLengthBytes: percentile(lengths, 0.5),
maxLengthBytes: lengths.at(-1) ?? 0,
uniqueIds: uniqueValues.length,
uniqueTotalUtf8Bytes: uniqueValues.reduce((sum, value) => sum + value, 0),
};
}
function uniqueById(chunks, idKey = "id") {
const unique = new Map();
for (const chunk of chunks) {
if (!unique.has(chunk[idKey])) unique.set(chunk[idKey], chunk);
}
return unique;
}
function dataBytes(chunks, key) {
return chunks.reduce((sum, chunk) => sum + chunk[key], 0);
}
function summarizeFile(file, measured, includeChunkArray) {
const { chunks } = measured;
const unique = uniqueById(chunks);
const fileMetric = {
path: file.path,
kind: file.kind,
sourceBytes: file.data.byteLength,
chunkCount: chunks.length,
uniqueChunkCount: unique.size,
storedChunkDataUtf8Bytes: dataBytes(chunks, "storedUtf8Bytes"),
uniqueChunkDataUtf8Bytes: dataBytes([...unique.values()], "storedUtf8Bytes"),
uniqueChunkRawBytes: dataBytes([...unique.values()], "rawBytes"),
rawChunkBytes: summarizeRawChunkBytes(chunks.map((chunk) => chunk.rawBytes)),
primaryId: summariseIdLengths(chunks, "id"),
legacyId: summariseIdLengths(chunks, "legacyId"),
chunkFingerprintSha256: measured.chunkFingerprintSha256,
timing: {
primarySplitAndIdElapsedMs: measured.primaryElapsedMs,
primarySplitAndIdCpuMs: measured.primaryCpuMs,
legacyIdElapsedMs: measured.legacyIdElapsedMs,
legacyIdCpuMs: measured.legacyIdCpuMs,
},
};
if (includeChunkArray) {
fileMetric.chunks = chunks.map(({ id, legacyId, rawBytes, storedUtf8Bytes, dataSha256, idUtf8Bytes, legacyIdUtf8Bytes }) => ({
id,
legacyId,
rawBytes,
storedUtf8Bytes,
dataSha256,
idUtf8Bytes,
legacyIdUtf8Bytes,
}));
}
return fileMetric;
}
function aggregateKind(stage, kind, fileResults, historyIds, previousIds) {
const files = fileResults.filter((item) => item.file.kind === kind);
const chunks = files.flatMap((item) => item.measured.chunks);
const unique = uniqueById(chunks);
const uniqueChunks = [...unique.values()];
const newVsHistory = uniqueChunks.filter((chunk) => !historyIds.has(chunk.id));
const newVsPrevious = previousIds === null ? null : uniqueChunks.filter((chunk) => !previousIds.has(chunk.id));
const storedBytes = dataBytes(chunks, "storedUtf8Bytes");
const primaryIds = summariseIdLengths(chunks, "id");
const legacyIds = summariseIdLengths(chunks, "legacyId");
return {
kind,
fileCount: files.length,
sourceBytes: files.reduce((sum, item) => sum + item.file.data.byteLength, 0),
chunkCount: chunks.length,
uniqueChunkCount: unique.size,
storedChunkDataUtf8Bytes: storedBytes,
uniqueChunkDataUtf8Bytes: dataBytes(uniqueChunks, "storedUtf8Bytes"),
uniqueChunkRawBytes: dataBytes(uniqueChunks, "rawBytes"),
rawChunkBytes: summarizeRawChunkBytes(chunks.map((chunk) => chunk.rawBytes)),
primaryId: primaryIds,
legacyId: legacyIds,
newUniqueChunksVsHistory: newVsHistory.length,
newUniqueChunkDataUtf8BytesVsHistory: dataBytes(newVsHistory, "storedUtf8Bytes"),
newUniqueChunkRawBytesVsHistory: dataBytes(newVsHistory, "rawBytes"),
newUniqueChunksVsPreviousStage: newVsPrevious === null ? null : newVsPrevious.length,
newUniqueChunkDataUtf8BytesVsPreviousStage: newVsPrevious === null ? null : dataBytes(newVsPrevious, "storedUtf8Bytes"),
newUniqueChunkRawBytesVsPreviousStage: newVsPrevious === null ? null : dataBytes(newVsPrevious, "rawBytes"),
primaryElapsedMs: files.reduce((sum, item) => sum + item.measured.primaryElapsedMs, 0),
primaryCpuMs: files.reduce((sum, item) => sum + item.measured.primaryCpuMs, 0),
legacyIdElapsedMs: files.reduce((sum, item) => sum + item.measured.legacyIdElapsedMs, 0),
legacyIdCpuMs: files.reduce((sum, item) => sum + item.measured.legacyIdCpuMs, 0),
};
}
async function runCondition(condition, stages, repeat, order, includeChunkArray) {
const primaryIdContents = new Map();
const legacyIdContents = new Map();
const historyIds = new Set();
let previousIds = null;
const stageResults = [];
for (const stage of stages) {
const fileResults = [];
for (const file of stage.files) {
const measured = await splitAndHashFile(condition, file, primaryIdContents, legacyIdContents);
fileResults.push({ file, measured });
}
const currentIds = new Set();
for (const item of fileResults) {
for (const chunk of item.measured.chunks) currentIds.add(chunk.id);
}
const kinds = [...new Set(fileResults.map((item) => item.file.kind))].sort();
const byKind = kinds.map((kind) => aggregateKind(stage, kind, fileResults, historyIds, previousIds));
const stageFingerprint = fileResults
.map((item) => `${item.file.path}\0${item.measured.chunkFingerprintSha256}`)
.sort()
.join("\n");
stageResults.push({
name: stage.name,
fingerprintSha256: sha256(Buffer.from(stageFingerprint, "utf8")),
byKind,
files: fileResults.map((item) => summarizeFile(item.file, item.measured, includeChunkArray)),
});
for (const id of currentIds) historyIds.add(id);
previousIds = currentIds;
}
return { repeat, conditionOrder: order, stages: stageResults };
}
function median(values) {
const sorted = [...values].sort((left, right) => left - right);
if (sorted.length === 0) return 0;
const middle = Math.floor(sorted.length / 2);
return sorted.length % 2 === 0 ? (sorted[middle - 1] + sorted[middle]) / 2 : sorted[middle];
}
function orderForRepeat(sizes, repeat) {
const offset = repeat % sizes.length;
return [...sizes.slice(offset), ...sizes.slice(0, offset)];
}
function timingsForRuns(runs, stageName, kind) {
const samples = runs.map((run) => {
const stage = run.stages.find((item) => item.name === stageName);
const group = stage.byKind.find((item) => item.kind === kind);
return {
repeat: run.repeat,
primaryElapsedMs: group.primaryElapsedMs,
primaryCpuMs: group.primaryCpuMs,
legacyIdElapsedMs: group.legacyIdElapsedMs,
legacyIdCpuMs: group.legacyIdCpuMs,
};
});
return {
primarySplitAndIdElapsedMsMedian: median(samples.map((sample) => sample.primaryElapsedMs)),
primarySplitAndIdCpuMsMedian: median(samples.map((sample) => sample.primaryCpuMs)),
legacyIdElapsedMsMedian: median(samples.map((sample) => sample.legacyIdElapsedMs)),
legacyIdCpuMsMedian: median(samples.map((sample) => sample.legacyIdCpuMs)),
repeats: samples,
};
}
function timingsForFile(runs, stageName, filePath) {
const samples = runs.map((run) => {
const stage = run.stages.find((item) => item.name === stageName);
const file = stage.files.find((item) => item.path === filePath);
return { repeat: run.repeat, ...file.timing };
});
return {
primarySplitAndIdElapsedMsMedian: median(samples.map((sample) => sample.primarySplitAndIdElapsedMs)),
primarySplitAndIdCpuMsMedian: median(samples.map((sample) => sample.primarySplitAndIdCpuMs)),
legacyIdElapsedMsMedian: median(samples.map((sample) => sample.legacyIdElapsedMs)),
legacyIdCpuMsMedian: median(samples.map((sample) => sample.legacyIdCpuMs)),
repeats: samples,
};
}
function assertRepeatStability(runs, customChunkSize) {
const first = runs[0];
for (const run of runs.slice(1)) {
for (const stage of first.stages) {
const currentStage = run.stages.find((item) => item.name === stage.name);
if (currentStage.fingerprintSha256 !== stage.fingerprintSha256) {
throw new Error(`Chunk boundaries or IDs changed across repeats for customChunkSize=${customChunkSize}, stage=${stage.name}.`);
}
for (const file of stage.files) {
const currentFile = currentStage.files.find((item) => item.path === file.path);
if (currentFile.chunkFingerprintSha256 !== file.chunkFingerprintSha256) {
throw new Error(`Chunk IDs changed across repeats for ${stage.name}/${file.path} at customChunkSize=${customChunkSize}.`);
}
}
}
}
}
function compareCandidateBoundaries(candidates, sizes) {
const candidate40 = candidates.get(40);
const candidate60 = candidates.get(60);
if (!sizes.includes(40) || !sizes.includes(60)) return null;
for (const stage of candidate40.stages) {
const otherStage = candidate60.stages.find((item) => item.name === stage.name);
for (const file of stage.files) {
const other = otherStage.files.find((item) => item.path === file.path);
if (file.chunkFingerprintSha256 !== other.chunkFingerprintSha256) {
throw new Error(`customChunkSize=40 and 60 differ for ${stage.name}/${file.path}.`);
}
}
}
return {
customChunkSize40And60SamePerFileBoundariesAndIds: true,
comparedStages: candidate40.stages.length,
comparedFilesPerStage: candidate40.stages[0]?.files.length ?? 0,
};
}
async function main() {
const args = parseArguments(process.argv.slice(2));
if (args.help) {
process.stdout.write(`${usage()}\n`);
return;
}
const outputDirectory = path.resolve(args.output);
const corpusDirectory = path.join(outputDirectory, "corpus");
const resultPath = path.join(outputDirectory, "results.json");
await mkdir(outputDirectory, { recursive: true });
const startedAt = new Date().toISOString();
if (COMMONLIB_PACKAGE.version !== "0.1.44") {
throw new Error(`Expected @vrtmrz/livesync-commonlib 0.1.44, found ${COMMONLIB_PACKAGE.version}.`);
}
const manifest = await createCorpus(corpusDirectory, PROJECT_ROOT);
const manifestJson = JSON.stringify(manifest);
const corpusManifestSha256 = sha256(Buffer.from(manifestJson, "utf8"));
const stages = await loadCorpus(manifest, corpusDirectory);
const packageJson = JSON.parse(await readFile(path.join(PROJECT_ROOT, "package.json"), "utf8"));
if (ID_DERIVATION_KEY.length !== 64 || !/^[0-9a-f]{64}$/u.test(ID_DERIVATION_KEY)) {
throw new Error("The benchmark ID derivation fixture must be 64 lower-case hex characters.");
}
const conditions = new Map();
for (const size of args.sizes) {
const condition = await createCondition(size);
const warmupFile = stages[0].files.find((file) => file.kind === "markdown" && file.source.startsWith("repo:"));
if (!warmupFile) throw new Error("Corpus has no repository Markdown file for warm-up.");
await warmCondition(condition, warmupFile);
conditions.set(size, condition);
}
const runsBySize = new Map(args.sizes.map((size) => [size, []]));
for (let repeat = 0; repeat < args.repeats; repeat += 1) {
const order = orderForRepeat(args.sizes, repeat);
for (const size of order) {
const includeChunkArray = repeat === 0 && (size === 40 || size === 60);
const run = await runCondition(conditions.get(size), stages, repeat + 1, order, includeChunkArray);
runsBySize.get(size).push(run);
}
}
const candidates = new Map();
for (const size of args.sizes) {
const runs = runsBySize.get(size);
assertRepeatStability(runs, size);
const firstRun = runs[0];
const detailedStages = firstRun.stages.map((stage) => ({
name: stage.name,
fingerprintSha256: stage.fingerprintSha256,
byKind: stage.byKind.map((group) => {
const { primaryElapsedMs, primaryCpuMs, legacyIdElapsedMs, legacyIdCpuMs, ...metrics } = group;
return {
...metrics,
timing: timingsForRuns(runs, stage.name, group.kind),
};
}),
files: stage.files.map(({ timing, ...file }) => ({
...file,
timing: timingsForFile(runs, stage.name, file.path),
})),
}));
const candidate = {
customChunkSize: size,
chunkSizeCeilingBytes: Math.floor(102400 * (size + 1)),
stages: detailedStages,
runs: runs.map((run) => ({
repeat: run.repeat,
conditionOrder: run.conditionOrder,
stages: run.stages.map((stage) => ({ name: stage.name, fingerprintSha256: stage.fingerprintSha256 })),
})),
};
candidates.set(size, candidate);
}
const comparisons = compareCandidateBoundaries(candidates, args.sizes);
const results = {
schemaVersion: 1,
benchmark: "local Rabin-Karp splitting and chunk-ID measurement",
startedAt,
completedAt: new Date().toISOString(),
runtime: {
node: process.version,
v8: process.versions.v8,
platform: process.platform,
architecture: process.arch,
osRelease: os.release(),
cpuModel: os.cpus()[0]?.model ?? "unknown",
logicalCpuCount: os.cpus().length,
projectVersion: packageJson.version,
commonlibVersion: COMMONLIB_PACKAGE.version,
variant: process.env.LIVESYNC_BENCH_VARIANT || "unlabelled",
commonlibChunksSha256: sha256(await readFile(path.join(COMMONLIB_ROOT, "dist/string_and_binary/chunks.js"))),
},
settings: {
splitter: "v3-rabin-karp",
minimumChunkSize: 20,
workerDisabled: true,
E2EEAlgorithm: "v2",
encrypt: true,
hashAlg: "xxhash64",
primaryIdDerivation: {
version: 1,
keyFixture: "fixed public benchmark fixture; 64 lower-case hex characters omitted",
},
legacyIdComparison: {
idDerivationVersion: 0,
algorithm: "encrypted xxhash64 with the fixed public legacy passphrase fixture",
passphrase: "omitted",
},
enableCompression: false,
useEden: false,
customChunkSizeValues: args.sizes,
warmup: "one actual repository Markdown file per condition; excluded from timings",
timings: "primary includes splitting and actual EntryManager prepareChunk hashing; legacy timing hashes the same emitted chunks separately",
},
corpus: {
seed: manifest.seed,
manifestSha256: corpusManifestSha256,
fileCount: manifest.files.length,
stages: manifest.stages.map((stage) => ({
name: stage.name,
directory: stage.directory,
fileCount: stage.files.length,
sourceBytes: stage.files.reduce((sum, file) => sum + file.bytes, 0),
})),
},
assertions: {
everyFileReconstructedFromSplitterOutput: true,
duplicateIdsHadIdenticalChunkContent: true,
sameBoundariesAndIdsAcrossRepeats: true,
...(comparisons ?? {}),
},
candidates: [...candidates.values()],
scope: "Local splitting and hashing only; no network or request timing is measured.",
};
await writeFile(resultPath, `${JSON.stringify(results, null, 2)}\n`);
process.stdout.write(`${JSON.stringify({ resultPath, corpusManifestSha256, candidateCount: args.sizes.length, repeats: args.repeats })}\n`);
}
main().catch((error) => {
process.stderr.write(`${error instanceof Error ? error.stack ?? error.message : String(error)}\n`);
process.exitCode = 1;
});
utils/chunk-size-benchmark/p2p.mjs
+159
View File
@@ -0,0 +1,159 @@
import assert from 'node:assert/strict';
import { createHash } from 'node:crypto';
import { spawn } from 'node:child_process';
import { promises as fs } from 'node:fs';
import { dirname, join, resolve } from 'node:path';
import { fileURLToPath } from 'node:url';
import { setTimeout as delay } from 'node:timers/promises';
import { createNewVaultSettings, REMOTE_P2P, SALT_OF_PASSPHRASE } from '@vrtmrz/livesync-commonlib/compat/common/types';
import { encryptString } from '@vrtmrz/livesync-commonlib/compat/encryption/stringEncryption';
const root = resolve(dirname(fileURLToPath(import.meta.url)), '../..');
const cli = join(root, 'src/apps/cli/dist/index.cjs');
const preload = join(root, 'utils/chunk-size-benchmark/datachannel-metrics.cjs');
const args = process.argv.slice(2);
const option = (key, fallback) => { const at = args.indexOf(`--${key}`); return at < 0 ? fallback : args[at + 1]; };
const sourceCli = resolve(option('source-cli', cli));
const targetCli = resolve(option('target-cli', cli));
const output = resolve(option('output', join(root, '.bench/chunk-size')));
const corpusPath = resolve(option('corpus', join(output, 'corpus/manifest.json')));
const sizes = option('sizes', '0,4,10,40').split(',').map(Number);
const repeats = Number(option('repeats', '3'));
const sha256 = (bytes) => createHash('sha256').update(bytes).digest('hex');
const cliSha256 = sha256(await fs.readFile(cli));
const sourceCliSha256 = sha256(await fs.readFile(sourceCli));
const targetCliSha256 = sha256(await fs.readFile(targetCli));
const environment = { ...process.env, NODE_OPTIONS: '--max-old-space-size=2048' };
async function runCli(vault, settings, command, commandArgs, logPath, extraEnvironment = {}, entrypoint = cli) {
const marker = '__BENCH_TIME__';
const started = performance.now();
const nodeArgs = extraEnvironment.LIVESYNC_BENCH_DATACHANNEL_JSON ? ['--require', preload] : [];
const child = spawn('/usr/bin/time', ['-f', `${marker}%U\t%S\t%M`, process.execPath, ...nodeArgs, entrypoint, vault, '--settings', settings, command, ...commandArgs], {
cwd: root, detached: true, stdio: ['ignore', 'pipe', 'pipe'], env: { ...environment, ...extraEnvironment },
});
let stdout = ''; let stderr = '';
child.stdout.on('data', (data) => { stdout += data; });
child.stderr.on('data', (data) => { stderr += data; });
const timeout = setTimeout(() => { try { process.kill(-child.pid, 'SIGTERM'); } catch (error) { if (error.code !== 'ESRCH') throw error; } }, 240000);
const code = await new Promise((accept, reject) => { child.once('error', reject); child.once('close', accept); });
clearTimeout(timeout);
await fs.writeFile(logPath, `${stdout}\n${stderr}`);
if (code !== 0) throw new Error(`CLI ${command} exited ${code}; ${logPath}\n${stderr.slice(-2000)}`);
const line = stderr.split(/\r?\n/).find((row) => row.startsWith(marker));
assert(line, 'GNU time output is missing');
const [userSeconds, systemSeconds, maxResidentSetKiB] = line.slice(marker.length).split('\t').map(Number);
return { elapsedMs: performance.now() - started, userCpuMs: userSeconds * 1000, systemCpuMs: systemSeconds * 1000, maxResidentSetKiB };
}
async function startHost(vault, settings, logPath, metricsPath) {
const child = spawn(process.execPath, ['--require', preload, sourceCli, vault, '--settings', settings, 'p2p-host'], { cwd: root, stdio: ['ignore', 'pipe', 'pipe'], env: { ...environment, LIVESYNC_BENCH_DATACHANNEL_JSON: metricsPath } });
let combined = ''; let ended = false; let failure;
const closed = new Promise((accept) => { child.once('error', (error) => { failure = error; ended = true; accept(); }); child.once('close', () => { ended = true; accept(); }); });
child.stdout.on('data', (data) => { combined += data; }); child.stderr.on('data', (data) => { combined += data; });
const stop = async () => {
if (!ended) child.kill('SIGTERM');
const timeout = setTimeout(() => { if (!ended) child.kill('SIGKILL'); }, 10000);
await closed; clearTimeout(timeout); await fs.writeFile(logPath, combined);
};
const started = performance.now();
try {
while (!combined.includes('P2P host is running')) {
if (ended || performance.now() - started > 25000) throw failure || new Error(`P2P host did not start: ${combined.slice(-2500)}`);
await delay(100);
}
} catch (error) { await stop(); throw error; }
return { stop, readyMs: performance.now() - started };
}
async function settingsFor(size, room, name) {
return {
...createNewVaultSettings(), isConfigured: true, remoteType: REMOTE_P2P,
encrypt: true, passphrase: 'chunk-size-benchmark-only-passphrase', usePathObfuscation: true, encryptInternalMetadata: true,
idDerivationVersion: 1, idDerivationKey: '', encryptedIdDerivationKey: await encryptString('0123456789abcdef'.repeat(4), '*' + SALT_OF_PASSPHRASE),
E2EEAlgorithm: 'v2', hashAlg: 'xxhash64', chunkSplitterVersion: 'v3-rabin-karp', minimumChunkSize: 20, customChunkSize: size,
enableCompression: false, useEden: false, useIndexedDBAdapter: false, handleFilenameCaseSensitive: false, pathNormalizationVersion: 0,
autoAcceptCompatibleTweak: false, readChunksOnline: false, liveSync: false, syncOnStart: false, syncOnSave: false, periodicReplication: false,
syncOnFileOpen: false, syncMaxSizeInMB: 50,
P2P_Enabled: true, P2P_AutoStart: false, P2P_AutoBroadcast: false, P2P_IsHeadless: true,
P2P_AppID: 'livesync-chunk-size-benchmark', P2P_roomID: room, P2P_passphrase: 'chunk-size-local-relay-fixture',
P2P_relays: 'ws://127.0.0.1:14094/', P2P_AutoAcceptingPeers: '~.*', P2P_AutoDenyingPeers: '',
P2P_turnServers: '', P2P_turnUsername: '', P2P_turnCredential: '', P2P_DevicePeerName: name,
};
}
async function copyStage(stage, vault, previous) {
const old = new Map((previous?.files || []).map((file) => [file.path, file.sha256]));
const changed = [];
for (const file of stage.files) {
if (old.get(file.path) === file.sha256) continue;
const from = join(stage.directory, file.path); const to = join(vault, file.path);
await fs.mkdir(dirname(to), { recursive: true }); await fs.copyFile(from, to);
const stat = await fs.stat(from); await fs.utimes(to, stat.atime, stat.mtime); changed.push(file.path);
}
return changed;
}
await fs.mkdir(join(output, 'p2p'), { recursive: true });
const manifest = JSON.parse(await fs.readFile(corpusPath, 'utf8'));
for (const stage of manifest.stages) stage.directory = resolve(dirname(corpusPath), stage.directory);
const corpusDigest = sha256(JSON.stringify(manifest.stages.map((stage) => ({ name: stage.name, files: stage.files }))));
const resultPath = join(output, 'p2p/results.json');
let results = [];
try { const saved = JSON.parse(await fs.readFile(resultPath, 'utf8')); assert.equal(saved.corpusDigest, corpusDigest); assert.equal(saved.sourceCliSha256 ?? saved.cliSha256, sourceCliSha256); assert.equal(saved.targetCliSha256 ?? saved.cliSha256, targetCliSha256); results = saved.results; } catch (error) { if (error.code !== 'ENOENT') throw error; }
const save = () => fs.writeFile(resultPath, JSON.stringify({ version: '1.0.39', commonlib: '0.1.44', variant: process.env.LIVESYNC_BENCH_VARIANT || 'unlabelled', cliSha256, sourceCliSha256, targetCliSha256, corpusDigest,
scope: 'Local WebRTC direct path; timed target command includes process start-up, peer discovery, connection establishment, pull and reverse synchronisation. DataChannel payload bytes are measured using werift counters observed through a preload on both endpoints; they exclude transport overhead, transport retransmissions, and relay signalling. ICE byte counters are retained as diagnostics and report zero in this runtime. GNU time covers the target process; the preload also records CPU/RSS on each endpoint. Source mirroring and host start-up are recorded separately. E2EE V2 settings with independent IDs; compression disabled.', results }, null, 2));
for (let repeat = 0; repeat < repeats; repeat++) {
const rotation = repeat % sizes.length; const order = [...sizes.slice(rotation), ...sizes.slice(0, rotation)];
for (const size of order) {
const name = `p2p-c${size}-r${repeat + 1}`;
if (results.some((result) => result.name === name && result.complete)) continue;
const work = join(output, 'p2p', name); await fs.mkdir(join(work, 'logs'), { recursive: true });
const a = join(work, 'source'); const b = join(work, 'target');
await fs.rm(a, { recursive: true, force: true }); await fs.rm(b, { recursive: true, force: true });
await fs.mkdir(a, { recursive: true }); await fs.mkdir(b, { recursive: true });
const room = `chunk-size-${Date.now().toString(36)}-${name}`;
const hostName = `host-${name}`;
const settingsA = join(work, 'source-settings.json'); const settingsB = join(work, 'target-settings.json');
await fs.writeFile(settingsA, JSON.stringify(await settingsFor(size, room, hostName)));
await fs.writeFile(settingsB, JSON.stringify(await settingsFor(size, room, `target-${name}`)));
const result = { name, customChunkSize: size, repeat: repeat + 1, stages: [], complete: false };
results.push(result); await save();
let previous;
try {
for (const stage of manifest.stages) {
const log = (label) => join(work, 'logs', `${stage.name}-${label}.txt`);
const changedFiles = await copyStage(stage, a, previous);
const mirror = await runCli(a, settingsA, 'mirror', [], log('mirror'), {}, sourceCli);
const hostMetricsPath = join(work, `${stage.name}-host-datachannel.json`);
const targetMetricsPath = join(work, `${stage.name}-target-datachannel.json`);
const host = await startHost(a, settingsA, log('host'), hostMetricsPath);
const statsPath = join(work, `${stage.name}-stats.jsonl`); await fs.rm(statsPath, { force: true });
let sync;
try { sync = await runCli(b, settingsB, 'p2p-sync', [hostName, '120'], log('sync'), { LIVESYNC_P2P_STATS_JSONL: statsPath, LIVESYNC_BENCH_DATACHANNEL_JSON: targetMetricsPath }, targetCli); }
finally { await host.stop(); }
const rows = (await fs.readFile(statsPath, 'utf8')).trim().split(/\r?\n/);
const stats = JSON.parse(rows.at(-1));
assert(stats.candidatePathCollected, 'ICE statistics were not collected');
assert(!stats.selectedPath.includes('relay'), `Unexpected TURN path: ${stats.selectedPath}`);
const dataChannel = { host: JSON.parse(await fs.readFile(hostMetricsPath, 'utf8')), target: JSON.parse(await fs.readFile(targetMetricsPath, 'utf8')) };
assert(dataChannel.host.payloadBytesSent > 0 && dataChannel.target.payloadBytesSent > 0, 'DataChannel counters must record both directions');
const materialisedByPull = [];
for (const file of stage.files) {
let bytes; try { bytes = await fs.readFile(join(b, file.path)); } catch (error) { if (error.code !== 'ENOENT') throw error; }
if (!bytes || sha256(bytes) !== file.sha256) {
const to = join(work, 'verification', file.path); await fs.mkdir(dirname(to), { recursive: true });
const measurement = await runCli(b, settingsB, 'pull', [file.path, to], log(`verify-${file.path.replace(/[^a-zA-Z0-9]/g, '-')}`), {}, targetCli);
bytes = await fs.readFile(to); materialisedByPull.push({ path: file.path, measurement });
}
assert.equal(sha256(bytes), file.sha256, `${name} ${stage.name} ${file.path}`);
}
result.stages.push({ name: stage.name, changedFiles, mirror, hostReadyMs: host.readyMs, sync, stats, dataChannel, verifiedFiles: stage.files.length, materialisedByPull });
await save(); console.log(JSON.stringify({ case: name, stage: stage.name, syncMs: Math.round(sync.elapsedMs), selectedPath: stats.selectedPath, hostPayloadBytesSent: dataChannel.host.payloadBytesSent, targetPayloadBytesSent: dataChannel.target.payloadBytesSent }));
previous = stage;
}
result.complete = true; await save();
} catch (error) { result.error = String(error); await save(); throw error; }
}
}
console.log(`Completed P2P measurements: ${resultPath}`);
utils/chunk-size-benchmark/rabin-window.test.mjs
+20
View File
@@ -0,0 +1,20 @@
import assert from 'node:assert/strict';
import { dirname, join } from 'node:path';
import { fileURLToPath, pathToFileURL } from 'node:url';
import test from 'node:test';
const packageRoot = dirname(fileURLToPath(import.meta.resolve('@vrtmrz/livesync-commonlib/package.json')));
const { splitPiecesRabinKarp } = await import(pathToFileURL(join(packageRoot, 'dist/string_and_binary/chunks.js')).href);
test('zero windows lose the contribution of a marker in the preceding chunk', async () => {
const blockBytes = 300 * 1024;
const input = new Uint8Array(blockBytes * 3);
for (let end = blockBytes; end <= input.length; end += blockBytes) input[end - 1] = 1;
const pieces = await splitPiecesRabinKarp(new Blob([input], { type: 'application/octet-stream' }), 1100 * 1024, false, 20, 'markers.bin', false);
const chunks = [];
for await (const encoded of pieces()) chunks.push(Buffer.from(encoded, 'base64'));
assert.deepEqual(Buffer.concat(chunks), Buffer.from(input), 'all input bytes must survive splitting');
// A 48-byte zero window hashes to zero; each isolated marker hashes to one.
// Markers are further apart than the 256 KiB minimum and below the maximum.
assert.deepEqual(chunks.map((chunk) => chunk.length), [blockBytes, blockBytes, blockBytes]);
});
utils/chunk-size-benchmark/remote.mjs
+210
View File
@@ -0,0 +1,210 @@
import assert from 'node:assert/strict';
import { createHash } from 'node:crypto';
import { spawn } from 'node:child_process';
import { promises as fs } from 'node:fs';
import http from 'node:http';
import { dirname, join, resolve } from 'node:path';
import { fileURLToPath } from 'node:url';
import { setTimeout as delay } from 'node:timers/promises';
import { createNewVaultSettings, REMOTE_COUCHDB, REMOTE_MINIO, SALT_OF_PASSPHRASE } from '@vrtmrz/livesync-commonlib/compat/common/types';
import { encryptString } from '@vrtmrz/livesync-commonlib/compat/encryption/stringEncryption';
import { S3Client, CreateBucketCommand, ListObjectsV2Command, DeleteObjectsCommand, DeleteBucketCommand } from '@aws-sdk/client-s3';
const here = dirname(fileURLToPath(import.meta.url));
const root = resolve(here, '../..');
const cli = join(root, 'src/apps/cli/dist/index.cjs');
const args = process.argv.slice(2);
const option = (key, fallback) => { const at = args.indexOf(`--${key}`); return at < 0 ? fallback : args[at + 1]; };
const sourceCli = resolve(option('source-cli', cli));
const targetCli = resolve(option('target-cli', cli));
const output = resolve(option('output', join(root, '.bench/chunk-size')));
const corpusPath = resolve(option('corpus', join(output, 'corpus/manifest.json')));
const sizes = option('sizes', '0,4,10,20,40').split(',').map(Number);
const providers = option('providers', 'couchdb,s3').split(',');
const repeats = Number(option('repeats', '3'));
const rttMs = Number(option('rtt-ms', '0'));
const user = 'bench';
const password = 'benchmark-only-password';
const accessKeyId = 'bench-access-key';
const secretAccessKey = 'benchmark-only-secret-key';
const couchUri = 'http://127.0.0.1:15994';
const s3Uri = 'http://127.0.0.1:19094';
const auth = `Basic ${Buffer.from(`${user}:${password}`).toString('base64')}`;
const key = '0123456789abcdef'.repeat(4);
const sha256 = (value) => createHash('sha256').update(value).digest('hex');
const sourceCliSha256 = sha256(await fs.readFile(sourceCli));
const targetCliSha256 = sha256(await fs.readFile(targetCli));
const credentials = { accessKeyId, secretAccessKey };
const s3 = new S3Client({ endpoint: s3Uri, region: 'us-east-1', forcePathStyle: true, credentials });
function emptyCounters() { return { requests: 0, requestBodyBytes: 0, responseBodyBytes: 0, maxRequestBodyBytes: 0, byMethod: {}, byOperation: {} }; }
async function startProxy(backend) {
let counters = emptyCounters();
const server = http.createServer(async (request, response) => {
counters.requests++;
counters.byMethod[request.method] = (counters.byMethod[request.method] || 0) + 1;
const url = new URL(request.url, backend);
const operation = url.pathname.split('/').find((part) => part.startsWith('_')) || (url.searchParams.has('list-type') ? 'list' : 'object');
counters.byOperation[operation] = (counters.byOperation[operation] || 0) + 1;
if (rttMs) await delay(rttMs / 2);
const target = new URL(request.url, backend);
const upstream = http.request(target, { method: request.method, headers: request.headers }, async (incoming) => {
if (rttMs) await delay(rttMs / 2);
response.writeHead(incoming.statusCode, incoming.headers);
incoming.on('data', (chunk) => { counters.responseBodyBytes += chunk.length; });
incoming.pipe(response);
});
upstream.on('error', (error) => { response.writeHead(502); response.end(String(error)); });
let requestBytes = 0;
request.on('data', (chunk) => { counters.requestBodyBytes += chunk.length; requestBytes += chunk.length; });
request.on('end', () => { counters.maxRequestBodyBytes = Math.max(counters.maxRequestBodyBytes, requestBytes); });
request.pipe(upstream);
});
await new Promise((accept, reject) => { server.once('error', reject); server.listen(0, '127.0.0.1', accept); });
return { uri: `http://127.0.0.1:${server.address().port}`, reset: () => { counters = emptyCounters(); }, snapshot: () => structuredClone(counters), close: () => new Promise((accept) => server.close(accept)) };
}
async function couch(path, method = 'GET', body) {
const response = await fetch(`${couchUri}${path}`, { method, headers: { Authorization: auth, 'Content-Type': 'application/json' }, body: body === undefined ? undefined : JSON.stringify(body) });
if (!response.ok) throw new Error(`CouchDB ${method} ${path}: ${response.status} ${await response.text()}`);
return response.json();
}
async function ready() {
for (let attempt = 0; attempt < 30; attempt++) {
try { await couch('/_up'); await s3.send(new ListObjectsV2Command({ Bucket: 'missing-benchmark-bucket' })).catch((error) => { if (error.$metadata?.httpStatusCode !== 404) throw error; }); return; }
catch (error) { if (attempt === 29) throw error; await delay(1000); }
}
}
async function runCli(directory, settings, command, commandArgs, logPath, entrypoint = cli) {
const marker = '__BENCH_TIME__';
const started = performance.now();
const child = spawn('/usr/bin/time', ['-f', `${marker}%U\t%S\t%M`, process.execPath, entrypoint, directory, '--settings', settings, command, ...commandArgs], { cwd: root, stdio: ['ignore', 'pipe', 'pipe'], env: { ...process.env, NODE_OPTIONS: '--max-old-space-size=2048' } });
let stdout = ''; let stderr = '';
child.stdout.on('data', (data) => { stdout += data; });
child.stderr.on('data', (data) => { stderr += data; });
const timeout = setTimeout(() => child.kill('SIGTERM'), 180000);
const code = await new Promise((accept, reject) => { child.once('error', reject); child.once('exit', accept); });
clearTimeout(timeout);
const elapsedMs = performance.now() - started;
await fs.writeFile(logPath, `${stdout}\n${stderr}`);
if (code !== 0) throw new Error(`CLI ${command} exited ${code}; ${logPath}\n${stderr.slice(-2200)}`);
const line = stderr.split(/\r?\n/).find((row) => row.startsWith(marker));
assert(line, 'GNU time output is missing');
const [userSeconds, systemSeconds, maxResidentSetKiB] = line.slice(marker.length).split('\t').map(Number);
return { elapsedMs, userCpuMs: userSeconds * 1000, systemCpuMs: systemSeconds * 1000, maxResidentSetKiB };
}
async function settingsFor(provider, endpoint, resource, size) {
return {
...createNewVaultSettings(), isConfigured: true, remoteType: provider === 'couchdb' ? REMOTE_COUCHDB : REMOTE_MINIO,
couchDB_URI: endpoint, couchDB_DBNAME: resource, couchDB_USER: user, couchDB_PASSWORD: password,
endpoint, bucket: resource, accessKey: accessKeyId, secretKey: secretAccessKey, region: 'us-east-1', forcePathStyle: true,
encrypt: true, passphrase: 'chunk-size-benchmark-only-passphrase', usePathObfuscation: true, encryptInternalMetadata: true,
idDerivationVersion: 1, idDerivationKey: '', encryptedIdDerivationKey: await encryptString(key, '*' + SALT_OF_PASSPHRASE), E2EEAlgorithm: 'v2', hashAlg: 'xxhash64',
chunkSplitterVersion: 'v3-rabin-karp', minimumChunkSize: 20, customChunkSize: size, autoAcceptCompatibleTweak: false,
enableCompression: false, useEden: false, useIndexedDBAdapter: false, handleFilenameCaseSensitive: false, pathNormalizationVersion: 0,
readChunksOnline: false, liveSync: false, syncOnStart: false, syncOnSave: false, periodicReplication: false,
syncOnFileOpen: false, syncMaxSizeInMB: 50, P2P_Enabled: false,
};
}
async function copyStage(stage, vault, previous) {
const old = new Map((previous?.files || []).map((file) => [file.path, file.sha256]));
const changed = [];
for (const file of stage.files) {
if (old.get(file.path) === file.sha256) continue;
const from = join(stage.directory, file.path); const to = join(vault, file.path);
await fs.mkdir(dirname(to), { recursive: true }); await fs.copyFile(from, to);
const stat = await fs.stat(from); await fs.utimes(to, stat.atime, stat.mtime); changed.push(file.path);
}
return changed;
}
async function inspect(provider, resource, previousIds) {
if (provider === 'couchdb') {
const info = await couch(`/${resource}`);
const all = await couch(`/${resource}/_all_docs?include_docs=true`);
const chunks = all.rows.filter(({ doc }) => doc?.type === 'leaf');
const added = chunks.filter(({ id }) => !previousIds.has(id));
const ids = chunks.map(({ id }) => id);
const jsonBytes = ({ doc }) => Buffer.byteLength(JSON.stringify(doc));
return { ids, metrics: { documentCount: info.doc_count, chunkCount: chunks.length, newChunkCount: added.length, chunkJsonBytes: chunks.reduce((sum, row) => sum + jsonBytes(row), 0), newChunkJsonBytes: added.reduce((sum, row) => sum + jsonBytes(row), 0), largestChunkJsonBytes: Math.max(0, ...chunks.map(jsonBytes)), largestDocumentJsonBytes: Math.max(0, ...all.rows.map(jsonBytes)), storedChunkDataBytes: chunks.reduce((sum, { doc }) => sum + Buffer.byteLength(doc.data || ''), 0), sizes: info.sizes, idLengthMin: Math.min(...ids.map((id) => id.length)), idLengthMax: Math.max(...ids.map((id) => id.length)) } };
}
const objects = []; let token;
do { const result = await s3.send(new ListObjectsV2Command({ Bucket: resource, ContinuationToken: token })); objects.push(...result.Contents || []); token = result.NextContinuationToken; } while (token);
return { ids: objects.map(({ Key }) => Key), metrics: { objectCount: objects.length, storedObjectBytes: objects.reduce((sum, { Size }) => sum + Size, 0), newObjectCount: objects.filter(({ Key }) => !previousIds.has(Key)).length } };
}
async function removeResource(provider, resource) {
if (provider === 'couchdb') { await couch(`/${resource}`, 'DELETE'); return; }
let token;
do {
const listing = await s3.send(new ListObjectsV2Command({ Bucket: resource, ContinuationToken: token }));
if (listing.Contents?.length) await s3.send(new DeleteObjectsCommand({ Bucket: resource, Delete: { Objects: listing.Contents.map(({ Key }) => ({ Key })) } }));
token = listing.NextContinuationToken;
} while (token);
await s3.send(new DeleteBucketCommand({ Bucket: resource }));
}
await fs.mkdir(join(output, 'remote'), { recursive: true });
const manifest = JSON.parse(await fs.readFile(corpusPath, 'utf8'));
for (const stage of manifest.stages) stage.directory = resolve(dirname(corpusPath), stage.directory);
const digest = sha256(JSON.stringify(manifest.stages.map((stage) => ({ name: stage.name, files: stage.files }))));
const resultPath = join(output, 'remote', `results-rtt${rttMs}.json`);
let results = [];
try { const saved = JSON.parse(await fs.readFile(resultPath, 'utf8')); assert.equal(saved.corpusDigest, digest); assert.equal(saved.sourceCliSha256, sourceCliSha256); assert.equal(saved.targetCliSha256, targetCliSha256); results = saved.results; } catch (error) { if (error.code !== 'ENOENT') throw error; }
const proxies = { couchdb: await startProxy(couchUri), s3: await startProxy(s3Uri) };
const save = async () => fs.writeFile(resultPath, JSON.stringify({ version: '1.0.39', commonlib: '0.1.44', variant: process.env.LIVESYNC_BENCH_VARIANT || 'unlabelled', sourceCliSha256, targetCliSha256, corpusDigest: digest, rttMs, mode: 'local services; HTTP body bytes exclude headers; full chunk download; E2EE V2 with independent IDs; compression disabled', results }, null, 2));
try {
await ready();
for (let repeat = 0; repeat < repeats; repeat++) {
const rotation = repeat % sizes.length; const order = [...sizes.slice(rotation), ...sizes.slice(0, rotation)];
for (const provider of (repeat % 2 ? [...providers].reverse() : providers)) for (const size of order) {
const name = `${provider}-c${size}-r${repeat + 1}`;
if (results.some((result) => result.name === name && result.complete)) continue;
const resource = `chunks-${Date.now().toString(36)}-${name}`;
const work = join(output, 'remote', name); await fs.mkdir(join(work, 'logs'), { recursive: true });
const a = join(work, 'source'); const b = join(work, 'target');
await fs.rm(a, { recursive: true, force: true }); await fs.rm(b, { recursive: true, force: true });
await fs.mkdir(a, { recursive: true }); await fs.mkdir(b, { recursive: true });
const settingsA = join(work, 'source-settings.json'); const settingsB = join(work, 'target-settings.json');
const settings = await settingsFor(provider, proxies[provider].uri, resource, size);
await fs.writeFile(settingsA, JSON.stringify(settings)); await fs.writeFile(settingsB, JSON.stringify(settings));
if (provider === 'couchdb') await couch(`/${resource}`, 'PUT'); else await s3.send(new CreateBucketCommand({ Bucket: resource }));
const result = { name, provider, customChunkSize: size, repeat: repeat + 1, stages: [], complete: false };
results.push(result); await save();
try {
let previous; let ids = new Set();
for (const stage of manifest.stages) {
const changedFiles = await copyStage(stage, a, previous);
const log = (label) => join(work, 'logs', `${stage.name}-${label}.txt`);
const mirror = await runCli(a, settingsA, 'mirror', [], log('mirror'), sourceCli);
proxies[provider].reset();
const upload = await runCli(a, settingsA, 'sync', [], log('upload'), sourceCli); upload.http = proxies[provider].snapshot();
const remote = await inspect(provider, resource, ids); ids = new Set(remote.ids);
proxies[provider].reset();
const download = await runCli(b, settingsB, 'sync', [], log('download'), targetCli); download.http = proxies[provider].snapshot();
const verified = []; const materialisedByPull = [];
for (const file of stage.files) {
let bytes; try { bytes = await fs.readFile(join(b, file.path)); } catch (error) { if (error.code !== 'ENOENT') throw error; }
if (!bytes || sha256(bytes) !== file.sha256) {
const target = join(work, 'verification', file.path); await fs.mkdir(dirname(target), { recursive: true });
const measurement = await runCli(b, settingsB, 'pull', [file.path, target], log(`verify-${file.path.replace(/[^a-zA-Z0-9]/g, '-')}`), targetCli);
bytes = await fs.readFile(target); materialisedByPull.push({ path: file.path, measurement });
}
assert.equal(sha256(bytes), file.sha256, `${name} ${stage.name} ${file.path}`); verified.push(file.path);
}
result.stages.push({ name: stage.name, changedFiles, mirror, upload, download, remote: remote.metrics, verifiedFiles: verified.length, materialisedByPull });
await save(); console.log(JSON.stringify({ case: name, stage: stage.name, uploadMs: Math.round(upload.elapsedMs), downloadMs: Math.round(download.elapsedMs), requestBytes: upload.http.requestBodyBytes, responseBytes: download.http.responseBodyBytes, remote: remote.metrics }));
previous = stage;
}
result.complete = true; await save();
} catch (error) { result.error = String(error); await save(); throw error; }
finally { await removeResource(provider, resource); }
}
}
} finally { await Promise.all(Object.values(proxies).map((proxy) => proxy.close())); s3.destroy(); }
console.log(`Completed remote measurements: ${resultPath}`);
utils/chunk-size-benchmark/report.mjs
+85
View File
@@ -0,0 +1,85 @@
import assert from 'node:assert/strict';
import { promises as fs } from 'node:fs';
import { join, resolve } from 'node:path';
const directory = resolve(process.argv[2] || '.bench/rabin-window-fixed');
const baselineDirectory = resolve(process.argv[3] || '.bench/chunk-size');
const load = async (file, base = directory) => JSON.parse(await fs.readFile(join(base, file), 'utf8'));
const summary = await load('summary.json');
const baseline = await load('summary.json', baselineDirectory);
const comparison = await load('comparison.json');
const validation = await load('validation.json');
const boundary = await load('cloudant-json-boundary/results.json');
const consumer = await load('context/validated-consumer.json');
const recovery = await load('context/cli-bundle-recovery.json');
assert.equal(validation.completedRemoteCases, 30);
assert.equal(validation.completedP2PCases, 12);
assert.equal(summary.corpus.manifestSha256, baseline.corpus.manifestSha256);
const mib = bytes => (bytes / 1048576).toFixed(3);
const second = milliseconds => (milliseconds / 1000).toFixed(2);
const getRemote = (data, provider, size, stage) => data.remote.find(row => row.provider === provider && row.customChunkSize === size && row.stage === stage);
const getP2P = (data, size, stage) => data.p2p.find(row => row.customChunkSize === size && row.stage === stage);
const getLocal = (data, size, stage) => data.local.find(row => row.customChunkSize === size && row.stage === stage);
const decrease = (before, after) => ((1 - after / before) * 100).toFixed(1);
const lines = [];
const paragraph = text => lines.push(text, '');
paragraph('Rabin–Karp窓処理修正後の実測（Self-hosted LiveSync 1.0.39 / Commonlib 0.1.44の未公開修正パッケージ）');
paragraph('窓処理は、今回の挿入後のチャンク再利用を妨げていた原因の一つだった。最小限の修正をCommonlibへ加え、前回と完全に同じ入力・設定・候補を再測定した。今回の混在コーパスでは、通常の新規推奨値を4（上限500 KiB）へ揃える案が有力になった。Cloudant対応は接続先ごとの推奨値ではなく、将来の同期先も含むvault共通のデータ制約として扱う必要がある。ただし数値4だけで全設定のCloudant互換性を保証する結論ではない。初期値、推奨値、UI、および移行処理はまだ変更していない。');
paragraph(`LiveSyncの基点は ${consumer.liveSyncBaseRevision}、Commonlibの基点は ${consumer.commonlibBaseRevision}。Commonlib所有リポジトリーの作業ツリーで修正し、単体・境界・リリースツール・パッケージ検証を通したtarballをLiveSyncへfile依存として入れた。公開版0.1.44を置換して公開したものではない。候補の実体は ${consumer.packageIntegrity}、インストール済みchunks.jsのSHA-256は ${consumer.installedChunksSha256}。`);
paragraph('修正はsrc/string_and_binary/chunks.tsの窓更新条件を pos >= start + windowSize から pos >= windowSize へ変える1行と、その目的を示すコメントである。境界でstartが進んでも、hashの48バイト窓を連続して更新する。形式、ID導出方法、設定キー、最小・平均・最大チャンク目標は変更していない。');
paragraph('先に3件の振る舞いの回帰テストを追加し、未修正の公開0.1.44相当ソースで3件とも期待した境界assertionが失敗することを確認した。バイナリ300/300/300 KiBの目印列は従来300/256/256/88 KiBとなり、挿入後の同一末尾チャンクを再利用できなかった。テキストの目印列でも過去のチャンクの影響が残った。修正後は3件とも成功し、再構成した全バイトも一致した。Commonlib全123ファイル・2,416テスト、型、boundary、release tools、packed consumerが成功した。LiveSyncはnpm 10クリーンインストール、通常npm ci、check、単体1,297成功・1 skipped、およびObsidian bundle/CLI/WebApp/WebPeerビルドが成功した。Community lintの既存51 warningsは残っている。');
paragraph(`入力は前回と同じ15ファイル、初期合計 ${mib(summary.corpus.stages[0].sourceBytes)} MiB。通常のプロジェクトMarkdown10本、約512 KiBと約5 MiBの生成Markdown、PNG2枚、および固定seedの8 MiB高エントロピーバイナリである。manifest SHA-256は ${summary.corpus.manifestSha256}。段階は初回、4 KiBのバイナリ上書き等、続いてバイナリ途中4 KiB挿入と生成Markdownへの段落挿入。設定はRabin–Karp、minimumChunkSize=20、E2EE V2、独立ID鍵version 1、パス・内部Metadata暗号化、チャンク圧縮なし、Edenなし、全chunk取得で固定した。`);
paragraph('ローカル分割は0,1,4,10,20,40,60を各3回。HTTPはCouchDB 3.5.0とRustFSを0,4,10,20,40各3回、P2Pは0,4,10,40各3回。同じホスト上の生成テストvaultとタスク専用コンテナを使用し、WAN遅延やパケットロスは加えていない。CPU負荷をかける検証・測定は同時に実行していない。Node v24.18.0 / Linux arm64 / Neoverse-N1 / 論理CPU3である。');
paragraph('以下はHTTPの実送信本文の3回中央値。ヘッダー・TLSは含めない。同期秒数はCLI起動を含み、事前のmirrorと転送後のファイル復元・SHA照合を含まない。OSキャッシュやサービスは各回に再起動していない。S3のJournalパック圧縮はenableCompressionによるチャンク圧縮とは別である。');
lines.push('remote | value | 初期送信MiB | 上書き送信MiB | 挿入送信MiB | 初期送信秒 | 初期受信秒 | 初期HTTP要求数');
for (const provider of ['couchdb', 's3']) for (const size of [0, 4, 10, 20, 40]) {
const initial = getRemote(summary, provider, size, 'initial');
const overwrite = getRemote(summary, provider, size, 'overwrite');
const insert = getRemote(summary, provider, size, 'insert');
lines.push([provider, size, mib(initial.upload_requestBodyBytes.median), mib(overwrite.upload_requestBodyBytes.median), mib(insert.upload_requestBodyBytes.median), second(initial.upload_elapsedMs.median), second(initial.download_elapsedMs.median), initial.upload_requests.median].join(' | '));
}
lines.push('');
paragraph('P2Pは全回host<->host経路。werift 0.24.4のcandidate-pair bytesカウンターは0のため採用せず、RTCDataChannelの実送信payloadを両端で観測した。下表は送信元host側で、RPC制御とアプリ層再送を含み、SCTP/DTLS/UDP/IPのヘッダー、下位層再送、Nostr signallingは含まない。HTTP本文量と完全に同じ層ではない。同期時間は探索・接続・双方向同期を含む。');
lines.push('value | 初期送信MiB | 上書き送信MiB | 挿入送信MiB 中央値 [min..max] | 初期同期秒 | 上書き同期秒 | 挿入同期秒');
for (const size of [0, 4, 10, 40]) {
const initial = getP2P(summary, size, 'initial'); const overwrite = getP2P(summary, size, 'overwrite'); const insert = getP2P(summary, size, 'insert');
lines.push([size, mib(initial.host_payloadBytesSent.median), mib(overwrite.host_payloadBytesSent.median), `${mib(insert.host_payloadBytesSent.median)} [${mib(insert.host_payloadBytesSent.min)}..${mib(insert.host_payloadBytesSent.max)}]`, second(initial.elapsedMs.median), second(overwrite.elapsedMs.median), second(insert.elapsedMs.median)].join(' | '));
}
lines.push('');
paragraph('値4の途中挿入における修正前→修正後の比較は次のとおり。');
lines.push('対象 | 修正前MiB | 修正後MiB | 減少率');
const oldRaw = getLocal(baseline, 4, 'insert').binaryNewRawBytes; const newRaw = getLocal(summary, 4, 'insert').binaryNewRawBytes;
lines.push(`新しく必要なバイナリの生データ | ${mib(oldRaw)} | ${mib(newRaw)} | ${decrease(oldRaw, newRaw)}%`);
for (const provider of ['couchdb', 's3']) {
const oldValue = getRemote(baseline, provider, 4, 'insert').upload_requestBodyBytes.median; const newValue = getRemote(summary, provider, 4, 'insert').upload_requestBodyBytes.median;
lines.push(`${provider}送信本文 | ${mib(oldValue)} | ${mib(newValue)} | ${decrease(oldValue, newValue)}%`);
}
const oldPeer = getP2P(baseline, 4, 'insert').host_payloadBytesSent.median; const newPeer = getP2P(summary, 4, 'insert').host_payloadBytesSent.median;
lines.push(`P2P host payload | ${mib(oldPeer)} | ${mib(newPeer)} | ${decrease(oldPeer, newPeer)}%`, '');
paragraph('通常の4 MiB未満のMarkdown11本×3段階は、修正後も7候補の全てで分割境界とIDが一致した。customChunkSizeを上げてもこの領域のID数は減らない。新しい独立ID鍵方式のIDは67文字で、旧暗号化xxhash64比較は14〜16文字、中央値16文字だった。ID命名の変更がIDを短くしたという前提は置けない。');
paragraph('0/1ではバイナリ・大きいテキストの最小値が最大値へクランプされ、100/200 KiBの固定分割になるため、窓修正後も挿入時の再同期の利点は出ない。小さい局所上書きだけなら0が有利だが、今回の挿入を含む編集では4の方が大幅に少なく済む。40と60は実装上の4 MiB上限に達し、全15ファイル×3段階で分割境界とIDが完全一致した。60の重複するネットワーク測定は省いた。');
for (const provider of ['couchdb', 's3']) {
const four = getRemote(summary, provider, 4, 'initial').upload_requestBodyBytes.median;
const forty = getRemote(summary, provider, 40, 'initial').upload_requestBodyBytes.median;
paragraph(`${provider}で4から40へ増やした初回送信量の削減は ${((1 - forty / four) * 100).toFixed(3)}%程度だった。上書き・挿入では40の送信量が大きく、今回の条件から大きいチャンクを維持する性能上の根拠は得られなかった。`);
}
paragraph('S3ではどの候補も初回7オブジェクト・14 HTTP要求だった。4と40は編集時もそれぞれ1オブジェクト追加・6 HTTP要求で、chunk数の削減がそのままPUT削減になる構成ではない。AWS等の実サービスの課金やWAN性能を測った値ではない。P2Pでは回ごとの再送等の変動があるため、中央値だけでなくsummary.jsonのmin/maxも参照する。150msの断片再要求との因果は計測しておらず、この処理は変更していない。');
paragraph('Cloudantの現行IBM公式limits（2026-10-23確認）は、単一JSON文書1 MB、リクエスト全体64MB、添付10 MB。https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-limits 。chunksは添付ではなくJSON文書として送る。モデルには保守的に1,000,000 / 64,000,000 bytesを使った。通常の暗号化条件ではCouchDBに文書上限だけをかけ、4は15ファイルの復元まで成功、10は413となった。4の最大保存文書は910,461 bytes、通常HTTP測定の最大リクエストは約10.27 MBで、両方の上限内だった。');
paragraph('別の境界probeでは、実際の分割器・圧縮・暗号化関数を用い、レプリケーションのrevisionフィールドを含む合成leafを送信した。結果は次のとおり。');
lines.push('入力 | value | 暗号化 | 圧縮 | JSON文書bytes | CouchDB HTTP status');
for (const row of boundary.cases) lines.push([row.kind, row.customChunkSize, row.encryption, row.compression, row.jsonBytes, row.status].join(' | '));
lines.push('');
paragraph('値4のバイナリはraw 512,000 bytes → base64 682,668 bytes → E2EE V1/V2で約910.5 KBとなった。値5のE2EE V2は1,092,573 bytesとなり、文書上限で413が返った。したがって暗号化バイナリに対して4は整数の候補として上限に近く、余裕を残す候補である。');
paragraph('ただし、CouchDB 3.5.0のmax_document_sizeはparsed document bodyをcouch_ejson_size:encoded_sizeで概算し、文字列はUTF-8長に2を足すだけでJSON escapeを数えない。このためNULの未暗号化テキストはwire上3,072,213 bytesでも、モデルの1 MB文書制限では受理された。Cloudant内部の算定が完全に同一であるとは確認していない。単純にwire JSONが1 MB超なら必ずCloudantの文書制限で失敗する、とこの結果から断定してはいけない。公式CouchDBソースはcontext/couch_doc-3.5.0.erlとcontext/couch_ejson_size-3.5.0.erlに保存した。');
paragraph('一括リクエストは別の制限である。分割器が作った500 KiBの異なる制御文字テキスト25個をまとめると76,805,358 bytesとなり、64,000,000-byte request制限で413になった。20個なら61,444,288 bytesで受理された。これは合成bulk requestの境界probeであり、アプリケーションの通常同期全体で同じbulkが組まれると実証したテストではない。現在の通常Couch同期のbatch_size=25では、暗号化文書が1 MB以内なら25 MB程度だが、暗号化しないテキストのescape膨張には件数制限だけでは十分でない。optional bulk chunk sendには独自のbyte制限もあるので、その経路と通常Pouch replicationの経路を区別する必要がある。');
paragraph('CloudantモデルはローカルCouchDBで制限を模したものであり、IBM Cloudantサービス自体の同期・性能は測定していない。全probe後に両制限設定を復元し、合成文書のDBも削除した。Cloudantの互換性を約束するなら、leafだけでなくMetadataを含む最終文書とバッチの実サイズも方針の対象にするのが妥当である。');
paragraph(`旧版と修正版の混在は、同じ旧依存リンクからビルドを復元し、旧entryが前回の保存物と完全一致することを確認して実測した（${recovery.originalEntryBitIdenticalToBaselineArchive}）。旧→新、新→旧の各3段階×15ファイル、計${validation.mixedVersionVerifiedFileStages}件でSHA-256が一致した。チャンクの保存形式とID方式は引き続き読める。再利用効率を揃えるには書込み端末の分割実装も揃える方がよい。最初の互換性試行は旧bundleの付随CJS不足で実行前に失敗し、そのログと復元記録を保存した。復元後の全試行は成功した。`);
const cutover = comparison.cutover.find(row => row.customChunkSize === 40);
paragraph(`窓の修正で分割境界が変わるため、既存ファイルを再保存する初回には追加チャンクが多く必要になり得る。ローカルの同じ初期データを値40で比較すると、新方式のunique raw ${mib(cutover.totalNewAlgorithmUniqueRawBytes)} MiB中、旧初期chunkに存在しない部分は ${mib(cutover.bytesNotPresentInOldAlgorithmInitialChunks)} MiBだった。これは内容/ID照合による算定で、実際のアップグレード時の転送量を測定した値ではない。以後の同じ方式での編集に対する改善とは分けて扱う。`);
paragraph('設定整理への提案は、通常の新規作成・推奨値をremote種別によらず4へ揃える方向で進め、旧設定の省略値や既存remoteで合意済みのtweakを一律に書き換えないことである。60と10を標準候補として分ける根拠は今回弱い。一方、Cloudantへ将来同期する意図を問うなら、現在の接続先ではなくvault全体の互換方針として扱い、全書込み端末で守る必要がある。暗号化・JSON化・バッチ化後の上限を内部で導出し、ユーザーへ組合せの数字を選ばせることは避けたい。');
paragraph('現在のremote profile URIは接続欄だけを含み、customChunkSizeは共通設定である。通常のprofile切替でその値を変えないが、起動時のCloudant接続先補正やpreferred tweak採用では変わる経路がある。保存済みleafは転送時に再分割されない。Cloudant対応を後から有効化して値だけ下げても、既存の大きいleafがそのまま残るため、互換方針を変える移行では既存DBの検査・再構成も別途設計する必要がある。これは今回未実装の設計案である。');
paragraph(`検証結果は、主測定42ケース・126段階×15ファイル=${validation.verifiedFileStages}件のSHA-256一致、保存後設定${validation.checkedPersistedConfigurations}件の条件一致、混在版${validation.mixedVersionVerifiedFileStages}件のSHA-256一致。通常Markdown、40/60同一、全分割の再構成、反復でのID・境界再現性、および同じIDの内容一致も成功した。`);
paragraph('保存物は生JSON、summary/CSV、comparison.json、validation.json、全入力corpus、各caseのログ・設定・P2Pカウンター、scripts、contextの完全な旧/新CLI bundleとlockfile、commonlibの修正差分・回帰テスト・検証ログ・tarball、およびartifacts.sha256である。巨大な作業用DB複製は含めず、再生成できる入力と証拠を保存する。旧baselineは親ディレクトリーにそのまま残している。');
paragraph('再現時はLiveSync固定コミットとCommonlib固定コミットの作業ツリーを用意し、Commonlibのproduction.patchとsource内の回帰テストを反映する。Commonlibの元リポジトリーで検証し、保存済みtarballと同じ実体をLiveSyncにfile依存として入れる。scriptsをLiveSyncのutils/chunk-size-benchmarkへ置き、npm 10 ci、通常npm ci、check、unit、全app buildの順に実行する。旧版の再構築には公開0.1.44パッケージを使い、実行本体だけでなくdist全体を別ディレクトリーへ残す。新しい出力先とLIVESYNC_BENCH_VARIANT=rolling-window-fixedを使ってlocal/remote/cloudant probes/compatibility/p2pを逐次実行し、summarise、compare、verify-results、reportの順に集計する。ローカル専用fixtureの起動・停止にもcomposeのproject名を指定する。');
await fs.writeFile(join(directory, 'report.txt'), lines.join('\n'));
console.log(JSON.stringify({ report: join(directory, 'report.txt'), bytes: Buffer.byteLength(lines.join('\n')) }));
utils/chunk-size-benchmark/summarise.mjs
+83
View File
@@ -0,0 +1,83 @@
import assert from 'node:assert/strict';
import { promises as fs } from 'node:fs';
import { join, resolve } from 'node:path';
const directory = resolve(process.argv[2] || '.bench/chunk-size');
const local = JSON.parse(await fs.readFile(join(directory, 'results.json'), 'utf8'));
const optionalJson = async (path) => { try { return JSON.parse(await fs.readFile(path, 'utf8')); } catch (error) { if (error.code === 'ENOENT') return undefined; throw error; } };
const remote = await optionalJson(join(directory, 'remote/results-rtt0.json'));
const p2p = await optionalJson(join(directory, 'p2p/results.json'));
const median = (numbers) => { const sorted = [...numbers].sort((a, b) => a - b); const mid = Math.floor(sorted.length / 2); return sorted.length % 2 ? sorted[mid] : (sorted[mid - 1] + sorted[mid]) / 2; };
const stats = (numbers) => ({ n: numbers.length, median: median(numbers), min: Math.min(...numbers), max: Math.max(...numbers) });
let normalMarkdownFileStages = 0;
for (const stage of local.candidates[0].stages) {
for (const file of stage.files.filter((file) => file.kind === 'markdown' && file.sourceBytes < 4 * 1024 * 1024)) {
const fingerprints = local.candidates.map((candidate) => candidate.stages.find((entry) => entry.name === stage.name).files.find((entry) => entry.path === file.path).chunkFingerprintSha256);
assert.equal(new Set(fingerprints).size, 1, `${stage.name}/${file.path}`); normalMarkdownFileStages++;
}
}
const localRows = local.candidates.flatMap((candidate) => candidate.stages.map((stage) => {
const binary = stage.byKind.find((group) => group.kind === 'binary');
const markdown = stage.byKind.find((group) => group.kind === 'markdown');
const random = stage.files.find((file) => file.path === 'benchmark/random-8m.bin');
return {
customChunkSize: candidate.customChunkSize, stage: stage.name, rawCeilingBytes: candidate.chunkSizeCeilingBytes,
allChunks: stage.files.reduce((sum, file) => sum + file.chunkCount, 0), binaryChunks: binary.chunkCount,
randomBinaryChunks: random.chunkCount, randomBinaryCpuMs: random.timing.primarySplitAndIdCpuMsMedian,
binaryNewRawBytes: binary.newUniqueChunkRawBytesVsHistory, markdownNewRawBytes: markdown.newUniqueChunkRawBytesVsHistory,
splitAndIdCpuMs: binary.timing.primarySplitAndIdCpuMsMedian + markdown.timing.primarySplitAndIdCpuMsMedian,
};
}));
function grouped(entries, makeRow) {
const groups = new Map();
for (const entry of entries) for (const stage of entry.stages) {
const key = `${entry.provider || 'p2p'}/${entry.customChunkSize}/${stage.name}`;
const group = groups.get(key) || { provider: entry.provider || 'p2p', customChunkSize: entry.customChunkSize, stage: stage.name, samples: [] };
group.samples.push(stage); groups.set(key, group);
}
return [...groups.values()].map(makeRow);
}
const remoteRows = grouped((remote?.results || []).filter((entry) => entry.complete), (group) => {
const result = { provider: group.provider, customChunkSize: group.customChunkSize, stage: group.stage, repeats: group.samples.length };
for (const direction of ['mirror', 'upload', 'download']) {
for (const metric of ['elapsedMs', 'userCpuMs', 'systemCpuMs', 'maxResidentSetKiB']) result[`${direction}_${metric}`] = stats(group.samples.map((stage) => stage[direction][metric]));
if (direction !== 'mirror') for (const metric of ['requests', 'requestBodyBytes', 'responseBodyBytes', 'maxRequestBodyBytes']) {
const values = group.samples.map((stage) => stage[direction].http[metric]).filter(Number.isFinite);
if (values.length) result[`${direction}_${metric}`] = stats(values);
}
}
for (const metric of ['objectCount', 'storedObjectBytes', 'chunkCount', 'newChunkCount', 'chunkJsonBytes', 'largestChunkJsonBytes', 'largestDocumentJsonBytes']) {
const values = group.samples.map((stage) => stage.remote[metric]).filter(Number.isFinite);
if (values.length) result[metric] = stats(values);
}
result.verifiedFilesPerSample = group.samples.map((stage) => stage.verifiedFiles);
return result;
});
const p2pRows = grouped((p2p?.results || []).filter((entry) => entry.complete), (group) => {
const result = { provider: 'p2p', customChunkSize: group.customChunkSize, stage: group.stage, repeats: group.samples.length };
for (const metric of ['elapsedMs', 'userCpuMs', 'systemCpuMs', 'maxResidentSetKiB']) result[metric] = stats(group.samples.map((stage) => stage.sync[metric]));
result.currentRoundTripTime = stats(group.samples.map((stage) => stage.stats.selectedPair.currentRoundTripTime));
for (const side of ['host', 'target']) for (const metric of ['payloadBytesSent', 'messagesSent']) result[`${side}_${metric}`] = stats(group.samples.map((stage) => stage.dataChannel[side][metric]));
for (const side of ['host', 'target']) for (const metric of ['userCpuMs', 'systemCpuMs', 'maxResidentSetKiB']) result[`${side}_${metric}`] = stats(group.samples.map((stage) => stage.dataChannel[side].processUsage[metric]));
result.paths = [...new Set(group.samples.map((stage) => stage.stats.selectedPath))];
result.verifiedFilesPerSample = group.samples.map((stage) => stage.verifiedFiles);
return result;
});
const summary = { version: local.runtime.projectVersion, commonlib: local.runtime.commonlibVersion, variant: local.runtime.variant, commonlibChunksSha256: local.runtime.commonlibChunksSha256, corpus: local.corpus,
assertions: { ...local.assertions, normalMarkdownIdenticalAcrossAllSevenValues: true, normalMarkdownFileStages }, local: localRows, remote: remoteRows, p2p: p2pRows };
await fs.writeFile(join(directory, 'summary.json'), JSON.stringify(summary, null, 2));
function csv(rows, fields) {
const cell = (value) => JSON.stringify(typeof value === 'object' && value !== null ? value.median : value ?? '');
return `${fields.join(',')}\n${rows.map((row) => fields.map((field) => cell(row[field])).join(',')).join('\n')}\n`;
}
await fs.writeFile(join(directory, 'local-summary.csv'), csv(localRows, Object.keys(localRows[0])));
if (remoteRows.length) await fs.writeFile(join(directory, 'remote-summary.csv'), csv(remoteRows, Object.keys(remoteRows[0]).filter((key) => key !== 'verifiedFilesPerSample')));
if (p2pRows.length) await fs.writeFile(join(directory, 'p2p-summary.csv'), csv(p2pRows, Object.keys(p2pRows[0]).filter((key) => !['paths', 'verifiedFilesPerSample'].includes(key))));
console.log(JSON.stringify({ directory, completeRemoteCases: remote?.results.filter((entry) => entry.complete).length || 0, completeP2PCases: p2p?.results.filter((entry) => entry.complete).length || 0, assertions: summary.assertions }));
utils/chunk-size-benchmark/verify-results.mjs
+82
View File
@@ -0,0 +1,82 @@
import assert from 'node:assert/strict';
import { createHash } from 'node:crypto';
import { promises as fs } from 'node:fs';
import { join, resolve } from 'node:path';
const directory = resolve(process.argv[2] || '.bench/rabin-window-fixed');
const load = async name => JSON.parse(await fs.readFile(join(directory, name), 'utf8'));
const local = await load('results.json');
const summary = await load('summary.json');
const remote = await load('remote/results-rtt0.json');
const p2p = await load('p2p/results.json');
const manifest = await load('corpus/manifest.json');
const sha256 = value => createHash('sha256').update(value).digest('hex');
assert.equal(remote.corpusDigest, p2p.corpusDigest);
assert.equal(remote.corpusDigest, sha256(JSON.stringify(manifest.stages.map(stage => ({ name: stage.name, files: stage.files })))));
assert.equal(local.corpus.manifestSha256, sha256(JSON.stringify(manifest)));
assert.equal(remote.results.length, 30);
assert.equal(p2p.results.length, 12);
let verified = 0;
let configurations = 0;
for (const [subdirectory, result] of [['remote', remote], ['p2p', p2p]]) {
assert(result.variant === 'rolling-window-fixed');
const groups = new Map();
for (const entry of result.results) {
assert(entry.complete && !entry.error, entry.name);
assert.equal(entry.stages.length, 3);
const key = `${entry.provider || 'p2p'}/${entry.customChunkSize}`;
groups.set(key, (groups.get(key) || 0) + 1);
for (const stage of entry.stages) { assert.equal(stage.verifiedFiles, 15); verified += stage.verifiedFiles; }
for (const side of ['source', 'target']) {
const settings = await load(join(subdirectory, entry.name, `${side}-settings.json`));
assert.equal(settings.customChunkSize, entry.customChunkSize);
assert.equal(settings.chunkSplitterVersion, 'v3-rabin-karp');
assert.equal(settings.minimumChunkSize, 20);
assert.equal(settings.idDerivationVersion, 1);
assert.equal(settings.encrypt, true);
assert.equal(settings.E2EEAlgorithm, 'v2');
assert.equal(settings.useEden, false);
assert.equal(settings.enableCompression, false);
assert.equal(settings.autoAcceptCompatibleTweak, false);
assert.equal(settings.batch_size, 25);
configurations++;
}
}
assert([...groups.values()].every(count => count === 3));
}
assert.equal(verified, 1890);
assert.equal(configurations, 84);
let mixedVerified = 0;
const compatibility = [];
for (const name of ['compatibility-old-to-new', 'compatibility-new-to-old']) {
const result = await load(`${name}/remote/results-rtt0.json`);
const completed = result.results.filter(entry => entry.complete);
assert.equal(completed.length, 1);
assert.notEqual(result.sourceCliSha256, result.targetCliSha256);
const entry = completed[0];
assert(entry.complete && !entry.error);
assert.equal(entry.stages.length, 3);
for (const stage of entry.stages) { assert.equal(stage.verifiedFiles, 15); mixedVerified += stage.verifiedFiles; }
compatibility.push({ name, sourceCliSha256: result.sourceCliSha256, targetCliSha256: result.targetCliSha256, verifiedFiles: 45, retainedIncompleteSetupAttempts: result.results.filter(entry => !entry.complete).length });
}
const model = await load('cloudant-model/constraint-result.json');
assert(model.configurationRestored && model.sizeRejectionObserved);
assert(model.cases.some(entry => entry.customChunkSize === 4 && entry.complete));
assert(model.cases.some(entry => entry.customChunkSize === 10 && !entry.complete));
const boundary = await load('cloudant-json-boundary/results.json');
assert(boundary.configurationRestored && boundary.requestConfigurationRestored);
assert.equal(boundary.cases.length, 8);
assert.equal(boundary.batches.length, 2);
assert(boundary.batches.some(entry => entry.count === 25 && entry.status === 413 && !entry.accepted));
assert(boundary.batches.some(entry => entry.count === 20 && entry.accepted));
assert(Object.values(local.assertions).every(value => value === true || typeof value === 'number'));
const result = { completedRemoteCases: 30, completedP2PCases: 12, repeatsPerCondition: 3,
verifiedFileStages: verified, checkedPersistedConfigurations: configurations,
sameCorpusDigestForRemoteAndP2P: true, localAssertions: summary.assertions,
compatibility, mixedVersionVerifiedFileStages: mixedVerified,
cloudantConstraintModel: { candidate4Completed: true, candidate10RejectedWith413: true, fixtureConfigurationRestored: true },
cloudantJSONBoundary: { cases: boundary.cases.length, batches: boundary.batches.length, request25RejectedWith413: true, request20Accepted: true, fixtureConfigurationRestored: true, cloudantServiceMeasured: false },
rabinWindowDiagnostic: { command: 'node --test utils/chunk-size-benchmark/rabin-window.test.mjs', exitCode: 0, productionFixApplied: true },
};
await fs.writeFile(join(directory, 'validation.json'), JSON.stringify(result, null, 2));
console.log(JSON.stringify(result));
utils/couchdb/livesync-commonlib.ts
+2 -2
View File
@@ -1,5 +1,5 @@
// Keep CouchDB database-version negotiation isolated from Setup URI generation.
// The exact release must match utils/livesync-commonlib-version.ts; the setup
// tool suite checks every static specifier before release.
export { checkRemoteVersion } from "npm:@vrtmrz/livesync-commonlib@0.1.34/compat/pouchdb/negotiation";
export { PouchDB } from "npm:@vrtmrz/livesync-commonlib@0.1.34/compat/pouchdb/pouchdb-browser";
export { checkRemoteVersion } from "npm:@vrtmrz/livesync-commonlib@0.2.0/compat/pouchdb/negotiation";
export { PouchDB } from "npm:@vrtmrz/livesync-commonlib@0.2.0/compat/pouchdb/pouchdb-browser";
utils/flyio/deno.lock
Generated
+50 -7
View File
@@ -1,9 +1,25 @@
{
"version": "5",
"specifiers": {
"npm:@vrtmrz/livesync-commonlib@0.1.34": "0.1.34"
"npm:@vrtmrz/livesync-commonlib@0.2.0": "0.2.0"
},
"npm": {
"@aws-crypto/sha256-js@5.2.0": {
"integrity": "sha512-FFQQyu7edu4ufvIZ+OadFpHHOt+eSTBaYaki44c+akjg7qZg9oOQeLlk77F6tSYqjDAFClrHJk9tMf0HdVyOvA==",
"dependencies": [
"@aws-crypto/util",
"@aws-sdk/types",
"tslib"
]
},
"@aws-crypto/util@5.2.0": {
"integrity": "sha512-4RkU9EsI6ZpBve5fseQlGNUWKMa1RLPQ1dnjnQoe07ldfIzcsGb5hC5W0Dm7u423KWzawlrpbjXBrXCEv9zazQ==",
"dependencies": [
"@aws-sdk/types",
"@smithy/util-utf8",
"tslib"
]
},
"@aws-sdk/checksums@3.1000.18": {
"integrity": "sha512-IImkbEyXdV6/uaF5r6Wkk+8718mQw1ll83j0a4a30R3JM/rHVFdWAiT4jtJpFjJiIwM/oJ6SxIxr0z2TaQUGqw==",
"dependencies": [
@@ -230,6 +246,12 @@
"tslib"
]
},
"@smithy/is-array-buffer@2.2.0": {
"integrity": "sha512-GGP3O9QFD24uGeAXYUjwSTXARoqpZykHadOmA8G5vfJPK0/DC67qa//0qvqrJzL1xc8WQWX7/yc7fwudjPHPhA==",
"dependencies": [
"tslib"
]
},
"@smithy/md5-js@4.4.10": {
"integrity": "sha512-XI5xhWxRkWuiLNj0/Z30vLTPpZe9UQcg57Ox/n4vzGmFiHSrD+xAmx6ubEcWt6u69IOH+4LAucpaZk5AMxB8oQ==",
"dependencies": [
@@ -267,6 +289,13 @@
"tslib"
]
},
"@smithy/util-buffer-from@2.2.0": {
"integrity": "sha512-IJdWBbTcMQ6DA0gdNhh/BwrLkDR+ADW5Kr1aZmd4k3DIF6ezMV4R2NIAmT08wQJ3yUK82thHWmC/TnK/wpMMIA==",
"dependencies": [
"@smithy/is-array-buffer",
"tslib"
]
},
"@smithy/util-retry@4.5.10": {
"integrity": "sha512-hYu5ieq8myuO29xCQV0IIhicRm1aO0lcNBeMcF1mVVAVQC0oylPGKFliMq5NbxtdOvRxJWCMuir2gwKI91f1lw==",
"dependencies": [
@@ -274,6 +303,13 @@
"tslib"
]
},
"@smithy/util-utf8@2.3.0": {
"integrity": "sha512-R8Rdn8Hy72KKcebgLiv8jQcQkXoLMOGGv5uI1/k0l+snqkOzQ1R0ChUBCxWMlBsFMekWjq0wRudIweFs7sKT5A==",
"dependencies": [
"@smithy/util-buffer-from",
"tslib"
]
},
"@trystero-p2p/core@0.25.3": {
"integrity": "sha512-lQKNq/ha+vF6kQZrpaXJGzlzxLF/Fhizoy5dwJUYQlkqRrbPup/Jov2EpPX/CPr2X+f79HxVzonkeni3MxmCuQ=="
},
@@ -284,9 +320,10 @@
"@trystero-p2p/core"
]
},
"@vrtmrz/livesync-commonlib@0.1.34": {
"integrity": "sha512-EeVpeFg3W43cpN+x0BZvFj9fN+eQFil9vohvmLLV1z/8bCTT0Hsq/bfRV59oL0qFNggPetkwyCZUusdfjajNOw==",
"@vrtmrz/livesync-commonlib@0.2.0": {
"integrity": "sha512-K24Pkke+FTjZI7DjIV7qkZk+BmC9VebN3da+2cVD9SWOgoGl7OvDyXTHDafbrmmM4vi4ydaydAEh9glO1xHF6A==",
"dependencies": [
"@aws-crypto/sha256-js",
"@aws-sdk/client-s3",
"@smithy/fetch-http-handler",
"@smithy/md5-js",
@@ -294,6 +331,7 @@
"@smithy/types",
"@smithy/util-retry",
"@trystero-p2p/nostr",
"@vrtmrz/value-store",
"diff-match-patch",
"events",
"fflate",
@@ -305,9 +343,11 @@
"pouchdb-adapter-idb",
"pouchdb-adapter-indexeddb",
"pouchdb-adapter-memory",
"pouchdb-checkpointer",
"pouchdb-core",
"pouchdb-errors",
"pouchdb-find",
"pouchdb-generate-replication-id",
"pouchdb-mapreduce",
"pouchdb-merge",
"pouchdb-replication",
@@ -317,6 +357,9 @@
"xxhash-wasm-102@npm:xxhash-wasm@1.1.0"
]
},
"@vrtmrz/value-store@0.1.0": {
"integrity": "sha512-M62f1j0bnJkqpgI5sYxnkXUAOSDEwQbYrLf+w58gCCm4sjIXDmFzDJd2duXabRbktBlACt9P99yjN/9gOAPxrA=="
},
"abstract-leveldown@2.7.2": {
"integrity": "sha512-+OVvxH2rHVEhWLdbudP6p0+dNMXu8JA1CbhP19T8paTYAcX7oJ4OVjT+ZUVpv7mITxXHqDMej+GdqXBmXkw09w==",
"dependencies": [
@@ -497,8 +540,8 @@
],
"deprecated": true
},
"minimatch@10.2.5": {
"integrity": "sha512-MULkVLfKGYDFYejP07QOurDLLQpcjk7Fw+7jXS2R2czRQzR56yHRveU5NDJEOviH+hETZKSkIk5c+T23GjFUMg==",
"minimatch@10.2.6": {
"integrity": "sha512-vpLQEs+VLCr1nU0BXS07maYoFwlDAH0gngQuuttxIwutDFEMHq2blX+8vpgxDdK3J1PwjCJiep77OitTZ4Ll1A==",
"dependencies": [
"brace-expansion"
]
@@ -509,8 +552,8 @@
"whatwg-url"
]
},
"octagonal-wheels@0.1.54": {
"integrity": "sha512-Je3ancYhjKX7UY2K19T/qTjG8C9nK8YVrACr5naIf78mN4bbjQkYyWmlj+ooifV/moWVsQrp4fEWz/7mv6It3A==",
"octagonal-wheels@0.1.55": {
"integrity": "sha512-mHsQv/h4Tw4ZGJCmTxXFlgeRvPuL9s47C0arD8Njqld+DKr3ppaWjVteEBP+9bWFSOpv/a1u2yY3LYNFGmuexg==",
"dependencies": [
"idb"
]
utils/flyio/generate_setupuri.test.ts
+14 -4
View File
@@ -1,6 +1,7 @@
import {
decodeSettingsFromSetupURI,
DEFAULT_SETTINGS,
PREFERRED_SETTING_SELF_HOSTED,
} from "../setup/livesync-commonlib.ts";
function assert(condition: unknown, message: string): asserts condition {
@@ -9,8 +10,14 @@ function assert(condition: unknown, message: string): asserts condition {
Deno.test("generates a current self-hosted Setup URI through the published Commonlib contract", async () => {
const scriptPath = new URL("./generate_setupuri.ts", import.meta.url);
const candidateConfig = Deno.env.get("LIVESYNC_TEST_COMMONLIB_CONFIG");
const command = new Deno.Command(Deno.execPath(), {
args: ["run", "-A", scriptPath.pathname],
args: [
"run",
"-A",
...(candidateConfig ? ["--config", candidateConfig, "--no-lock"] : []),
scriptPath.pathname,
],
env: {
hostname: "https://couch.example.test",
username: "alice",
@@ -38,7 +45,9 @@ Deno.test("generates a current self-hosted Setup URI through the published Commo
const recoveryCode = stdout.match(/sls-id-v1:[0-9a-f]{64}/u)?.[0];
assert(recoveryCode, "the generator did not print an ID recovery code");
assert(
(effectiveSettings as typeof effectiveSettings & { idDerivationKey?: string }).idDerivationKey ===
(effectiveSettings as typeof effectiveSettings & {
idDerivationKey?: string;
}).idDerivationKey ===
recoveryCode.slice("sls-id-v1:".length),
"the CouchDB Setup URI did not contain the generated ID key",
);
@@ -47,8 +56,9 @@ Deno.test("generates a current self-hosted Setup URI through the published Commo
"the CouchDB Setup URI left the imported device unconfigured",
);
assert(
effectiveSettings.customChunkSize === 60,
"the Setup URI did not use the current self-hosted chunk-size recommendation",
effectiveSettings.customChunkSize ===
PREFERRED_SETTING_SELF_HOSTED.customChunkSize,
"the Setup URI did not use the self-hosted chunk-size preference from Commonlib",
);
assert(
effectiveSettings.chunkSplitterVersion === "v3-rabin-karp",
utils/livesync-commonlib-version.ts
+1 -1
View File
@@ -2,4 +2,4 @@
// Commonlib registry release. Static npm specifiers cannot interpolate this
// value, so livesync-commonlib-version.test.ts verifies the domain-specific
// facades against it.
export const LIVESYNC_COMMONLIB_VERSION = "0.1.34";
export const LIVESYNC_COMMONLIB_VERSION = "0.2.0";
utils/setup/generate_setup_uri.test.ts
+4 -3
View File
@@ -1,6 +1,7 @@
import {
decodeSettingsFromSetupURI,
DEFAULT_SETTINGS,
PREFERRED_JOURNAL_SYNC,
} from "./livesync-commonlib.ts";
import { generateSetupURI } from "./generate_setup_uri.ts";
@@ -55,8 +56,8 @@ Deno.test("generates an Object Storage Setup URI with a selected S3 profile", as
"the Setup URI left the imported device unconfigured",
);
assert(
effective.customChunkSize === 10,
"the journal chunk-size preset was not applied",
effective.customChunkSize === PREFERRED_JOURNAL_SYNC.customChunkSize,
"the Object Storage Setup URI did not use its Commonlib journal chunk-size preference",
);
assert(
effective.liveSync,
@@ -122,7 +123,7 @@ Deno.test("generates a random-room P2P Setup URI without copying a device identi
);
assert(
effective.customChunkSize === 0,
"the P2P profile inherited the self-hosted CouchDB chunk-size recommendation",
"the P2P Setup URI did not retain its zero chunk-size value",
);
assert(
effective.sendChunksBulkMaxSize === 1,
utils/setup/livesync-commonlib.ts
+6 -6
View File
@@ -5,10 +5,10 @@ export {
decodeSettingsFromSetupURI,
encodeTimeBoundSetupURI,
isTimeBoundSetupURIUsableNow,
} from "npm:@vrtmrz/livesync-commonlib@0.1.34/setup-uri";
export type { TimeBoundSetupURIMode } from "npm:@vrtmrz/livesync-commonlib@0.1.34/setup-uri";
export { generateP2PRoomId } from "npm:@vrtmrz/livesync-commonlib@0.1.34/compat/common/utils";
export { upsertRemoteConfigurationInPlace } from "npm:@vrtmrz/livesync-commonlib@0.1.34/remote-configurations";
} from "npm:@vrtmrz/livesync-commonlib@0.2.0/setup-uri";
export type { TimeBoundSetupURIMode } from "npm:@vrtmrz/livesync-commonlib@0.2.0/setup-uri";
export { generateP2PRoomId } from "npm:@vrtmrz/livesync-commonlib@0.2.0/compat/common/utils";
export { upsertRemoteConfigurationInPlace } from "npm:@vrtmrz/livesync-commonlib@0.2.0/remote-configurations";
export {
createNewVaultSettings,
DEFAULT_SETTINGS,
@@ -16,5 +16,5 @@ export {
PREFERRED_BASE,
PREFERRED_JOURNAL_SYNC,
PREFERRED_SETTING_SELF_HOSTED,
} from "npm:@vrtmrz/livesync-commonlib@0.1.34/settings";
export type { ObsidianLiveSyncSettings } from "npm:@vrtmrz/livesync-commonlib@0.1.34/settings";
} from "npm:@vrtmrz/livesync-commonlib@0.2.0/settings";
export type { ObsidianLiveSyncSettings } from "npm:@vrtmrz/livesync-commonlib@0.2.0/settings";