Integrate encrypted internal metadata in LiveSync

This commit is contained in:
vorotamoroz
2026-09-27 09:51:53 +00:00
parent 7b3b6ff854
commit 6c50505096
23 changed files with 1220 additions and 61 deletions
@@ -57,6 +57,8 @@ Keep configured-state inference separate from new-Vault initialisation. If an ex
- On resume, clear `versionUpFlash` and persist that fail-closed change before recording the current `VER` as acknowledged. If saving fails, restore the gate. Reapply settings only after the marker has advanced so that the previously configured synchronisation behaviour can resume without reconstruction.
- Preserve the original legacy review message as a structured reason when no more specific database or settings-schema reason is available. Escape it before including it in Markdown UI.
- Continue to reject a remote version document which is newer than the running implementation. That receiver-side check is independent of the local upgrade review.
- From remote generation 13, assess the `used_features` list in that document as a separate compatibility dimension. A client must recognise every listed feature before it interprets the database or runs maintenance which depends on Metadata. Declare a feature before writing its representation, and retain the declaration while older data may depend on it. An unknown identifier is reported as text without requiring a descriptive label in that client.
- Do not advance the device-local `VER` acknowledgement merely because a remote feature is introduced. The remote generation and its feature list govern remote admission; `VER` remains the local compatibility review gate. Connecting to a generation-12 database does not promote it solely because the client understands generation 13.
### Onboarding activation and initialisation
@@ -90,7 +92,7 @@ Keep configured-state inference separate from new-Vault initialisation. If an ex
- Accepted new-device and existing-device setup cannot enable ordinary processing before the selected Rebuild or Fetch has been reserved.
- An older installation cannot dismiss evidence that a newer implementation or settings schema has already been used on the device.
- The Obsidian-specific dialogue depends only on a host-neutral compatibility result and the injected confirmation capability. Commonlib remains responsible for settings migration, device-local storage, and the replication gate.
- A future incompatible database change must increment `VER`, provide an actionable review message, verify the remote version negotiation, and test both the pending and acknowledged states. A major SemVer increase without those changes has no database-compatibility effect.
- A future local database change which requires compatibility review must increment `VER`, provide an actionable review message, and test both the pending and acknowledged states. A new remote representation must declare its feature before use and verify remote admission independently. A major SemVer increase alone has no database-compatibility effect.
## Verification
@@ -0,0 +1,198 @@
---
date: 2026-09-27
commonlib-version: "0.1.30"
self-hosted-livesync-version: "1.0.32"
status: unreleased
---
# Internal Metadata encryption and remote feature changes
This document defines the LiveSync integration of Commonlib's remote feature
contract and encrypted Metadata for Hidden File Sync and Customisation Sync.
It describes unreleased behaviour being implemented in this branch.
Commonlib's companion `docs/remote-feature-compatibility.md` is the
source of truth for the wire document, identifiers, validation, and shared
assessment. This document owns the application behaviour, settings, Doctor
recommendation, and verification of the Obsidian and CLI integrations.
## Scope and settings
Add `encryptInternalMetadata` to the shared encryption settings. A genuinely new
Vault or CLI configuration defaults to true. Existing stored settings and old
Setup URI or QR imports complete an absent value as false. Ordinary partial
setting updates retain the current value.
The preference applies to CouchDB with E2EE V2 and Property Encryption enabled.
Show the preference as unavailable and explain its prerequisites when they are
absent. Keep Journal and P2P's existing
transport protection and avoid unrelated setting mismatches for those remotes.
Use the existing HKDF Metadata representation to protect path, creation and
modification times, size, and Chunk references for obfuscated internal entries.
Keep the `i:`, `ix:`, and supported legacy `ps:` document IDs, path conversion,
and content Chunk representation. Read encrypted Metadata independently of the
write preference, including after that preference is disabled.
The protection leaves document IDs, namespaces, revisions, deletion state,
document counts, and ciphertext lengths visible. It does not encrypt device or
Vault names stored in separate participant records.
## Enabling the preference
Changing the preference does not automatically reconstruct a database or gather
all devices' data. It affects subsequent Metadata writes. Unchanged documents
and old revisions can retain plaintext; mixed plaintext and encrypted Metadata
are a supported transition state.
Strongly recommend the existing manual remote Rebuild workflow when the person
wants existing Metadata protected as well. The person prepares the authoritative
data for that workflow. Describe this distinction in the setting, Doctor reason,
and operational documentation. Do not advertise complete historical protection
merely because the preference is enabled.
Copy the preference with the other encryption settings when preparing a remote
profile. Recreate a connection when its effective encryption settings change.
Use the existing Tweak assessment and manual mismatch resolution; do not change
the remote's shared policy silently when importing or loading settings.
## Doctor
Use Commonlib's existing conditional recommendation rules. Recommend true when
the selected CouchDB settings have E2EE V2 and Property Encryption enabled and
the new preference is false. Do not require Hidden File Sync or Customisation
Sync to be active before offering the recommendation.
Retain the existing E2EE V2 recommendation for a legacy algorithm. After that
change, ensure the newly applicable Metadata recommendation is not hidden by a
premature `doctorProcessedVersion` update. Advance the Doctor rule revision so
an older completed consultation does not suppress this new recommendation.
Apply the preference only when the person accepts the recommendation. Include
the existing-data limitation, the manual Rebuild recommendation, and the need
for compatible clients in the explanation. Do not set `requireRebuild` or
`requireRebuildLocal` for this rule: the current host wrapper can schedule those
operations and restart. `recommendRebuild` currently exists only as an unused
rule field, so setting it alone does not display an explanation.
## Receiving a changed version document
The existing path is:
1. Commonlib receives a CouchDB replication change and calls
`parseSynchroniseResult` with the received documents.
2. The replication service feature passes them to
`ReplicateResultProcessor.enqueueAll`.
3. `processIfNonDocumentChange` recognises `type: versioninfo` and requests active
Replicator retirement when `version > VER`.
4. The owner closes admission, requests transfer cancellation, drains its work,
and closes the instance. The result callback does not wait for that transition.
Retain this observation path, but use Commonlib's complete assessment of the
identified control document. A changed `used_features` list must be inspected
even when the numeric version is unchanged. A mere revision change, list reorder,
or duplicate known identifier does not require an interruption.
Inspect the entire batch's control information before passing any file entries
to normal or optional processing. Recognise the fixed control-document ID and
validate its type and contents. A deleted or malformed control document is a
rejection, not a successful empty update.
When all requirements remain supported, refresh the assessment and affected
shared-setting checks; a newly added feature retires the current writer so its
next admission rechecks the shared Tweak policy. When a requirement is
unknown or incompatible, synchronously record the block for the affected
database, stop admitting new reflection and database operations, and request
retirement through the existing owner. Notify with the unknown identifiers as
text, using a generic message when no descriptive label exists.
File application and remote transfer have separate lifetimes. Requesting owner
retirement alone is not the application block. Keep the block separate from
temporary lifecycle suspension so an ordinary resume event cannot clear it.
Queued or waiting work checks it before starting another write; notifications
from an old physical database must not affect its replacement.
Do not await `onCloseActiveReplication` inside the callback which delivered the
change. That callback can belong to work which retirement must drain. Establish
the block immediately, request retirement without awaiting it there, and let the
owner perform cancellation and close in its existing order.
## Persistence and recovery boundaries
A CouchDB replication notification can arrive after the documents have entered
the local DB. Already-started network and filesystem operations may settle.
This feature does not promise rollback or atomic revocation of those operations.
Preserve pending work or durable reconciliation information when stopping. A
checkpoint may already include the documents which have not reached the Vault.
Do not drop those documents or depend on an ordinary reconnect to send them
again. A compatible client must reassess and reprocess or explicitly reacquire
them before lifting the applicable block.
Persist a blocked pending-work snapshot before the received-change callback
settles, while requesting owner retirement separately to avoid a circular wait.
Restore compatibility checks before replication result application and the next
ordinary synchronisation. Retain observed feature requirements with the pending
work snapshot so that a shortened version-document list does not release a
blocked local database on restart. A dismissed Notice or a changed connection
does not establish that the affected local data has become interpretable.
An older local generation without feature declarations remains readable during
normal application and cleaned-remote recovery; remote migration remains the
responsibility of the replication admission check.
Garbage Collection V3 is a beta manual operation which begins with an ordinary
bidirectional synchronisation. That admission checks the remote feature
contract; no additional per-step GC checks are introduced. The separate
cleaned-remote recovery path checks the local version document before its
first Chunk-reference count because it does not start with that synchronisation.
Use the same Commonlib assessment at the CLI, Fast Fetch, and direct-access
boundaries. The Obsidian result processor is one consumer, not the only place
which determines compatibility. Keep unrelated Vaults and databases operational.
## Verification and documentation
Keep focused tests for settings defaults and imports, the Doctor condition
matrix, acceptance and dismissal, connection replacement, and absence of an
automatic Rebuild, Fetch, or restart for this rule.
Add deterministic runtime tests for feature-only changes, version documents
first and last in a batch, unknown-name presentation, duplicate notifications,
queued and waiting reflection, stale database callbacks, restart, checkpointed
but unapplied documents, and cancellation without a circular wait.
Before this implementation, the host processor was checked with a focused unit probe: a numeric
incompatibility requests retirement, but the processor still applies a note in
the same batch when its host remains ready. A same-version document with an
unknown feature does not request retirement. These observations motivated the
new checks.
Extend real Obsidian Hidden File Sync and Customisation Sync scenarios and the
CLI interoperability checks. Inspect raw CouchDB documents as well as restored
files. Exercise an active connection when another client changes the feature
requirements, and verify that previously accepted data survives the stop.
Pending-work restoration and recovery with a compatible client are separate
boundaries.
The local packed Commonlib 0.1.30 candidate passed Commonlib unit and boundary
tests and the LiveSync build, type checks, and unit tests. Real Obsidian 1.12.7
passed the Hidden File Sync, Customisation Sync, and encrypted CLI-to-Obsidian
scenarios. The dedicated active-connection scenario changed a generation 12
remote to generation 13 with an unknown feature whilst continuous replication
was running. The local control document arrived, the active Replicator retired,
a subsequent replication was refused, and an earlier accepted note stayed in
the Vault. Focused host tests cover both batch orders, pending-work snapshots,
restart, stale callbacks, and the older local-generation case. Recovery after
upgrading to a future client that understands the unknown feature has not been
exercised in real Obsidian.
Keep the primary-language settings and troubleshooting guides, the
database-compatibility ADR, and Unreleased notes aligned with this behaviour.
Keep the detailed shared protocol in Commonlib and link to it after publication;
do not maintain another copy of its wire schema here. Translations are a separate
change. Update tested-version evidence when the implementation and its
validation have been accepted.
Related application contracts: [Replicator architecture](replicator_architecture.md),
[Tweak compatibility](tweak_compatibility.md), and
[database compatibility](../adr/2026_07_release_notes_and_database_compatibility.md).
+8
View File
@@ -245,6 +245,14 @@ Setting key: usePathObfuscation
In default, the path of the file is not obfuscated to improve the performance. If you enable this, the path of the file will be obfuscated. This is useful when you want to hide the path of the file.
#### Encrypt internal file Metadata
Setting key: encryptInternalMetadata
For CouchDB, this encrypts paths, times, sizes, and Chunk references in the Metadata used by Hidden File Sync and Customisation Sync. It requires E2EE V2 and **Property Encryption**. New Vaults enable the preference by default, but it has no effect until those prerequisites are enabled. Existing Vaults and older Setup URIs and QR codes keep it disabled unless you enable it.
Enabling the preference protects future Metadata writes. Existing Metadata and earlier revisions can remain readable in the remote database. If you want to protect existing Metadata too, prepare the authoritative data, update every device to a compatible version, and manually Rebuild the remote database. LiveSync does not gather data or start a Rebuild when you change this preference. Plaintext and encrypted Metadata can coexist during the transition. Document IDs, revision information, document counts, and ciphertext lengths remain visible.
#### Encryption Algorithm
Setting key: E2EEAlgorithm
+6
View File
@@ -96,6 +96,8 @@ Current releases automatically align compatible settings which control how new c
A missing legacy file-name case setting means case-insensitive handling. It matches an explicit disabled setting and does not require a rebuild for that difference. An explicitly enabled setting can use different document IDs and still requires a compatibility decision against either value. Other configuration differences shown in the dialogue must still be resolved.
If the mismatch names **Encrypt internal file Metadata**, update every device before accepting that preference. It affects subsequent Metadata writes for Hidden File Sync and Customisation Sync; it does not automatically protect existing Metadata. A manual remote Rebuild is strongly recommended if you need to protect existing paths, times, sizes, and Chunk references.
The `Sync now` command keeps routine replication progress quiet so that it is convenient to assign to a keyboard shortcut; assign one in Obsidian if that suits your workflow. A quiet command may still open this dialogue when a mismatch or another decision requires your attention.
The available actions depend on when the mismatch is found:
@@ -109,6 +111,10 @@ The available actions depend on when the mismatch is found:
Historic defect notices and renamed controls are retained in the [0.25 release history](releases/0.25.md) and [legacy release history](releases/legacy.md), rather than in the current troubleshooting path.
## The remote database uses an unknown feature
When a notice identifies an unknown feature, update this device and every other client of the same CouchDB database, including the CLI. The notice includes the feature identifier even if this version has no descriptive name for it. Synchronisation and pending file reflection pause because an older client may not interpret the Metadata and its Chunk references correctly. The cleaned-remote recovery path also checks compatibility before counting Chunk references. Do not remove the feature name from the remote version document to bypass the check. After updating, reconnect and review any pending file changes before running Garbage Collection.
## Setup and settings questions
### Share a configuration with another device
+1
View File
@@ -85,6 +85,7 @@
"test:e2e:obsidian:security-seed-reconnect": "tsx test/e2e-obsidian/scripts/security-seed-reconnect.ts",
"test:e2e:obsidian:hidden-file-snippet-sync": "tsx test/e2e-obsidian/scripts/hidden-file-snippet-sync.ts",
"test:e2e:obsidian:customisation-sync": "tsx test/e2e-obsidian/scripts/customisation-sync.ts",
"test:e2e:obsidian:remote-feature-change": "tsx test/e2e-obsidian/scripts/remote-feature-change.ts",
"test:e2e:obsidian:setting-markdown-export": "tsx test/e2e-obsidian/scripts/setting-markdown-export.ts",
"test:e2e:obsidian:upgrade-from-stable": "tsx test/e2e-obsidian/scripts/upgrade-from-stable.ts",
"test:e2e:obsidian:local-suite": "tsx test/e2e-obsidian/scripts/local-suite.ts",
@@ -1,4 +1,5 @@
import type { RemoteDBSettings } from "@vrtmrz/livesync-commonlib/compat/common/types";
import { usesEncryptedInternalMetadata } from "@vrtmrz/livesync-commonlib/replication";
type EndpointProjection = readonly [kind: "url" | "invalid-url", value: string];
@@ -77,6 +78,7 @@ export function getCouchDBReplicatorConfigurationIdentity(settings: RemoteDBSett
settings.useRequestAPI,
settings.disableRequestURI,
projectRemoteSecurity(settings),
usesEncryptedInternalMetadata(settings),
settings.enableCompression,
]);
}
@@ -67,6 +67,22 @@ describe("active Replicator configuration identity", () => {
);
});
it("recreates the CouchDB connection when internal Metadata encryption becomes effective", () => {
const active = configuredSettings({ usePathObfuscation: true, encryptInternalMetadata: false });
const enabled = { ...active, encryptInternalMetadata: true };
expect(getCouchDBReplicatorConfigurationIdentity(enabled)).not.toBe(
getCouchDBReplicatorConfigurationIdentity(active)
);
const inactive = { ...active, usePathObfuscation: false };
expect(getCouchDBReplicatorConfigurationIdentity({ ...inactive, encryptInternalMetadata: true })).toBe(
getCouchDBReplicatorConfigurationIdentity(inactive)
);
expect(getObjectStorageReplicatorConfigurationIdentity(enabled)).toBe(
getObjectStorageReplicatorConfigurationIdentity(active)
);
});
it("projects only the active CouchDB authentication mode", () => {
const basic = configuredSettings({ useJWT: false, jwtKey: "inactive-a" });
expect(getCouchDBReplicatorConfigurationIdentity({ ...basic, jwtKey: "inactive-b" })).toBe(
@@ -569,6 +569,7 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
}
}
// Internal Metadata encryption affects future Metadata writes and is not a rebuild requirement.
isNeedRebuildLocal() {
return this.isSomeDirty([
"useIndexedDBAdapter",
@@ -5,6 +5,7 @@ import {
DEFAULT_SETTINGS,
LOG_LEVEL_NOTICE,
type ObsidianLiveSyncSettings,
type EncryptionSettings,
LOG_LEVEL_VERBOSE,
} from "@vrtmrz/livesync-commonlib/compat/common/types";
import { Menu, type ButtonComponent } from "@/deps.ts";
@@ -31,11 +32,13 @@ import {
import { ConnectionStringParser } from "@vrtmrz/livesync-commonlib/compat/common/ConnectionString";
import type { RemoteConfigurationResult } from "@vrtmrz/livesync-commonlib/compat/common/ConnectionString";
import SetupRemote from "@/modules/features/SetupWizard/dialogs/SetupRemote.svelte";
import SetupRemoteE2EE from "@/modules/features/SetupWizard/dialogs/SetupRemoteE2EE.svelte";
import SetupRemoteCouchDB from "@/modules/features/SetupWizard/dialogs/SetupRemoteCouchDB.svelte";
import SetupRemoteBucket from "@/modules/features/SetupWizard/dialogs/SetupRemoteBucket.svelte";
import type {
SetupRemoteCouchDBInitialData,
SetupRemoteCouchDBResultType,
SetupRemoteE2EEResultType,
} from "@/modules/features/SetupWizard/dialogs/setupDialogTypes.ts";
import { syncActivatedRemoteSettings } from "./remoteConfigBuffer.ts";
@@ -116,7 +119,35 @@ export function paneRemoteConfig(
.onClick(async () => {
const setupManager = this.core.getModule(SetupManager);
const originalSettings = getSettingsFromEditingSettings(this.editingSettings);
await setupManager.onlyE2EEConfiguration(UserMode.Update, originalSettings);
const e2eeConf = await setupManager.dialogManager.openWithExplicitCancel<
SetupRemoteE2EEResultType,
EncryptionSettings
>(SetupRemoteE2EE, originalSettings);
if (e2eeConf === "cancelled") {
return;
}
const onlyInternalMetadataPreferenceChanged =
originalSettings.encryptInternalMetadata !== e2eeConf.encryptInternalMetadata &&
originalSettings.encrypt === e2eeConf.encrypt &&
originalSettings.passphrase === e2eeConf.passphrase &&
originalSettings.E2EEAlgorithm === e2eeConf.E2EEAlgorithm &&
originalSettings.usePathObfuscation === e2eeConf.usePathObfuscation;
if (onlyInternalMetadataPreferenceChanged) {
await this.services.setting.applyPartial(
{ encryptInternalMetadata: e2eeConf.encryptInternalMetadata },
true
);
this.editingSettings.encryptInternalMetadata = e2eeConf.encryptInternalMetadata;
if (this.initialSettings) {
this.initialSettings.encryptInternalMetadata = e2eeConf.encryptInternalMetadata;
}
this.requestUpdate();
} else {
await setupManager.onConfirmApplySettingsFromWizard(
{ ...originalSettings, ...e2eeConf },
UserMode.Update
);
}
updateE2EESummary();
})
.setButtonText("Configure")
@@ -243,6 +274,7 @@ export function paneRemoteConfig(
...DEFAULT_SETTINGS,
encrypt: this.editingSettings.encrypt,
usePathObfuscation: this.editingSettings.usePathObfuscation,
encryptInternalMetadata: this.editingSettings.encryptInternalMetadata,
passphrase: this.editingSettings.passphrase,
configPassphraseStore: this.editingSettings.configPassphraseStore,
});
@@ -2,6 +2,7 @@ import { afterEach, describe, expect, it, vi } from "vitest";
const runtime = vi.hoisted(() => ({
buttonClasses: [] as string[],
clickHandlers: [] as Array<() => Promise<void> | void>,
panels: [] as Array<{ destroy: ReturnType<typeof vi.fn> }>,
settingClasses: [] as string[],
}));
@@ -51,7 +52,8 @@ vi.mock("./LiveSyncSetting.ts", () => ({
setDestructive() {
return this;
},
onClick() {
onClick(callback: () => Promise<void> | void) {
runtime.clickHandlers.push(callback);
return this;
},
setButtonText() {
@@ -97,6 +99,7 @@ vi.mock("@vrtmrz/livesync-commonlib/compat/common/ConnectionString", () => ({
},
}));
vi.mock("@/modules/features/SetupWizard/dialogs/SetupRemote.svelte", () => ({ default: {} }));
vi.mock("@/modules/features/SetupWizard/dialogs/SetupRemoteE2EE.svelte", () => ({ default: {} }));
vi.mock("@/modules/features/SetupWizard/dialogs/SetupRemoteCouchDB.svelte", () => ({ default: {} }));
vi.mock("@/modules/features/SetupWizard/dialogs/SetupRemoteBucket.svelte", () => ({ default: {} }));
vi.mock("@/modules/features/SetupWizard/dialogs/SetupRemoteP2P.svelte", () => ({ default: {} }));
@@ -114,6 +117,7 @@ function createPanelElement(): HTMLElement {
afterEach(() => {
runtime.buttonClasses.length = 0;
runtime.clickHandlers.length = 0;
runtime.panels.length = 0;
runtime.settingClasses.length = 0;
vi.clearAllMocks();
@@ -148,4 +152,56 @@ describe("paneRemoteConfig", () => {
expect(runtime.panels[0].destroy).toHaveBeenCalledOnce();
});
it("applies an internal Metadata preference change without scheduling setup initialisation", async () => {
const originalSettings = {
encrypt: true,
passphrase: "passphrase",
E2EEAlgorithm: "v2",
usePathObfuscation: true,
encryptInternalMetadata: false,
remoteConfigurations: {},
};
const applyPartial = vi.fn(async () => {});
const onConfirmApplySettingsFromWizard = vi.fn(async () => {});
const setupManager = {
dialogManager: {
openWithExplicitCancel: vi.fn(async () => ({
encrypt: true,
passphrase: "passphrase",
E2EEAlgorithm: "v2",
usePathObfuscation: true,
encryptInternalMetadata: true,
})),
},
onConfirmApplySettingsFromWizard,
};
const host = {
editingSettings: { ...originalSettings },
initialSettings: { ...originalSettings },
services: { setting: { applyPartial } },
core: {
settings: { ...originalSettings },
getModule: vi.fn(() => setupManager),
},
lifetimeComponent: { register: vi.fn() },
requestUpdate: vi.fn(),
};
const addPanel = vi.fn((_parent: HTMLElement, heading: string) => ({
then(callback: (paneEl: HTMLElement) => void) {
if (heading === "E2EE Configuration") {
callback(createPanelElement());
}
},
}));
paneRemoteConfig.call(host as never, {} as HTMLElement, { addPanel } as never);
await runtime.clickHandlers[0]();
expect(applyPartial).toHaveBeenCalledWith({ encryptInternalMetadata: true }, true);
expect(onConfirmApplySettingsFromWizard).not.toHaveBeenCalled();
expect(host.editingSettings.encryptInternalMetadata).toBe(true);
expect(host.initialSettings.encryptInternalMetadata).toBe(true);
expect(host.requestUpdate).toHaveBeenCalledOnce();
});
});
@@ -26,7 +26,8 @@
passphrase: "",
E2EEAlgorithm: DEFAULT_SETTINGS.E2EEAlgorithm,
usePathObfuscation: true,
} as EncryptionSettings;
encryptInternalMetadata: true,
};
let encryptionSettings = $state<EncryptionSettings>({ ...default_encryption });
@@ -42,6 +43,11 @@
if (!encryptionSettings.encrypt) return true;
return encryptionSettings.passphrase.trim().length >= 1;
});
let canEncryptInternalMetadata = $derived(
encryptionSettings.encrypt &&
encryptionSettings.E2EEAlgorithm === E2EEAlgorithms.V2 &&
encryptionSettings.usePathObfuscation
);
function commit() {
setResult(pickEncryptionSettings(encryptionSettings));
@@ -87,6 +93,21 @@
</InfoNote>
{/if}
<InputRow label="Encrypt internal file Metadata">
<input
type="checkbox"
bind:checked={encryptionSettings.encryptInternalMetadata}
disabled={!canEncryptInternalMetadata}
/>
</InputRow>
<InfoNote>
This option applies only to CouchDB and requires End-to-End Encryption, the V2 algorithm, and Property Encryption
(Obfuscate Properties). The remote type is selected later in this setup wizard.
<br />
It protects Metadata written after the option is enabled; existing Metadata is not rewritten. A manual remote
Rebuild is strongly recommended to protect existing Metadata.
</InfoNote>
<ExtraItems title={translateMessage("Advanced")}>
<InputRow label={translateMessage("Encryption Algorithm")}>
<select bind:value={encryptionSettings.E2EEAlgorithm} disabled={!encryptionSettings.encrypt}>
@@ -1,6 +1,7 @@
import {
SYNCINFO_ID,
VER,
VERSIONING_DOCID,
type EntryVersionInfo,
type AnyEntry,
type EntryDoc,
type EntryLeaf,
@@ -8,6 +9,11 @@ import {
type MetaEntry,
type ObsidianLiveSyncSettings,
} from "@vrtmrz/livesync-commonlib/compat/common/types";
import {
assessRemoteFeatureDocument,
describeRemoteFeatureRejection,
REMOTE_FEATURE_GENERATION,
} from "@vrtmrz/livesync-commonlib/replication";
import { isChunk } from "@vrtmrz/livesync-commonlib/compat/common/typeUtils";
import {
LOG_LEVEL_DEBUG,
@@ -63,6 +69,10 @@ interface ReplicateResultProcessorContext {
readonly services: ReplicateResultProcessorServices;
}
type ReplicateResultProcessorState = {
databaseId?: string;
observedFeatures?: string[];
highestObservedVersion?: number;
invalidControlObserved?: boolean;
queued: PouchDB.Core.ExistingDocument<EntryDoc>[];
processing: PouchDB.Core.ExistingDocument<EntryDoc>[];
};
@@ -73,6 +83,10 @@ function shortenRev(rev: string | undefined): string {
if (!rev) return "undefined";
return rev.length > 10 ? rev.substring(0, 10) : rev;
}
function getPhysicalDatabaseId(database: PouchDB.Database<EntryDoc>): Promise<string | undefined> {
const identified = database as PouchDB.Database<EntryDoc> & { id?: () => Promise<string> };
return typeof identified.id === "function" ? identified.id() : Promise.resolve(undefined);
}
export class ReplicateResultProcessor {
private log(message: string, level: LOG_LEVEL = LOG_LEVEL_INFO) {
Logger(`[ReplicateResultProcessor] ${message}`, level);
@@ -115,6 +129,89 @@ export class ReplicateResultProcessor {
// If true, the processing queue processor bails the loop.
private _suspended: boolean = false;
// A temporary lifecycle resume cannot make an unknown remote format safe to apply.
private _compatibilityBlocked = false;
private _assessingDatabase = false;
private _physicalDatabase?: PouchDB.Database<EntryDoc>;
private _observedFeatures = new Set<string>();
private _highestObservedVersion = 0;
private _invalidControlObserved = false;
public get isCompatibilityBlocked() {
return this._compatibilityBlocked;
}
private refreshPhysicalDatabase() {
const current = this.localDatabase.localDatabase;
if (current === this._physicalDatabase) return;
if (this._physicalDatabase) {
this._compatibilityBlocked = false;
this._assessingDatabase = true;
this._observedFeatures.clear();
this._highestObservedVersion = 0;
this._invalidControlObserved = false;
this._queuedChanges = [];
this._processingChanges = [];
this._restoreFromSnapshot = undefined;
this.updateProcessingActivity();
}
this._physicalDatabase = current;
}
private shouldStopApplication(sourceDatabase: PouchDB.Database<EntryDoc>) {
return (
this._compatibilityBlocked || this._assessingDatabase || sourceDatabase !== this.localDatabase.localDatabase
);
}
private blockForIncompatibleVersion(document: unknown, recordObservation = true) {
const assessment = assessRemoteFeatureDocument(document);
const hadObservedVersion = this._highestObservedVersion > 0;
let newFeaturesAdded = false;
if (recordObservation) {
let changed = false;
if (assessment.status === "invalid-control") {
changed = !this._invalidControlObserved;
this._invalidControlObserved = true;
} else {
const version = (document as EntryVersionInfo).version;
if (version > this._highestObservedVersion) {
this._highestObservedVersion = version;
changed = true;
}
if (assessment.status === "supported" || assessment.status === "unknown-features") {
for (const feature of assessment.status === "supported"
? assessment.usedFeatures
: ((document as EntryVersionInfo).used_features ?? [])) {
if (this._observedFeatures.has(feature)) continue;
this._observedFeatures.add(feature);
changed = true;
newFeaturesAdded = true;
}
}
}
if (changed) this.triggerTakeSnapshot();
}
if (assessment.status === "supported" || assessment.status === "older-generation") {
// A live writer must recheck the shared Tweak policy after another
// client starts using a newly declared representation.
if (
assessment.status === "supported" &&
hadObservedVersion &&
newFeaturesAdded &&
!this._assessingDatabase
) {
this.context.requestActiveReplicatorRetirement();
}
return;
}
if (this._compatibilityBlocked) return;
this._compatibilityBlocked = true;
this.updateProcessingActivity();
this.log(describeRemoteFeatureRejection(assessment), LOG_LEVEL_NOTICE);
this.context.requestActiveReplicatorRetirement();
}
/**
* Whether the application accepts replicated documents being applied.
*
@@ -135,6 +232,8 @@ export class ReplicateResultProcessor {
public get isSuspended() {
return (
this._suspended ||
this._compatibilityBlocked ||
this._assessingDatabase ||
!this.acceptsResultApplication ||
this.context.currentSettings().suspendParseReplicationResult ||
this.services.appLifecycle.isSuspended()
@@ -145,17 +244,38 @@ export class ReplicateResultProcessor {
* Take a snapshot of the current processing state.
* This snapshot is stored in the KV database for recovery on restart.
*/
protected async _takeSnapshot() {
const snapshot = {
queued: this._queuedChanges.slice(),
processing: this._processingChanges.slice(),
} satisfies ReplicateResultProcessorState;
await this.context.getKeyValueDB().set(KV_KEY_REPLICATION_RESULT_PROCESSOR_SNAPSHOT, snapshot);
this.log(
`Snapshot taken. Queued: ${snapshot.queued.length}, Processing: ${snapshot.processing.length}`,
LOG_LEVEL_DEBUG
);
this.reportStatus();
private _snapshotWriter: Promise<void> = Promise.resolve();
protected _takeSnapshot(): Promise<void> {
// A blocked-batch flush must follow any earlier throttled write, or an
// older snapshot could replace the queue after the replication callback.
const write = this._snapshotWriter
.catch((): void => undefined)
.then(async () => {
const physicalDatabase = this.localDatabase.localDatabase;
const databaseId = await getPhysicalDatabaseId(physicalDatabase);
if (physicalDatabase !== this.localDatabase.localDatabase) return;
const snapshot = {
...(databaseId ? { databaseId } : {}),
observedFeatures: [...this._observedFeatures],
highestObservedVersion: this._highestObservedVersion,
invalidControlObserved: this._invalidControlObserved,
queued: this._queuedChanges.slice(),
processing: this._processingChanges.slice(),
} satisfies ReplicateResultProcessorState;
await this.context.getKeyValueDB().set(KV_KEY_REPLICATION_RESULT_PROCESSOR_SNAPSHOT, snapshot);
this.log(
`Snapshot taken. Queued: ${snapshot.queued.length}, Processing: ${snapshot.processing.length}`,
LOG_LEVEL_DEBUG
);
this.reportStatus();
});
this._snapshotWriter = write;
return write;
}
public async persistBlockedSnapshot(): Promise<void> {
if (this._compatibilityBlocked) await this._takeSnapshot();
}
/**
* Trigger taking a snapshot.
@@ -172,10 +292,42 @@ export class ReplicateResultProcessor {
* Restore from snapshot.
*/
public async restoreFromSnapshot() {
const physicalDatabase = this.localDatabase.localDatabase;
// Replication may have checkpointed a version document before its accompanying
// file changes reached the Vault. Assess the persisted requirement first.
let versionInfo: unknown;
try {
versionInfo = await this.localDatabase.getRaw(VERSIONING_DOCID);
} catch (error) {
if (!isNotFoundError(error)) throw error;
}
if (physicalDatabase !== this.localDatabase.localDatabase) return;
const snapshot = await this.context
.getKeyValueDB()
.get<ReplicateResultProcessorState>(KV_KEY_REPLICATION_RESULT_PROCESSOR_SNAPSHOT);
if (snapshot) {
if (physicalDatabase !== this.localDatabase.localDatabase) return;
const databaseId = await getPhysicalDatabaseId(physicalDatabase);
if (snapshot && (!snapshot.databaseId || !databaseId || snapshot.databaseId === databaseId)) {
for (const feature of snapshot.observedFeatures ?? []) this._observedFeatures.add(feature);
this._highestObservedVersion = Math.max(this._highestObservedVersion, snapshot.highestObservedVersion ?? 0);
this._invalidControlObserved ||= snapshot.invalidControlObserved === true;
}
if (versionInfo !== undefined) this.blockForIncompatibleVersion(versionInfo);
if (this._invalidControlObserved) this.blockForIncompatibleVersion(null, false);
if (this._highestObservedVersion > 0) {
this.blockForIncompatibleVersion(
{
_id: VERSIONING_DOCID,
type: "versioninfo",
version: this._highestObservedVersion,
...(this._highestObservedVersion >= REMOTE_FEATURE_GENERATION
? { used_features: [...this._observedFeatures] }
: {}),
},
false
);
}
if (snapshot && (!snapshot.databaseId || !databaseId || snapshot.databaseId === databaseId)) {
// Restoring the snapshot re-runs processing for both queued and processing items.
const newQueue = [...snapshot.processing, ...snapshot.queued, ...this._queuedChanges];
this._queuedChanges = [];
@@ -186,6 +338,9 @@ export class ReplicateResultProcessor {
);
// await this._takeSnapshot();
}
this._assessingDatabase = false;
this.updateProcessingActivity();
this.triggerProcessQueue();
}
private _restoreFromSnapshot: Promise<void> | undefined = undefined;
@@ -195,7 +350,9 @@ export class ReplicateResultProcessor {
* @returns Promise that resolves when restoration is complete.
*/
public restoreFromSnapshotOnce() {
this.refreshPhysicalDatabase();
if (!this._restoreFromSnapshot) {
this._assessingDatabase = true;
this._restoreFromSnapshot = this.restoreFromSnapshot();
}
return this._restoreFromSnapshot;
@@ -229,7 +386,17 @@ export class ReplicateResultProcessor {
* @param changes Changes to enqueue
*/
public enqueueAll(changes: PouchDB.Core.ExistingDocument<EntryDoc>[]) {
public enqueueAll(changes: PouchDB.Core.ExistingDocument<EntryDoc>[], sourceDatabase?: PouchDB.Database<EntryDoc>) {
if (sourceDatabase && sourceDatabase !== this.localDatabase.localDatabase) return;
const previousPhysicalDatabase = this._physicalDatabase;
this.refreshPhysicalDatabase();
if (previousPhysicalDatabase && previousPhysicalDatabase !== this._physicalDatabase) {
fireAndForget(() => this.restoreFromSnapshotOnce());
}
// Inspect every control document before a note in this batch can start applying.
for (const change of changes) {
if (change?._id === VERSIONING_DOCID) this.blockForIncompatibleVersion(change);
}
for (const change of changes) {
// Check if the change is not a document change (e.g., chunk, versioninfo, syncinfo), and processed it directly.
const isProcessed = this.processIfNonDocumentChange(change);
@@ -274,16 +441,8 @@ export class ReplicateResultProcessor {
this.log(`Processed chunk: ${shortenId(change._id)}`, LOG_LEVEL_DEBUG);
return true;
}
if (change.type == "versioninfo") {
if (change._id === VERSIONING_DOCID) {
this.log(`Version info document received: ${change._id}`, LOG_LEVEL_VERBOSE);
if (change.version > VER) {
// Fence and retire the active publication through its owner.
this.context.requestActiveReplicatorRetirement();
this.log(
`Remote database updated to incompatible version. update your Self-hosted LiveSync plugin.`,
LOG_LEVEL_NOTICE
);
}
return true;
}
if (
@@ -409,11 +568,12 @@ export class ReplicateResultProcessor {
// (per-document serialisation caps concurrency).
const releaser = await this._semaphore.acquire();
releaser();
if (this.isSuspended) break;
// Dequeue the next change
const doc = this._queuedChanges.shift();
if (doc) {
this._processingChanges.push(doc);
void this.parseDocumentChange(doc);
void this.parseDocumentChange(doc, this.localDatabase.localDatabase);
}
// Take snapshot (to be restored on next startup if needed)
this.triggerTakeSnapshot();
@@ -429,8 +589,12 @@ export class ReplicateResultProcessor {
* @param change
* @returns
*/
async parseDocumentChange(change: PouchDB.Core.ExistingDocument<EntryDoc>) {
async parseDocumentChange(
change: PouchDB.Core.ExistingDocument<EntryDoc>,
sourceDatabase: PouchDB.Database<EntryDoc> = this.localDatabase.localDatabase
) {
try {
if (this.shouldStopApplication(sourceDatabase)) return;
if (isAnyNote(change)) {
const docMtime = change.mtime ?? 0;
const maxMTime = this.context.currentSettings().maxMTimeForReflectEvents;
@@ -447,6 +611,7 @@ export class ReplicateResultProcessor {
}
// If the document is a virtual document, process it in the virtual document processor.
if (await this.services.replication.processVirtualDocument(change)) return;
if (this.shouldStopApplication(sourceDatabase)) return;
// If the document is version info, check compatibility and return.
if (isAnyNote(change)) {
const docPath = this.getPath(change);
@@ -454,6 +619,7 @@ export class ReplicateResultProcessor {
this.log(`Skipped: ${docPath}`, LOG_LEVEL_VERBOSE);
return;
}
if (this.shouldStopApplication(sourceDatabase)) return;
const size = change.size;
// Note that this size check depends size that in metadata, not the actual content size.
if (this.services.vault.isFileSizeTooLarge(size)) {
@@ -463,11 +629,20 @@ export class ReplicateResultProcessor {
);
return;
}
return await this.applyToDatabase(change);
return await this.applyToDatabase(change, sourceDatabase);
}
this.log(`Skipped unexpected non-note document: ${change._id}`, LOG_LEVEL_INFO);
return;
} finally {
// An in-flight parse may have started before the control document arrived.
// Retain it even if a later asynchronous boundary stopped application.
if (
this._compatibilityBlocked &&
sourceDatabase === this.localDatabase.localDatabase &&
!this._queuedChanges.includes(change)
) {
this._queuedChanges.push(change);
}
// Remove from processing queue
this._processingChanges = this._processingChanges.filter((e) => e !== change);
try {
@@ -487,12 +662,16 @@ export class ReplicateResultProcessor {
}
// Phase 2: apply the document to database
protected applyToDatabase(doc: PouchDB.Core.ExistingDocument<AnyEntry>) {
protected applyToDatabase(
doc: PouchDB.Core.ExistingDocument<AnyEntry>,
sourceDatabase: PouchDB.Database<EntryDoc> = this.localDatabase.localDatabase
) {
return this.withCounting(async () => {
let releaser: Awaited<ReturnType<typeof this._semaphore.acquire>> | undefined = undefined;
try {
releaser = await this._semaphore.acquire();
await this._applyToDatabase(doc);
if (this.shouldStopApplication(sourceDatabase)) return;
await this._applyToDatabase(doc, sourceDatabase);
} catch (e) {
this.log(`Error while processing replication result`, LOG_LEVEL_NOTICE);
this.logError(e);
@@ -506,12 +685,16 @@ export class ReplicateResultProcessor {
}
// Phase 2.1: process the document and apply to storage
// This function is serialized per document to avoid race-condition for the same document.
private _applyToDatabase(doc_: PouchDB.Core.ExistingDocument<AnyEntry>) {
private _applyToDatabase(
doc_: PouchDB.Core.ExistingDocument<AnyEntry>,
sourceDatabase: PouchDB.Database<EntryDoc>
) {
const dbDoc = doc_ as LoadedEntry; // It has no `data`
const path = this.getPath(dbDoc);
return serialized(`replication-process:${dbDoc._id}`, async () => {
const docNote = `${path} (${shortenId(dbDoc._id)}, ${shortenRev(dbDoc._rev)})`;
const isRequired = await this.checkIsChangeRequiredForDatabaseProcessing(dbDoc);
if (this.shouldStopApplication(sourceDatabase)) return;
if (!isRequired) {
this.log(`Skipped (Not latest): ${docNote}`, LOG_LEVEL_VERBOSE);
return;
@@ -525,6 +708,7 @@ export class ReplicateResultProcessor {
const doc = isDeleted
? { ...dbDoc, data: "" }
: await this.localDatabase.getDBEntryFromMeta({ ...dbDoc }, false, true);
if (this.shouldStopApplication(sourceDatabase)) return;
if (!doc) {
// Failed to gather content
this.log(`Failed to gather content of ${docNote}`, LOG_LEVEL_NOTICE);
@@ -535,9 +719,10 @@ export class ReplicateResultProcessor {
// Already processed
this.log(`Processed by other processor: ${docNote}`, LOG_LEVEL_DEBUG);
} else if (this.services.vault.isValidPath(this.getPath(doc))) {
if (this.shouldStopApplication(sourceDatabase)) return;
// Apply to storage if the path is valid
try {
const reflected = await this.applyToStorage(doc as MetaEntry);
const reflected = await this.applyToStorage(doc as MetaEntry, sourceDatabase);
if (!reflected) {
this.reportVaultReflectionFailure(doc as MetaEntry);
return;
@@ -558,9 +743,15 @@ export class ReplicateResultProcessor {
* @param entry
* @returns
*/
protected applyToStorage(entry: MetaEntry) {
protected applyToStorage(
entry: MetaEntry,
sourceDatabase: PouchDB.Database<EntryDoc> = this.localDatabase.localDatabase
) {
return this.withCounting(
() => this.services.replication.processSynchroniseResult(entry),
() =>
this.shouldStopApplication(sourceDatabase)
? Promise.resolve(false)
: this.services.replication.processSynchroniseResult(entry),
this.services.replication.storageApplyingCount
);
}
@@ -1,7 +1,12 @@
import { promiseWithResolvers } from "octagonal-wheels/promises";
import { reactiveSource } from "octagonal-wheels/dataobject/reactive";
import { describe, expect, it, vi } from "vitest";
import { VER, type EntryDoc, type FilePathWithPrefix } from "@vrtmrz/livesync-commonlib/compat/common/types";
import {
VERSIONING_DOCID,
type EntryDoc,
type FilePathWithPrefix,
} from "@vrtmrz/livesync-commonlib/compat/common/types";
import { ENCRYPTED_INTERNAL_METADATA_FEATURE, REMOTE_FEATURE_GENERATION } from "@vrtmrz/livesync-commonlib/replication";
import {
isValidFilenameInAndroid,
isValidFilenameInWidows,
@@ -37,6 +42,10 @@ type SetupOptions = {
isValidPath?: (path: string) => boolean;
processSynchroniseResult?: (entry: unknown) => Promise<boolean>;
setSnapshot?: (key: string, value: unknown) => Promise<unknown>;
getSnapshot?: (key: string) => Promise<unknown>;
localVersionInfo?: unknown;
isTargetFile?: (path: string) => Promise<boolean>;
databaseId?: string;
};
function setup(options: SetupOptions = {}) {
@@ -47,6 +56,9 @@ function setup(options: SetupOptions = {}) {
const isReady = vi.fn(() => options.applicationReady ?? true);
const isValidPath = vi.fn(options.isValidPath ?? (() => true));
const getDBEntryFromMeta = vi.fn(async (entry: object) => ({ ...entry, data: "x" }));
const localPhysicalDatabase = {
...(options.databaseId ? { id: vi.fn(async () => options.databaseId) } : {}),
} as PouchDB.Database<EntryDoc>;
const core = {
services: {
appLifecycle: { isReady, isSuspended: () => false },
@@ -62,14 +74,21 @@ function setup(options: SetupOptions = {}) {
},
replicator: { onCloseActiveReplication, runBoundedLocalApplicationActivity },
vault: {
isTargetFile: vi.fn(async () => true),
isTargetFile: vi.fn(options.isTargetFile ?? (async () => true)),
isFileSizeTooLarge: vi.fn(() => false),
isValidPath,
},
},
kvDB: { set: setSnapshot },
kvDB: { set: setSnapshot, get: vi.fn(options.getSnapshot ?? (async () => undefined)) },
localDatabase: {
getRaw: vi.fn(async (id: string) => ({ _id: id, _rev: "1-test" })),
localDatabase: localPhysicalDatabase,
getRaw: vi.fn(async (id: string) => {
if (id === VERSIONING_DOCID) {
if (options.localVersionInfo === undefined) throw { status: 404 };
return options.localVersionInfo;
}
return { _id: id, _rev: "1-test" };
}),
getDBEntryFromMeta,
},
};
@@ -88,6 +107,9 @@ function setup(options: SetupOptions = {}) {
} as never);
return {
getDBEntryFromMeta,
isTargetFile: core.services.vault.isTargetFile,
localPhysicalDatabase,
localDatabase: core.localDatabase,
isReady,
isValidPath,
onCloseActiveReplication,
@@ -145,9 +167,9 @@ describe("ReplicateResultProcessor", () => {
});
}
expect(processSynchroniseResult).toHaveBeenCalledTimes(11);
expect(processSynchroniseResult.mock.calls.some(([entry]) =>
(entry as { _id: string })._id === "unrelated-queue"
)).toBe(true);
expect(
processSynchroniseResult.mock.calls.some(([entry]) => (entry as { _id: string })._id === "unrelated-queue")
).toBe(true);
});
it("suspends result application while the application is not ready", () => {
@@ -199,10 +221,10 @@ describe("ReplicateResultProcessor", () => {
it("retires active ownership when a newer remote version is observed", async () => {
const { onCloseActiveReplication, processor } = setup();
const versionInfo = {
_id: "versioninfo",
_id: VERSIONING_DOCID,
_rev: "1-test",
type: "versioninfo",
version: VER + 1,
version: REMOTE_FEATURE_GENERATION + 1,
} as unknown as PouchDB.Core.ExistingDocument<EntryDoc>;
processor.enqueueAll([versionInfo]);
@@ -210,6 +232,251 @@ describe("ReplicateResultProcessor", () => {
await vi.waitFor(() => expect(onCloseActiveReplication).toHaveBeenCalledOnce());
});
it("continues applying documents after restoring a legacy local version document", async () => {
const { onCloseActiveReplication, processor, processSynchroniseResult } = setup({
localVersionInfo: {
_id: VERSIONING_DOCID,
type: "versioninfo",
version: 11,
},
});
await processor.restoreFromSnapshotOnce();
processor.enqueueAll([note("legacy-database-note")]);
await vi.waitFor(() => expect(processSynchroniseResult).toHaveBeenCalledOnce());
expect(processor.isSuspended).toBe(false);
expect(onCloseActiveReplication).not.toHaveBeenCalled();
});
it.each(["first", "last"] as const)(
"holds an entire received batch when an unknown feature is %s",
async (position) => {
const { onCloseActiveReplication, processor, processSynchroniseResult } = setup();
const versionInfo = {
_id: VERSIONING_DOCID,
_rev: "2-unknown",
type: "versioninfo",
version: REMOTE_FEATURE_GENERATION,
used_features: ["future-format-v7"],
} as PouchDB.Core.ExistingDocument<EntryDoc>;
const document = note("pending-after-compatibility-change");
processor.enqueueAll(position === "first" ? [versionInfo, document] : [document, versionInfo]);
await vi.waitFor(() => expect(onCloseActiveReplication).toHaveBeenCalledOnce());
await Promise.resolve();
expect(processor.isSuspended).toBe(true);
expect(processSynchroniseResult).not.toHaveBeenCalled();
processor.resume();
await Promise.resolve();
expect(processSynchroniseResult).not.toHaveBeenCalled();
}
);
it("continues when a newly received feature is supported", async () => {
const { onCloseActiveReplication, processor, processSynchroniseResult } = setup();
const versionInfo = {
_id: VERSIONING_DOCID,
_rev: "2-supported",
type: "versioninfo",
version: REMOTE_FEATURE_GENERATION,
used_features: [ENCRYPTED_INTERNAL_METADATA_FEATURE],
} as PouchDB.Core.ExistingDocument<EntryDoc>;
processor.enqueueAll([versionInfo, note("supported-update")]);
await vi.waitFor(() => expect(processSynchroniseResult).toHaveBeenCalledOnce());
expect(onCloseActiveReplication).not.toHaveBeenCalled();
});
it("rechecks shared writer settings when a supported feature is added to an active database", async () => {
const { onCloseActiveReplication, processor } = setup({
localVersionInfo: {
_id: VERSIONING_DOCID,
type: "versioninfo",
version: REMOTE_FEATURE_GENERATION,
used_features: [],
},
});
await processor.restoreFromSnapshotOnce();
const versionInfo = {
_id: VERSIONING_DOCID,
_rev: "2-supported",
type: "versioninfo",
version: REMOTE_FEATURE_GENERATION,
used_features: [ENCRYPTED_INTERNAL_METADATA_FEATURE],
} as PouchDB.Core.ExistingDocument<EntryDoc>;
processor.enqueueAll([versionInfo]);
processor.enqueueAll([{ ...versionInfo, _rev: "3-same-features" }]);
expect(onCloseActiveReplication).toHaveBeenCalledOnce();
expect(processor.isSuspended).toBe(false);
});
it("retains an in-flight note when a feature change arrives during an asynchronous check", async () => {
const targetCheck = promiseWithResolvers<boolean>();
const { isTargetFile, onCloseActiveReplication, processor, processSynchroniseResult } = setup({
isTargetFile: async () => targetCheck.promise,
});
const pending = note("in-flight");
processor.enqueueAll([pending]);
await vi.waitFor(() => expect(isTargetFile).toHaveBeenCalledOnce());
processor.enqueueAll([
{
_id: VERSIONING_DOCID,
_rev: "2-unknown",
type: "versioninfo",
version: REMOTE_FEATURE_GENERATION,
used_features: ["future-format-v7"],
} as PouchDB.Core.ExistingDocument<EntryDoc>,
]);
targetCheck.resolve(true);
await vi.waitFor(() => expect(processor["_processingChanges"]).toHaveLength(0));
expect(processor["_queuedChanges"]).toContain(pending);
expect(processSynchroniseResult).not.toHaveBeenCalled();
expect(onCloseActiveReplication).toHaveBeenCalledOnce();
});
it("restores a checkpointed note only after assessing the persisted version document", async () => {
const pending = note("checkpointed");
const { onCloseActiveReplication, processor, processSynchroniseResult } = setup({
localVersionInfo: {
_id: VERSIONING_DOCID,
type: "versioninfo",
version: REMOTE_FEATURE_GENERATION,
used_features: ["future-format-v7"],
},
getSnapshot: async () => ({ processing: [pending], queued: [] }),
});
await processor.restoreFromSnapshotOnce();
expect(processor.isSuspended).toBe(true);
expect(processor["_queuedChanges"]).toContain(pending);
expect(processSynchroniseResult).not.toHaveBeenCalled();
expect(onCloseActiveReplication).toHaveBeenCalledOnce();
});
it("retains an observed unknown feature when the local version list is later shortened", async () => {
let snapshot: unknown;
const first = setup({
databaseId: "same-database",
setSnapshot: async (_key, value) => {
snapshot = value;
},
});
const pending = note("checkpointed-after-list-change");
first.processor.enqueueAll([
{
_id: VERSIONING_DOCID,
_rev: "2-unknown",
type: "versioninfo",
version: REMOTE_FEATURE_GENERATION,
used_features: ["future-format-v7"],
} as PouchDB.Core.ExistingDocument<EntryDoc>,
pending,
]);
await vi.waitFor(() => expect(snapshot).toBeDefined());
const resumed = setup({
databaseId: "same-database",
localVersionInfo: {
_id: VERSIONING_DOCID,
type: "versioninfo",
version: REMOTE_FEATURE_GENERATION,
used_features: [],
},
getSnapshot: async () => snapshot,
});
await resumed.processor.restoreFromSnapshotOnce();
expect(resumed.processor.isSuspended).toBe(true);
expect(resumed.processor["_queuedChanges"]).toContain(pending);
expect(resumed.processSynchroniseResult).not.toHaveBeenCalled();
expect(resumed.onCloseActiveReplication).toHaveBeenCalledOnce();
});
it("does not restore pending work belonging to a replaced physical database", async () => {
const pending = note("old-database");
const { processor, processSynchroniseResult } = setup({
databaseId: "replacement-database",
getSnapshot: async () => ({ databaseId: "retired-database", processing: [pending], queued: [] }),
});
await processor.restoreFromSnapshotOnce();
expect(processor["_queuedChanges"]).toHaveLength(0);
expect(processSynchroniseResult).not.toHaveBeenCalled();
expect(processor.isSuspended).toBe(false);
});
it("reports unknown identifiers and retires only once for repeated notifications", () => {
const log = vi.fn((_message: unknown, _level?: number) => undefined);
setGlobalLogFunction(log);
try {
const { onCloseActiveReplication, processor } = setup();
const versionInfo = {
_id: VERSIONING_DOCID,
_rev: "2-unknown",
type: "versioninfo",
version: REMOTE_FEATURE_GENERATION,
used_features: ["future-format-v7"],
} as PouchDB.Core.ExistingDocument<EntryDoc>;
processor.enqueueAll([versionInfo]);
processor.enqueueAll([versionInfo]);
expect(onCloseActiveReplication).toHaveBeenCalledOnce();
expect(log).toHaveBeenCalledWith(
"[ReplicateResultProcessor] Unknown features are in use: future-format-v7",
LOG_LEVEL_NOTICE,
undefined
);
} finally {
setGlobalLogFunction(defaultLogger);
}
});
it("ignores a late feature notification from a replaced physical database", () => {
const { onCloseActiveReplication, processor } = setup();
const oldPhysicalDatabase = {} as PouchDB.Database<EntryDoc>;
const versionInfo = {
_id: VERSIONING_DOCID,
_rev: "2-unknown",
type: "versioninfo",
version: REMOTE_FEATURE_GENERATION,
used_features: ["future-format-v7"],
} as PouchDB.Core.ExistingDocument<EntryDoc>;
processor.enqueueAll([versionInfo], oldPhysicalDatabase);
expect(processor.isSuspended).toBe(false);
expect(onCloseActiveReplication).not.toHaveBeenCalled();
});
it("reassesses compatibility for a replacement local database", async () => {
const { localDatabase, onCloseActiveReplication, processor } = setup();
processor.enqueueAll([
{
_id: VERSIONING_DOCID,
_rev: "2-unknown",
type: "versioninfo",
version: REMOTE_FEATURE_GENERATION,
used_features: ["future-format-v7"],
} as PouchDB.Core.ExistingDocument<EntryDoc>,
]);
expect(processor.isSuspended).toBe(true);
localDatabase.localDatabase = {} as PouchDB.Database<EntryDoc>;
await processor.restoreFromSnapshotOnce();
expect(processor.isSuspended).toBe(false);
expect(onCloseActiveReplication).toHaveBeenCalledOnce();
});
it("scans normal-file metadata without loading chunk documents and requeues it", async () => {
const documents = [
{ _id: "first", _rev: "1-a", type: "plain", path: "first.md" },
@@ -1,4 +1,8 @@
import type { ObsidianLiveSyncSettings } from "@vrtmrz/livesync-commonlib/compat/common/types";
import {
VERSIONING_DOCID,
type EntryDoc,
type ObsidianLiveSyncSettings,
} from "@vrtmrz/livesync-commonlib/compat/common/types";
import { assessTweakCompatibility } from "@vrtmrz/livesync-commonlib/settings";
import { LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, Logger } from "octagonal-wheels/common/logger";
import { skipIfDuplicated } from "octagonal-wheels/concurrency/lock";
@@ -7,12 +11,27 @@ import { LiveSyncCouchDBReplicator } from "@vrtmrz/livesync-commonlib/compat/rep
import {
CENTRAL_COMPATIBILITY_REJECTION_REASONS,
REPLICATION_PROGRESS_PRESENTATIONS,
assessRemoteFeatureDocument,
describeRemoteFeatureRejection,
type ReplicatorInstance,
type ReplicationFailureRequest,
} from "@vrtmrz/livesync-commonlib/replication";
import { $msg } from "@/common/translation";
import { usesLegacyIndexedDBAdapter } from "@/common/compatibilitySettings";
import type { LiveSyncBaseCore } from "@/LiveSyncBaseCore";
import type PouchDB from "pouchdb-core";
async function canInterpretCleanupDatabase(db: PouchDB.Database<EntryDoc>): Promise<boolean> {
try {
const assessment = assessRemoteFeatureDocument(await db.get(VERSIONING_DOCID));
if (assessment.status === "supported" || assessment.status === "older-generation") return true;
Logger(`Database cleanup cancelled: ${describeRemoteFeatureRejection(assessment)}`, LOG_LEVEL_NOTICE);
} catch (error) {
Logger("Database cleanup cancelled: feature compatibility could not be checked.", LOG_LEVEL_NOTICE);
Logger(error, LOG_LEVEL_INFO);
}
return false;
}
type CentralCompatibilityRecoveryServices = Pick<
LiveSyncBaseCore["services"],
@@ -60,6 +79,7 @@ export function createCentralCompatibilityRecovery(context: CentralCompatibility
) {
Logger("The remote database has been cleaned.", showProgress ? LOG_LEVEL_NOTICE : LOG_LEVEL_INFO);
await skipIfDuplicated("cleanup", async () => {
if (!(await canInterpretCleanupDatabase(context.getLocalDatabase().localDatabase))) return;
const count = await purgeUnreferencedChunks(context.getLocalDatabase().localDatabase, true);
const message = `The remote database has been cleaned up.
To synchronize, this device must be also cleaned up. ${count} chunk(s) will be erased from this device.
@@ -1,5 +1,5 @@
import { describe, expect, it, vi } from "vitest";
import type { ObsidianLiveSyncSettings } from "@vrtmrz/livesync-commonlib/compat/common/types";
import { VERSIONING_DOCID, type ObsidianLiveSyncSettings } from "@vrtmrz/livesync-commonlib/compat/common/types";
import { assessTweakCompatibility } from "@vrtmrz/livesync-commonlib/settings";
import { defaultLogger, LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, setGlobalLogFunction } from "octagonal-wheels/common/logger";
import {
@@ -24,6 +24,49 @@ import { LiveSyncCouchDBReplicator } from "@vrtmrz/livesync-commonlib/compat/rep
import { createCentralCompatibilityRecovery } from "./centralCompatibilityRecovery";
describe("central compatibility recovery", () => {
it("does not count chunks for cleanup when local feature requirements are unknown", async () => {
chunkMocks.purgeUnreferencedChunks.mockClear();
const confirmWithMessage = vi.fn(async () => "Dismiss");
const recovery = createCentralCompatibilityRecovery({
confirm: { confirmWithMessage },
getLocalDatabase: () => ({
localDatabase: {
get: vi.fn(async (id: string) => ({
_id: id,
type: "versioninfo",
version: 13,
used_features: ["future-format-v7"],
})),
},
}),
services: { replicator: {} },
} as never);
await recovery.reconcileCleanedRemote(true, {} as ObsidianLiveSyncSettings, {} as never);
expect(chunkMocks.purgeUnreferencedChunks).not.toHaveBeenCalled();
expect(confirmWithMessage).not.toHaveBeenCalled();
});
it("allows cleanup counting for a legacy local version document", async () => {
chunkMocks.purgeUnreferencedChunks.mockClear();
const confirmWithMessage = vi.fn(async () => "Dismiss");
const recovery = createCentralCompatibilityRecovery({
confirm: { confirmWithMessage },
getLocalDatabase: () => ({
localDatabase: {
get: vi.fn(async (id: string) => ({ _id: id, type: "versioninfo", version: 11 })),
},
}),
services: { replicator: {} },
} as never);
await recovery.reconcileCleanedRemote(true, {} as ObsidianLiveSyncSettings, {} as never);
expect(chunkMocks.purgeUnreferencedChunks).toHaveBeenCalledWith(expect.anything(), true);
expect(confirmWithMessage).toHaveBeenCalledOnce();
});
it("passes the failed attempt's exact tweak assessment to mismatch resolution", async () => {
const setting = { customChunkSize: 0 };
const preferredTweakValue = { customChunkSize: 60 };
@@ -292,7 +335,9 @@ describe("central compatibility recovery", () => {
});
const runFiniteReplicationActivity = vi.fn(async (task: () => unknown) => await task());
const openOneShotReplication = vi.fn(async () => true);
const remoteDatabase = { close: vi.fn(async () => undefined) };
const remoteDatabase = {
close: vi.fn(async () => undefined),
};
const close = vi.fn(async () => undefined);
const activeReplicator = Object.assign(new LiveSyncCouchDBReplicator({} as never), {
connectRemoteCouchDBWithSetting: vi.fn(async () => ({ db: remoteDatabase, close })),
@@ -303,7 +348,12 @@ describe("central compatibility recovery", () => {
const runWithActiveReplicatorContext = vi.fn(async (task: (context: unknown) => unknown) =>
task(expectedContext)
);
const localDatabase = { localDatabase: {}, clearCaches: vi.fn() };
const localDatabase = {
localDatabase: {
get: vi.fn(async () => ({ _id: VERSIONING_DOCID, type: "versioninfo", version: 12 })),
},
clearCaches: vi.fn(),
};
const getLocalDatabase = vi.fn(() => localDatabase);
const recovery = createCentralCompatibilityRecovery({
confirm: { confirmWithMessage: vi.fn(async () => "Cleanup") },
@@ -335,7 +385,7 @@ describe("central compatibility recovery", () => {
activityFinished.mock.invocationCallOrder[0]
);
expect(chunkMocks.balanceChunkPurgedDBs).toHaveBeenCalledOnce();
expect(getLocalDatabase).toHaveBeenCalledTimes(2);
expect(getLocalDatabase).toHaveBeenCalled();
expect(close).toHaveBeenCalledOnce();
expect(close.mock.invocationCallOrder[0]).toBeLessThan(activityFinished.mock.invocationCallOrder[0]);
});
+8 -6
View File
@@ -98,19 +98,21 @@ export function useReplicationFeature<TContext extends ServiceContext, TCommands
clearHandlers();
return Promise.resolve(true);
});
services.databaseEvents.onDatabaseInitialised.addHandler(() => {
fireAndForget(() => resultProcessor.restoreFromSnapshotOnce());
return Promise.resolve(true);
services.databaseEvents.onDatabaseInitialised.addHandler(async () => {
await resultProcessor.restoreFromSnapshotOnce();
return true;
});
services.appLifecycle.onSettingLoaded.addHandler(initialiseAutomaticReplicationTriggers);
services.replication.parseSynchroniseResult.addHandler((documents) => {
resultProcessor.enqueueAll(documents);
return Promise.resolve(true);
services.replication.parseSynchroniseResult.addHandler(async (documents, sourceDatabase) => {
resultProcessor.enqueueAll(documents, sourceDatabase);
await resultProcessor.persistBlockedSnapshot();
return true;
});
services.replication.onBeforeReplicate.addHandler(onlinePreflight, 10);
services.replication.onPrepareCentralRemoteReplication.addHandler(securitySeedPreflight);
services.replication.onBeforeReplicate.addHandler(async () => {
await resultProcessor.restoreFromSnapshotOnce();
if (resultProcessor.isCompatibilityBlocked) return false;
unresolvedErrorManager.clearErrors();
return true;
}, 100);
@@ -1,6 +1,7 @@
import { describe, expect, it, vi } from "vitest";
import { createServiceContext } from "@vrtmrz/livesync-commonlib/context";
import { VER, type EntryDoc } from "@vrtmrz/livesync-commonlib/compat/common/types";
import { VERSIONING_DOCID, type EntryDoc } from "@vrtmrz/livesync-commonlib/compat/common/types";
import { REMOTE_FEATURE_GENERATION } from "@vrtmrz/livesync-commonlib/replication";
import { promiseWithResolvers } from "octagonal-wheels/promises";
import { useReplicationFeature } from "./index";
@@ -19,8 +20,12 @@ type SetupOptions = {
};
function setup(options: SetupOptions = {}) {
const defaultLocalDatabase = {
localDatabase: {},
getRaw: vi.fn(async () => { throw { status: 404 }; }),
};
const {
getLocalDatabase = () => ({}),
getLocalDatabase = () => defaultLocalDatabase,
keyValueDB = {
kvDB: {
get: vi.fn(async () => undefined),
@@ -39,7 +44,7 @@ function setup(options: SetupOptions = {}) {
API: { isMobile: vi.fn(() => false), isOnline: true },
appLifecycle: {
getUnresolvedMessages: { addHandler: vi.fn() },
isReady: true,
isReady: vi.fn(() => true),
isSuspended: vi.fn(() => false),
onSettingLoaded: { addHandler: vi.fn() },
},
@@ -48,6 +53,7 @@ function setup(options: SetupOptions = {}) {
keyValueDB,
path: { getPath: vi.fn((entry: { path: string }) => entry.path) },
replication: {
replicationResultCount: { value: 0 },
onBeforeReplicate: {
addHandler: vi.fn((handler: BooleanHandler, priority = 0) => {
beforeReplicateHandlers.set(priority, handler);
@@ -165,10 +171,10 @@ describe("replication serviceFeature composition", () => {
const onCloseActiveReplication = vi.fn(() => retirement.promise);
const harness = setup({ onCloseActiveReplication });
const versionInfo = {
_id: "versioninfo",
_id: VERSIONING_DOCID,
_rev: "1-test",
type: "versioninfo",
version: VER + 1,
version: REMOTE_FEATURE_GENERATION + 1,
} as unknown as PouchDB.Core.ExistingDocument<EntryDoc>;
expect(harness.parseHandler).toBeDefined();
@@ -177,4 +183,59 @@ describe("replication serviceFeature composition", () => {
retirement.resolve(true);
});
it("persists a blocked batch before its replication callback settles", async () => {
const writeFinished = promiseWithResolvers<void>();
const writes: Array<{ queued: Array<{ _id: string }> }> = [];
const { parseHandler } = setup({
keyValueDB: {
kvDB: {
get: vi.fn(async () => undefined),
set: vi.fn(async (_key, value) => {
writes.push(value as { queued: Array<{ _id: string }> });
await writeFinished.promise;
}),
},
},
});
const pending = {
_id: "checkpointed-note",
_rev: "1-test",
type: "plain",
path: "checkpointed-note.md",
} as PouchDB.Core.ExistingDocument<EntryDoc>;
const unknownVersion = {
_id: VERSIONING_DOCID,
_rev: "2-unknown",
type: "versioninfo",
version: REMOTE_FEATURE_GENERATION,
used_features: ["future-format-v7"],
} as PouchDB.Core.ExistingDocument<EntryDoc>;
let callbackSettled = false;
const callback = parseHandler!([pending, unknownVersion]).then((result) => {
callbackSettled = true;
return result;
});
await vi.waitFor(() => expect(writes.length).toBeGreaterThan(0));
expect(callbackSettled).toBe(false);
writeFinished.resolve();
await expect(callback).resolves.toBe(true);
expect(writes[writes.length - 1]?.queued.map((entry) => entry._id)).toContain(pending._id);
});
it("refuses another replication after observing an unknown feature locally", async () => {
const { beforeReplicateHandlers, parseHandler } = setup();
const unknownVersion = {
_id: VERSIONING_DOCID,
_rev: "2-unknown",
type: "versioninfo",
version: REMOTE_FEATURE_GENERATION,
used_features: ["future-format-v7"],
} as PouchDB.Core.ExistingDocument<EntryDoc>;
await parseHandler!([unknownVersion]);
await expect(beforeReplicateHandlers.get(100)!(false)).resolves.toBe(false);
});
});
+2
View File
@@ -200,6 +200,8 @@ This proves in real Obsidian the plug-in behaviour shared by supported platforms
`test:e2e:obsidian:customisation-sync` runs a two-vault Customisation Sync workflow. It scans a real snippet CSS file, config JSON file, and sample plug-in fixture into per-file Customisation Sync data, synchronises the entries through CouchDB, applies them on the second vault, verifies the resulting `.obsidian` files, propagates a snippet update, and verifies deletion of the source-vault snippet sync data without confusing it with the target vault's own applied copy.
`test:e2e:obsidian:remote-feature-change` starts real Obsidian with continuous CouchDB replication, then changes the remote version document from generation 12 to generation 13 with an unknown feature. It waits for the control document to reach the local database and the active Replicator to retire, checks that another replication is refused, and verifies that an already accepted Vault note remains intact. It is a focused test outside `test:e2e:obsidian:local-suite`; pending-work recovery with a future compatible client remains a separate validation boundary.
`test:e2e:obsidian:setting-markdown-export` enables setting Markdown export, waits for the generated Markdown file in the vault, and verifies that credentials are omitted when `writeCredentialsForSettingSync=false`.
`test:e2e:obsidian:upgrade-from-stable` is the release-acceptance upgrade workflow. It installs the exact published 0.25.83 artefacts into an isolated Vault, verifies their pinned SHA-256 values, and then replaces only the plug-in artefacts with the current target while retaining the same Vault and isolated Obsidian profile. The first run downloads the old release into the ignored `_testdata/releases` cache; every later run verifies the cached bytes before use.
@@ -210,6 +210,7 @@ async function configureLiveSyncCli(
encrypt: true,
passphrase: e2eePassphrase,
usePathObfuscation: true,
encryptInternalMetadata: true,
doctorProcessedVersion: "0.25.27",
isConfigured: true,
});
@@ -323,6 +324,7 @@ async function main(): Promise<void> {
encrypt: true,
passphrase: e2eePassphrase,
usePathObfuscation: true,
encryptInternalMetadata: true,
E2EEAlgorithm: "v2",
}
),
@@ -1,10 +1,13 @@
import { mkdir, readFile, rm, writeFile } from "node:fs/promises";
import { dirname, join } from "node:path";
import { VERSIONING_DOCID } from "@vrtmrz/livesync-commonlib/compat/common/types";
import { ENCRYPTED_INTERNAL_METADATA_FEATURE, REMOTE_FEATURE_GENERATION } from "@vrtmrz/livesync-commonlib/replication";
import { evalObsidianJson } from "../runner/cli.ts";
import {
assertCouchDbReachable,
createCouchDbDatabase,
deleteCouchDbDatabase,
fetchCouchDbDocument,
loadCouchDbConfig,
makeUniqueDatabaseName,
waitForCouchDbDocs,
@@ -159,6 +162,10 @@ async function startConfiguredSession(
dbName: context.dbName,
};
const customisationSettings = {
encrypt: true,
passphrase: "internal-metadata-e2e-secret",
usePathObfuscation: true,
encryptInternalMetadata: true,
deviceAndVaultName: deviceName,
usePluginSync: true,
usePluginSyncV2: true,
@@ -486,6 +493,18 @@ async function main(): Promise<void> {
(target) => ids.has(target.id) && target.children.every((childId) => ids.has(childId))
);
});
for (const target of [entry, configEntry, ...pluginEntries]) {
const remoteEntry = await fetchCouchDbDocument(context.couchDb, context.dbName, target.id);
if (!remoteEntry.path?.startsWith("/\\:") || remoteEntry.children?.length !== 0 ||
remoteEntry.ctime !== 0 || remoteEntry.mtime !== 0 || remoteEntry.size !== 0) {
throw new Error(`Customisation Sync Metadata was not encrypted for ${target.id}.`);
}
}
const versionInfo = await fetchCouchDbDocument(context.couchDb, context.dbName, VERSIONING_DOCID);
if (versionInfo.version !== REMOTE_FEATURE_GENERATION ||
!(versionInfo.used_features as unknown[] | undefined)?.includes(ENCRYPTED_INTERNAL_METADATA_FEATURE)) {
throw new Error("The remote feature list does not declare encrypted internal Metadata.");
}
await session.app.stop();
session = await startConfiguredSession(context, vaultB, targetDeviceName);
@@ -1,5 +1,7 @@
import { mkdir, readFile, rm, writeFile } from "node:fs/promises";
import { dirname, join } from "node:path";
import { VERSIONING_DOCID } from "@vrtmrz/livesync-commonlib/compat/common/types";
import { ENCRYPTED_INTERNAL_METADATA_FEATURE, REMOTE_FEATURE_GENERATION } from "@vrtmrz/livesync-commonlib/replication";
import {
assertLocatorHasMinimumTouchTarget,
assertLocatorWithinSafeArea,
@@ -11,6 +13,7 @@ import {
assertCouchDbReachable,
createCouchDbDatabase,
deleteCouchDbDatabase,
fetchCouchDbDocument,
loadCouchDbConfig,
makeUniqueDatabaseName,
waitForCouchDbDocs,
@@ -324,6 +327,10 @@ async function startConfiguredSession(
dbName: context.dbName,
};
const hiddenFileSettings = {
encrypt: true,
passphrase: "internal-metadata-e2e-secret",
usePathObfuscation: true,
encryptInternalMetadata: true,
syncInternalFiles: true,
syncInternalFilesBeforeReplication: true,
watchInternalFileChanges: false,
@@ -360,6 +367,16 @@ async function uploadHiddenFile(
const ids = new Set(docs.map((doc) => doc._id));
return ids.has(entry.id) && entry.children.every((childId) => ids.has(childId));
});
const remoteEntry = await fetchCouchDbDocument(context.couchDb, context.dbName, entry.id);
if (!remoteEntry.path?.startsWith("/\\:") || remoteEntry.children?.length !== 0 ||
remoteEntry.ctime !== 0 || remoteEntry.mtime !== 0 || remoteEntry.size !== 0) {
throw new Error(`Hidden File Sync Metadata was not encrypted for ${entry.id}.`);
}
const versionInfo = await fetchCouchDbDocument(context.couchDb, context.dbName, VERSIONING_DOCID);
if (versionInfo.version !== REMOTE_FEATURE_GENERATION ||
!(versionInfo.used_features as unknown[] | undefined)?.includes(ENCRYPTED_INTERNAL_METADATA_FEATURE)) {
throw new Error("The remote feature list does not declare encrypted internal Metadata.");
}
return entry;
}
@@ -0,0 +1,178 @@
import { mkdir, readFile, writeFile } from "node:fs/promises";
import { dirname, join } from "node:path";
import { VERSIONING_DOCID } from "@vrtmrz/livesync-commonlib/compat/common/types";
import { evalObsidianJson } from "../runner/cli.ts";
import {
assertCouchDbReachable,
createCouchDbDatabase,
deleteCouchDbDatabase,
fetchCouchDbDocument,
loadCouchDbConfig,
makeUniqueDatabaseName,
putCouchDbDocument,
} from "../runner/couchdb.ts";
import { discoverObsidianCli, requireObsidianBinary } from "../runner/environment.ts";
import {
assertE2eCompatibilityMarker,
configureCouchDb,
createE2eCouchDbPluginData,
createE2eObsidianDeviceLocalState,
prepareRemote,
pushLocalChanges,
waitForLiveSyncCoreReady,
waitForLocalDatabaseEntry,
} from "../runner/liveSyncWorkflow.ts";
import { startObsidianLiveSyncSession, type ObsidianLiveSyncSession } from "../runner/session.ts";
import { createTemporaryVault } from "../runner/vault.ts";
const acceptedPath = "E2E/remote-feature/accepted.md";
const acceptedContent = "Accepted before the remote feature changed.\n";
const unknownFeature = "future-format-v7";
type FeatureState = {
version: number | null;
features: string[];
hasActiveReplicator: boolean;
};
async function readFeatureState(cliBinary: string, env: NodeJS.ProcessEnv): Promise<FeatureState> {
return await evalObsidianJson<FeatureState>(
cliBinary,
[
"(async()=>{",
"const core=app.plugins.plugins['obsidian-livesync'].core;",
`const id=${JSON.stringify(VERSIONING_DOCID)};`,
"const info=await core.localDatabase.getRaw(id).catch(()=>null);",
"return JSON.stringify({",
"version:typeof info?.version==='number'?info.version:null,",
"features:Array.isArray(info?.used_features)?info.used_features:[],",
"hasActiveReplicator:!!core.services.replicator.getActiveReplicator(),",
"});",
"})()",
].join(""),
env
);
}
async function waitForState(
cliBinary: string,
env: NodeJS.ProcessEnv,
predicate: (state: FeatureState) => boolean,
description: string
): Promise<FeatureState> {
const deadline = Date.now() + 20_000;
let state = await readFeatureState(cliBinary, env);
while (!predicate(state) && Date.now() < deadline) {
await new Promise((resolve) => setTimeout(resolve, 250));
state = await readFeatureState(cliBinary, env);
}
if (!predicate(state)) throw new Error(`Timed out waiting for ${description}: ${JSON.stringify(state)}`);
return state;
}
async function main(): Promise<void> {
const binary = requireObsidianBinary();
const cli = discoverObsidianCli();
if (!cli.binary) throw new Error(`Could not find obsidian-cli. Checked paths: ${cli.checked.join(", ")}`);
const couchDb = await loadCouchDbConfig();
const dbName = makeUniqueDatabaseName(couchDb.dbPrefix, "remote-feature-change");
const vault = await createTemporaryVault();
let session: ObsidianLiveSyncSession | undefined;
try {
await assertCouchDbReachable(couchDb);
await createCouchDbDatabase(couchDb, dbName);
const couchDbSettings = {
uri: couchDb.uri,
username: couchDb.username,
password: couchDb.password,
dbName,
};
const settings = {
encrypt: false,
usePathObfuscation: false,
encryptInternalMetadata: false,
liveSync: false,
};
session = await startObsidianLiveSyncSession({
binary,
cliBinary: cli.binary,
vault,
startupGraceMs: Number(process.env.E2E_OBSIDIAN_STARTUP_GRACE_MS ?? 1000),
pluginData: createE2eCouchDbPluginData(couchDbSettings, settings),
localStorageEntries: createE2eObsidianDeviceLocalState(vault.name),
});
await waitForLiveSyncCoreReady(cli.binary, session.cliEnv);
await assertE2eCompatibilityMarker(cli.binary, session.cliEnv);
await configureCouchDb(cli.binary, session.cliEnv, couchDbSettings, settings);
await prepareRemote(cli.binary, session.cliEnv);
const fullPath = join(vault.path, acceptedPath);
await mkdir(dirname(fullPath), { recursive: true });
await writeFile(fullPath, acceptedContent, "utf-8");
await waitForLocalDatabaseEntry(cli.binary, session.cliEnv, acceptedPath);
await pushLocalChanges(cli.binary, session.cliEnv);
const initialVersion = await fetchCouchDbDocument(couchDb, dbName, VERSIONING_DOCID);
if (initialVersion.version !== 12 || "used_features" in initialVersion) {
throw new Error(
`An inactive feature unexpectedly changed the remote contract: ${JSON.stringify(initialVersion)}`
);
}
const start = await evalObsidianJson<{ status: string }>(
cli.binary,
[
"(async()=>{",
"const core=app.plugins.plugins['obsidian-livesync'].core;",
"await core.services.setting.applyExternalSettings({liveSync:true},true);",
"await core.services.control.applySettings();",
"const result=await core.services.replication.startContinuous({trigger:'daemon',interaction:{kind:'forbidden'}});",
"return JSON.stringify(result);",
"})()",
].join(""),
session.cliEnv
);
if (start.status !== "completed")
throw new Error(`Continuous replication did not start: ${JSON.stringify(start)}`);
await waitForState(cli.binary, session.cliEnv, (state) => state.hasActiveReplicator, "an active Replicator");
await putCouchDbDocument(couchDb, dbName, {
...initialVersion,
version: 13,
used_features: [unknownFeature],
});
const observed = await waitForState(
cli.binary,
session.cliEnv,
(state) => state.version === 13 && state.features.includes(unknownFeature) && !state.hasActiveReplicator,
"the live feature change and Replicator retirement"
);
const replicated = await evalObsidianJson<boolean>(
cli.binary,
"(async()=>JSON.stringify(!!(await app.plugins.plugins['obsidian-livesync'].core.services.replication.replicate(true))))()",
session.cliEnv
);
if (replicated) throw new Error("An unknown remote feature was admitted for another replication.");
const acceptedAfterStop = await readFile(fullPath, "utf-8");
if (acceptedAfterStop !== acceptedContent)
throw new Error("Previously accepted Vault content changed on stop.");
console.log(
`Active feature change retired the Replicator and kept accepted content: ${JSON.stringify(observed)}`
);
} finally {
await session?.app.stop();
await vault.dispose();
if (process.env.E2E_OBSIDIAN_KEEP_COUCHDB !== "true") {
await deleteCouchDbDatabase(couchDb, dbName).catch((error: unknown) => {
console.warn(error instanceof Error ? error.message : error);
});
}
}
}
main().catch((error: unknown) => {
console.error(error instanceof Error ? error.stack : error);
process.exit(1);
});
+7
View File
@@ -12,6 +12,13 @@ Earlier releases remain available in the 1.0 release history, the 1.0 preview hi
## Unreleased
### Privacy and compatibility
#### New Feature
- Hidden File Sync and Customisation Sync can now encrypt their paths, times, sizes, and Chunk references in CouchDB Metadata when E2EE V2 and Property Encryption are enabled. The preference is enabled for new Vaults and remains off for existing configurations until selected. It protects future writes; protecting existing Metadata also requires a manual remote Rebuild after all devices have been updated.
- CouchDB records the features its data uses. Clients now stop synchronisation and pending file reflection when they encounter an unknown feature, and show its identifier so that the required update can be identified.
## 1.0.32
27th September, 2026