feat: restore opt-in review harness

This commit is contained in:
vorotamoroz
2026-07-20 10:50:08 +00:00
parent 71f1b63b9a
commit f70779c352
14 changed files with 907 additions and 5 deletions
+67
View File
@@ -0,0 +1,67 @@
# Review Harness
The Review Harness is an opt-in, real-device review tool for Self-hosted LiveSync maintainers. It replaces the disabled legacy Test Pane with fixed, auditable scenarios, a device-local one-shot continuation, and a Markdown report which can be pasted into a pull request.
It supplies supporting evidence rather than a release gate by itself. Unit, integration, Compose, CLI E2E, and real-Obsidian E2E remain authoritative for the boundaries they own.
## Enabling the Harness
1. Use a dedicated test Vault.
2. Enable **Power users → Enable Developers' Debug Tools**.
3. Restart Obsidian.
4. Run **Self-hosted LiveSync: Open review harness** from the command palette.
The command and view are registered only when developer tools are enabled. Disabling the setting and restarting removes the entry point from an ordinary user session.
## Scenarios and access
| Scenario | Mode | Access | Purpose |
| --- | --- | --- | --- |
| Settings lifecycle | Automatic | Read-only | Exposes the seven synchronisation choices as typed observations and, for a genuinely new Vault, compares the selected recommendations with Commonlib's new-Vault contract. Existing-Vault setting preservation remains an automated migration and `settings-ui` E2E responsibility. |
| Compatibility review boundary | Guided | Device-local state | Observes the current dedicated compatibility controller and opens its actual review action. The Harness does not restore Change Log acknowledgement, `lastReadUpdates`, or a separate manual Pass or Fail result. |
| P2P composition | Automatic | Read-only | Confirms that the live P2P result resolves one replicator bound to the active Obsidian services. Peer discovery, replacement, reconnection, and transfer remain unit, CLI, and Compose E2E responsibilities. |
| Vault fixture round trip | Automatic, explicit | Dedicated Vault fixtures | After explicit confirmation, creates, reads, modifies, renames, and removes one owned fixture tree. General Vault reflection remains covered by its separate real-Obsidian E2E workflow. |
**Automatic** runs only the two read-only local observations. **Full review** also starts the guided compatibility observation and asks before the Vault fixture scenario writes anything. A scenario can also be run individually.
These observations deliberately do not repeat stronger automated workflows. They exist to record which contracts the current real-device composition exposes while a maintainer reviews an immutable artefact.
### Vault fixture boundary
The Vault scenario owns only `__self-hosted-livesync-review-harness__`. It refuses to run if that path already exists, so it cannot assume ownership of a user's existing file or folder. Once it creates the root, it removes the complete owned tree from a `finally` block whether the round trip passes or fails.
The Harness accepts no path, command, code, remote configuration, or credential through plug-in data or its continuation state. New write scenarios must use a distinct fixed fixture root, describe their side effects in the interface, require confirmation, and clean up in `finally`.
## Restart continuation
The restart action writes a small device-local record under `review-harness-v1`, then asks Obsidian to reload. The record permits only:
- the fixed `compatibility-review` scenario;
- the fixed `awaiting-restart` stage;
- a canonical ISO request time; and
- a request identifier derived exactly as `compatibility-review-<request time>`.
On the next settings load, the Harness deletes the record before parsing and acting on it. A valid record reopens the Harness after layout is ready and leaves the compatibility observation waiting for the reviewer. Invalid state is removed and reported as a failed continuation. The record is not stored in `data.json`, copied through a Setup URI, or synchronised to another device.
The compatibility controller does not require an Obsidian restart to acknowledge a pause. This continuation belongs to the review tool and proves its one-shot reload boundary; it is not a second compatibility lifecycle.
## Reports and privacy
**Copy Markdown report** includes:
- the plug-in and Obsidian versions;
- the platform, user agent, and viewport;
- each scenario's status and bounded summary; and
- a bounded event transcript.
The formatter has no inputs for Vault identifiers, paths, file names, file contents, remote configuration, or secrets. The report is copied locally and is never transmitted by the plug-in. Review the environment information before posting it because a user agent and viewport may identify a device or operating system.
Unexpected runtime errors are written to the local LiveSync log, while copied reports retain only a generic failure summary. This prevents an adapter error from copying a local path or file name into a pull request by accident.
## Automated real-Obsidian coverage
External automation drives the stable `data-testid` attributes beginning with `review-harness-`. The dedicated workflow checks only Harness-owned behaviour: debug-only registration, consume-before-use continuation handling, fixed Vault fixture clean-up, report privacy, and mobile layout.
Compatibility dialogue behaviour and persistence belong to `test:e2e:obsidian:settings-ui`. Real P2P transport belongs to the Compose P2P suite. General Vault reflection belongs to `test:e2e:obsidian:vault-reflection`. Keeping those responsibilities separate prevents the review tool from becoming a second, weaker copy of the acceptance suite.
The mobile checks use `app.emulateMobile(true)`, a representative viewport, and safe-area and touch-target assertions. They do not claim to reproduce native operating-system overlays.
+1
View File
@@ -45,6 +45,7 @@
"test:e2e:obsidian:smoke": "tsx test/e2e-obsidian/scripts/smoke.ts",
"test:e2e:obsidian:dialog-mounts": "tsx test/e2e-obsidian/scripts/dialog-mounts.ts",
"test:e2e:obsidian:settings-ui": "tsx test/e2e-obsidian/scripts/settings-ui.ts",
"test:e2e:obsidian:review-harness": "tsx test/e2e-obsidian/scripts/review-harness.ts",
"test:e2e:obsidian:vault-reflection": "tsx test/e2e-obsidian/scripts/vault-reflection.ts",
"test:e2e:obsidian:couchdb-upload": "tsx test/e2e-obsidian/scripts/couchdb-upload.ts",
"test:e2e:obsidian:cli-to-obsidian-sync": "tsx test/e2e-obsidian/scripts/cli-to-obsidian-sync.ts",
@@ -0,0 +1,159 @@
import { ItemView, Notice, Setting, type WorkspaceLeaf } from "@/deps";
import {
REVIEW_HARNESS_SCENARIOS,
type ReviewHarnessScenarioId,
type ReviewHarnessScenarioStatus,
} from "./reviewHarnessContract";
import type { ReviewHarnessController } from "./reviewHarnessController";
export const VIEW_TYPE_REVIEW_HARNESS = "self-hosted-livesync-review-harness";
const STATUS_LABELS: Record<ReviewHarnessScenarioStatus, string> = {
idle: "Not run",
queued: "Queued",
running: "Running",
"waiting-for-user": "Waiting for review",
passed: "Passed",
failed: "Failed",
cancelled: "Cancelled",
};
export class ReviewHarnessView extends ItemView {
override icon = "test-tube-2";
override navigation = true;
private unsubscribe: (() => void) | undefined;
constructor(
leaf: WorkspaceLeaf,
private readonly controller: ReviewHarnessController
) {
super(leaf);
}
getViewType(): string {
return VIEW_TYPE_REVIEW_HARNESS;
}
getDisplayText(): string {
return "Self-hosted LiveSync review harness";
}
override async onOpen(): Promise<void> {
this.unsubscribe = this.controller.subscribe(() => this.render());
this.render();
await Promise.resolve();
}
override async onClose(): Promise<void> {
this.unsubscribe?.();
this.unsubscribe = undefined;
await Promise.resolve();
}
private addActionButton(
setting: Setting,
label: string,
testId: string,
action: () => void | Promise<void>,
cta = false
): void {
setting.addButton((button) => {
button.setButtonText(label);
if (cta) button.setCta();
button.buttonEl.dataset.testid = testId;
button.onClick(() => void action());
});
}
private renderScenario(id: ReviewHarnessScenarioId): void {
const scenario = REVIEW_HARNESS_SCENARIOS.find((candidate) => candidate.id === id)!;
const snapshot = this.controller.snapshot();
const result = snapshot.results[id];
const setting = new Setting(this.contentEl)
.setName(scenario.title)
.setDesc(`${scenario.description} Mode: ${scenario.mode}. Access: ${scenario.access}.`)
.setClass("sls-review-harness__scenario");
setting.settingEl.dataset.testid = `review-harness-scenario-${id}`;
this.addActionButton(
setting,
id === "compatibility-review" ? "Start review" : "Run",
`review-harness-run-${id}`,
() => this.controller.runScenario(id)
);
const resultEl = this.contentEl.createDiv({ cls: "sls-review-harness__result" });
resultEl.dataset.testid = `review-harness-result-${id}`;
resultEl.createEl("strong", { text: `${STATUS_LABELS[result.status]}: ` });
resultEl.appendText(result.detail);
if (result.observations.length > 0) {
const observations = resultEl.createEl("ul");
for (const observation of result.observations) observations.createEl("li", { text: observation });
}
if (id === "compatibility-review" && result.status === "waiting-for-user") {
const actions = new Setting(this.contentEl).setClass("sls-review-harness__actions");
this.addActionButton(
actions,
"Open compatibility review",
"review-harness-open-compatibility-review",
() => this.controller.openCompatibilityReview(),
true
);
this.addActionButton(actions, "Restart and return", "review-harness-restart", () =>
this.controller.prepareCompatibilityReviewRestart()
);
}
}
private render(): void {
this.contentEl.empty();
this.contentEl.addClass("sls-review-harness");
this.contentEl.dataset.testid = "review-harness";
this.contentEl.createEl("h2", { text: "Self-hosted LiveSync review harness" });
this.contentEl.createEl("p", {
text: "Use a dedicated test Vault. Read-only scenarios are labelled. The Vault round-trip scenario writes only after confirmation, owns one fixed fixture tree, and removes it in a finally block. The Harness never accepts arbitrary commands, paths, code, or remote credentials.",
cls: "sls-review-harness__warning",
});
this.contentEl.createEl("p", {
text: "Automatic scenarios inspect local contracts. The guided compatibility review uses the same device-local pause and explicit action as normal start-up. Real P2P transport remains covered by the Compose E2E suite.",
});
const snapshot = this.controller.snapshot();
if (snapshot.continuationError) {
this.contentEl.createEl("p", {
text: `Continuation error: ${snapshot.continuationError}`,
cls: "sls-review-harness__error",
});
} else if (snapshot.resumedRequestId) {
const resumed = this.contentEl.createEl("p", {
text: "The one-shot restart continuation was consumed. Complete the guided review below.",
cls: "sls-review-harness__resumed",
});
resumed.dataset.testid = "review-harness-resumed";
}
const suiteActions = new Setting(this.contentEl)
.setName("Review suite")
.setDesc(snapshot.running ? `Running ${snapshot.current ?? "scenario"}.` : "Choose the scope to run.")
.setClass("sls-review-harness__actions");
this.addActionButton(suiteActions, "Automatic", "review-harness-run-automatic", () =>
this.controller.runAutomaticScenarios()
);
this.addActionButton(suiteActions, "Full review", "review-harness-run-full", () =>
this.controller.runAllScenarios()
);
this.addActionButton(suiteActions, "Copy Markdown report", "review-harness-copy-report", async () => {
await this.controller.copyReport();
new Notice("Review Harness Markdown report copied.");
});
this.contentEl.createEl("h3", { text: "Scenarios" });
for (const { id } of REVIEW_HARNESS_SCENARIOS) this.renderScenario(id);
this.contentEl.createEl("p", {
text: "Reports are copied locally and are not transmitted. They omit Vault identifiers, paths, contents, remote configuration, and secrets.",
cls: "sls-review-harness__privacy",
});
}
}
@@ -12,7 +12,7 @@ export const REVIEW_HARNESS_SCENARIOS = [
id: "settings-lifecycle",
title: "Settings lifecycle",
description:
"Checks whether loaded settings retain the seven synchronisation choices and apply new-Vault recommendations only to a genuinely new Vault.",
"Checks whether loaded settings expose the seven synchronisation choices and apply new-Vault recommendations only to a genuinely new Vault.",
mode: "automatic",
access: "read-only",
},
@@ -78,7 +78,7 @@ describe("Review Harness contract", () => {
]);
});
it("checks that an existing Vault keeps the seven synchronisation choices typed and intact", () => {
it("checks that an existing Vault exposes the seven synchronisation choices as booleans", () => {
const result = inspectSettingsLifecycle({
migration: migration(),
settings: preservedSyncSettings,
@@ -227,13 +227,16 @@ export class ReviewHarnessController {
this.current = "compatibility-review";
this.notify();
try {
const pauseWasPending = this.runtime.getCompatibilityPause() !== undefined;
await this.runtime.openCompatibilityReview();
const inspection = this.inspectCompatibilityReview();
this.results["compatibility-review"] =
inspection.status === "passed" && !this.runtime.getCompatibilityPause()
? {
status: "passed",
detail: "The device-local compatibility pause was reviewed and cleared.",
detail: pauseWasPending
? "The device-local compatibility pause was reviewed and cleared."
: inspection.detail,
observations: inspection.observations,
}
: inspection.status === "failed"
@@ -253,6 +253,19 @@ describe("ReviewHarnessController", () => {
expect(controller.snapshot().results["compatibility-review"].status).toBe("waiting-for-user");
});
it("does not claim that a compatibility pause was cleared when none was pending", async () => {
const runtime = createRuntime();
runtime.compatibilityPause = undefined;
const controller = new ReviewHarnessController(runtime);
await controller.openCompatibilityReview();
expect(controller.snapshot().results["compatibility-review"]).toMatchObject({
status: "passed",
detail: "No compatibility review is pending on this device.",
});
});
it("copies a Markdown report after compatibility evidence is recorded", async () => {
const runtime = createRuntime();
const controller = new ReviewHarnessController(runtime);
+6 -1
View File
@@ -42,6 +42,7 @@ import { useSetupManagerHandlersFeature } from "./serviceFeatures/setupObsidian/
import { useP2PReplicatorFeature } from "@vrtmrz/livesync-commonlib/compat/replication/trystero/useP2PReplicatorFeature";
import { useP2PReplicatorCommands } from "@vrtmrz/livesync-commonlib/compat/replication/trystero/useP2PReplicatorCommands";
import { useP2PReplicatorUI } from "./serviceFeatures/useP2PReplicatorUI.ts";
import { useReviewHarness } from "./serviceFeatures/useReviewHarness.ts";
import { createOpenReplicationUI, createOpenRebuildUI } from "./features/P2PSync/P2PReplicator/P2PReplicationUI.ts";
import { useCompatibilityReview } from "./serviceFeatures/compatibilityReview.ts";
import { createObsidianCompatibilityReviewUi } from "./serviceFeatures/compatibilityReviewObsidian.ts";
@@ -194,7 +195,11 @@ export default class ObsidianLiveSyncPlugin extends Plugin {
useOfflineScanner(core);
useRedFlagFeatures(core);
useCheckRemoteSize(core);
useCompatibilityReview(core, createObsidianCompatibilityReviewUi(core.confirm));
const compatibilityReview = useCompatibilityReview(
core,
createObsidianCompatibilityReviewUi(core.confirm)
);
useReviewHarness(core, this, replicator, compatibilityReview);
// p2pReplicatorResult = useP2PReplicator(core, [
// VIEW_TYPE_P2P,
// (leaf: any) => new P2PReplicatorPaneView(leaf, core, p2pReplicatorResult!),
+125
View File
@@ -0,0 +1,125 @@
import { NEW_VAULT_SETTINGS } from "@vrtmrz/livesync-commonlib/settings";
import { LOG_LEVEL_NOTICE } from "octagonal-wheels/common/logger";
import type { UseP2PReplicatorResult } from "@vrtmrz/livesync-commonlib/compat/replication/trystero/UseP2PReplicatorResult";
import type ObsidianLiveSyncPlugin from "@/main";
import type { LiveSyncCore } from "@/main";
import type { WorkspaceLeaf } from "@/deps";
import {
ReviewHarnessController,
REVIEW_HARNESS_STATE_KEY,
type ReviewHarnessRuntime,
} from "@/features/ReviewHarness/reviewHarnessController";
import { ReviewHarnessView, VIEW_TYPE_REVIEW_HARNESS } from "@/features/ReviewHarness/ReviewHarnessView";
import type { ReviewHarnessScenarioResult } from "@/features/ReviewHarness/reviewHarnessContract";
import {
REVIEW_HARNESS_FIXTURE_ROOT,
runReviewHarnessVaultRoundTrip,
} from "@/features/ReviewHarness/reviewHarnessVaultFixture";
import type { CompatibilityReviewController } from "./compatibilityReview";
async function runVaultRoundTrip(plugin: ObsidianLiveSyncPlugin): Promise<ReviewHarnessScenarioResult> {
const vault = plugin.app.vault;
return runReviewHarnessVaultRoundTrip({
confirmFixtureAccess: async () =>
(await plugin.core.services.UI.confirm.askYesNoDialog(
"This scenario creates, reads, modifies, renames, and removes one owned fixture tree. Use a dedicated test Vault. Continue?",
{
title: "Review Harness: Vault fixture access",
defaultOption: "No",
}
)) === "yes",
fixtureRootExists: () => vault.getAbstractFileByPath(REVIEW_HARNESS_FIXTURE_ROOT) !== null,
createFixtureRoot: async () => {
await vault.createFolder(REVIEW_HARNESS_FIXTURE_ROOT);
},
createFile: (path, content) => vault.create(path, content),
readFile: (file) => vault.read(file),
modifyFile: (file, content) => vault.modify(file, content),
renameFile: (file, path) => vault.rename(file, path),
filePath: (file) => file.path,
removeFixtureRoot: async () => {
const fixtureRoot = vault.getAbstractFileByPath(REVIEW_HARNESS_FIXTURE_ROOT);
if (fixtureRoot) await plugin.app.fileManager.trashFile(fixtureRoot);
},
});
}
export function useReviewHarness(
core: LiveSyncCore,
plugin: ObsidianLiveSyncPlugin,
p2p: UseP2PReplicatorResult,
compatibilityReview: CompatibilityReviewController
): ReviewHarnessController {
const services = core.services;
const runtime: ReviewHarnessRuntime = {
now: () => new Date(),
getSettings: () => services.setting.currentSettings(),
getNewVaultSettings: () => NEW_VAULT_SETTINGS,
getSettingsMigrationState: () => services.setting.getSettingsMigrationState(),
isCompatibilityReviewInitialised: () => compatibilityReview.initialised,
getCompatibilityPause: () => compatibilityReview.pendingPause,
openCompatibilityReview: () => compatibilityReview.openReview(),
getP2PComposition: () => ({
first: p2p.replicator,
second: p2p.replicator,
expectedServices: services,
}),
runVaultRoundTrip: () => runVaultRoundTrip(plugin),
readContinuation: () => services.setting.getSmallConfig(REVIEW_HARNESS_STATE_KEY),
writeContinuation: (value) => services.setting.setSmallConfig(REVIEW_HARNESS_STATE_KEY, value),
deleteContinuation: () => services.setting.deleteSmallConfig(REVIEW_HARNESS_STATE_KEY),
restart: () => services.appLifecycle.performRestart(),
reportError: (error) => services.API.addLog(error, LOG_LEVEL_NOTICE),
copyText: async (value) => {
if (!activeWindow.navigator.clipboard) throw new Error("Clipboard access is unavailable on this device.");
await activeWindow.navigator.clipboard.writeText(value);
},
getEnvironment: () => ({
pluginVersion: services.API.getPluginVersion(),
obsidianVersion: services.API.getAppVersion(),
platform: services.API.getPlatform(),
userAgent: activeWindow.navigator.userAgent || "unavailable",
viewport:
typeof activeWindow.innerWidth === "number" && typeof activeWindow.innerHeight === "number"
? `${activeWindow.innerWidth}x${activeWindow.innerHeight}`
: "unavailable",
}),
};
const controller = new ReviewHarnessController(runtime);
let continuationConsumed = false;
let registered = false;
let openAfterLayout = false;
services.appLifecycle.onSettingLoaded.addHandler(() => {
if (!continuationConsumed) {
continuationConsumed = true;
controller.consumeContinuation();
}
const snapshot = controller.snapshot();
openAfterLayout = snapshot.resumedRequestId !== null || snapshot.continuationError !== null;
if (!services.setting.currentSettings().enableDebugTools || registered) return Promise.resolve(true);
registered = true;
services.API.registerWindow(
VIEW_TYPE_REVIEW_HARNESS,
(leaf: WorkspaceLeaf) => new ReviewHarnessView(leaf, controller)
);
services.API.addCommand({
id: "open-review-harness",
name: "Open review harness",
callback: () => {
void services.API.showWindow(VIEW_TYPE_REVIEW_HARNESS);
},
});
return Promise.resolve(true);
});
services.appLifecycle.onLayoutReady.addHandler(() => {
if (openAfterLayout && services.setting.currentSettings().enableDebugTools) {
void services.API.showWindow(VIEW_TYPE_REVIEW_HARNESS);
}
return Promise.resolve(true);
});
return controller;
}
@@ -0,0 +1,158 @@
import { describe, expect, it, vi } from "vitest";
import type { CompatibilityPause } from "@/common/databaseCompatibility.ts";
import { REVIEW_HARNESS_STATE_KEY } from "@/features/ReviewHarness/reviewHarnessController.ts";
import { useReviewHarness } from "./useReviewHarness.ts";
const VIEW_TYPE_REVIEW_HARNESS = "self-hosted-livesync-review-harness";
vi.mock("@/features/ReviewHarness/ReviewHarnessView", () => ({
VIEW_TYPE_REVIEW_HARNESS: "self-hosted-livesync-review-harness",
ReviewHarnessView: class {},
}));
function compatibilityPause(): CompatibilityPause {
return {
resumable: true,
reasons: [
{
source: "settings-schema",
sourceVersion: 9,
currentVersion: 10,
isFromFutureSchema: false,
resumable: true,
},
],
};
}
function createFixture(options: { enableDebugTools?: boolean; continuation?: string } = {}) {
const settingLoadedHandlers: Array<() => Promise<boolean>> = [];
const layoutReadyHandlers: Array<() => Promise<boolean>> = [];
const local = new Map<string, string>();
if (options.continuation) local.set(REVIEW_HARNESS_STATE_KEY, options.continuation);
const settings = {
enableDebugTools: options.enableDebugTools ?? true,
liveSync: false,
syncOnSave: false,
syncOnEditorSave: true,
syncOnStart: false,
syncOnFileOpen: true,
syncAfterMerge: false,
periodicReplication: true,
};
const services = {} as Record<string, unknown>;
const replicator = { env: { services } };
const api = {
registerWindow: vi.fn(),
addCommand: vi.fn(),
showWindow: vi.fn().mockResolvedValue(undefined),
addLog: vi.fn(),
getPluginVersion: () => "1.0.0-rc.0",
getAppVersion: () => "1.12.7",
getPlatform: () => "desktop",
};
Object.assign(services, {
setting: {
currentSettings: () => settings,
getSettingsMigrationState: () => ({
sourceVersion: 9,
targetVersion: 10,
isNewVault: false,
isFromFutureSchema: false,
changed: true,
requiresSyncReview: true,
reviewReasons: [],
}),
getSmallConfig: (key: string) => local.get(key) ?? "",
setSmallConfig: (key: string, value: string) => local.set(key, value),
deleteSmallConfig: (key: string) => local.delete(key),
},
appLifecycle: {
onSettingLoaded: {
addHandler: vi.fn((handler: () => Promise<boolean>) => settingLoadedHandlers.push(handler)),
},
onLayoutReady: {
addHandler: vi.fn((handler: () => Promise<boolean>) => layoutReadyHandlers.push(handler)),
},
performRestart: vi.fn(),
},
API: api,
UI: {
confirm: { askYesNoDialog: vi.fn().mockResolvedValue("no") },
},
});
let pause: CompatibilityPause | undefined = compatibilityPause();
const compatibilityReview = {
initialised: true,
get pendingPause() {
return pause;
},
openReview: vi.fn(async () => {
pause = undefined;
}),
};
const core = { services };
const plugin = {
core,
app: {
vault: {},
fileManager: {},
},
};
const controller = useReviewHarness(core as never, plugin as never, { replicator } as never, compatibilityReview as never);
return {
controller,
api,
settings,
local,
settingLoadedHandlers,
layoutReadyHandlers,
compatibilityReview,
};
}
describe("Review Harness composition", () => {
it("does not register the command or view when developer debug tools are disabled", async () => {
const fixture = createFixture({ enableDebugTools: false });
await fixture.settingLoadedHandlers[0]();
expect(fixture.api.registerWindow).not.toHaveBeenCalled();
expect(fixture.api.addCommand).not.toHaveBeenCalled();
});
it("removes a one-shot continuation before reopening the Harness after layout", async () => {
const requestedAt = "2026-07-18T11:59:00.000Z";
const continuation = JSON.stringify({
formatVersion: 1,
requestId: `compatibility-review-${requestedAt}`,
scenarioId: "compatibility-review",
stage: "awaiting-restart",
requestedAt,
});
const fixture = createFixture({ continuation });
await fixture.settingLoadedHandlers[0]();
expect(fixture.local.has(REVIEW_HARNESS_STATE_KEY)).toBe(false);
expect(fixture.controller.snapshot().resumedRequestId).toBe(`compatibility-review-${requestedAt}`);
expect(fixture.api.registerWindow).toHaveBeenCalledWith(VIEW_TYPE_REVIEW_HARNESS, expect.any(Function));
await fixture.layoutReadyHandlers[0]();
expect(fixture.api.showWindow).toHaveBeenCalledWith(VIEW_TYPE_REVIEW_HARNESS);
});
it("uses the actual compatibility controller as the guided review boundary", async () => {
const fixture = createFixture();
await fixture.controller.runScenario("compatibility-review");
expect(fixture.controller.snapshot().results["compatibility-review"].status).toBe("waiting-for-user");
await fixture.controller.openCompatibilityReview();
expect(fixture.compatibilityReview.openReview).toHaveBeenCalledOnce();
expect(fixture.controller.snapshot().results["compatibility-review"].status).toBe("passed");
});
});
+49
View File
@@ -339,6 +339,55 @@ body {
-webkit-text-security: disc;
}
.sls-review-harness {
box-sizing: border-box;
max-width: 100%;
overflow-x: clip;
padding-bottom: max(1rem, env(safe-area-inset-bottom));
}
.sls-review-harness .setting-item,
.sls-review-harness .setting-item-control {
flex-wrap: wrap;
max-width: 100%;
}
.sls-review-harness .setting-item-info {
min-width: min(16rem, 100%);
}
.sls-review-harness .setting-item-control {
gap: 0.5rem;
}
.sls-review-harness button {
min-height: 44px;
max-width: 100%;
white-space: normal;
}
.sls-review-harness__result,
.sls-review-harness__warning,
.sls-review-harness__privacy,
.sls-review-harness__error,
.sls-review-harness__resumed {
max-width: 100%;
overflow-wrap: anywhere;
}
.sls-review-harness__result {
margin: 0 0 1rem;
}
.sls-review-harness__warning,
.sls-review-harness__error {
color: var(--text-error);
}
.sls-review-harness__resumed {
color: var(--text-success);
}
span.ls-mark-cr::after {
user-select: none;
content: "↲";
+5 -1
View File
@@ -62,6 +62,7 @@ npm run test:e2e:obsidian:cli-help -- vaults verbose
npm run test:e2e:obsidian:smoke
npm run test:e2e:obsidian:dialog-mounts
npm run test:e2e:obsidian:settings-ui
npm run test:e2e:obsidian:review-harness
npm run test:e2e:obsidian:vault-reflection
npm run test:e2e:obsidian:couchdb-upload
npm run test:e2e:obsidian:cli-to-obsidian-sync
@@ -85,7 +86,9 @@ npm run test:e2e:obsidian:local-suite:services
The mobile pass uses Obsidian's `app.emulateMobile(true)`, a 390 by 844 CSS-pixel viewport, and explicit iPhone-style safe-area insets of 47 pixels at the top and 34 pixels at the bottom. The public `@vrtmrz/obsidian-test-session` layout assertions require each modal to remain within the viewport and safe area without horizontal overflow. They also require the Obsidian Close control to remain within the safe area and provide at least a 44 by 44 CSS-pixel touch target. The runner clicks that control to verify actionability, then completes the explicit cancellation path. These simulated checks cover deterministic layout and interaction boundaries; they do not claim to reproduce a native operating-system overlay.
`test:e2e:obsidian:local-suite` builds the plug-in and, unless `LIVESYNC_CLI_COMMAND` selects an external CLI, the local LiveSync CLI. It then runs discovery, smoke, Svelte dialogue mounting, settings UI, vault reflection, CouchDB upload, CLI-to-Obsidian synchronisation, Object Storage upload, startup scan, two-vault synchronisation, Hidden File Sync, Customisation Sync, and setting Markdown export in sequence. Start the local CouchDB and MinIO fixtures before running it, or use `test:e2e:obsidian:local-suite:services` to let the wrapper stop leftover fixtures, start fresh fixtures, and stop them again after the run.
`test:e2e:obsidian:review-harness` exercises only the boundaries owned by the opt-in maintainer Harness. It retains a real compatibility pause, uses the fixed Harness restart action to persist a device-local continuation and reload Obsidian, and requires the Harness to delete that state before reopening. It also runs the bounded local observations, confirms the dedicated Vault fixture root is removed, captures the copied privacy-bounded Markdown report, and checks the Harness layout and touch targets in mobile test mode. Compatibility explanation and persistence details remain owned by `settings-ui`, real P2P transfer remains owned by the Compose suite, and general Vault reflection remains owned by `vault-reflection`; the Harness test does not duplicate those workflows.
`test:e2e:obsidian:local-suite` builds the plug-in and, unless `LIVESYNC_CLI_COMMAND` selects an external CLI, the local LiveSync CLI. It then runs discovery, smoke, Svelte dialogue mounting, settings UI, the Review Harness, Vault reflection, CouchDB upload, CLI-to-Obsidian synchronisation, Object Storage upload, startup scan, two-vault synchronisation, Hidden File Sync, Customisation Sync, and setting Markdown export in sequence. Start the local CouchDB and MinIO fixtures before running it, or use `test:e2e:obsidian:local-suite:services` to let the wrapper stop leftover fixtures, start fresh fixtures, and stop them again after the run.
`test:e2e:obsidian:couchdb-upload` reuses the CouchDB variables from `.test.env` or the process environment. It expects a reachable CouchDB service, creates a unique database, starts from configured plug-in data without the device-local compatibility marker, and verifies the copied-or-restored Vault explanation in the actual compatibility dialogue. It captures the summary and details, resumes explicitly, confirms that the marker was recorded, creates a note in real Obsidian, commits the note into the local database, runs one-shot synchronisation, and verifies that the remote database contains both the metadata document and its chunk documents.
@@ -150,6 +153,7 @@ Useful environment variables:
- `E2E_OBSIDIAN_SMOKE_TIMEOUT_MS`: smoke timeout in milliseconds.
- `E2E_OBSIDIAN_DIALOG_TIMEOUT_MS`: timeout for a representative Svelte dialogue to mount, expose its principal controls, and close; default is 10 seconds.
- `E2E_OBSIDIAN_SETTINGS_TIMEOUT_MS`: timeout for the settings pane and its deletion controls to become visible; default is 10 seconds.
- `E2E_OBSIDIAN_REVIEW_HARNESS_TIMEOUT_MS`: timeout for Review Harness view and action boundaries; default is 15 seconds.
- `E2E_OBSIDIAN_READY_TIMEOUT_MS`: plug-in readiness timeout in milliseconds.
- `E2E_OBSIDIAN_CLI_READY_TIMEOUT_MS`: timeout for waiting until the vault-side Obsidian CLI exposes the plug-in catalogue.
- `E2E_OBSIDIAN_CLI_TIMEOUT_MS`: timeout for each `obsidian-cli` invocation.
+1
View File
@@ -15,6 +15,7 @@ const testSteps: Step[] = [
{ name: "smoke", args: ["run", "test:e2e:obsidian:smoke"] },
{ name: "Svelte dialogue mounts", args: ["run", "test:e2e:obsidian:dialog-mounts"] },
{ name: "settings UI", args: ["run", "test:e2e:obsidian:settings-ui"] },
{ name: "Review Harness", args: ["run", "test:e2e:obsidian:review-harness"] },
{ name: "vault reflection", args: ["run", "test:e2e:obsidian:vault-reflection"] },
{ name: "CouchDB upload", args: ["run", "test:e2e:obsidian:couchdb-upload"] },
{
+317
View File
@@ -0,0 +1,317 @@
import {
assertLocatorHasMinimumTouchTarget,
assertLocatorWithinSafeArea,
assertNoHorizontalOverflow,
} from "@vrtmrz/obsidian-test-session";
import { CURRENT_SETTING_VERSION } from "@vrtmrz/livesync-commonlib/compat/common/models/setting.const";
import { REVIEW_HARNESS_STATE_KEY } from "../../../src/features/ReviewHarness/reviewHarnessController.ts";
import { REVIEW_HARNESS_FIXTURE_ROOT } from "../../../src/features/ReviewHarness/reviewHarnessVaultFixture.ts";
import { discoverObsidianCli, requireObsidianBinary } from "../runner/environment.ts";
import { waitForLiveSyncCoreReady } from "../runner/liveSyncWorkflow.ts";
import { iPhoneSafeArea, setObsidianMobileTestMode } from "../runner/mobileUi.ts";
import { startObsidianLiveSyncSession, type ObsidianLiveSyncSession } from "../runner/session.ts";
import { captureObsidianDialogue, obsidianRemoteDebuggingPort, withObsidianPage } from "../runner/ui.ts";
import { createTemporaryVault } from "../runner/vault.ts";
const uiTimeoutMs = Number(process.env.E2E_OBSIDIAN_REVIEW_HARNESS_TIMEOUT_MS ?? 15000);
type ObsidianTestApp = {
commands?: { executeCommandById(commandId: string): boolean };
plugins?: { plugins: Record<string, unknown> };
vault?: { getAbstractFileByPath(path: string): unknown | null };
};
type ReviewHarnessTestGlobal = typeof globalThis & {
app?: ObsidianTestApp;
reviewHarnessCopiedReport?: string;
};
async function openHarness(): Promise<void> {
const opened = await withObsidianPage(obsidianRemoteDebuggingPort(), async (page) => {
return await page.evaluate(
(commandId) => (globalThis as ReviewHarnessTestGlobal).app?.commands?.executeCommandById(commandId) === true,
"obsidian-livesync:open-review-harness"
);
});
if (!opened) throw new Error("The Review Harness command was not registered.");
}
async function waitForHarness(): Promise<void> {
await withObsidianPage(obsidianRemoteDebuggingPort(), async (page) => {
await page.locator('[data-testid="review-harness"]').waitFor({ state: "visible", timeout: uiTimeoutMs });
});
}
async function keepCompatibilityPaused(): Promise<void> {
await withObsidianPage(obsidianRemoteDebuggingPort(), async (page) => {
const summary = page.locator(".modal-container").filter({
has: page.locator(".modal-title").filter({
hasText: "Synchronisation paused for compatibility review",
}),
});
await summary.waitFor({ state: "visible", timeout: uiTimeoutMs });
await summary.getByRole("button", { name: "Keep synchronisation paused" }).click({ timeout: uiTimeoutMs });
await summary.waitFor({ state: "hidden", timeout: uiTimeoutMs });
});
}
async function runAutomaticScenarios(): Promise<void> {
await withObsidianPage(obsidianRemoteDebuggingPort(), async (page) => {
const harness = page.locator('[data-testid="review-harness"]');
await harness.locator('[data-testid="review-harness-run-automatic"]').click({ timeout: uiTimeoutMs });
for (const id of ["settings-lifecycle", "p2p-composition"]) {
await harness
.locator(`[data-testid="review-harness-result-${id}"]`)
.getByText("Passed:", { exact: false })
.waitFor({ state: "visible", timeout: uiTimeoutMs });
}
});
}
async function runVaultFixture(): Promise<string> {
await withObsidianPage(obsidianRemoteDebuggingPort(), async (page) => {
await page
.locator('[data-testid="review-harness-run-vault-round-trip"]')
.click({ timeout: uiTimeoutMs });
const confirmation = page.locator(".modal-container").filter({
has: page.getByText("Review Harness: Vault fixture access", { exact: true }),
});
await confirmation.waitFor({ state: "visible", timeout: uiTimeoutMs });
});
const screenshot = await captureObsidianDialogue(
obsidianRemoteDebuggingPort(),
"review-harness-vault-confirmation.png",
async (page) => {
const confirmation = page.locator(".modal-container").filter({
has: page.getByText("Review Harness: Vault fixture access", { exact: true }),
});
await confirmation.waitFor({ state: "visible", timeout: uiTimeoutMs });
await assertNoHorizontalOverflow(page, confirmation, { label: "Vault fixture confirmation" });
}
);
await withObsidianPage(obsidianRemoteDebuggingPort(), async (page) => {
const harness = page.locator('[data-testid="review-harness"]');
const confirmation = page.locator(".modal-container").filter({
has: page.getByText("Review Harness: Vault fixture access", { exact: true }),
});
await confirmation.getByRole("button", { name: "Yes" }).click({ timeout: uiTimeoutMs });
await harness
.locator('[data-testid="review-harness-result-vault-round-trip"]')
.getByText("Passed:", { exact: false })
.waitFor({ state: "visible", timeout: uiTimeoutMs });
const fixtureRemoved = await page.evaluate(
(root) => (globalThis as ReviewHarnessTestGlobal).app?.vault?.getAbstractFileByPath(root) === null,
REVIEW_HARNESS_FIXTURE_ROOT
);
if (!fixtureRemoved) throw new Error("The Review Harness fixture root remained after the scenario.");
});
return screenshot;
}
async function restartAndResumeHarness(): Promise<string> {
await withObsidianPage(obsidianRemoteDebuggingPort(), async (page) => {
const harness = page.locator('[data-testid="review-harness"]');
await harness
.locator('[data-testid="review-harness-run-compatibility-review"]')
.click({ timeout: uiTimeoutMs });
await harness
.locator('[data-testid="review-harness-result-compatibility-review"]')
.getByText("Waiting for review:", { exact: false })
.waitFor({ state: "visible", timeout: uiTimeoutMs });
await harness.locator('[data-testid="review-harness-restart"]').click({ timeout: uiTimeoutMs });
});
await withObsidianPage(obsidianRemoteDebuggingPort(), async (page) => {
await page.waitForFunction(
() => {
const plugin = (globalThis as ReviewHarnessTestGlobal).app?.plugins?.plugins["obsidian-livesync"];
if (typeof plugin !== "object" || plugin === null || !("core" in plugin)) return false;
const core = (plugin as { core: { services: { appLifecycle: { isReady(): boolean } } } }).core;
return core.services.appLifecycle.isReady();
},
undefined,
{ timeout: uiTimeoutMs * 2 }
);
});
await keepCompatibilityPaused();
await waitForHarness();
return await captureObsidianDialogue(
obsidianRemoteDebuggingPort(),
"review-harness-resumed.png",
async (page) => {
const harness = page.locator('[data-testid="review-harness"]');
await harness
.locator('[data-testid="review-harness-resumed"]')
.waitFor({ state: "visible", timeout: uiTimeoutMs });
const continuationRemoved = await page.evaluate((stateKey) => {
const plugin = (globalThis as ReviewHarnessTestGlobal).app?.plugins?.plugins["obsidian-livesync"];
if (typeof plugin !== "object" || plugin === null || !("core" in plugin)) {
throw new Error("Self-hosted LiveSync is unavailable after restart.");
}
const core = (plugin as { core: { services: { setting: { getSmallConfig(key: string): string } } } })
.core;
return core.services.setting.getSmallConfig(stateKey) === "";
}, REVIEW_HARNESS_STATE_KEY);
if (!continuationRemoved) throw new Error("The one-shot continuation was not removed before use.");
await assertNoHorizontalOverflow(page, harness, { label: "resumed Review Harness" });
}
);
}
async function completeResumedCompatibilityStep(): Promise<void> {
await withObsidianPage(obsidianRemoteDebuggingPort(), async (page) => {
const harness = page.locator('[data-testid="review-harness"]');
await harness
.locator('[data-testid="review-harness-open-compatibility-review"]')
.click({ timeout: uiTimeoutMs });
const summary = page.locator(".modal-container").filter({
has: page.locator(".modal-title").filter({
hasText: "Synchronisation paused for compatibility review",
}),
});
await summary.waitFor({ state: "visible", timeout: uiTimeoutMs });
await summary.getByRole("button", { name: "Resume synchronisation" }).click({ timeout: uiTimeoutMs });
await summary.waitFor({ state: "hidden", timeout: uiTimeoutMs });
await harness
.locator('[data-testid="review-harness-result-compatibility-review"]')
.getByText("The device-local compatibility pause was reviewed and cleared.", { exact: false })
.waitFor({ state: "visible", timeout: uiTimeoutMs });
});
}
async function copyAndReadReport(): Promise<string> {
return await withObsidianPage(obsidianRemoteDebuggingPort(), async (page) => {
await page.evaluate(`
globalThis.reviewHarnessCopiedReport = undefined;
navigator.clipboard.writeText = function (value) {
globalThis.reviewHarnessCopiedReport = value;
return Promise.resolve();
};
`);
await page.locator('[data-testid="review-harness-copy-report"]').click({ timeout: uiTimeoutMs });
await page.waitForFunction(
() => typeof (globalThis as ReviewHarnessTestGlobal).reviewHarnessCopiedReport === "string",
undefined,
{ timeout: uiTimeoutMs }
);
return await page.evaluate(
() => (globalThis as ReviewHarnessTestGlobal).reviewHarnessCopiedReport ?? ""
);
});
}
async function verifyMobileHarness(): Promise<string> {
await setObsidianMobileTestMode(obsidianRemoteDebuggingPort(), true, uiTimeoutMs);
await withObsidianPage(obsidianRemoteDebuggingPort(), async (page) => {
await page.evaluate(async (viewType) => {
const plugin = (globalThis as ReviewHarnessTestGlobal).app?.plugins?.plugins["obsidian-livesync"];
if (typeof plugin !== "object" || plugin === null || !("core" in plugin)) {
throw new Error("Self-hosted LiveSync is unavailable in mobile test mode.");
}
const core = (plugin as {
core: { services: { API: { showWindow(type: string): Promise<void> } } };
}).core;
await core.services.API.showWindow(viewType);
}, "self-hosted-livesync-review-harness");
});
return await captureObsidianDialogue(
obsidianRemoteDebuggingPort(),
"review-harness-mobile.png",
async (page) => {
const harness = page.locator('[data-testid="review-harness"]');
await harness.waitFor({ state: "visible", timeout: uiTimeoutMs });
await assertNoHorizontalOverflow(page, harness, { label: "mobile Review Harness" });
const heading = harness.getByRole("heading", { name: "Self-hosted LiveSync review harness" });
await assertLocatorWithinSafeArea(page, heading, {
label: "mobile Review Harness heading",
safeAreaInsets: iPhoneSafeArea,
});
for (const testId of [
"review-harness-run-automatic",
"review-harness-run-full",
"review-harness-copy-report",
]) {
await assertLocatorHasMinimumTouchTarget(page, harness.locator(`[data-testid="${testId}"]`), {
label: testId,
});
}
}
);
}
async function main(): Promise<void> {
const binary = requireObsidianBinary();
const cli = discoverObsidianCli();
if (!cli.binary) throw new Error(`Could not find obsidian-cli. Checked paths: ${cli.checked.join(", ")}`);
const vault = await createTemporaryVault();
let session: ObsidianLiveSyncSession | undefined;
try {
session = await startObsidianLiveSyncSession({
binary,
cliBinary: cli.binary,
vault,
startupGraceMs: Number(process.env.E2E_OBSIDIAN_STARTUP_GRACE_MS ?? 1000),
pluginData: {
doctorProcessedVersion: "0.25.27",
settingVersion: CURRENT_SETTING_VERSION,
isConfigured: true,
additionalSuffixOfDatabaseName: "",
enableDebugTools: true,
notifyThresholdOfRemoteStorageSize: 0,
P2P_Enabled: false,
P2P_AutoStart: false,
liveSync: false,
syncOnSave: false,
syncOnEditorSave: true,
syncOnStart: false,
syncOnFileOpen: true,
syncAfterMerge: false,
periodicReplication: true,
},
});
await waitForLiveSyncCoreReady(cli.binary, session.cliEnv);
await keepCompatibilityPaused();
await openHarness();
await waitForHarness();
const initialScreenshot = await captureObsidianDialogue(
obsidianRemoteDebuggingPort(),
"review-harness-initial.png",
async (page) => {
const harness = page.locator('[data-testid="review-harness"]');
await harness.waitFor({ state: "visible", timeout: uiTimeoutMs });
await assertNoHorizontalOverflow(page, harness, { label: "Review Harness" });
}
);
await runAutomaticScenarios();
const vaultConfirmationScreenshot = await runVaultFixture();
const resumedScreenshot = await restartAndResumeHarness();
await completeResumedCompatibilityStep();
const report = await copyAndReadReport();
if (!report.includes("## Self-hosted LiveSync Review Harness report")) {
throw new Error("The copied Review Harness report was not Markdown evidence.");
}
for (const forbidden of [vault.name, REVIEW_HARNESS_FIXTURE_ROOT]) {
if (report.includes(forbidden)) throw new Error(`The Review Harness report exposed local state: ${forbidden}`);
}
const mobileScreenshot = await verifyMobileHarness();
console.log(
`Review Harness passed one-shot, fixture, report, and mobile checks. Screenshots: ${[
initialScreenshot,
vaultConfirmationScreenshot,
resumedScreenshot,
mobileScreenshot,
].join(", ")}`
);
} finally {
if (session) await session.app.stop();
await vault.dispose();
}
}
main().catch((error: unknown) => {
console.error(error instanceof Error ? error.stack : error);
process.exit(1);
});