Compare commits

..

23 Commits

Author SHA1 Message Date
vorotamoroz 236f9c434e Add P2P signalling-only netem benchmark 2026-07-08 10:04:03 +00:00
vorotamoroz 708bd187d8 Add P2P split-container netem stress fixture 2026-07-08 10:03:35 +00:00
vorotamoroz 46081a5f47 Record P2P selected candidate paths 2026-07-08 10:03:11 +00:00
vorotamoroz e5aa8d5c88 Add CouchDB netem shim benchmark 2026-07-08 10:02:41 +00:00
vorotamoroz abae4bda28 Add netem smoke fixture for benchmark simulation 2026-07-08 07:38:54 +00:00
vorotamoroz dc9b4fd37e Document CLI benchmark comparison model 2026-07-08 07:03:23 +00:00
vorotamoroz cb1fb0f874 Lower benchmark relay file descriptor limit 2026-07-08 06:54:42 +00:00
vorotamoroz 2cca2355fd Stabilise CLI P2P smoke readiness 2026-07-08 06:48:17 +00:00
vorotamoroz 5e090a3971 Run CLI network smoke on pull requests 2026-07-08 06:35:26 +00:00
vorotamoroz fd251346e3 Add Compose-based CLI network benchmarks 2026-07-08 06:30:12 +00:00
vorotamoroz 6a9918677f Merge pull request #996 from vrtmrz/0_25_80
releasing 0.25.80
2026-07-07 21:21:34 +09:00
vorotamoroz 2d42b92a89 bump 2026-07-07 12:15:58 +00:00
vorotamoroz a2b794c520 Merge pull request #995 from vrtmrz/fix_conflict_issues
Fix conflict issues
2026-07-07 21:13:00 +09:00
vorotamoroz 6f9446f447 Merge remote-tracking branch 'origin/main' into fix_conflict_issues 2026-07-07 12:11:38 +00:00
vorotamoroz 3f9cd67b1c Merge pull request #992 from vrtmrz/fix_989
fix: enable hidden file sync before overwrite setup
2026-07-07 21:10:09 +09:00
vorotamoroz dbcbf2c5ca fix: document safer conflict preservation 2026-07-07 12:05:40 +00:00
vorotamoroz eed0fca8d3 fix: preserve ambiguous conflict candidates 2026-07-07 12:00:24 +00:00
vorotamoroz 05e031b90b update submodule pointer 2026-07-07 11:31:29 +00:00
vorotamoroz bec767c13f add notes 2026-07-07 11:30:36 +00:00
vorotamoroz 8046a777af fix: refine conflict merge policy 2026-07-07 11:29:51 +00:00
vorotamoroz 0c58b0c513 test: reproduce conflict issue reports 2026-07-07 11:07:09 +00:00
vorotamoroz b66e227a02 update doc
update submodule pointer
2026-07-07 10:54:49 +00:00
vorotamoroz af6df84b5d fix: enable hidden file sync before overwrite setup 2026-07-03 05:09:55 +00:00
53 changed files with 2357 additions and 652 deletions
+3 -1
View File
@@ -16,7 +16,9 @@ pouchdb-browser.js
production/
# Test coverage and reports
coverage/
coverage/
test/bench-network/bench-results/
src/apps/cli/testdeno/bench-results/
# Local environment / secrets
.env
@@ -0,0 +1,84 @@
# Run the Compose-packaged CLI P2P smoke benchmark.
#
# This workflow is intentionally non-required at first. It exercises the local
# Compose package for CouchDB + Nostr relay + CLI runner, and uploads the
# benchmark JSON results for inspection.
name: cli-p2p-compose-smoke
on:
pull_request:
paths:
- '.github/workflows/cli-p2p-compose-smoke.yml'
- 'package.json'
- 'package-lock.json'
- 'src/apps/cli/**'
- 'test/bench-network/**'
workflow_dispatch:
inputs:
cases:
description: 'Comma-separated benchmark cases'
required: false
default: 'couchdb-baseline,p2p-direct-local'
md_files:
description: 'Markdown file count'
required: false
default: '2'
bin_files:
description: 'Binary file count'
required: false
default: '1'
couchdb_rtt_ms:
description: 'Requested CouchDB RTT in milliseconds'
required: false
default: '20'
permissions:
contents: read
jobs:
smoke:
runs-on: ubuntu-latest
timeout-minutes: 45
steps:
- name: Checkout
uses: actions/checkout@v4
with:
submodules: recursive
- name: Show Docker versions
run: |
docker --version
docker compose version
- name: Run Compose P2P smoke benchmark
env:
BENCH_CASES: ${{ inputs.cases || 'couchdb-baseline,p2p-direct-local' }}
BENCH_MD_FILE_COUNT: ${{ inputs.md_files || '2' }}
BENCH_MD_MIN_SIZE_BYTES: '128'
BENCH_MD_MAX_SIZE_BYTES: '256'
BENCH_BIN_FILE_COUNT: ${{ inputs.bin_files || '1' }}
BENCH_BIN_SIZE_BYTES: '512'
BENCH_COUCHDB_RTT_MS: ${{ inputs.couchdb_rtt_ms || '20' }}
BENCH_SYNC_TIMEOUT: '180'
BENCH_PEERS_TIMEOUT: '20'
LIVESYNC_P2P_RELAY_READY_TIMEOUT_MS: '60000'
BENCH_LIVESYNC_TEST_TEE: '0'
run: docker compose -f test/bench-network/compose.yml run --rm bench-runner
- name: Show Compose diagnostics
if: failure()
run: |
docker compose -f test/bench-network/compose.yml ps
docker compose -f test/bench-network/compose.yml logs --no-color couchdb nostr-relay
- name: Upload benchmark results
if: always()
uses: actions/upload-artifact@v4
with:
name: cli-p2p-compose-smoke-results
path: test/bench-network/bench-results/**
if-no-files-found: warn
- name: Stop Compose services
if: always()
run: docker compose -f test/bench-network/compose.yml down -v --remove-orphans
+14
View File
@@ -148,6 +148,20 @@ Hence, the new feature should be implemented as follows:
- **Service Hub** (`src/modules/services/`): Central service registry using dependency injection
- **Common Library** (`src/lib/`): Platform-independent sync logic, shared with other tools
### Conflict Merge Policy
Markdown conflict auto-merge should behave like a conservative three-way merge. The guiding rule is to merge changes when they touch non-overlapping regions, and to keep a manual conflict when the edits overlap semantically.
When in doubt, prefer the safer outcome: preserve data, keep the conflict visible, and ask the user rather than silently discarding content or choosing one side.
- If one side deletes a line and the other side leaves that same line unchanged, treat it as a safe deletion. The deleted line must not be reintroduced into the merged result.
- If one side inserts new content in a different region while the other side deletes an unchanged old region, preserve the insertion and the deletion.
- If one side deletes a line and the other side modifies that same line, keep the conflict for user resolution.
- If both sides insert different content at the same position, keep both insertions in a deterministic order unless the surrounding deletion context indicates that they are competing replacements.
- Avoid resolving conflicts by simply choosing the newest revision unless the user has explicitly selected that behaviour.
This policy is intentionally aligned with the conflict checkboxes and compatibility settings: automatic merge should remove avoidable prompts, but it must not silently choose between overlapping user intentions.
### File Structure Conventions
- **Platform-specific code**: Use `.platform.ts` suffix (replaced with `.obsidian.ts` in production builds via esbuild)
+1 -7
View File
@@ -153,7 +153,7 @@ Show verbose log. Please enable when you report the logs
Self-hosted LiveSync supports multiple remote connection profiles under **Remote Server** -> **Remote Databases**. This allows you to save and switch between multiple databases or bucket configurations in a single vault.
- ** Add new connection**: Create a new connection profile by launching the setup dialogue.
- **📥 Import connection**: Paste a connection string (e.g., `sls+https://...`, `sls+s3://...`, `sls+webdav://...`, `sls+p2p://...`) to import a remote configuration profile.
- **📥 Import connection**: Paste a connection string (e.g., `sls+https://...`, `sls+s3://...`, `sls+p2p://...`) to import a remote configuration profile.
- **🔧 Configure**: Open the setup dialogue to edit settings for the selected connection profile.
- **✅ Activate**: Select and activate this profile as the current active remote.
- **🗑️ Delete**: Remove this connection profile from the list.
@@ -164,12 +164,6 @@ Setting key: remoteType
The active remote server type. This is automatically projected to the legacy configuration when you activate a connection profile.
#### WebDAV Connection URI (Experimental)
Setting key: webDAVactiveConnectionURI
The active WebDAV connection URI used by experimental Journal synchronisation storage. This is configured in the **WebDAV Configuration** setup dialogue as a single `sls+webdav://...` URI. Optional query parameters include `prefix`, `useProxy=true`, and `insecure=true` for local HTTP testing. This feature is still a proof of concept for pluggable Journal storage backends and should be treated as experimental.
### 2. Notification
#### Notify when the estimated remote storage size exceeds on start up
+1 -1
View File
@@ -1,7 +1,7 @@
{
"id": "obsidian-livesync",
"name": "Self-hosted LiveSync",
"version": "0.25.79",
"version": "0.25.80",
"minAppVersion": "1.7.2",
"description": "Community implementation of self-hosted livesync. Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.",
"author": "vorotamoroz",
+5 -5
View File
@@ -1,12 +1,12 @@
{
"name": "obsidian-livesync",
"version": "0.25.79",
"version": "0.25.80",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "obsidian-livesync",
"version": "0.25.79",
"version": "0.25.80",
"license": "MIT",
"workspaces": [
"src/apps/cli",
@@ -16210,7 +16210,7 @@
},
"src/apps/cli": {
"name": "self-hosted-livesync-cli",
"version": "0.25.79-cli",
"version": "0.25.80-cli",
"dependencies": {
"chokidar": "^4.0.0",
"minimatch": "^10.2.5",
@@ -16236,7 +16236,7 @@
},
"src/apps/webapp": {
"name": "livesync-webapp",
"version": "0.25.79-webapp",
"version": "0.25.80-webapp",
"dependencies": {
"octagonal-wheels": "^0.1.47"
},
@@ -16251,7 +16251,7 @@
}
},
"src/apps/webpeer": {
"version": "0.25.79-webpeer",
"version": "0.25.80-webpeer",
"dependencies": {
"octagonal-wheels": "^0.1.47"
},
+1 -6
View File
@@ -1,6 +1,6 @@
{
"name": "obsidian-livesync",
"version": "0.25.79",
"version": "0.25.80",
"description": "Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.",
"main": "main.js",
"type": "module",
@@ -64,11 +64,6 @@
"test:docker-s3:start": "npm run test:docker-s3:up && sleep 3 && npm run test:docker-s3:init",
"test:docker-s3:down": "npx dotenv-cli -e .env -e .test.env -- ./test/shell/minio-stop.sh",
"test:docker-s3:stop": "npm run test:docker-s3:down",
"test:docker-webdav:up": "npx dotenv-cli -e .env -e .test.env -- ./test/shell/webdav-start.sh",
"test:docker-webdav:init": "npx dotenv-cli -e .env -e .test.env -- ./test/shell/webdav-init.sh",
"test:docker-webdav:start": "npm run test:docker-webdav:up && sleep 3 && npm run test:docker-webdav:init",
"test:docker-webdav:down": "npx dotenv-cli -e .env -e .test.env -- ./test/shell/webdav-stop.sh",
"test:docker-webdav:stop": "npm run test:docker-webdav:down",
"test:docker-p2p:up": "npx dotenv-cli -e .env -e .test.env -- ./test/shell/p2p-start.sh",
"test:docker-p2p:init": "npx dotenv-cli -e .env -e .test.env -- ./test/shell/p2p-init.sh",
"test:docker-p2p:start": "npm run test:docker-p2p:up && sleep 3 && npm run test:docker-p2p:init",
+78
View File
@@ -4,12 +4,21 @@ import type { ServiceContext } from "@lib/services/base/ServiceBase";
import { LiveSyncTrysteroReplicator } from "@lib/replication/trystero/LiveSyncTrysteroReplicator";
import { compatGlobal } from "@lib/common/coreEnvFunctions.ts";
import { LiveSyncError } from "@lib/common/LSError";
import { getPeerConnectionStats } from "@lib/rpc/transports/DiagRTCPeerConnections.utils";
import { appendFile } from "node:fs/promises";
type CLIP2PPeer = {
peerId: string;
name: string;
};
type CandidateSummary = {
id: string | "unknown";
candidateType: string | "unknown";
protocol: string | "unknown";
relayProtocol: string | "unknown";
};
function delay(ms: number): Promise<void> {
return new Promise((resolve) => compatGlobal.setTimeout(resolve, ms));
}
@@ -81,6 +90,74 @@ function resolvePeer(peers: CLIP2PPeer[], peerToken: string): CLIP2PPeer | undef
return undefined;
}
function getReportValue<T extends string | number>(
report: Record<string, unknown> | undefined,
key: string
): T | "unknown" {
const value = report?.[key];
return typeof value === "string" || typeof value === "number" ? (value as T) : "unknown";
}
function summariseCandidate(reports: unknown[], candidateId: string | "unknown"): CandidateSummary | undefined {
if (candidateId === "unknown") {
return undefined;
}
const report = reports.map((r) => r as Record<string, unknown>).find((r) => r.id === candidateId);
if (!report) {
return undefined;
}
return {
id: candidateId,
candidateType: getReportValue<string>(report, "candidateType"),
protocol: getReportValue<string>(report, "protocol"),
relayProtocol: getReportValue<string>(report, "relayProtocol"),
};
}
async function writePeerConnectionStatsIfRequested(
replicator: LiveSyncTrysteroReplicator,
peer: CLIP2PPeer
): Promise<void> {
const outputPath = process.env.LIVESYNC_P2P_STATS_JSONL?.trim();
if (!outputPath) {
return;
}
const peerConnection = replicator.rawHost?.room?.getPeers()[peer.peerId];
const stats = peerConnection ? await getPeerConnectionStats(`cli-p2p-${peer.peerId}`, peerConnection) : undefined;
const localCandidate = summariseCandidate(stats?.reports ?? [], stats?.localCandidateId ?? "unknown");
const remoteCandidate = summariseCandidate(stats?.reports ?? [], stats?.remoteCandidateId ?? "unknown");
const selectedPath =
localCandidate && remoteCandidate
? `${localCandidate.candidateType}<->${remoteCandidate.candidateType}`
: "unknown";
const payload = {
generatedAt: new Date().toISOString(),
command: "p2p-sync",
peerId: peer.peerId,
peerName: peer.name,
candidatePathCollected: !!stats?.selectedPair,
selectedPath,
selectedPair: stats
? {
id: stats.selectedPairId,
state: stats.state,
currentRoundTripTime: stats.currentRoundTripTime,
totalRoundTripTime: stats.totalRoundTripTime,
requestsSent: stats.requestsSent,
responsesReceived: stats.responsesReceived,
packetsDiscardedOnSend: stats.packetsDiscardedOnSend,
bytesSent: stats.bytesSent,
bytesReceived: stats.bytesReceived,
}
: undefined,
localCandidate,
remoteCandidate,
};
await appendFile(outputPath, `${JSON.stringify(payload)}\n`, "utf8");
}
export async function syncWithPeer(
core: LiveSyncBaseCore<ServiceContext, never>,
peerToken: string,
@@ -118,6 +195,7 @@ export async function syncWithPeer(
: LiveSyncError.fromError(err ?? "P2P sync failed while requesting remote sync");
}
await writePeerConnectionStatsIfRequested(replicator, targetPeer);
return targetPeer;
} finally {
await replicator.close();
+1 -21
View File
@@ -4,7 +4,7 @@ import * as os from "os";
import * as processSetting from "@lib/API/processSetting";
import { ConnectionStringParser } from "@lib/common/ConnectionString";
import { configURIBase } from "@lib/common/models/shared.const";
import { DEFAULT_SETTINGS, REMOTE_COUCHDB, REMOTE_MINIO, REMOTE_P2P, REMOTE_WEBDAV } from "@lib/common/types";
import { DEFAULT_SETTINGS, REMOTE_COUCHDB, REMOTE_MINIO, REMOTE_P2P } from "@lib/common/types";
import { describe, expect, it, vi, beforeEach, afterEach } from "vitest";
import { runCommand } from "./runCommand";
import type { CLIOptions } from "./types";
@@ -205,22 +205,6 @@ const protocolFixtures: ProtocolFixture[] = [
expect(settings.P2P_AppID).toBe("self-hosted-livesync");
},
},
{
protocol: "webdav",
connectionString: ConnectionStringParser.serialize({
type: "webdav",
settings: {
webDAVactiveConnectionURI:
"sls+webdav://user:pass@webdav.example.com/dav?prefix=vault%2F&headers=x-test%3A1&useProxy=true",
},
}),
assertProjectedFields: (settings) => {
expect(settings.remoteType).toBe(REMOTE_WEBDAV);
expect(settings.webDAVactiveConnectionURI).toBe(
"sls+webdav://user:pass@webdav.example.com/dav?prefix=vault%2F&headers=x-test%3A1&useProxy=true"
);
},
},
];
describe("runCommand abnormal cases", () => {
@@ -580,10 +564,6 @@ describe("runCommand abnormal cases", () => {
"p2p",
"sls+p2p://room-abc?passphrase=pass-123&relays=wss%3A%2F%2Frelay.example&appId=self-hosted-livesync",
] as const,
[
"webdav",
"sls+webdav://user:pass@webdav.example.com/dav?prefix=vault%2F&headers=x-test%3A1&useProxy=true",
] as const,
])("remote command round-trip works for %s", async (_protocol, initialConnStr) => {
const core = createCoreMock();
+1 -1
View File
@@ -1,7 +1,7 @@
{
"name": "self-hosted-livesync-cli",
"private": true,
"version": "0.25.79-cli",
"version": "0.25.80-cli",
"main": "dist/index.cjs",
"type": "module",
"scripts": {
+30 -3
View File
@@ -1,10 +1,11 @@
import { TempDir } from "./helpers/temp.ts";
import { applyRemoteSyncSettings, initSettingsFile } from "./helpers/settings.ts";
import { assertFilesEqual, runCliOrFail } from "./helpers/cli.ts";
import { startCouchdb, stopCouchdb } from "./helpers/docker.ts";
import { createCouchdbDatabase, startCouchdb, stopCouchdb } from "./helpers/docker.ts";
import { createDeterministicDataset, type DatasetEntry } from "./helpers/dataset.ts";
type BenchmarkConfig = {
caseName: string;
couchdbBackendUri: string;
couchdbProxyUri: string;
couchdbUser: string;
@@ -21,6 +22,10 @@ type BenchmarkConfig = {
requestedRttMs: number;
passphrase: string;
encrypt: boolean;
managedCouchdb: boolean;
simulationTier: string;
networkProfile: string;
networkModel: string;
};
function readEnvString(name: string, fallback: string): string {
@@ -70,6 +75,7 @@ function formatBytes(value: number): string {
function buildConfig(): BenchmarkConfig {
return {
caseName: readEnvString("BENCH_CASE", "couchdb-baseline"),
couchdbBackendUri: readEnvString("BENCH_COUCHDB_BACKEND_URI", "http://127.0.0.1:5989"),
couchdbProxyUri: readEnvString("BENCH_COUCHDB_URI", "http://127.0.0.1:15989"),
couchdbUser: readEnvString("BENCH_COUCHDB_USER", readEnvString("username", "admin")),
@@ -86,6 +92,10 @@ function buildConfig(): BenchmarkConfig {
requestedRttMs: Math.floor(readEnvNumber("BENCH_COUCHDB_RTT_MS", 50)),
passphrase: readEnvString("BENCH_PASSPHRASE", `bench-${Date.now()}`),
encrypt: readEnvBool("BENCH_ENCRYPT", true),
managedCouchdb: readEnvBool("BENCH_COUCHDB_MANAGED", true),
simulationTier: readEnvString("BENCH_SIMULATION_TIER", "1"),
networkProfile: readEnvString("BENCH_NETWORK_PROFILE", "http-latency-proxy"),
networkModel: readEnvString("BENCH_NETWORK_MODEL", "local-http-proxy"),
};
}
@@ -200,7 +210,17 @@ async function main(): Promise<void> {
await initSettingsFile(settingsA);
await initSettingsFile(settingsB);
await startCouchdb(config.couchdbBackendUri, config.couchdbUser, config.couchdbPassword, config.couchdbDbname);
if (config.managedCouchdb) {
await startCouchdb(config.couchdbBackendUri, config.couchdbUser, config.couchdbPassword, config.couchdbDbname);
} else {
console.log(`[INFO] using externally managed CouchDB: ${config.couchdbBackendUri}`);
await createCouchdbDatabase(
config.couchdbBackendUri,
config.couchdbUser,
config.couchdbPassword,
config.couchdbDbname
);
}
const proxy = startCouchdbProxy({
backendUri: config.couchdbBackendUri,
@@ -265,10 +285,15 @@ async function main(): Promise<void> {
}
const result = {
caseName: config.caseName,
mode: "couchdb-cli-benchmark",
couchdbBackendUri: config.couchdbBackendUri,
couchdbProxyUri: config.couchdbProxyUri,
couchdbDbname: config.couchdbDbname,
managedCouchdb: config.managedCouchdb,
simulationTier: config.simulationTier,
networkProfile: config.networkProfile,
networkModel: config.networkModel,
rttRequestedMs: config.requestedRttMs,
proxyApplied: proxy.applied,
proxyNote: proxy.note,
@@ -300,7 +325,9 @@ async function main(): Promise<void> {
);
} finally {
await proxy.stop();
await stopCouchdb().catch(() => {});
if (config.managedCouchdb) {
await stopCouchdb().catch(() => {});
}
}
}
@@ -0,0 +1,133 @@
type SweepResult = {
name: string;
runner: "p2p" | "couchdb";
rttMs?: number;
result: Record<string, unknown>;
};
function readEnvString(name: string, fallback: string): string {
const value = Deno.env.get(name)?.trim();
return value && value.length > 0 ? value : fallback;
}
function timestamp(): string {
const d = new Date();
const pad = (n: number) => String(n).padStart(2, "0");
return (
`${d.getUTCFullYear()}${pad(d.getUTCMonth() + 1)}${pad(d.getUTCDate())}-` +
`${pad(d.getUTCHours())}${pad(d.getUTCMinutes())}${pad(d.getUTCSeconds())}`
);
}
function parseRttList(raw: string): number[] {
const values = raw
.split(",")
.map((value) => Number(value.trim()))
.filter((value) => Number.isFinite(value) && value > 0)
.map((value) => Math.floor(value));
if (values.length === 0) {
throw new Error(`BENCH_SWEEP_RTT_MS must contain at least one positive number, got '${raw}'`);
}
return values;
}
function buildBaseEnv(): Record<string, string> {
return {
BENCH_MD_FILE_COUNT: readEnvString("BENCH_MD_FILE_COUNT", "20"),
BENCH_MD_MIN_SIZE_BYTES: readEnvString("BENCH_MD_MIN_SIZE_BYTES", "512"),
BENCH_MD_MAX_SIZE_BYTES: readEnvString("BENCH_MD_MAX_SIZE_BYTES", "2048"),
BENCH_BIN_FILE_COUNT: readEnvString("BENCH_BIN_FILE_COUNT", "5"),
BENCH_BIN_SIZE_BYTES: readEnvString("BENCH_BIN_SIZE_BYTES", "8192"),
BENCH_SYNC_TIMEOUT: readEnvString("BENCH_SYNC_TIMEOUT", "300"),
BENCH_PEERS_TIMEOUT: readEnvString("BENCH_PEERS_TIMEOUT", "60"),
BENCH_SEED: readEnvString("BENCH_SEED", "livesync-benchmark-seed"),
LIVESYNC_TEST_TEE: readEnvString("BENCH_LIVESYNC_TEST_TEE", "0"),
};
}
async function runBenchmark(options: {
taskName: "bench:p2p" | "bench:couchdb";
name: string;
outputDir: string;
env: Record<string, string>;
}): Promise<Record<string, unknown>> {
const resultPath = `${options.outputDir}/${options.name}.json`;
const env = {
...Deno.env.toObject(),
...options.env,
BENCH_RESULT_JSON: resultPath,
};
console.log(`[latency-sweep] running ${options.name}`);
const child = new Deno.Command("deno", {
args: ["task", options.taskName],
cwd: import.meta.dirname,
env,
stdin: "null",
stdout: "inherit",
stderr: "inherit",
}).spawn();
const status = await child.status;
if (status.code !== 0) {
throw new Error(`benchmark failed: ${options.name} (exit ${status.code})`);
}
return JSON.parse(await Deno.readTextFile(resultPath)) as Record<string, unknown>;
}
async function main(): Promise<void> {
const outRoot = readEnvString("BENCH_SWEEP_ROOT", `${import.meta.dirname}/bench-results`);
const outputDir = `${outRoot}/latency-sweep-${timestamp()}`;
const rtts = parseRttList(readEnvString("BENCH_SWEEP_RTT_MS", "20,50,100,150,300"));
const base = buildBaseEnv();
await Deno.mkdir(outputDir, { recursive: true });
const results: SweepResult[] = [];
if (readEnvString("BENCH_SWEEP_INCLUDE_P2P", "true") !== "false") {
const p2pResult = await runBenchmark({
taskName: "bench:p2p",
name: "p2p-direct-local",
outputDir,
env: {
...base,
BENCH_CASE: "p2p-direct-local",
BENCH_TURN_SERVERS: "",
},
});
results.push({ name: "p2p-direct-local", runner: "p2p", result: p2pResult });
}
for (const rtt of rtts) {
const name = `couchdb-rtt-${rtt}ms`;
const couchdbResult = await runBenchmark({
taskName: "bench:couchdb",
name,
outputDir,
env: {
...base,
BENCH_CASE: name,
BENCH_COUCHDB_RTT_MS: String(rtt),
},
});
results.push({ name, runner: "couchdb", rttMs: rtt, result: couchdbResult });
}
const summary = {
generatedAt: new Date().toISOString(),
outputDir,
note:
"This sweep models additional remote CouchDB request latency through the existing HTTP proxy. It is not a full netem model of jitter, loss, MTU, bandwidth, or VPN encapsulation.",
rtts,
results,
};
await Deno.writeTextFile(`${outputDir}/summary.json`, JSON.stringify(summary, null, 2));
console.log(JSON.stringify(summary, null, 2));
console.log(`[latency-sweep] result directory: ${outputDir}`);
}
if (import.meta.main) {
main().catch((error) => {
console.error("[Fatal Error]", error);
Deno.exit(1);
});
}
@@ -0,0 +1,312 @@
type BenchmarkCase = {
name: string;
runner: "p2p" | "couchdb";
description: string;
dataPath: string;
trustBoundary: string;
env: Record<string, string>;
};
function readEnvString(name: string, fallback: string): string {
const value = Deno.env.get(name)?.trim();
return value && value.length > 0 ? value : fallback;
}
function readEnvInteger(name: string, fallback: number): number {
const value = readEnvString(name, String(fallback));
const parsed = Number(value);
if (!Number.isInteger(parsed) || parsed < 1) {
throw new Error(`${name} must be a positive integer, got '${value}'`);
}
return parsed;
}
function timestamp(): string {
const d = new Date();
const pad = (n: number) => String(n).padStart(2, "0");
return (
`${d.getUTCFullYear()}${pad(d.getUTCMonth() + 1)}${pad(d.getUTCDate())}-` +
`${pad(d.getUTCHours())}${pad(d.getUTCMinutes())}${pad(d.getUTCSeconds())}`
);
}
function buildBaseEnv(): Record<string, string> {
return {
BENCH_MD_FILE_COUNT: readEnvString("BENCH_MD_FILE_COUNT", "20"),
BENCH_MD_MIN_SIZE_BYTES: readEnvString("BENCH_MD_MIN_SIZE_BYTES", "512"),
BENCH_MD_MAX_SIZE_BYTES: readEnvString("BENCH_MD_MAX_SIZE_BYTES", "2048"),
BENCH_BIN_FILE_COUNT: readEnvString("BENCH_BIN_FILE_COUNT", "5"),
BENCH_BIN_SIZE_BYTES: readEnvString("BENCH_BIN_SIZE_BYTES", "8192"),
BENCH_SYNC_TIMEOUT: readEnvString("BENCH_SYNC_TIMEOUT", "300"),
BENCH_PEERS_TIMEOUT: readEnvString("BENCH_PEERS_TIMEOUT", "60"),
BENCH_SEED: readEnvString("BENCH_SEED", "livesync-benchmark-seed"),
LIVESYNC_TEST_TEE: readEnvString("BENCH_LIVESYNC_TEST_TEE", "0"),
};
}
function buildCases(): BenchmarkCase[] {
const base = buildBaseEnv();
const couchdbRtt = readEnvString("BENCH_COUCHDB_RTT_MS", "20");
const tetheringVpnRtt = readEnvString("BENCH_TETHERING_VPN_RTT_MS", "120");
const localTurnServers = readEnvString("BENCH_LOCAL_TURN_SERVERS", "turn:127.0.0.1:3478");
const shimCouchdbUri = readEnvString("BENCH_SHIM_COUCHDB_URI", "http://couchdb-shim:5984");
const signallingShimRelay = readEnvString("BENCH_SIGNAL_SHIM_RELAY", "ws://p2p-signalling-shim:7777/");
return [
{
name: "couchdb-baseline",
runner: "couchdb",
description: "Standard self-hosted CouchDB path through a local latency proxy.",
dataPath: "Device A -> CouchDB -> Device B",
trustBoundary: "CouchDB operator and network path",
env: {
...base,
BENCH_CASE: "couchdb-baseline",
BENCH_COUCHDB_RTT_MS: couchdbRtt,
},
},
{
name: "p2p-direct-local",
runner: "p2p",
description: "Preferred direct WebRTC P2P path with Nostr signalling and TURN disabled.",
dataPath: "Device A -> Device B",
trustBoundary: "Nostr relay for signalling metadata; no TURN relay",
env: {
...base,
BENCH_CASE: "p2p-direct-local",
BENCH_TURN_SERVERS: "",
BENCH_SIMULATION_TIER: "1",
BENCH_NETWORK_PROFILE: "local-direct",
BENCH_NETWORK_MODEL: "local-runner-webrtc",
BENCH_P2P_CANDIDATE_PATH_VERIFICATION: "turn-disabled-but-selected-ice-pair-not-collected",
},
},
{
name: "couchdb-tethering-vpn-proxy",
runner: "couchdb",
description:
"Approximate smartphone tethering/VPN remote-database path using an HTTP latency proxy. This does not model loss, jitter, MTU, or VPN encapsulation.",
dataPath: "Device A -> VPN/network path -> CouchDB -> VPN/network path -> Device B",
trustBoundary: "VPN/network path and CouchDB operator",
env: {
...base,
BENCH_CASE: "couchdb-tethering-vpn-proxy",
BENCH_COUCHDB_RTT_MS: tetheringVpnRtt,
},
},
{
name: "couchdb-netem-home-wifi",
runner: "couchdb",
description:
"Tier 2 CouchDB path through the Compose netem TCP shim using the home-wifi profile.",
dataPath: "Device A -> netem TCP shim -> CouchDB -> netem TCP shim -> Device B",
trustBoundary: "CouchDB operator and constrained network shim",
env: {
...base,
BENCH_CASE: "couchdb-netem-home-wifi",
BENCH_COUCHDB_BACKEND_URI: shimCouchdbUri,
BENCH_COUCHDB_RTT_MS: "1",
BENCH_SIMULATION_TIER: "2",
BENCH_NETWORK_PROFILE: "home-wifi",
BENCH_NETWORK_MODEL: "compose-netem-tcp-shim",
},
},
{
name: "couchdb-netem-tethering-vpn",
runner: "couchdb",
description:
"Tier 2 CouchDB path through the Compose netem TCP shim using a tethering-vpn profile.",
dataPath: "Device A -> netem TCP shim -> CouchDB -> netem TCP shim -> Device B",
trustBoundary: "CouchDB operator and constrained smartphone/VPN-like network shim",
env: {
...base,
BENCH_CASE: "couchdb-netem-tethering-vpn",
BENCH_COUCHDB_BACKEND_URI: shimCouchdbUri,
BENCH_COUCHDB_RTT_MS: "1",
BENCH_SIMULATION_TIER: "2",
BENCH_NETWORK_PROFILE: "tethering-vpn",
BENCH_NETWORK_MODEL: "compose-netem-tcp-shim",
},
},
{
name: "p2p-smartphone-vpn-direct",
runner: "p2p",
description:
"Direct P2P case name for smartphone tethering/VPN measurements. In this local runner it is unshaped and should be treated as a wiring check unless executed on that network.",
dataPath: "Device A -> Device B when WebRTC direct connectivity succeeds",
trustBoundary: "Smartphone/VPN routing policy plus Nostr signalling metadata",
env: {
...base,
BENCH_CASE: "p2p-smartphone-vpn-direct",
BENCH_TURN_SERVERS: "",
BENCH_SIMULATION_TIER: "unmeasured",
BENCH_NETWORK_PROFILE: "smartphone-vpn-direct-placeholder",
BENCH_NETWORK_MODEL: "local-runner-no-netem",
BENCH_P2P_CANDIDATE_PATH_VERIFICATION:
"structural-placeholder-only; selected ICE pair may be collected, but the path is not shaped",
},
},
{
name: "p2p-signalling-netem-home-wifi",
runner: "p2p",
description:
"Tier 2 P2P path with only the Nostr signalling relay accessed through the home-wifi netem shim.",
dataPath: "Device A -> Device B over WebRTC DataChannel; Nostr signalling through netem shim",
trustBoundary: "Nostr signalling metadata through constrained network shim; no TURN relay",
env: {
...base,
BENCH_CASE: "p2p-signalling-netem-home-wifi",
BENCH_RELAY: signallingShimRelay,
BENCH_TURN_SERVERS: "",
BENCH_SIMULATION_TIER: "2",
BENCH_NETWORK_PROFILE: "home-wifi",
BENCH_NETWORK_MODEL: "compose-netem-signalling-shim",
BENCH_P2P_CANDIDATE_PATH_VERIFICATION:
"selected ICE pair collected; only Nostr signalling path is shaped",
},
},
{
name: "p2p-signalling-netem-tethering-vpn",
runner: "p2p",
description:
"Tier 2 P2P path with only the Nostr signalling relay accessed through the tethering-vpn netem shim.",
dataPath: "Device A -> Device B over WebRTC DataChannel; Nostr signalling through netem shim",
trustBoundary: "Nostr signalling metadata through constrained smartphone/VPN-like network shim; no TURN relay",
env: {
...base,
BENCH_CASE: "p2p-signalling-netem-tethering-vpn",
BENCH_RELAY: signallingShimRelay,
BENCH_TURN_SERVERS: "",
BENCH_SIMULATION_TIER: "2",
BENCH_NETWORK_PROFILE: "tethering-vpn",
BENCH_NETWORK_MODEL: "compose-netem-signalling-shim",
BENCH_P2P_CANDIDATE_PATH_VERIFICATION:
"selected ICE pair collected; only Nostr signalling path is shaped",
},
},
{
name: "p2p-user-turn",
runner: "p2p",
description: "Optional fallback path through a local user-controlled TURN server.",
dataPath: "Device A -> user-controlled TURN -> Device B",
trustBoundary: "User-controlled TURN server",
env: {
...base,
BENCH_CASE: "p2p-user-turn",
BENCH_TURN_SERVERS: localTurnServers,
BENCH_SIMULATION_TIER: "1",
BENCH_NETWORK_PROFILE: "local-turn-fallback",
BENCH_NETWORK_MODEL: "local-runner-webrtc-turn-configured",
BENCH_P2P_CANDIDATE_PATH_VERIFICATION:
"turn-configured; selected ICE pair may still be direct or relayed, so interpret the recorded candidate types",
},
},
];
}
async function runCase(
testCase: BenchmarkCase,
outputDir: string,
repeatIndex: number,
repeatCount: number
): Promise<Record<string, unknown>> {
const suffix = repeatCount > 1 ? `-r${String(repeatIndex).padStart(2, "0")}` : "";
const resultPath = `${outputDir}/${testCase.name}${suffix}.json`;
const taskName = testCase.runner === "p2p" ? "bench:p2p" : "bench:couchdb";
const env = {
...Deno.env.toObject(),
...testCase.env,
BENCH_RESULT_JSON: resultPath,
BENCH_REPEAT_INDEX: String(repeatIndex),
BENCH_REPEAT_COUNT: String(repeatCount),
};
const repeatLabel = repeatCount > 1 ? ` (${repeatIndex}/${repeatCount})` : "";
console.log(`[bench-cases] running ${testCase.name}${repeatLabel}: ${testCase.description}`);
const command = new Deno.Command("deno", {
args: ["task", taskName],
cwd: import.meta.dirname,
env,
stdin: "null",
stdout: "inherit",
stderr: "inherit",
});
const child = command.spawn();
const status = await child.status;
if (status.code !== 0) {
throw new Error(`case failed: ${testCase.name} (exit ${status.code})`);
}
const result = JSON.parse(await Deno.readTextFile(resultPath)) as Record<string, unknown>;
return {
...testCase,
repeatIndex,
repeatCount,
resultPath,
result,
};
}
function selectCases(allCases: BenchmarkCase[]): BenchmarkCase[] {
const requested = readEnvString("BENCH_CASES", "couchdb-baseline,p2p-direct-local");
const names = requested
.split(",")
.map((v) => v.trim())
.filter((v) => v.length > 0);
const byName = new Map(allCases.map((c) => [c.name, c]));
return names.map((name) => {
const found = byName.get(name);
if (!found) {
throw new Error(`Unknown BENCH_CASES entry '${name}'. Available: ${allCases.map((c) => c.name).join(", ")}`);
}
return found;
});
}
async function main(): Promise<void> {
const outRoot = readEnvString("BENCH_CASES_ROOT", `${import.meta.dirname}/bench-results`);
const outputDir = `${outRoot}/cases-${timestamp()}`;
await Deno.mkdir(outputDir, { recursive: true });
const allCases = buildCases();
const cases = selectCases(allCases);
const repeatCount = readEnvInteger("BENCH_REPEAT_COUNT", 1);
await Deno.writeTextFile(
`${outputDir}/case-manifest.json`,
JSON.stringify(
{
generatedAt: new Date().toISOString(),
repeatCount,
selectedCases: cases,
availableCases: allCases,
},
null,
2
)
);
const results: Record<string, unknown>[] = [];
for (const testCase of cases) {
for (let repeatIndex = 1; repeatIndex <= repeatCount; repeatIndex++) {
results.push(await runCase(testCase, outputDir, repeatIndex, repeatCount));
}
}
const summary = {
generatedAt: new Date().toISOString(),
outputDir,
repeatCount,
results,
};
await Deno.writeTextFile(`${outputDir}/summary.json`, JSON.stringify(summary, null, 2));
console.log(JSON.stringify(summary, null, 2));
console.log(`[bench-cases] result directory: ${outputDir}`);
}
if (import.meta.main) {
main().catch((error) => {
console.error("[Fatal Error]", error);
Deno.exit(1);
});
}
@@ -0,0 +1,435 @@
import { join } from "@std/path";
import { startCliInBackground } from "./helpers/backgroundCli.ts";
import { assertFilesEqual, runCliOrFail } from "./helpers/cli.ts";
import { createDeterministicDataset, type DatasetEntry } from "./helpers/dataset.ts";
import { discoverPeer } from "./helpers/p2p.ts";
import { applyP2pSettings, applyP2pTestTweaks, initSettingsFile } from "./helpers/settings.ts";
type Role = "host" | "client";
type NetemSummary = {
enabled: boolean;
profile: string;
interface: string;
delayMs: number;
jitterMs: number;
lossPercent: number;
bandwidthMbit: number;
mtu: number;
tcQdisc?: string;
ipAddr?: string;
ipRoute?: string;
};
type HostReady = {
generatedAt: string;
totalFiles: number;
totalBytes: number;
mdFileCount: number;
binFileCount: number;
mirrorElapsedMs: number;
netem: NetemSummary;
};
type P2PConnectionStats = {
candidatePathCollected: boolean;
selectedPath: string;
localCandidate?: { candidateType: string; protocol: string; relayProtocol: string };
remoteCandidate?: { candidateType: string; protocol: string; relayProtocol: string };
};
function errorToRecord(error: unknown): Record<string, unknown> {
if (error instanceof Error) {
return {
name: error.name,
message: error.message,
stack: error.stack,
};
}
return {
name: "UnknownError",
message: String(error),
};
}
function readEnvString(name: string, fallback: string): string {
const value = Deno.env.get(name)?.trim();
return value && value.length > 0 ? value : fallback;
}
function readEnvNumber(name: string, fallback: number): number {
const raw = Deno.env.get(name);
if (raw === undefined || raw.trim() === "") {
return fallback;
}
const parsed = Number(raw);
if (!Number.isFinite(parsed) || parsed < 0) {
throw new Error(`${name} must be a non-negative number, got '${raw}'`);
}
return parsed;
}
function nowMs(): number {
return performance.now();
}
async function commandOutput(command: string, args: string[]): Promise<string> {
const output = await new Deno.Command(command, {
args,
stdin: "null",
stdout: "piped",
stderr: "piped",
}).output();
const stdout = new TextDecoder().decode(output.stdout);
const stderr = new TextDecoder().decode(output.stderr);
if (!output.success) {
throw new Error(`${command} ${args.join(" ")} failed\nstdout: ${stdout}\nstderr: ${stderr}`);
}
return stdout.trim();
}
async function commandOk(command: string, args: string[]): Promise<void> {
await commandOutput(command, args);
}
async function applyNetemIfRequested(): Promise<NetemSummary> {
const enabled = readEnvString("BENCH_NETEM_ENABLED", "0") === "1";
const profile = readEnvString("NETEM_PROFILE", "home-wifi");
const iface = readEnvString("NETEM_INTERFACE", "eth0");
const delayMs = readEnvNumber("NETEM_DELAY_MS", 20);
const jitterMs = readEnvNumber("NETEM_JITTER_MS", 5);
const lossPercent = readEnvNumber("NETEM_LOSS_PERCENT", 0.1);
const bandwidthMbit = readEnvNumber("NETEM_BANDWIDTH_MBIT", 100);
const mtu = readEnvNumber("NETEM_MTU", 1500);
const summary: NetemSummary = {
enabled,
profile,
interface: iface,
delayMs,
jitterMs,
lossPercent,
bandwidthMbit,
mtu,
};
if (!enabled) {
return summary;
}
await commandOk("ip", ["link", "set", "dev", iface, "mtu", String(mtu)]);
await new Deno.Command("tc", { args: ["qdisc", "del", "dev", iface, "root"] }).output();
await commandOk("tc", [
"qdisc",
"add",
"dev",
iface,
"root",
"netem",
"delay",
`${delayMs}ms`,
`${jitterMs}ms`,
"loss",
`${lossPercent}%`,
"rate",
`${bandwidthMbit}mbit`,
]);
summary.tcQdisc = await commandOutput("tc", ["qdisc", "show", "dev", iface]);
summary.ipAddr = await commandOutput("ip", ["addr", "show", iface]);
summary.ipRoute = await commandOutput("ip", ["route"]);
return summary;
}
async function waitForFile(path: string, timeoutMs: number): Promise<void> {
const started = Date.now();
while (Date.now() - started < timeoutMs) {
try {
const stat = await Deno.stat(path);
if (stat.isFile) {
return;
}
} catch {
// wait
}
await new Promise((resolve) => setTimeout(resolve, 250));
}
throw new Error(`Timed out waiting for ${path}`);
}
async function readJsonFile<T>(path: string): Promise<T> {
return JSON.parse(await Deno.readTextFile(path)) as T;
}
function pickSampleFiles(entries: DatasetEntry[]): DatasetEntry[] {
const unique = new Map<string, DatasetEntry>();
for (const entry of [entries.find((e) => e.kind === "md"), entries.find((e) => e.kind === "bin"), entries.at(-1)]) {
if (entry) {
unique.set(entry.relativePath, entry);
}
}
return [...unique.values()];
}
async function readLatestP2PConnectionStats(path: string): Promise<P2PConnectionStats | undefined> {
try {
const lines = (await Deno.readTextFile(path))
.split(/\r?\n/)
.map((line) => line.trim())
.filter((line) => line.length > 0);
return lines.length === 0 ? undefined : (JSON.parse(lines.at(-1)!) as P2PConnectionStats);
} catch {
return undefined;
}
}
function buildCommonConfig() {
const runId = readEnvString("BENCH_SPLIT_RUN_ID", readEnvString("BENCH_ROOM_ID", "bench-split-run"));
const baseWorkRoot = readEnvString("BENCH_SPLIT_WORK_ROOT", "/p2p-work");
return {
runId,
workRoot: join(baseWorkRoot, runId),
resultRoot: readEnvString("BENCH_SPLIT_RESULT_ROOT", "/workspace/src/apps/cli/testdeno/bench-results"),
relay: readEnvString("BENCH_RELAY", "ws://nostr-relay:7777/"),
appId: readEnvString("BENCH_APP_ID", "self-hosted-livesync-cli-benchmark"),
roomId: readEnvString("BENCH_ROOM_ID", "bench-split-room"),
passphrase: readEnvString("BENCH_PASSPHRASE", "bench-split-passphrase"),
turnServers: readEnvString("BENCH_TURN_SERVERS", ""),
datasetDirName: readEnvString("BENCH_DATASET_DIR", "bench-dataset"),
datasetSeed: readEnvString("BENCH_SEED", "livesync-benchmark-seed"),
mdFileCount: Math.floor(readEnvNumber("BENCH_MD_FILE_COUNT", 20)),
mdMinSizeBytes: Math.floor(readEnvNumber("BENCH_MD_MIN_SIZE_BYTES", 512)),
mdMaxSizeBytes: Math.floor(readEnvNumber("BENCH_MD_MAX_SIZE_BYTES", 2048)),
binFileCount: Math.floor(readEnvNumber("BENCH_BIN_FILE_COUNT", 5)),
binSizeBytes: Math.floor(readEnvNumber("BENCH_BIN_SIZE_BYTES", 8192)),
peersTimeoutSeconds: readEnvNumber("BENCH_PEERS_TIMEOUT", 60),
syncTimeoutSeconds: readEnvNumber("BENCH_SYNC_TIMEOUT", 300),
nodeTimeoutMs: readEnvNumber("BENCH_SPLIT_NODE_TIMEOUT_MS", 360_000),
profile: readEnvString("BENCH_NETWORK_PROFILE", readEnvString("NETEM_PROFILE", "split-compose")),
};
}
async function prepareP2PSettings(
settingsPath: string,
peerName: string,
config: ReturnType<typeof buildCommonConfig>
) {
await initSettingsFile(settingsPath);
await applyP2pSettings(
settingsPath,
config.roomId,
config.passphrase,
config.appId,
config.relay,
"~.*",
config.turnServers
);
await applyP2pTestTweaks(settingsPath, peerName, config.passphrase);
}
async function runHost(): Promise<void> {
const config = buildCommonConfig();
const netem = await applyNetemIfRequested();
await Deno.mkdir(config.workRoot, { recursive: true });
await Deno.mkdir(config.resultRoot, { recursive: true });
const hostVault = join(config.workRoot, "vault-host");
const hostSettings = join(config.workRoot, "settings-host.json");
await Deno.mkdir(hostVault, { recursive: true });
await prepareP2PSettings(hostSettings, "p2p-split-host", config);
const seedFiles = await createDeterministicDataset({
rootDir: hostVault,
datasetDirName: config.datasetDirName,
seed: config.datasetSeed,
mdCount: config.mdFileCount,
mdMinSizeBytes: config.mdMinSizeBytes,
mdMaxSizeBytes: config.mdMaxSizeBytes,
binCount: config.binFileCount,
binSizeBytes: config.binSizeBytes,
});
await Deno.writeTextFile(
join(config.workRoot, "sample-files.json"),
JSON.stringify(pickSampleFiles(seedFiles.entries), null, 2)
);
const mirrorStart = nowMs();
await runCliOrFail(hostVault, "--settings", hostSettings, "mirror");
const mirrorElapsedMs = Number((nowMs() - mirrorStart).toFixed(1));
const hostReady: HostReady = {
generatedAt: new Date().toISOString(),
totalFiles: seedFiles.totalFiles,
totalBytes: seedFiles.totalBytes,
mdFileCount: seedFiles.mdCount,
binFileCount: seedFiles.binCount,
mirrorElapsedMs,
netem,
};
await Deno.writeTextFile(join(config.workRoot, "host-ready.json"), JSON.stringify(hostReady, null, 2));
const host = startCliInBackground(hostVault, "--settings", hostSettings, "p2p-host");
try {
await host.waitUntilContains("P2P host is running", 20_000);
await Deno.writeTextFile(
join(config.workRoot, "p2p-host-ready.json"),
JSON.stringify({ generatedAt: new Date().toISOString() })
);
await waitForFile(join(config.workRoot, "client-done.json"), config.nodeTimeoutMs);
} finally {
await host.stop();
}
}
async function runClient(): Promise<void> {
const config = buildCommonConfig();
const netem = await applyNetemIfRequested();
await Deno.mkdir(config.resultRoot, { recursive: true });
await waitForFile(join(config.workRoot, "host-ready.json"), config.nodeTimeoutMs);
await waitForFile(join(config.workRoot, "p2p-host-ready.json"), config.nodeTimeoutMs);
const clientVault = join(config.workRoot, "vault-client");
const clientSettings = join(config.workRoot, "settings-client.json");
const statsPath = join(config.workRoot, "p2p-connection-stats.jsonl");
await Deno.mkdir(clientVault, { recursive: true });
await prepareP2PSettings(clientSettings, "p2p-split-client", config);
const hostReady = await readJsonFile<HostReady>(join(config.workRoot, "host-ready.json"));
const timestamp = new Date().toISOString().replace(/[-:]/g, "").slice(0, 15);
const outputDir = join(config.resultRoot, `p2p-split-${config.profile}-${timestamp}`);
await Deno.mkdir(outputDir, { recursive: true });
const previousStatsPath = Deno.env.get("LIVESYNC_P2P_STATS_JSONL");
Deno.env.set("LIVESYNC_P2P_STATS_JSONL", statsPath);
let stage = "peer-discovery";
let peerDiscoveryCommandElapsedMs: number | undefined;
let syncElapsedMs: number | undefined;
try {
const peerDiscoveryCommandStart = nowMs();
const peer = await discoverPeer(clientVault, clientSettings, config.peersTimeoutSeconds);
peerDiscoveryCommandElapsedMs = Number((nowMs() - peerDiscoveryCommandStart).toFixed(1));
stage = "p2p-sync";
const syncStart = nowMs();
await runCliOrFail(
clientVault,
"--settings",
clientSettings,
"p2p-sync",
peer.id,
String(config.syncTimeoutSeconds)
);
syncElapsedMs = Number((nowMs() - syncStart).toFixed(1));
stage = "sample-verification";
const samples = await readJsonFile<DatasetEntry[]>(join(config.workRoot, "sample-files.json"));
for (const sample of samples) {
const pulledPath = join(config.workRoot, `pulled-${sample.relativePath.replaceAll("/", "_")}`);
await runCliOrFail(clientVault, "--settings", clientSettings, "pull", sample.relativePath, pulledPath);
await assertFilesEqual(
sample.absolutePath,
pulledPath,
`sample file mismatch after split sync: ${sample.relativePath}`
);
}
const p2pConnectionStats = await readLatestP2PConnectionStats(statsPath);
const result = {
ok: true,
generatedAt: new Date().toISOString(),
caseName: "p2p-split-compose",
mode: "p2p-split-compose-benchmark",
runId: config.runId,
simulationTier: Deno.env.get("BENCH_NETEM_ENABLED") === "1" ? "2" : "1",
networkProfile: config.profile,
networkModel:
Deno.env.get("BENCH_NETEM_ENABLED") === "1" ? "split-compose-egress-netem" : "split-compose-no-netem",
relay: config.relay,
turnServers: config.turnServers,
turnEnabled: config.turnServers.trim().length > 0,
p2pCandidatePathVerified: p2pConnectionStats?.candidatePathCollected === true,
p2pConnectionStats,
hostNetem: hostReady.netem,
clientNetem: netem,
totalFiles: hostReady.totalFiles,
totalBytes: hostReady.totalBytes,
mdFileCount: hostReady.mdFileCount,
binFileCount: hostReady.binFileCount,
mirrorElapsedMs: hostReady.mirrorElapsedMs,
peerDiscoveryTimeoutSeconds: config.peersTimeoutSeconds,
peerDiscoveryCommandElapsedMs,
syncElapsedMs,
throughputBytesPerSec: Number((hostReady.totalBytes / (syncElapsedMs / 1000)).toFixed(2)),
throughputMiBPerSec: Number((hostReady.totalBytes / (syncElapsedMs / 1000) / 1024 / 1024).toFixed(4)),
};
await Deno.writeTextFile(join(outputDir, "summary.json"), JSON.stringify(result, null, 2));
await Deno.writeTextFile(
join(config.workRoot, "client-done.json"),
JSON.stringify({ generatedAt: new Date().toISOString(), outputDir, ok: true })
);
console.log(JSON.stringify(result, null, 2));
} catch (error) {
const p2pConnectionStats = await readLatestP2PConnectionStats(statsPath);
const result = {
ok: false,
generatedAt: new Date().toISOString(),
caseName: "p2p-split-compose",
mode: "p2p-split-compose-benchmark",
runId: config.runId,
simulationTier: Deno.env.get("BENCH_NETEM_ENABLED") === "1" ? "2" : "1",
networkProfile: config.profile,
networkModel:
Deno.env.get("BENCH_NETEM_ENABLED") === "1" ? "split-compose-egress-netem" : "split-compose-no-netem",
relay: config.relay,
turnServers: config.turnServers,
turnEnabled: config.turnServers.trim().length > 0,
p2pCandidatePathVerified: p2pConnectionStats?.candidatePathCollected === true,
p2pConnectionStats,
hostNetem: hostReady.netem,
clientNetem: netem,
totalFiles: hostReady.totalFiles,
totalBytes: hostReady.totalBytes,
mdFileCount: hostReady.mdFileCount,
binFileCount: hostReady.binFileCount,
mirrorElapsedMs: hostReady.mirrorElapsedMs,
peerDiscoveryTimeoutSeconds: config.peersTimeoutSeconds,
peerDiscoveryCommandElapsedMs,
syncElapsedMs,
failure: {
stage,
...errorToRecord(error),
},
};
await Deno.writeTextFile(join(outputDir, "summary.json"), JSON.stringify(result, null, 2));
await Deno.writeTextFile(
join(config.workRoot, "client-done.json"),
JSON.stringify({ generatedAt: new Date().toISOString(), outputDir, ok: false })
);
console.log(JSON.stringify(result, null, 2));
throw error;
} finally {
if (previousStatsPath === undefined) {
Deno.env.delete("LIVESYNC_P2P_STATS_JSONL");
} else {
Deno.env.set("LIVESYNC_P2P_STATS_JSONL", previousStatsPath);
}
}
}
async function main(): Promise<void> {
const role = readEnvString("BENCH_P2P_SPLIT_ROLE", "") as Role;
if (role === "host") {
await runHost();
return;
}
if (role === "client") {
await runClient();
return;
}
throw new Error("BENCH_P2P_SPLIT_ROLE must be 'host' or 'client'");
}
if (import.meta.main) {
main().catch((error) => {
console.error("[Fatal Error]", error);
Deno.exit(1);
});
}
+207 -90
View File
@@ -1,15 +1,23 @@
import { TempDir } from "./helpers/temp.ts";
import { applyP2pSettings, applyP2pTestTweaks, initSettingsFile } from "./helpers/settings.ts";
import { startCliInBackground } from "./helpers/backgroundCli.ts";
import { discoverPeer, maybeStartLocalRelay, stopLocalRelayIfStarted } from "./helpers/p2p.ts";
import {
discoverPeer,
maybeStartCoturn,
maybeStartLocalRelay,
stopCoturnIfStarted,
stopLocalRelayIfStarted,
} from "./helpers/p2p.ts";
import { assertFilesEqual, runCliOrFail } from "./helpers/cli.ts";
import { createDeterministicDataset, type DatasetEntry } from "./helpers/dataset.ts";
type BenchmarkConfig = {
caseName: string;
relay: string;
appId: string;
roomId: string;
passphrase: string;
turnServers: string;
datasetDirName: string;
datasetSeed: string;
mdFileCount: number;
@@ -19,6 +27,42 @@ type BenchmarkConfig = {
binSizeBytes: number;
peersTimeoutSeconds: number;
syncTimeoutSeconds: number;
simulationTier: string;
networkProfile: string;
networkModel: string;
candidatePathVerification: string;
};
type P2PConnectionStats = {
generatedAt: string;
command: string;
peerId: string;
peerName: string;
candidatePathCollected: boolean;
selectedPath: string;
selectedPair?: {
id: string;
state: string;
currentRoundTripTime: number | "unknown";
totalRoundTripTime: number | "unknown";
requestsSent: number | "unknown";
responsesReceived: number | "unknown";
packetsDiscardedOnSend: number | "unknown";
bytesSent: number | "unknown";
bytesReceived: number | "unknown";
};
localCandidate?: {
id: string;
candidateType: string;
protocol: string;
relayProtocol: string;
};
remoteCandidate?: {
id: string;
candidateType: string;
protocol: string;
relayProtocol: string;
};
};
function readEnvString(name: string, fallback: string): string {
@@ -61,10 +105,12 @@ function formatBytes(value: number): string {
function buildConfig(): BenchmarkConfig {
return {
caseName: readEnvString("BENCH_CASE", "p2p-direct-local"),
relay: readEnvString("BENCH_RELAY", "ws://localhost:4000/"),
appId: readEnvString("BENCH_APP_ID", "self-hosted-livesync-cli-benchmark"),
roomId: readEnvString("BENCH_ROOM_ID", `bench-room-${Date.now()}`),
passphrase: readEnvString("BENCH_PASSPHRASE", `bench-${Date.now()}`),
turnServers: readEnvString("BENCH_TURN_SERVERS", ""),
datasetDirName: readEnvString("BENCH_DATASET_DIR", "bench-dataset"),
datasetSeed: readEnvString("BENCH_SEED", "livesync-benchmark-seed"),
mdFileCount: Math.floor(readEnvNumber("BENCH_MD_FILE_COUNT", 1500)),
@@ -74,6 +120,10 @@ function buildConfig(): BenchmarkConfig {
binSizeBytes: Math.floor(readEnvNumber("BENCH_BIN_SIZE_BYTES", 100 * 1024)),
peersTimeoutSeconds: readEnvNumber("BENCH_PEERS_TIMEOUT", 20),
syncTimeoutSeconds: readEnvNumber("BENCH_SYNC_TIMEOUT", 240),
simulationTier: readEnvString("BENCH_SIMULATION_TIER", "1"),
networkProfile: readEnvString("BENCH_NETWORK_PROFILE", "local-direct"),
networkModel: readEnvString("BENCH_NETWORK_MODEL", "local-runner-webrtc"),
candidatePathVerification: readEnvString("BENCH_P2P_CANDIDATE_PATH_VERIFICATION", "not-collected"),
};
}
@@ -102,115 +152,182 @@ function pickSampleFiles(entries: DatasetEntry[]): DatasetEntry[] {
return [...unique.values()];
}
async function readLatestP2PConnectionStats(statsPath: string): Promise<P2PConnectionStats | undefined> {
try {
const text = await Deno.readTextFile(statsPath);
const lines = text
.split(/\r?\n/)
.map((line) => line.trim())
.filter((line) => line.length > 0);
if (lines.length === 0) {
return undefined;
}
return JSON.parse(lines[lines.length - 1]) as P2PConnectionStats;
} catch {
return undefined;
}
}
async function main(): Promise<void> {
const config = buildConfig();
const resultPath = readOptionalResultPath();
const relayStarted = await maybeStartLocalRelay(config.relay);
const coturnStarted = await maybeStartCoturn(config.turnServers);
await using workDir = await TempDir.create("livesync-cli-p2p-bench");
const hostVault = workDir.join("vault-host");
const clientVault = workDir.join("vault-client");
const hostSettings = workDir.join("settings-host.json");
const clientSettings = workDir.join("settings-client.json");
const p2pStatsPath = workDir.join("p2p-connection-stats.jsonl");
const previousStatsPath = Deno.env.get("LIVESYNC_P2P_STATS_JSONL");
Deno.env.set("LIVESYNC_P2P_STATS_JSONL", p2pStatsPath);
await Promise.all([
Deno.mkdir(hostVault, { recursive: true }),
Deno.mkdir(clientVault, { recursive: true }),
initSettingsFile(hostSettings),
initSettingsFile(clientSettings),
]);
await Promise.all([
applyP2pSettings(hostSettings, config.roomId, config.passphrase, config.appId, config.relay, "~.*"),
applyP2pSettings(clientSettings, config.roomId, config.passphrase, config.appId, config.relay, "~.*"),
]);
await Promise.all([
applyP2pTestTweaks(hostSettings, "p2p-bench-host", config.passphrase),
applyP2pTestTweaks(clientSettings, "p2p-bench-client", config.passphrase),
]);
const seedFiles = await createDeterministicDataset({
rootDir: hostVault,
datasetDirName: config.datasetDirName,
seed: config.datasetSeed,
mdCount: config.mdFileCount,
mdMinSizeBytes: config.mdMinSizeBytes,
mdMaxSizeBytes: config.mdMaxSizeBytes,
binCount: config.binFileCount,
binSizeBytes: config.binSizeBytes,
});
const mirrorStart = nowMs();
await runCliOrFail(hostVault, "--settings", hostSettings, "mirror");
const mirrorElapsed = nowMs() - mirrorStart;
const host = startCliInBackground(hostVault, "--settings", hostSettings, "p2p-host");
try {
const hostReadyStart = nowMs();
await host.waitUntilContains("P2P host is running", 20000);
const hostReadyElapsed = nowMs() - hostReadyStart;
await Promise.all([
Deno.mkdir(hostVault, { recursive: true }),
Deno.mkdir(clientVault, { recursive: true }),
initSettingsFile(hostSettings),
initSettingsFile(clientSettings),
]);
const peerDiscoveryStart = nowMs();
const peer = await discoverPeer(clientVault, clientSettings, config.peersTimeoutSeconds);
const peerDiscoveryElapsed = nowMs() - peerDiscoveryStart;
await Promise.all([
applyP2pSettings(
hostSettings,
config.roomId,
config.passphrase,
config.appId,
config.relay,
"~.*",
config.turnServers
),
applyP2pSettings(
clientSettings,
config.roomId,
config.passphrase,
config.appId,
config.relay,
"~.*",
config.turnServers
),
]);
const syncStart = nowMs();
await runCliOrFail(
clientVault,
"--settings",
clientSettings,
"p2p-sync",
peer.id,
String(config.syncTimeoutSeconds)
);
const syncElapsed = nowMs() - syncStart;
await Promise.all([
applyP2pTestTweaks(hostSettings, "p2p-bench-host", config.passphrase),
applyP2pTestTweaks(clientSettings, "p2p-bench-client", config.passphrase),
]);
const sampleFiles = pickSampleFiles(seedFiles.entries);
for (const sample of sampleFiles) {
const pulledPath = workDir.join(`pulled-${sample.relativePath.replaceAll("/", "_")}`);
await runCliOrFail(clientVault, "--settings", clientSettings, "pull", sample.relativePath, pulledPath);
await assertFilesEqual(
sample.absolutePath,
pulledPath,
`sample file mismatch after sync: ${sample.relativePath}`
);
}
const result = {
mode: "p2p-cli-benchmark",
relay: config.relay,
appId: config.appId,
roomId: config.roomId,
datasetSeed: config.datasetSeed,
const seedFiles = await createDeterministicDataset({
rootDir: hostVault,
datasetDirName: config.datasetDirName,
peerId: peer.id,
peerName: peer.name,
totalFiles: seedFiles.totalFiles,
totalBytes: seedFiles.totalBytes,
mdFileCount: seedFiles.mdCount,
binFileCount: seedFiles.binCount,
mirrorElapsedMs: Number(mirrorElapsed.toFixed(1)),
hostReadyElapsedMs: Number(hostReadyElapsed.toFixed(1)),
peerDiscoveryElapsedMs: Number(peerDiscoveryElapsed.toFixed(1)),
syncElapsedMs: Number(syncElapsed.toFixed(1)),
throughputBytesPerSec: Number((seedFiles.totalBytes / (syncElapsed / 1000)).toFixed(2)),
throughputMiBPerSec: Number((seedFiles.totalBytes / (syncElapsed / 1000) / 1024 / 1024).toFixed(4)),
};
seed: config.datasetSeed,
mdCount: config.mdFileCount,
mdMinSizeBytes: config.mdMinSizeBytes,
mdMaxSizeBytes: config.mdMaxSizeBytes,
binCount: config.binFileCount,
binSizeBytes: config.binSizeBytes,
});
if (resultPath) {
await Deno.writeTextFile(resultPath, JSON.stringify(result, null, 2));
const mirrorStart = nowMs();
await runCliOrFail(hostVault, "--settings", hostSettings, "mirror");
const mirrorElapsed = nowMs() - mirrorStart;
const host = startCliInBackground(hostVault, "--settings", hostSettings, "p2p-host");
try {
const hostReadyStart = nowMs();
await host.waitUntilContains("P2P host is running", 20000);
const hostReadyElapsed = nowMs() - hostReadyStart;
const peerDiscoveryCommandStart = nowMs();
const peer = await discoverPeer(clientVault, clientSettings, config.peersTimeoutSeconds);
const peerDiscoveryCommandElapsed = nowMs() - peerDiscoveryCommandStart;
const syncStart = nowMs();
await runCliOrFail(
clientVault,
"--settings",
clientSettings,
"p2p-sync",
peer.id,
String(config.syncTimeoutSeconds)
);
const syncElapsed = nowMs() - syncStart;
const sampleFiles = pickSampleFiles(seedFiles.entries);
for (const sample of sampleFiles) {
const pulledPath = workDir.join(`pulled-${sample.relativePath.replaceAll("/", "_")}`);
await runCliOrFail(clientVault, "--settings", clientSettings, "pull", sample.relativePath, pulledPath);
await assertFilesEqual(
sample.absolutePath,
pulledPath,
`sample file mismatch after sync: ${sample.relativePath}`
);
}
const p2pConnectionStats = await readLatestP2PConnectionStats(p2pStatsPath);
const result = {
caseName: config.caseName,
mode: "p2p-cli-benchmark",
relay: config.relay,
turnServers: config.turnServers,
turnEnabled: config.turnServers.trim().length > 0,
simulationTier: config.simulationTier,
networkProfile: config.networkProfile,
networkModel: config.networkModel,
p2pCandidatePathVerified: p2pConnectionStats?.candidatePathCollected === true,
p2pCandidatePathVerification: p2pConnectionStats?.candidatePathCollected
? "selected ICE candidate pair collected from RTCPeerConnection.getStats"
: config.candidatePathVerification,
p2pCandidatePathNote: p2pConnectionStats?.candidatePathCollected
? "The selected ICE candidate pair was collected by the CLI benchmark. Interpret the path from the candidate types; do not infer TURN use from configuration alone."
: config.turnServers.trim().length > 0
? "TURN is configured, so the selected WebRTC path may be direct, server-reflexive, or relayed. The selected ICE candidate pair was not exported by this run."
: "TURN is disabled, so a TURN-relayed path is not expected. The selected ICE candidate pair was not exported by this run.",
p2pConnectionStats,
appId: config.appId,
roomId: config.roomId,
datasetSeed: config.datasetSeed,
datasetDirName: config.datasetDirName,
peerId: peer.id,
peerName: peer.name,
totalFiles: seedFiles.totalFiles,
totalBytes: seedFiles.totalBytes,
mdFileCount: seedFiles.mdCount,
binFileCount: seedFiles.binCount,
mirrorElapsedMs: Number(mirrorElapsed.toFixed(1)),
hostReadyElapsedMs: Number(hostReadyElapsed.toFixed(1)),
peerDiscoveryTimeoutSeconds: config.peersTimeoutSeconds,
peerDiscoveryCommandElapsedMs: Number(peerDiscoveryCommandElapsed.toFixed(1)),
peerDiscoveryNote:
"p2p-peers waits for the requested timeout before printing discovered peers, so this is command duration, not first-peer latency.",
syncElapsedMs: Number(syncElapsed.toFixed(1)),
throughputBytesPerSec: Number((seedFiles.totalBytes / (syncElapsed / 1000)).toFixed(2)),
throughputMiBPerSec: Number((seedFiles.totalBytes / (syncElapsed / 1000) / 1024 / 1024).toFixed(4)),
};
if (resultPath) {
await Deno.writeTextFile(resultPath, JSON.stringify(result, null, 2));
}
console.log(JSON.stringify(result, null, 2));
console.error(
`[Benchmark] mirrored ${seedFiles.totalFiles} files (${formatBytes(
seedFiles.totalBytes
)}) in ${formatMs(mirrorElapsed)}, ` +
`synced in ${formatMs(syncElapsed)} ` +
`(${result.throughputBytesPerSec} B/s, ${result.throughputMiBPerSec} MiB/s)`
);
} finally {
await host.stop();
}
console.log(JSON.stringify(result, null, 2));
console.error(
`[Benchmark] mirrored ${seedFiles.totalFiles} files (${formatBytes(seedFiles.totalBytes)}) in ${formatMs(mirrorElapsed)}, ` +
`synced in ${formatMs(syncElapsed)} ` +
`(${result.throughputBytesPerSec} B/s, ${result.throughputMiBPerSec} MiB/s)`
);
} finally {
await host.stop();
if (previousStatsPath === undefined) {
Deno.env.delete("LIVESYNC_P2P_STATS_JSONL");
} else {
Deno.env.set("LIVESYNC_P2P_STATS_JSONL", previousStatsPath);
}
await stopCoturnIfStarted(coturnStarted);
await stopLocalRelayIfStarted(relayStarted);
}
}
+3
View File
@@ -17,6 +17,9 @@
"test:p2p-upload-download": "deno test --env-file=.test.env -A --no-check test-p2p-upload-download-repro.ts",
"bench:p2p": "deno run --env-file=.test.env -A --no-check bench-p2p.ts",
"bench:couchdb": "deno run --env-file=.test.env -A --no-check bench-couchdb.ts",
"bench:cases": "deno run --env-file=.test.env -A --no-check bench-network-cases.ts",
"bench:latency-sweep": "deno run --env-file=.test.env -A --no-check bench-latency-sweep.ts",
"bench:p2p-split-node": "deno run --env-file=.test.env -A --no-check bench-p2p-split-node.ts",
"bench:item1": "bash ./bench-run-item1.sh",
"bench:item1:full": "BENCH_MD_FILE_COUNT=1500 BENCH_MD_MIN_SIZE_BYTES=1024 BENCH_MD_MAX_SIZE_BYTES=20480 BENCH_BIN_FILE_COUNT=500 BENCH_BIN_SIZE_BYTES=102400 BENCH_COUCHDB_RTT_MS=50 bash ./bench-run-item1.sh",
"test:e2e-couchdb": "deno test --env-file=.test.env -A --no-check test-e2e-two-vaults-couchdb.ts",
+1 -1
View File
@@ -9,7 +9,7 @@ function sleep(ms: number): Promise<void> {
}
async function connectWithTimeout(hostname: string, port: number, timeoutMs: number): Promise<void> {
let timer: number | undefined;
let timer: ReturnType<typeof setTimeout> | undefined;
try {
const connPromise = Deno.connect({ hostname, port });
const timeoutPromise = new Promise<never>((_, reject) => {
+8 -4
View File
@@ -76,8 +76,10 @@ export async function discoverPeer(
}
export async function maybeStartLocalRelay(relay: string): Promise<boolean> {
if (!isLocalP2pRelay(relay)) return false;
await startP2pRelay();
const shouldStart = isLocalP2pRelay(relay);
if (shouldStart) {
await startP2pRelay();
}
const endpoint = parseRelayEndpoint(relay);
await waitForPort(endpoint.hostname, endpoint.port, {
timeoutMs: Number(Deno.env.get("LIVESYNC_P2P_RELAY_READY_TIMEOUT_MS") ?? "15000"),
@@ -86,8 +88,10 @@ export async function maybeStartLocalRelay(relay: string): Promise<boolean> {
});
// Docker proxy accepts TCP connections instantly before the container's internal process is fully ready.
// Wait an additional few seconds to ensure strfry is actually accepting WebSockets.
await sleep(3000);
return true;
if (shouldStart) {
await sleep(3000);
}
return shouldStart;
}
export async function stopLocalRelayIfStarted(started: boolean): Promise<void> {
+1 -1
View File
@@ -1,7 +1,7 @@
{
"name": "livesync-webapp",
"private": true,
"version": "0.25.79-webapp",
"version": "0.25.80-webapp",
"type": "module",
"description": "Browser-based Self-hosted LiveSync using FileSystem API",
"scripts": {
+1 -1
View File
@@ -1,7 +1,7 @@
{
"name": "webpeer",
"private": true,
"version": "0.25.79-webpeer",
"version": "0.25.80-webpeer",
"type": "module",
"scripts": {
"dev": "vite",
@@ -51,6 +51,7 @@ import { EVENT_SETTING_SAVED, eventHub } from "@/common/events.ts";
import { Semaphore } from "octagonal-wheels/concurrency/semaphore";
import type { LiveSyncCore } from "@/main.ts";
import { tryGetFilePath } from "@lib/common/utils.doc.ts";
import { configureHiddenFileSyncMode } from "./configureHiddenFileSyncMode.ts";
type SyncDirection = "push" | "pull" | "safe" | "pullForce" | "pushForce";
declare global {
@@ -1832,43 +1833,35 @@ ${messageFetch}${messageOverwrite}${messageMerge}
}
async configureHiddenFileSync(mode: keyof OPTIONAL_SYNC_FEATURES) {
if (
mode != "FETCH" &&
mode != "OVERWRITE" &&
mode != "MERGE" &&
mode != "DISABLE" &&
mode != "DISABLE_HIDDEN"
) {
return;
}
if (mode == "DISABLE" || mode == "DISABLE_HIDDEN") {
// await this.core.$allSuspendExtraSync();
await this.core.services.setting.applyPartial(
{
syncInternalFiles: false,
},
true
);
// this.core.settings.syncInternalFiles = false;
// await this.core.saveSettings();
return;
}
this._log("Gathering files for enabling Hidden File Sync", LOG_LEVEL_NOTICE);
if (mode == "FETCH") {
await this.initialiseInternalFileSync("pullForce", true);
} else if (mode == "OVERWRITE") {
await this.initialiseInternalFileSync("pushForce", true);
} else if (mode == "MERGE") {
await this.initialiseInternalFileSync("safe", true);
}
await this.core.services.setting.applyPartial(
{
useAdvancedMode: true,
syncInternalFiles: true,
const result = await configureHiddenFileSyncMode(mode, {
disable: async () => {
// await this.core.$allSuspendExtraSync();
await this.core.services.setting.applyPartial(
{
syncInternalFiles: false,
},
true
);
// this.core.settings.syncInternalFiles = false;
// await this.core.saveSettings();
},
true
);
enable: async () => {
this._log("Gathering files for enabling Hidden File Sync", LOG_LEVEL_NOTICE);
await this.core.services.setting.applyPartial(
{
useAdvancedMode: true,
syncInternalFiles: true,
},
true
);
},
initialise: async (direction) => {
await this.initialiseInternalFileSync(direction, true);
},
});
if (result == "ignored" || result == "disabled") {
return;
}
// this.plugin.settings.useAdvancedMode = true;
// this.plugin.settings.syncInternalFiles = true;
@@ -0,0 +1,45 @@
type HiddenFileSyncMode = "FETCH" | "OVERWRITE" | "MERGE" | "DISABLE" | "DISABLE_HIDDEN";
type HiddenFileSyncDirection = "pullForce" | "pushForce" | "safe";
type ConfigureHiddenFileSyncHandlers = {
disable: () => Promise<void>;
enable: () => Promise<void>;
initialise: (direction: HiddenFileSyncDirection) => Promise<void>;
};
export type ConfigureHiddenFileSyncResult = "ignored" | "disabled" | "enabled";
function getInitialiseDirection(mode: keyof OPTIONAL_SYNC_FEATURES): HiddenFileSyncDirection | false {
if (mode == "FETCH") return "pullForce";
if (mode == "OVERWRITE") return "pushForce";
if (mode == "MERGE") return "safe";
return false;
}
function isDisableMode(mode: keyof OPTIONAL_SYNC_FEATURES): boolean {
return mode == "DISABLE" || mode == "DISABLE_HIDDEN";
}
function isHiddenFileSyncMode(mode: keyof OPTIONAL_SYNC_FEATURES): mode is HiddenFileSyncMode {
return mode == "FETCH" || mode == "OVERWRITE" || mode == "MERGE" || isDisableMode(mode);
}
export async function configureHiddenFileSyncMode(
mode: keyof OPTIONAL_SYNC_FEATURES,
handlers: ConfigureHiddenFileSyncHandlers
): Promise<ConfigureHiddenFileSyncResult> {
if (!isHiddenFileSyncMode(mode)) {
return "ignored";
}
if (isDisableMode(mode)) {
await handlers.disable();
return "disabled";
}
const direction = getInitialiseDirection(mode);
if (direction === false) {
return "ignored";
}
await handlers.enable();
await handlers.initialise(direction);
return "enabled";
}
@@ -0,0 +1,47 @@
import { describe, expect, it, vi } from "vitest";
import { configureHiddenFileSyncMode } from "./configureHiddenFileSyncMode.ts";
describe("configureHiddenFileSyncMode", () => {
it.each([
["FETCH", "pullForce"],
["OVERWRITE", "pushForce"],
["MERGE", "safe"],
] as const)("enables hidden file sync before initialising %s", async (mode, direction) => {
const calls: string[] = [];
const result = await configureHiddenFileSyncMode(mode, {
disable: vi.fn(async () => {
calls.push("disable");
}),
enable: vi.fn(async () => {
calls.push("enable");
}),
initialise: vi.fn(async (actualDirection) => {
calls.push(`init:${actualDirection}`);
}),
});
expect(result).toBe("enabled");
expect(calls).toEqual(["enable", `init:${direction}`]);
});
it.each(["DISABLE", "DISABLE_HIDDEN"] as const)("disables hidden file sync immediately for %s", async (mode) => {
const calls: string[] = [];
const result = await configureHiddenFileSyncMode(mode, {
disable: vi.fn(async () => {
calls.push("disable");
}),
enable: vi.fn(async () => {
calls.push("enable");
}),
initialise: vi.fn(async (direction) => {
calls.push(`init:${direction}`);
}),
});
expect(result).toBe("disabled");
expect(calls).toEqual(["disable"]);
});
});
+1 -1
Submodule src/lib updated: 2ee5d2055a...a0efb7274e
+3 -3
View File
@@ -1,5 +1,5 @@
import { fireAndForget } from "octagonal-wheels/promises";
import { REMOTE_P2P, isJournalRemoteType, type RemoteDBSettings } from "@lib/common/types";
import { REMOTE_MINIO, REMOTE_P2P, type RemoteDBSettings } from "@lib/common/types";
import { LiveSyncCouchDBReplicator } from "@lib/replication/couchdb/LiveSyncReplicator";
import type { LiveSyncAbstractReplicator } from "@lib/replication/LiveSyncAbstractReplicator";
import { AbstractModule } from "@/modules/AbstractModule";
@@ -9,7 +9,7 @@ export class ModuleReplicatorCouchDB extends AbstractModule {
_anyNewReplicator(settingOverride: Partial<RemoteDBSettings> = {}): Promise<LiveSyncAbstractReplicator | false> {
const settings = { ...this.settings, ...settingOverride };
// If new remote types were added, add them here. Do not use `REMOTE_COUCHDB` directly for the safety valve.
if (isJournalRemoteType(settings.remoteType) || settings.remoteType == REMOTE_P2P) {
if (settings.remoteType == REMOTE_MINIO || settings.remoteType == REMOTE_P2P) {
return Promise.resolve(false);
}
return Promise.resolve(new LiveSyncCouchDBReplicator(this.core));
@@ -17,7 +17,7 @@ export class ModuleReplicatorCouchDB extends AbstractModule {
_everyAfterResumeProcess(): Promise<boolean> {
if (this.services.appLifecycle.isSuspended()) return Promise.resolve(true);
if (!this.services.appLifecycle.isReady()) return Promise.resolve(true);
if (!isJournalRemoteType(this.settings.remoteType) && this.settings.remoteType != REMOTE_P2P) {
if (this.settings.remoteType != REMOTE_MINIO && this.settings.remoteType != REMOTE_P2P) {
const LiveSyncEnabled = this.settings.liveSync;
const continuous = LiveSyncEnabled;
const eventualOnStart = !LiveSyncEnabled && this.settings.syncOnStart;
+2 -2
View File
@@ -1,4 +1,4 @@
import { isJournalRemoteType, type RemoteDBSettings } from "@lib/common/types";
import { REMOTE_MINIO, type RemoteDBSettings } from "@lib/common/types";
import { LiveSyncJournalReplicator } from "@lib/replication/journal/LiveSyncJournalReplicator";
import type { LiveSyncAbstractReplicator } from "@lib/replication/LiveSyncAbstractReplicator";
import type { LiveSyncCore } from "@/main";
@@ -7,7 +7,7 @@ import { AbstractModule } from "@/modules/AbstractModule";
export class ModuleReplicatorMinIO extends AbstractModule {
_anyNewReplicator(settingOverride: Partial<RemoteDBSettings> = {}): Promise<LiveSyncAbstractReplicator | false> {
const settings = { ...this.settings, ...settingOverride };
if (isJournalRemoteType(settings.remoteType)) {
if (settings.remoteType == REMOTE_MINIO) {
return Promise.resolve(new LiveSyncJournalReplicator(this.core));
}
return Promise.resolve(false);
@@ -12,7 +12,6 @@ import {
LEVEL_ADVANCED,
LEVEL_EDGE_CASE,
REMOTE_P2P,
isJournalRemoteType,
} from "@lib/common/types.ts";
import { delay, isObjectDifferent, sizeToHumanReadable } from "@lib/common/utils.ts";
import { versionNumberString2Number } from "@lib/string_and_binary/convert.ts";
@@ -65,7 +64,7 @@ import { panePatches } from "./PanePatches.ts";
import { paneMaintenance } from "./PaneMaintenance.ts";
import { compatGlobal } from "@lib/common/coreEnvFunctions.ts";
import { JournalSyncCore } from "@lib/replication/journal/JournalSyncCore.js";
import { createJournalStorageAdapter } from "@lib/replication/journal/objectstore/JournalStorageAdapterFactory.js";
import { MinioStorageAdapter } from "@lib/replication/journal/objectstore/MinioStorageAdapter.js";
// For creating a document
// const toc = new Set<string>();
@@ -531,10 +530,6 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
({
visibility: this.isConfiguredAs("remoteType", REMOTE_MINIO),
}) as OnUpdateResult;
onlyOnJournalSync = () =>
({
visibility: isJournalRemoteType(this.editingSettings.remoteType),
}) as OnUpdateResult;
onlyOnOnlyP2P = () =>
({
visibility: this.isConfiguredAs("remoteType", REMOTE_P2P),
@@ -542,14 +537,11 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
onlyOnCouchDBOrMinIO = () =>
({
visibility:
this.isConfiguredAs("remoteType", REMOTE_COUCHDB) ||
isJournalRemoteType(this.editingSettings.remoteType),
this.isConfiguredAs("remoteType", REMOTE_COUCHDB) || this.isConfiguredAs("remoteType", REMOTE_MINIO),
}) as OnUpdateResult;
// E2EE Function
checkWorkingPassphrase = async (): Promise<boolean> => {
if (isJournalRemoteType(this.editingSettings.remoteType)) {
return true;
}
if (this.editingSettings.remoteType == REMOTE_MINIO) return true;
const settingForCheck: RemoteDBSettings = {
...this.editingSettings,
@@ -864,7 +856,7 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
this.core.settings,
this.core.simpleStore,
this.core,
createJournalStorageAdapter(this.core.settings, this.core)
new MinioStorageAdapter(this.core.settings, this.core)
);
}
async resetRemoteBucket() {
@@ -163,7 +163,7 @@ export function paneMaintenance(
Logger(`Journal received history has been cleared.`, LOG_LEVEL_NOTICE);
})
)
.addOnUpdate(this.onlyOnJournalSync);
.addOnUpdate(this.onlyOnMinIO);
new Setting(paneEl)
.setName("Reset journal sent history")
@@ -185,7 +185,7 @@ export function paneMaintenance(
Logger(`Journal sent history has been cleared.`, LOG_LEVEL_NOTICE);
})
)
.addOnUpdate(this.onlyOnJournalSync);
.addOnUpdate(this.onlyOnMinIO);
});
void addPanel(paneEl, "Garbage Collection V3 (Beta)", (e) => e, this.onlyOnP2POrCouchDB).then((paneEl) => {
new Setting(paneEl)
@@ -336,7 +336,7 @@ export function paneMaintenance(
Logger(`Journal exchange history has been cleared.`, LOG_LEVEL_NOTICE);
})
)
.addOnUpdate(this.onlyOnJournalSync);
.addOnUpdate(this.onlyOnMinIO);
new Setting(paneEl)
.setName("Purge all journal counter")
@@ -351,7 +351,7 @@ export function paneMaintenance(
Logger(`Journal download/upload cache has been cleared.`, LOG_LEVEL_NOTICE);
})
)
.addOnUpdate(this.onlyOnJournalSync);
.addOnUpdate(this.onlyOnMinIO);
new Setting(paneEl)
.setName("Fresh Start Wipe")
@@ -374,7 +374,7 @@ export function paneMaintenance(
Logger(`Deleted all data on remote server`, LOG_LEVEL_NOTICE);
})
)
.addOnUpdate(this.onlyOnJournalSync);
.addOnUpdate(this.onlyOnMinIO);
});
void addPanel(paneEl, "Reset").then((paneEl) => {
@@ -2,7 +2,6 @@ import {
REMOTE_COUCHDB,
REMOTE_MINIO,
REMOTE_P2P,
REMOTE_WEBDAV,
DEFAULT_SETTINGS,
LOG_LEVEL_NOTICE,
type ObsidianLiveSyncSettings,
@@ -34,9 +33,7 @@ import SetupRemote from "@/modules/features/SetupWizard/dialogs/SetupRemote.svel
import SetupRemoteCouchDB from "@/modules/features/SetupWizard/dialogs/SetupRemoteCouchDB.svelte";
import SetupRemoteBucket from "@/modules/features/SetupWizard/dialogs/SetupRemoteBucket.svelte";
import SetupRemoteP2P from "@/modules/features/SetupWizard/dialogs/SetupRemoteP2P.svelte";
import SetupRemoteWebDAV from "@/modules/features/SetupWizard/dialogs/SetupRemoteWebDAV.svelte";
import { syncActivatedRemoteSettings } from "./remoteConfigBuffer.ts";
import { getRemoteConfigurationDescription } from "./remoteConfigDisplay.ts";
function getSettingsFromEditingSettings(editingSettings: AllSettings): ObsidianLiveSyncSettings {
const workObj = { ...editingSettings } as ObsidianLiveSyncSettings;
@@ -72,9 +69,6 @@ function serializeRemoteConfiguration(settings: ObsidianLiveSyncSettings): strin
if (settings.remoteType === REMOTE_P2P) {
return ConnectionStringParser.serialize({ type: "p2p", settings });
}
if (settings.remoteType === REMOTE_WEBDAV) {
return ConnectionStringParser.serialize({ type: "webdav", settings });
}
return ConnectionStringParser.serialize({ type: "couchdb", settings });
}
@@ -98,14 +92,6 @@ function suggestRemoteConfigurationName(parsed: RemoteConfigurationResult): stri
if (parsed.type === "s3") {
return `S3 ${parsed.settings.bucket || parsed.settings.endpoint}`;
}
if (parsed.type === "webdav") {
try {
const url = new URL(parsed.settings.webDAVactiveConnectionURI.replace(/^sls\+webdav:/, "https:"));
return `WebDAV ${url.host}`;
} catch {
return "Imported WebDAV";
}
}
return `P2P ${parsed.settings.P2P_roomID || "Remote"}`;
}
@@ -212,7 +198,7 @@ export function paneRemoteConfig(
};
const runRemoteSetup = async (
baseSettings: ObsidianLiveSyncSettings,
remoteType?: typeof REMOTE_COUCHDB | typeof REMOTE_MINIO | typeof REMOTE_P2P | typeof REMOTE_WEBDAV
remoteType?: typeof REMOTE_COUCHDB | typeof REMOTE_MINIO | typeof REMOTE_P2P
): Promise<ObsidianLiveSyncSettings | false> => {
const setupManager = this.core.getModule(SetupManager);
const dialogManager = setupManager.dialogManager;
@@ -224,13 +210,7 @@ export function paneRemoteConfig(
return false;
}
targetRemoteType =
method === "bucket"
? REMOTE_MINIO
: method === "p2p"
? REMOTE_P2P
: method === "webdav"
? REMOTE_WEBDAV
: REMOTE_COUCHDB;
method === "bucket" ? REMOTE_MINIO : method === "p2p" ? REMOTE_P2P : REMOTE_COUCHDB;
}
if (targetRemoteType === REMOTE_MINIO) {
@@ -249,14 +229,6 @@ export function paneRemoteConfig(
return { ...baseSettings, ...p2pConf, remoteType: REMOTE_P2P };
}
if (targetRemoteType === REMOTE_WEBDAV) {
const webDAVConf = await dialogManager.openWithExplicitCancel(SetupRemoteWebDAV, baseSettings);
if (webDAVConf === "cancelled" || typeof webDAVConf !== "object") {
return false;
}
return { ...baseSettings, ...webDAVConf, remoteType: REMOTE_WEBDAV };
}
const couchConf = await dialogManager.openWithExplicitCancel(SetupRemoteCouchDB, baseSettings);
if (couchConf === "cancelled" || typeof couchConf !== "object") {
return false;
@@ -359,7 +331,7 @@ export function paneRemoteConfig(
for (const config of Object.values(configs)) {
const row = new Setting(listContainer)
.setName(config.name)
.setDesc(getRemoteConfigurationDescription(config.uri));
.setDesc(config.uri.split("@").pop() || ""); // Show host part for privacy
if (config.id === this.editingSettings.activeConfigurationId) {
row.nameEl.addClass("sls-active-remote-name");
@@ -384,8 +356,6 @@ export function paneRemoteConfig(
workSettings.remoteType = REMOTE_COUCHDB;
} else if (parsed.type === "s3") {
workSettings.remoteType = REMOTE_MINIO;
} else if (parsed.type === "webdav") {
workSettings.remoteType = REMOTE_WEBDAV;
} else {
workSettings.remoteType = REMOTE_P2P;
}
@@ -496,8 +466,6 @@ export function paneRemoteConfig(
workSettings.remoteType = REMOTE_COUCHDB;
} else if (parsed.type === "s3") {
workSettings.remoteType = REMOTE_MINIO;
} else if (parsed.type === "webdav") {
workSettings.remoteType = REMOTE_WEBDAV;
} else {
workSettings.remoteType = REMOTE_P2P;
}
@@ -1,12 +0,0 @@
import { describe, expect, it } from "vitest";
import { getRemoteConfigurationDescription } from "./remoteConfigDisplay";
describe("getRemoteConfigurationDescription", () => {
it("should mask credentials and query parameters in WebDAV connection strings", () => {
expect(
getRemoteConfigurationDescription(
"sls+webdav://user:pass@example.com/dav?prefix=vault%2F&headers=Authorization%3A%20Bearer%20token"
)
).toBe("https://example.com/dav");
});
});
@@ -1,9 +1,4 @@
import {
pickBucketSyncSettings,
pickCouchDBSyncSettings,
pickP2PSyncSettings,
pickWebDAVSyncSettings,
} from "@lib/common/utils.ts";
import { pickBucketSyncSettings, pickCouchDBSyncSettings, pickP2PSyncSettings } from "@lib/common/utils.ts";
import type { ObsidianLiveSyncSettings } from "@lib/common/types.ts";
// Keep the setting dialogue buffer aligned with the current core settings before persisting other dirty keys.
@@ -18,6 +13,5 @@ export function syncActivatedRemoteSettings(
...pickBucketSyncSettings(source),
...pickCouchDBSyncSettings(source),
...pickP2PSyncSettings(source),
...pickWebDAVSyncSettings(source),
});
}
@@ -1,5 +1,5 @@
import { describe, expect, it } from "vitest";
import { DEFAULT_SETTINGS, REMOTE_COUCHDB, REMOTE_MINIO, REMOTE_WEBDAV } from "@lib/common/types";
import { DEFAULT_SETTINGS, REMOTE_COUCHDB, REMOTE_MINIO } from "@lib/common/types";
import { syncActivatedRemoteSettings } from "./remoteConfigBuffer";
describe("syncActivatedRemoteSettings", () => {
@@ -80,25 +80,4 @@ describe("syncActivatedRemoteSettings", () => {
expect(target.couchDB_PASSWORD).toBe("current-pass");
expect(target.couchDB_DBNAME).toBe("current-db");
});
it("should copy active WebDAV connection URI into the editing buffer", () => {
const target = {
...DEFAULT_SETTINGS,
remoteType: REMOTE_COUCHDB,
activeConfigurationId: "old-remote",
webDAVactiveConnectionURI: "",
};
const source = {
...DEFAULT_SETTINGS,
remoteType: REMOTE_WEBDAV,
activeConfigurationId: "remote-webdav",
webDAVactiveConnectionURI: "sls+webdav://user:pass@example.com/dav?prefix=vault%2F",
};
syncActivatedRemoteSettings(target, source);
expect(target.remoteType).toBe(REMOTE_WEBDAV);
expect(target.activeConfigurationId).toBe("remote-webdav");
expect(target.webDAVactiveConnectionURI).toBe("sls+webdav://user:pass@example.com/dav?prefix=vault%2F");
});
});
@@ -1,29 +0,0 @@
import { ConnectionStringParser } from "@lib/common/ConnectionString.ts";
export function getRemoteConfigurationDescription(uri: string): string {
try {
const parsed = ConnectionStringParser.parse(uri);
if (parsed.type === "couchdb") {
const url = new URL(parsed.settings.couchDB_URI);
url.username = "";
url.password = "";
url.search = "";
url.hash = "";
return url.toString();
}
if (parsed.type === "s3") {
return `${parsed.settings.endpoint.replace(/\/+$/g, "")}/${parsed.settings.bucket}`;
}
if (parsed.type === "webdav") {
const url = new URL(parsed.settings.webDAVactiveConnectionURI.replace(/^sls\+webdav:/, "https:"));
url.username = "";
url.password = "";
url.search = "";
url.hash = "";
return url.toString();
}
return `P2P ${parsed.settings.P2P_roomID || "Remote"}`;
} catch {
return uri.split("@").pop()?.split("?")[0] || "";
}
}
-33
View File
@@ -4,14 +4,12 @@ import {
type EncryptionSettings,
type ObsidianLiveSyncSettings,
type P2PSyncSetting,
type WebDAVSyncSetting,
DEFAULT_SETTINGS,
LOG_LEVEL_NOTICE,
LOG_LEVEL_VERBOSE,
REMOTE_COUCHDB,
REMOTE_MINIO,
REMOTE_P2P,
REMOTE_WEBDAV,
} from "@lib/common/types.ts";
import { isObjectDifferent } from "@lib/common/utils.ts";
import Intro from "./SetupWizard/dialogs/Intro.svelte";
@@ -26,7 +24,6 @@ import SetupRemote from "./SetupWizard/dialogs/SetupRemote.svelte";
import SetupRemoteCouchDB from "./SetupWizard/dialogs/SetupRemoteCouchDB.svelte";
import SetupRemoteBucket from "./SetupWizard/dialogs/SetupRemoteBucket.svelte";
import SetupRemoteP2P from "./SetupWizard/dialogs/SetupRemoteP2P.svelte";
import SetupRemoteWebDAV from "./SetupWizard/dialogs/SetupRemoteWebDAV.svelte";
import SetupRemoteE2EE from "./SetupWizard/dialogs/SetupRemoteE2EE.svelte";
import { decodeSettingsFromQRCodeData } from "@lib/API/processSetting.ts";
import { AbstractModule } from "@/modules/AbstractModule.ts";
@@ -41,7 +38,6 @@ import type {
SetupRemoteE2EEResultType,
SetupRemoteP2PResultType,
SetupRemoteResultType,
SetupRemoteWebDAVResultType,
UseSetupURIResultType,
} from "./SetupWizard/dialogs/setupDialogTypes.ts";
@@ -206,33 +202,6 @@ export class SetupManager extends AbstractModule {
return await this.onConfirmApplySettingsFromWizard(newSetting, userMode, activate);
}
/**
* Handles manual setup for WebDAV-compatible storage.
* @param userMode
* @param currentSetting
* @param activate Whether to activate WebDAV as remote type
* @returns Promise that resolves to true if setup completed successfully, false otherwise
*/
async onWebDAVManualSetup(
userMode: UserMode,
currentSetting: ObsidianLiveSyncSettings,
activate = true
): Promise<boolean> {
const webDAVConf = await this.dialogManager.openWithExplicitCancel<
SetupRemoteWebDAVResultType,
WebDAVSyncSetting
>(SetupRemoteWebDAV, currentSetting);
if (webDAVConf === "cancelled") {
this._log("Manual configuration cancelled.", LOG_LEVEL_NOTICE);
return await this.onOnboard(userMode);
}
const newSetting = { ...currentSetting, ...webDAVConf } as ObsidianLiveSyncSettings;
if (activate) {
newSetting.remoteType = REMOTE_WEBDAV;
}
return await this.onConfirmApplySettingsFromWizard(newSetting, userMode, activate);
}
/**
* Handles manual setup for P2P
* @param userMode
@@ -334,8 +303,6 @@ export class SetupManager extends AbstractModule {
return await this.onBucketManualSetup(userMode, currentSetting, true);
} else if (method === "p2p") {
return await this.onP2PManualSetup(userMode, currentSetting, true);
} else if (method === "webdav") {
return await this.onWebDAVManualSetup(userMode, currentSetting, true);
} else if (method === "cancelled") {
this._log("Manual configuration cancelled.", LOG_LEVEL_NOTICE);
if (userMode !== UserMode.Unknown) {
@@ -10,7 +10,6 @@
TYPE_COUCHDB,
TYPE_BUCKET,
TYPE_P2P,
TYPE_WEBDAV,
TYPE_CANCELLED,
type SetupRemoteResultType,
} from "./setupDialogTypes";
@@ -27,19 +26,12 @@
return "Continue to S3/MinIO/R2 setup";
} else if (userType === TYPE_P2P) {
return "Continue to Peer-to-Peer only setup";
} else if (userType === TYPE_WEBDAV) {
return "Continue to WebDAV setup";
} else {
return "Please select an option to proceed";
}
});
const canProceed = $derived.by(() => {
return (
userType === TYPE_COUCHDB ||
userType === TYPE_BUCKET ||
userType === TYPE_P2P ||
userType === TYPE_WEBDAV
);
return userType === TYPE_COUCHDB || userType === TYPE_BUCKET || userType === TYPE_P2P;
});
</script>
@@ -54,10 +46,6 @@
<Option selectedValue={TYPE_BUCKET} title="S3/MinIO/R2 Object Storage" bind:value={userType}>
Synchronisation utilising journal files. You must have set up an S3/MinIO/R2 compatible object storage.
</Option>
<Option selectedValue={TYPE_WEBDAV} title="WebDAV (Experimental)" bind:value={userType}>
Experimental synchronisation utilising journal files stored in a WebDAV collection. This is a proof of
concept for pluggable Journal storage backends.
</Option>
<Option selectedValue={TYPE_P2P} title="Peer-to-Peer only" bind:value={userType}>
This feature enables direct synchronisation between devices. No server is required, but both devices must be
online at the same time for synchronisation to occur, and some features may be limited. Internet connection
@@ -1,160 +0,0 @@
<script lang="ts">
import DialogHeader from "@lib/UI/components/DialogHeader.svelte";
import Guidance from "@lib/UI/components/Guidance.svelte";
import Decision from "@lib/UI/components/Decision.svelte";
import UserDecisions from "@lib/UI/components/UserDecisions.svelte";
import InfoNote from "@lib/UI/components/InfoNote.svelte";
import InputRow from "@lib/UI/components/InputRow.svelte";
import {
DEFAULT_SETTINGS,
PREFERRED_JOURNAL_SYNC,
RemoteTypes,
type ObsidianLiveSyncSettings,
type WebDAVSyncSetting,
} from "@lib/common/types";
import { copyTo, pickWebDAVSyncSettings } from "@lib/common/utils";
import { getDialogContext, type GuestDialogProps } from "@lib/UI/svelteDialog";
import { onMount } from "svelte";
import { TYPE_CANCELLED, type SetupRemoteWebDAVResultType } from "./setupDialogTypes";
const default_setting = pickWebDAVSyncSettings(DEFAULT_SETTINGS);
let syncSetting = $state<WebDAVSyncSetting>({ ...default_setting });
type Props = GuestDialogProps<SetupRemoteWebDAVResultType, WebDAVSyncSetting>;
const { setResult, getInitialData }: Props = $props();
onMount(() => {
if (getInitialData) {
const initialData = getInitialData();
if (initialData) {
copyTo(initialData, syncSetting);
}
}
});
let error = $state("");
let processing = $state(false);
const context = getDialogContext();
function parseURI(): URL | false {
const match = syncSetting.webDAVactiveConnectionURI.trim().match(/^sls\+webdav:(.*)$/);
if (!match) {
return false;
}
try {
return new URL(`https:${match[1]}`);
} catch {
return false;
}
}
const parsedURI = $derived.by(() => parseURI());
const canProceed = $derived.by(() => parsedURI !== false);
const isInsecure = $derived.by(() => parsedURI !== false && parsedURI.searchParams.get("insecure") === "true");
const usesInternalAPI = $derived.by(
() => parsedURI !== false && parsedURI.searchParams.get("useProxy") === "true"
);
function generateSetting() {
const trialSettings: WebDAVSyncSetting = {
...syncSetting,
webDAVactiveConnectionURI: syncSetting.webDAVactiveConnectionURI.trim(),
};
const trialRemoteSetting: ObsidianLiveSyncSettings = {
...DEFAULT_SETTINGS,
...PREFERRED_JOURNAL_SYNC,
remoteType: RemoteTypes.REMOTE_WEBDAV,
...trialSettings,
};
return trialRemoteSetting;
}
async function checkConnection() {
try {
processing = true;
const trialRemoteSetting = generateSetting();
const replicator = await context.services.replicator.getNewReplicator(trialRemoteSetting);
if (!replicator) {
return "Failed to create replicator instance.";
}
try {
const result = await replicator.tryConnectRemote(trialRemoteSetting, false);
if (result) {
return "";
}
return "Failed to connect to the server. Please check your settings.";
} catch (e) {
return `Failed to connect to the server: ${e}`;
}
} finally {
processing = false;
}
}
async function checkAndCommit() {
error = "";
try {
error = (await checkConnection()) || "";
if (!error) {
setResult(pickWebDAVSyncSettings(generateSetting()));
return;
}
} catch (e) {
error = `Error during connection test: ${e}`;
}
}
function commit() {
setResult(pickWebDAVSyncSettings(generateSetting()));
}
function cancel() {
setResult(TYPE_CANCELLED);
}
</script>
<DialogHeader title="WebDAV Configuration" />
<Guidance>
Please enter the WebDAV connection URI. This experimental setup stores Journal synchronisation files in a WebDAV
collection.
</Guidance>
<InputRow label="Connection URI">
<input
type="text"
name="webdav-connection-uri"
placeholder="sls+webdav://user:password@example.com/dav?prefix=vault%2F"
autocorrect="off"
autocapitalize="off"
spellcheck="false"
required
bind:value={syncSetting.webDAVactiveConnectionURI}
/>
</InputRow>
<InfoNote>
Use `sls+webdav://user:password@host/path`. Optional query parameters include `prefix`, `useProxy=true`, and
`insecure=true` for local HTTP testing.
</InfoNote>
<InfoNote warning visible={isInsecure}>Secure HTTPS connections are required on Obsidian Mobile.</InfoNote>
<InfoNote visible={usesInternalAPI}>
This connection uses Obsidian's internal API. It can help when direct WebDAV requests are blocked by CORS.
</InfoNote>
<InfoNote error visible={syncSetting.webDAVactiveConnectionURI.trim() !== "" && !canProceed}>
The connection URI must start with `sls+webdav://`.
</InfoNote>
<InfoNote error visible={error !== ""}>
{error}
</InfoNote>
{#if processing}
Checking connection... Please wait.
{:else}
<UserDecisions>
<Decision title="Test Settings and Continue" important disabled={!canProceed} commit={() => checkAndCommit()} />
<Decision title="Continue anyway" disabled={!canProceed} commit={() => commit()} />
<Decision title="Cancel" commit={() => cancel()} />
</UserDecisions>
{/if}
@@ -4,7 +4,6 @@ import type {
EncryptionSettings,
ObsidianLiveSyncSettings,
P2PConnectionInfo,
WebDAVSyncSetting,
} from "@lib/common/models/setting.type";
export const TYPE_IDENTICAL = "identical";
@@ -41,7 +40,6 @@ export const TYPE_CLOSE = "close";
export const TYPE_COUCHDB = "couchdb";
export const TYPE_BUCKET = "bucket";
export const TYPE_P2P = "p2p";
export const TYPE_WEBDAV = "webdav";
export type ResultTypeVault =
| typeof TYPE_IDENTICAL
@@ -95,12 +93,7 @@ export type SelectMethodExistingResultType =
| typeof TYPE_CONFIGURE_MANUALLY
| typeof TYPE_CANCELLED;
export type SetupRemoteResultType =
| typeof TYPE_COUCHDB
| typeof TYPE_BUCKET
| typeof TYPE_P2P
| typeof TYPE_WEBDAV
| typeof TYPE_CANCELLED;
export type SetupRemoteResultType = typeof TYPE_COUCHDB | typeof TYPE_BUCKET | typeof TYPE_P2P | typeof TYPE_CANCELLED;
export type UseSetupURIResultType = typeof TYPE_CANCELLED | ObsidianLiveSyncSettings;
@@ -112,6 +105,4 @@ export type SetupRemoteCouchDBResultType = typeof TYPE_CANCELLED | CouchDBConnec
export type SetupRemoteP2PResultType = typeof TYPE_CANCELLED | P2PConnectionInfo;
export type SetupRemoteWebDAVResultType = typeof TYPE_CANCELLED | WebDAVSyncSetting;
export type ScanQRCodeResultType = typeof TYPE_CLOSE;
+7 -5
View File
@@ -147,12 +147,14 @@ export class ObsidianAPIService extends InjectableAPIService<ObsidianServiceCont
: req instanceof Request && typeof req.method === "string"
? req.method
: "GET";
if (opts?.body) {
body = typeof opts.body === "string" ? opts.body : await new Response(opts.body).arrayBuffer();
} else if (typeof req !== "string" && req.body) {
body = await new Response(req.body).arrayBuffer();
if (typeof req !== "string") {
if (opts?.body) {
body = typeof opts.body === "string" ? opts.body : await new Response(opts.body).arrayBuffer();
} else if (req.body) {
body = await new Response(req.body).arrayBuffer();
}
} else {
body = undefined;
body = opts?.body as string;
}
const reqHeaders = new Headers(req instanceof Request ? req.headers : {});
@@ -1,63 +0,0 @@
import { describe, expect, it, vi } from "vitest";
const requestUrlMock = vi.hoisted(() => vi.fn());
vi.mock("@/deps", () => {
return {
requestUrl: requestUrlMock,
Notice: vi.fn(),
Platform: {},
};
});
vi.mock("@/deps.ts", () => {
return {
requestUrl: requestUrlMock,
Notice: vi.fn(),
Platform: {},
};
});
vi.mock("./ObsidianConfirm", () => ({
ObsidianConfirm: class {},
}));
import { ObsidianAPIService } from "./ObsidianAPIService.ts";
import type { ObsidianServiceContext } from "@lib/services/implements/obsidian/ObsidianServiceContext.ts";
describe("ObsidianAPIService.nativeFetch", () => {
it("should pass binary string-request bodies to requestUrl as ArrayBuffer", async () => {
requestUrlMock.mockResolvedValueOnce({
arrayBuffer: new TextEncoder().encode("ok").buffer,
headers: {},
status: 207,
});
const service = new ObsidianAPIService({} as ObsidianServiceContext);
const body = new TextEncoder().encode("payload");
const response = await service.nativeFetch("https://webdav.example.com/file", {
method: "PUT",
body: body as unknown as BodyInit,
headers: {
Depth: "1",
Authorization: "Basic dXNlcjpwYXNz",
},
});
expect(response.status).toBe(207);
expect(requestUrlMock).toHaveBeenCalledWith(
expect.objectContaining({
url: "https://webdav.example.com/file",
method: "PUT",
body: expect.any(ArrayBuffer),
headers: expect.objectContaining({
Depth: "1",
Authorization: "Basic dXNlcjpwYXNz",
}),
throw: false,
})
);
const requestBody = requestUrlMock.mock.calls[0][0].body as ArrayBuffer;
expect(new TextDecoder().decode(requestBody)).toBe("payload");
});
});
+1
View File
@@ -0,0 +1 @@
bench-results/
+8
View File
@@ -0,0 +1,8 @@
FROM alpine:3.22
RUN apk add --no-cache iproute2
COPY test/bench-network/netem-smoke.sh /usr/local/bin/livesync-netem-smoke
RUN chmod +x /usr/local/bin/livesync-netem-smoke
CMD ["livesync-netem-smoke"]
+35
View File
@@ -0,0 +1,35 @@
# syntax=docker/dockerfile:1
FROM node:24-slim
RUN apt-get update \
&& apt-get install -y --no-install-recommends ca-certificates curl unzip python3 make g++ iproute2 \
&& rm -rf /var/lib/apt/lists/*
ENV DENO_INSTALL=/usr/local
RUN curl -fsSL https://deno.land/install.sh | sh
WORKDIR /workspace
COPY package.json package-lock.json ./
COPY src/apps/cli/package.json ./src/apps/cli/package.json
COPY src/apps/webapp/package.json ./src/apps/webapp/package.json
COPY src/apps/webpeer/package.json ./src/apps/webpeer/package.json
RUN npm ci
COPY . .
RUN npm run build -w self-hosted-livesync-cli
WORKDIR /workspace/src/apps/cli/testdeno
RUN deno cache --lock=deno.lock \
bench-network-cases.ts \
bench-latency-sweep.ts \
bench-p2p-split-node.ts \
bench-p2p.ts \
bench-couchdb.ts
COPY test/bench-network/run-bench.sh /usr/local/bin/run-livesync-bench
RUN chmod +x /usr/local/bin/run-livesync-bench
CMD ["run-livesync-bench"]
+10
View File
@@ -0,0 +1,10 @@
# syntax=docker/dockerfile:1
FROM alpine:3.22
RUN apk add --no-cache iproute2 socat
COPY test/bench-network/netem-tcp-shim.sh /usr/local/bin/livesync-netem-tcp-shim
RUN chmod +x /usr/local/bin/livesync-netem-tcp-shim
CMD ["livesync-netem-tcp-shim"]
+263
View File
@@ -0,0 +1,263 @@
# Network benchmark package
This directory packages the CLI benchmark cases with Docker Compose. It is
intended for reproducible local benchmark runs where CouchDB, the Nostr
signalling relay, optional TURN, and the benchmark runner are fixed by the
Compose file.
## Quick smoke run
From the repository root:
```bash
docker compose -f test/bench-network/compose.yml run --rm bench-runner
```
By default this runs:
- `couchdb-baseline`
- `p2p-direct-local`
The dataset is intentionally small by default. Results are written to
`test/bench-network/bench-results/`.
## GitHub Actions smoke run
`.github/workflows/cli-p2p-compose-smoke.yml` provides a manual
`workflow_dispatch` smoke run for the same Compose package. It is intentionally
not a required check yet, because WebRTC peer discovery can still be slow or
environment-sensitive on GitHub-hosted runners. Keep the dataset small and use
the uploaded JSON artefact to inspect whether failures are caused by peer
discovery, synchronisation, CouchDB startup, or Docker networking.
## Select cases
```bash
BENCH_CASES=couchdb-baseline,p2p-direct-local,p2p-user-turn \
docker compose -f test/bench-network/compose.yml --profile turn run --rm bench-runner
```
Available local cases:
- `couchdb-baseline`
- `p2p-direct-local`
- `couchdb-tethering-vpn-proxy`
- `couchdb-netem-home-wifi`
- `couchdb-netem-tethering-vpn`
- `p2p-smartphone-vpn-direct`
- `p2p-user-turn`
Set `BENCH_REPEAT_COUNT` to run each selected case more than once. Repeated
results are written with suffixes such as `-r01`, `-r02`, and `-r03`, and the
summary records the repeat index for each run.
`p2p-smartphone-vpn-direct` is a structural case name. When it is run inside
this Compose package it is not a real smartphone tethering/VPN measurement; it
uses the local Compose network. Use it only for wiring checks unless the runner
is executed in an actual tethered/VPN environment.
## Comparison model
The primary local comparison is between a remote-database path and a direct P2P
path:
| Case | Data path | What is measured | What is not measured |
| ------------------ | ------------------------------------------- | ------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------- |
| `couchdb-baseline` | Device A -> CouchDB -> Device B | Two one-shot CLI synchronisation commands through a local HTTP latency proxy | Real WAN jitter, packet loss, bandwidth limits, VPN encapsulation, and server contention |
| `p2p-direct-local` | Device A -> Device B after Nostr signalling | One CLI P2P synchronisation command over WebRTC DataChannel with TURN disabled | Public relay operation, mobile carrier behaviour, TURN relay throughput, and first-peer discovery latency |
Use the CouchDB result as the remote-store baseline and the P2P result as the
direct-transfer comparison. The Nostr relay is used for signalling in the P2P
case, but synchronised note content is transferred over the WebRTC DataChannel.
The P2P result JSON records the selected WebRTC ICE candidate pair when the CLI
can collect it from `RTCPeerConnection.getStats()`. Interpret P2P paths from
the recorded candidate types rather than from TURN configuration alone. Do not
report P2P runs as Tier 2 constrained-network measurements until host and
client are captured under an equivalent shaped topology.
## Dataset and latency controls
```bash
BENCH_MD_FILE_COUNT=100 \
BENCH_MD_MIN_SIZE_BYTES=512 \
BENCH_MD_MAX_SIZE_BYTES=2048 \
BENCH_BIN_FILE_COUNT=25 \
BENCH_BIN_SIZE_BYTES=8192 \
BENCH_COUCHDB_RTT_MS=20 \
BENCH_PEERS_TIMEOUT=60 \
docker compose -f test/bench-network/compose.yml run --rm bench-runner
```
The current CouchDB latency model is the existing HTTP proxy inside
`bench-couchdb.ts`. It models a remote database path with additional request
latency, but it does not model packet loss, jitter, MTU, bandwidth limits,
bufferbloat, or VPN encapsulation.
For P2P runs, `BENCH_PEERS_TIMEOUT` is passed to `p2p-peers`. That command waits
for the requested observation window before printing discovered peers, so the
reported peer discovery command time should not be read as first-peer latency.
## Latency sweep
To run P2P once and CouchDB at several requested RTT values:
```bash
BENCH_COMMAND=latency-sweep \
BENCH_SWEEP_RTT_MS=20,50,100,150,300 \
BENCH_MD_FILE_COUNT=100 \
BENCH_MD_MIN_SIZE_BYTES=512 \
BENCH_MD_MAX_SIZE_BYTES=2048 \
BENCH_BIN_FILE_COUNT=25 \
BENCH_BIN_SIZE_BYTES=8192 \
BENCH_SYNC_TIMEOUT=300 \
BENCH_PEERS_TIMEOUT=60 \
docker compose -f test/bench-network/compose.yml run --rm bench-runner
```
This sweep is useful for finding where the remote CouchDB path falls behind the
local direct P2P path in the current HTTP-proxy latency model. It should not be
presented as a full smartphone/VPN model.
## Network emulation smoke
The optional `netem` profile checks whether a Linux runner can apply traffic
shaping inside a Compose-managed container. This is a fixture smoke test for a
second-tier simulation design; it does not produce synchronisation performance
results by itself.
```bash
docker compose -f test/bench-network/compose.yml --profile netem run --rm netem-smoke
```
The smoke writes `tc qdisc`, route, and interface details under
`test/bench-network/bench-results/`. Profile parameters can be overridden:
```bash
NETEM_PROFILE=tethering-vpn \
NETEM_DELAY_MS=140 \
NETEM_JITTER_MS=50 \
NETEM_LOSS_PERCENT=1.0 \
NETEM_BANDWIDTH_MBIT=10 \
NETEM_MTU=1380 \
docker compose -f test/bench-network/compose.yml --profile netem run --rm netem-smoke
```
## Split-container P2P emulation
The optional `p2p-split` profile runs the P2P host and client in separate
Compose services. Each service can apply `tc netem` to its own egress interface
and the client result records the selected WebRTC ICE candidate pair.
```bash
BENCH_MD_FILE_COUNT=2 \
BENCH_BIN_FILE_COUNT=1 \
BENCH_PEERS_TIMEOUT=10 \
BENCH_SPLIT_RUN_ID="$(date -u +%Y%m%d%H%M%S)" \
docker compose -f test/bench-network/compose.yml --profile p2p-split up \
--abort-on-container-exit --exit-code-from p2p-split-client \
p2p-split-host p2p-split-client
```
By default this uses the `home-wifi` profile (`20 ms` delay, `5 ms` jitter,
`0.1%` loss, `100 Mbit`, and `1500` MTU) on both P2P containers. Override the
same `NETEM_*` variables used by the TCP shim to model a stricter profile.
```bash
BENCH_MD_FILE_COUNT=100 \
BENCH_MD_MIN_SIZE_BYTES=512 \
BENCH_MD_MAX_SIZE_BYTES=2048 \
BENCH_BIN_FILE_COUNT=25 \
BENCH_BIN_SIZE_BYTES=8192 \
BENCH_PEERS_TIMEOUT=60 \
BENCH_SYNC_TIMEOUT=420 \
BENCH_SPLIT_RUN_ID="$(date -u +%Y%m%d%H%M%S)" \
BENCH_NETWORK_PROFILE=tethering-vpn \
NETEM_PROFILE=tethering-vpn \
NETEM_DELAY_MS=140 \
NETEM_JITTER_MS=50 \
NETEM_LOSS_PERCENT=1.0 \
NETEM_BANDWIDTH_MBIT=10 \
NETEM_MTU=1380 \
docker compose -f test/bench-network/compose.yml --profile p2p-split up \
--abort-on-container-exit --exit-code-from p2p-split-client \
p2p-split-host p2p-split-client
```
This is a Linux-only manual benchmark fixture, not a required pull-request CI
job. It shapes each P2P container's egress path, including signalling traffic,
and should be reported separately from the CouchDB TCP-shim measurements. The
result JSON includes `ok: true` for completed runs; failed runs still write a
summary with `ok: false` and a `failure` object before returning a non-zero
exit code.
Remove the shared work volume between repeated manual runs when you do not use
a unique `BENCH_SPLIT_RUN_ID`:
```bash
docker compose -f test/bench-network/compose.yml --profile p2p-split down --volumes
```
## P2P Signalling-Only Emulation
The optional `signalling-shim` profile shapes only the Nostr signalling relay
path. The P2P host and client run in the benchmark runner as usual, and the
configured relay URL points at a TCP netem shim in front of `nostr-relay`.
This is the preferred fixture when evaluating the hypothesis that P2P avoids a
constrained remote database data path while still depending on a signalling
server for rendezvous.
```bash
BENCH_CASES=p2p-signalling-netem-home-wifi \
docker compose -f test/bench-network/compose.yml --profile signalling-shim run --rm \
bench-runner-signalling-shim
```
For a stricter signalling path:
```bash
NETEM_PROFILE=tethering-vpn \
NETEM_DELAY_MS=140 \
NETEM_JITTER_MS=50 \
NETEM_LOSS_PERCENT=1.0 \
NETEM_BANDWIDTH_MBIT=10 \
NETEM_MTU=1380 \
BENCH_CASES=p2p-signalling-netem-tethering-vpn \
docker compose -f test/bench-network/compose.yml --profile signalling-shim run --rm \
bench-runner-signalling-shim
```
Use this separately from `p2p-split`. The `p2p-split` profile shapes each peer's
egress path, so it constrains both signalling and the selected WebRTC data
path. The `signalling-shim` profile constrains only relay access, which keeps
it focused on peer-to-signalling-server reachability rather than peer-to-peer
note-data transfer.
## Shimmed CouchDB benchmark
The optional `shim` profile runs a CouchDB benchmark through a TCP forwarding
container that applies `tc netem`. This is a manual Tier 2 synchronisation
measurement path; it is intentionally separate from required pull-request CI.
```bash
docker compose -f test/bench-network/compose.yml --profile shim run --rm bench-runner-shim
```
The default profile is `home-wifi`. A smartphone/VPN-like profile can be
requested by overriding both the shim parameters and the benchmark case:
```bash
NETEM_PROFILE=tethering-vpn \
NETEM_DELAY_MS=140 \
NETEM_JITTER_MS=50 \
NETEM_LOSS_PERCENT=1.0 \
NETEM_BANDWIDTH_MBIT=10 \
NETEM_MTU=1380 \
BENCH_CASES=couchdb-netem-tethering-vpn \
docker compose -f test/bench-network/compose.yml --profile shim run --rm bench-runner-shim
```
The benchmark result records `simulationTier`, `networkProfile`, and
`networkModel`. The shim also writes its applied `tc qdisc`, route, and
interface state under `test/bench-network/bench-results/`.
This shim currently measures the CouchDB path only. It does not shape or verify
the WebRTC P2P data path.
+329
View File
@@ -0,0 +1,329 @@
services:
couchdb:
image: couchdb:3.5.0
environment:
COUCHDB_USER: ${BENCH_COUCHDB_USER:-admin}
COUCHDB_PASSWORD: ${BENCH_COUCHDB_PASSWORD:-testpassword}
COUCHDB_SINGLE_NODE: "true"
healthcheck:
test:
[
"CMD-SHELL",
"curl -fsS -u ${BENCH_COUCHDB_USER:-admin}:${BENCH_COUCHDB_PASSWORD:-testpassword} http://127.0.0.1:5984/_up >/dev/null",
]
interval: 2s
timeout: 5s
retries: 30
nostr-relay:
image: ghcr.io/hoytech/strfry:latest
entrypoint: sh
command:
- -lc
- |
cat > /tmp/strfry.conf <<'EOF'
db = "./strfry-db/"
relay {
bind = "0.0.0.0"
port = 7777
nofiles = 65536
info {
name = "livesync bench relay"
description = "local relay for livesync compose benchmarks"
}
maxWebsocketPayloadSize = 131072
autoPingSeconds = 55
writePolicy {
plugin = ""
}
}
EOF
exec /app/strfry --config /tmp/strfry.conf relay
tmpfs:
- /app/strfry-db:rw,size=256m
healthcheck:
test: ["CMD-SHELL", "nc -z 127.0.0.1 7777"]
interval: 2s
timeout: 5s
retries: 30
coturn:
image: coturn/coturn:latest
command:
- --log-file=stdout
- --listening-port=3478
- --user=${BENCH_TURN_USERNAME:-testuser}:${BENCH_TURN_CREDENTIAL:-testpass}
- --realm=${BENCH_TURN_REALM:-livesync.test}
profiles:
- turn
bench-runner:
build:
context: ../..
dockerfile: test/bench-network/Dockerfile.runner
depends_on:
couchdb:
condition: service_healthy
nostr-relay:
condition: service_healthy
environment:
BENCH_COMMAND: ${BENCH_COMMAND:-cases}
BENCH_CASES: ${BENCH_CASES:-couchdb-baseline,p2p-direct-local}
BENCH_REPEAT_COUNT: ${BENCH_REPEAT_COUNT:-1}
BENCH_CASES_ROOT: /workspace/src/apps/cli/testdeno/bench-results
BENCH_SWEEP_ROOT: /workspace/src/apps/cli/testdeno/bench-results
BENCH_SWEEP_RTT_MS: ${BENCH_SWEEP_RTT_MS:-20,50,100,150,300}
BENCH_SWEEP_INCLUDE_P2P: ${BENCH_SWEEP_INCLUDE_P2P:-true}
BENCH_COUCHDB_MANAGED: "false"
BENCH_COUCHDB_BACKEND_URI: http://couchdb:5984
BENCH_COUCHDB_URI: http://127.0.0.1:15989
BENCH_COUCHDB_USER: ${BENCH_COUCHDB_USER:-admin}
BENCH_COUCHDB_PASSWORD: ${BENCH_COUCHDB_PASSWORD:-testpassword}
BENCH_RELAY: ws://nostr-relay:7777/
BENCH_LOCAL_TURN_SERVERS: turn:coturn:3478
BENCH_MD_FILE_COUNT: ${BENCH_MD_FILE_COUNT:-20}
BENCH_MD_MIN_SIZE_BYTES: ${BENCH_MD_MIN_SIZE_BYTES:-512}
BENCH_MD_MAX_SIZE_BYTES: ${BENCH_MD_MAX_SIZE_BYTES:-2048}
BENCH_BIN_FILE_COUNT: ${BENCH_BIN_FILE_COUNT:-5}
BENCH_BIN_SIZE_BYTES: ${BENCH_BIN_SIZE_BYTES:-8192}
BENCH_COUCHDB_RTT_MS: ${BENCH_COUCHDB_RTT_MS:-20}
BENCH_TETHERING_VPN_RTT_MS: ${BENCH_TETHERING_VPN_RTT_MS:-120}
BENCH_SYNC_TIMEOUT: ${BENCH_SYNC_TIMEOUT:-300}
BENCH_PEERS_TIMEOUT: ${BENCH_PEERS_TIMEOUT:-60}
LIVESYNC_P2P_RELAY_READY_TIMEOUT_MS: ${LIVESYNC_P2P_RELAY_READY_TIMEOUT_MS:-60000}
BENCH_LIVESYNC_TEST_TEE: ${BENCH_LIVESYNC_TEST_TEE:-0}
volumes:
- ./bench-results:/workspace/src/apps/cli/testdeno/bench-results
couchdb-shim:
build:
context: ../..
dockerfile: test/bench-network/Dockerfile.shim
profiles:
- shim
depends_on:
couchdb:
condition: service_healthy
cap_add:
- NET_ADMIN
environment:
NETEM_PROFILE: ${NETEM_PROFILE:-home-wifi}
NETEM_INTERFACE: ${NETEM_INTERFACE:-eth0}
NETEM_DELAY_MS: ${NETEM_DELAY_MS:-20}
NETEM_JITTER_MS: ${NETEM_JITTER_MS:-5}
NETEM_LOSS_PERCENT: ${NETEM_LOSS_PERCENT:-0.1}
NETEM_BANDWIDTH_MBIT: ${NETEM_BANDWIDTH_MBIT:-100}
NETEM_MTU: ${NETEM_MTU:-1500}
NETEM_RESULT_ROOT: /bench-results
SHIM_LISTEN_PORT: 5984
SHIM_TARGET_HOST: couchdb
SHIM_TARGET_PORT: 5984
volumes:
- ./bench-results:/bench-results
healthcheck:
test: ["CMD-SHELL", "nc -z 127.0.0.1 5984"]
interval: 2s
timeout: 5s
retries: 30
bench-runner-shim:
build:
context: ../..
dockerfile: test/bench-network/Dockerfile.runner
profiles:
- shim
depends_on:
couchdb-shim:
condition: service_healthy
environment:
BENCH_COMMAND: ${BENCH_COMMAND:-cases}
BENCH_CASES: ${BENCH_CASES:-couchdb-netem-home-wifi}
BENCH_REPEAT_COUNT: ${BENCH_REPEAT_COUNT:-1}
BENCH_CASES_ROOT: /workspace/src/apps/cli/testdeno/bench-results
BENCH_SWEEP_ROOT: /workspace/src/apps/cli/testdeno/bench-results
BENCH_COUCHDB_MANAGED: "false"
BENCH_COUCHDB_BACKEND_URI: http://couchdb-shim:5984
BENCH_SHIM_COUCHDB_URI: http://couchdb-shim:5984
BENCH_COUCHDB_URI: http://127.0.0.1:15989
BENCH_COUCHDB_USER: ${BENCH_COUCHDB_USER:-admin}
BENCH_COUCHDB_PASSWORD: ${BENCH_COUCHDB_PASSWORD:-testpassword}
BENCH_MD_FILE_COUNT: ${BENCH_MD_FILE_COUNT:-20}
BENCH_MD_MIN_SIZE_BYTES: ${BENCH_MD_MIN_SIZE_BYTES:-512}
BENCH_MD_MAX_SIZE_BYTES: ${BENCH_MD_MAX_SIZE_BYTES:-2048}
BENCH_BIN_FILE_COUNT: ${BENCH_BIN_FILE_COUNT:-5}
BENCH_BIN_SIZE_BYTES: ${BENCH_BIN_SIZE_BYTES:-8192}
BENCH_COUCHDB_RTT_MS: ${BENCH_COUCHDB_RTT_MS:-1}
BENCH_SYNC_TIMEOUT: ${BENCH_SYNC_TIMEOUT:-300}
BENCH_LIVESYNC_TEST_TEE: ${BENCH_LIVESYNC_TEST_TEE:-0}
volumes:
- ./bench-results:/workspace/src/apps/cli/testdeno/bench-results
p2p-signalling-shim:
build:
context: ../..
dockerfile: test/bench-network/Dockerfile.shim
profiles:
- signalling-shim
depends_on:
nostr-relay:
condition: service_healthy
cap_add:
- NET_ADMIN
environment:
NETEM_PROFILE: ${NETEM_PROFILE:-home-wifi}
NETEM_INTERFACE: ${NETEM_INTERFACE:-eth0}
NETEM_DELAY_MS: ${NETEM_DELAY_MS:-20}
NETEM_JITTER_MS: ${NETEM_JITTER_MS:-5}
NETEM_LOSS_PERCENT: ${NETEM_LOSS_PERCENT:-0.1}
NETEM_BANDWIDTH_MBIT: ${NETEM_BANDWIDTH_MBIT:-100}
NETEM_MTU: ${NETEM_MTU:-1500}
NETEM_RESULT_ROOT: /bench-results
SHIM_LISTEN_PORT: 7777
SHIM_TARGET_HOST: nostr-relay
SHIM_TARGET_PORT: 7777
volumes:
- ./bench-results:/bench-results
healthcheck:
test: ["CMD-SHELL", "nc -z 127.0.0.1 7777"]
interval: 2s
timeout: 5s
retries: 30
bench-runner-signalling-shim:
build:
context: ../..
dockerfile: test/bench-network/Dockerfile.runner
profiles:
- signalling-shim
depends_on:
p2p-signalling-shim:
condition: service_healthy
environment:
BENCH_COMMAND: ${BENCH_COMMAND:-cases}
BENCH_CASES: ${BENCH_CASES:-p2p-signalling-netem-home-wifi}
BENCH_REPEAT_COUNT: ${BENCH_REPEAT_COUNT:-1}
BENCH_CASES_ROOT: /workspace/src/apps/cli/testdeno/bench-results
BENCH_SIGNAL_SHIM_RELAY: ws://p2p-signalling-shim:7777/
BENCH_MD_FILE_COUNT: ${BENCH_MD_FILE_COUNT:-20}
BENCH_MD_MIN_SIZE_BYTES: ${BENCH_MD_MIN_SIZE_BYTES:-512}
BENCH_MD_MAX_SIZE_BYTES: ${BENCH_MD_MAX_SIZE_BYTES:-2048}
BENCH_BIN_FILE_COUNT: ${BENCH_BIN_FILE_COUNT:-5}
BENCH_BIN_SIZE_BYTES: ${BENCH_BIN_SIZE_BYTES:-8192}
BENCH_SYNC_TIMEOUT: ${BENCH_SYNC_TIMEOUT:-300}
BENCH_PEERS_TIMEOUT: ${BENCH_PEERS_TIMEOUT:-60}
LIVESYNC_P2P_RELAY_READY_TIMEOUT_MS: ${LIVESYNC_P2P_RELAY_READY_TIMEOUT_MS:-60000}
BENCH_LIVESYNC_TEST_TEE: ${BENCH_LIVESYNC_TEST_TEE:-0}
volumes:
- ./bench-results:/workspace/src/apps/cli/testdeno/bench-results
p2p-split-host:
build:
context: ../..
dockerfile: test/bench-network/Dockerfile.runner
profiles:
- p2p-split
depends_on:
nostr-relay:
condition: service_healthy
cap_add:
- NET_ADMIN
environment:
BENCH_COMMAND: p2p-split-node
BENCH_P2P_SPLIT_ROLE: host
BENCH_SPLIT_RUN_ID: ${BENCH_SPLIT_RUN_ID:-bench-split-run}
BENCH_SPLIT_WORK_ROOT: /p2p-work
BENCH_SPLIT_RESULT_ROOT: /workspace/src/apps/cli/testdeno/bench-results
BENCH_RELAY: ws://nostr-relay:7777/
BENCH_APP_ID: ${BENCH_APP_ID:-self-hosted-livesync-cli-benchmark}
BENCH_ROOM_ID: ${BENCH_ROOM_ID:-bench-split-room}
BENCH_PASSPHRASE: ${BENCH_PASSPHRASE:-bench-split-passphrase}
BENCH_TURN_SERVERS: ${BENCH_TURN_SERVERS:-}
BENCH_MD_FILE_COUNT: ${BENCH_MD_FILE_COUNT:-20}
BENCH_MD_MIN_SIZE_BYTES: ${BENCH_MD_MIN_SIZE_BYTES:-512}
BENCH_MD_MAX_SIZE_BYTES: ${BENCH_MD_MAX_SIZE_BYTES:-2048}
BENCH_BIN_FILE_COUNT: ${BENCH_BIN_FILE_COUNT:-5}
BENCH_BIN_SIZE_BYTES: ${BENCH_BIN_SIZE_BYTES:-8192}
BENCH_SYNC_TIMEOUT: ${BENCH_SYNC_TIMEOUT:-300}
BENCH_PEERS_TIMEOUT: ${BENCH_PEERS_TIMEOUT:-60}
BENCH_NETEM_ENABLED: ${BENCH_NETEM_ENABLED:-1}
BENCH_NETWORK_PROFILE: ${BENCH_NETWORK_PROFILE:-home-wifi}
NETEM_PROFILE: ${NETEM_PROFILE:-home-wifi}
NETEM_INTERFACE: ${NETEM_INTERFACE:-eth0}
NETEM_DELAY_MS: ${NETEM_DELAY_MS:-20}
NETEM_JITTER_MS: ${NETEM_JITTER_MS:-5}
NETEM_LOSS_PERCENT: ${NETEM_LOSS_PERCENT:-0.1}
NETEM_BANDWIDTH_MBIT: ${NETEM_BANDWIDTH_MBIT:-100}
NETEM_MTU: ${NETEM_MTU:-1500}
LIVESYNC_P2P_RELAY_READY_TIMEOUT_MS: ${LIVESYNC_P2P_RELAY_READY_TIMEOUT_MS:-60000}
LIVESYNC_TEST_TEE: ${BENCH_LIVESYNC_TEST_TEE:-0}
volumes:
- p2p-split-work:/p2p-work
- ./bench-results:/workspace/src/apps/cli/testdeno/bench-results
p2p-split-client:
build:
context: ../..
dockerfile: test/bench-network/Dockerfile.runner
profiles:
- p2p-split
depends_on:
nostr-relay:
condition: service_healthy
p2p-split-host:
condition: service_started
cap_add:
- NET_ADMIN
environment:
BENCH_COMMAND: p2p-split-node
BENCH_P2P_SPLIT_ROLE: client
BENCH_SPLIT_RUN_ID: ${BENCH_SPLIT_RUN_ID:-bench-split-run}
BENCH_SPLIT_WORK_ROOT: /p2p-work
BENCH_SPLIT_RESULT_ROOT: /workspace/src/apps/cli/testdeno/bench-results
BENCH_RELAY: ws://nostr-relay:7777/
BENCH_APP_ID: ${BENCH_APP_ID:-self-hosted-livesync-cli-benchmark}
BENCH_ROOM_ID: ${BENCH_ROOM_ID:-bench-split-room}
BENCH_PASSPHRASE: ${BENCH_PASSPHRASE:-bench-split-passphrase}
BENCH_TURN_SERVERS: ${BENCH_TURN_SERVERS:-}
BENCH_SYNC_TIMEOUT: ${BENCH_SYNC_TIMEOUT:-300}
BENCH_PEERS_TIMEOUT: ${BENCH_PEERS_TIMEOUT:-60}
BENCH_NETEM_ENABLED: ${BENCH_NETEM_ENABLED:-1}
BENCH_NETWORK_PROFILE: ${BENCH_NETWORK_PROFILE:-home-wifi}
NETEM_PROFILE: ${NETEM_PROFILE:-home-wifi}
NETEM_INTERFACE: ${NETEM_INTERFACE:-eth0}
NETEM_DELAY_MS: ${NETEM_DELAY_MS:-20}
NETEM_JITTER_MS: ${NETEM_JITTER_MS:-5}
NETEM_LOSS_PERCENT: ${NETEM_LOSS_PERCENT:-0.1}
NETEM_BANDWIDTH_MBIT: ${NETEM_BANDWIDTH_MBIT:-100}
NETEM_MTU: ${NETEM_MTU:-1500}
LIVESYNC_P2P_RELAY_READY_TIMEOUT_MS: ${LIVESYNC_P2P_RELAY_READY_TIMEOUT_MS:-60000}
LIVESYNC_TEST_TEE: ${BENCH_LIVESYNC_TEST_TEE:-0}
volumes:
- p2p-split-work:/p2p-work
- ./bench-results:/workspace/src/apps/cli/testdeno/bench-results
netem-smoke:
build:
context: ../..
dockerfile: test/bench-network/Dockerfile.netem
profiles:
- netem
cap_add:
- NET_ADMIN
environment:
NETEM_PROFILE: ${NETEM_PROFILE:-home-wifi}
NETEM_INTERFACE: ${NETEM_INTERFACE:-eth0}
NETEM_DELAY_MS: ${NETEM_DELAY_MS:-20}
NETEM_JITTER_MS: ${NETEM_JITTER_MS:-5}
NETEM_LOSS_PERCENT: ${NETEM_LOSS_PERCENT:-0.1}
NETEM_BANDWIDTH_MBIT: ${NETEM_BANDWIDTH_MBIT:-100}
NETEM_MTU: ${NETEM_MTU:-1500}
NETEM_RESULT_ROOT: /bench-results
volumes:
- ./bench-results:/bench-results
volumes:
p2p-split-work:
+71
View File
@@ -0,0 +1,71 @@
#!/usr/bin/env sh
set -eu
profile="${NETEM_PROFILE:-home-wifi}"
iface="${NETEM_INTERFACE:-eth0}"
delay_ms="${NETEM_DELAY_MS:-20}"
jitter_ms="${NETEM_JITTER_MS:-5}"
loss_percent="${NETEM_LOSS_PERCENT:-0.1}"
bandwidth_mbit="${NETEM_BANDWIDTH_MBIT:-100}"
mtu="${NETEM_MTU:-1500}"
out_root="${NETEM_RESULT_ROOT:-/bench-results}"
timestamp="$(date -u +%Y%m%d-%H%M%S)"
out_dir="${out_root}/netem-smoke-${timestamp}"
out_file="${out_dir}/summary.json"
mkdir -p "$out_dir"
if ! ip link show "$iface" >/dev/null 2>&1; then
echo "Network interface '$iface' was not found" >&2
ip addr >&2
exit 2
fi
ip link set dev "$iface" mtu "$mtu"
tc qdisc del dev "$iface" root >/dev/null 2>&1 || true
tc qdisc add dev "$iface" root netem \
delay "${delay_ms}ms" "${jitter_ms}ms" \
loss "${loss_percent}%" \
rate "${bandwidth_mbit}mbit"
json_lines() {
awk '
{
gsub(/\\/, "\\\\");
gsub(/"/, "\\\"");
printf "%s \"%s\"", (NR == 1 ? "" : ",\n"), $0;
}
'
}
ip_addr="$(ip addr show "$iface" | json_lines)"
ip_route="$(ip route | json_lines)"
tc_qdisc="$(tc qdisc show dev "$iface" | json_lines)"
cat > "$out_file" <<EOF
{
"simulationTier": 2,
"mode": "netem-smoke",
"profile": "$profile",
"interface": "$iface",
"netem": {
"delayMs": $delay_ms,
"jitterMs": $jitter_ms,
"lossPercent": $loss_percent,
"bandwidthMbit": $bandwidth_mbit,
"mtu": $mtu
},
"ipAddr": [
$ip_addr
],
"ipRoute": [
$ip_route
],
"tcQdisc": [
$tc_qdisc
]
}
EOF
cat "$out_file"
echo "[netem-smoke] result file: $out_file"
+80
View File
@@ -0,0 +1,80 @@
#!/usr/bin/env sh
set -eu
profile="${NETEM_PROFILE:-home-wifi}"
iface="${NETEM_INTERFACE:-eth0}"
delay_ms="${NETEM_DELAY_MS:-20}"
jitter_ms="${NETEM_JITTER_MS:-5}"
loss_percent="${NETEM_LOSS_PERCENT:-0.1}"
bandwidth_mbit="${NETEM_BANDWIDTH_MBIT:-100}"
mtu="${NETEM_MTU:-1500}"
listen_port="${SHIM_LISTEN_PORT:-5984}"
target_host="${SHIM_TARGET_HOST:-couchdb}"
target_port="${SHIM_TARGET_PORT:-5984}"
out_root="${NETEM_RESULT_ROOT:-/bench-results}"
timestamp="$(date -u +%Y%m%d-%H%M%S)"
out_dir="${out_root}/netem-shim-${profile}-${timestamp}"
out_file="${out_dir}/summary.json"
json_lines() {
awk '
{
gsub(/\\/, "\\\\");
gsub(/"/, "\\\"");
printf "%s \"%s\"", (NR == 1 ? "" : ",\n"), $0;
}
'
}
mkdir -p "$out_dir"
if ! ip link show "$iface" >/dev/null 2>&1; then
echo "Network interface '$iface' was not found" >&2
ip addr >&2
exit 2
fi
ip link set dev "$iface" mtu "$mtu"
tc qdisc del dev "$iface" root >/dev/null 2>&1 || true
tc qdisc add dev "$iface" root netem \
delay "${delay_ms}ms" "${jitter_ms}ms" \
loss "${loss_percent}%" \
rate "${bandwidth_mbit}mbit"
ip_addr="$(ip addr show "$iface" | json_lines)"
ip_route="$(ip route | json_lines)"
tc_qdisc="$(tc qdisc show dev "$iface" | json_lines)"
cat > "$out_file" <<EOF
{
"simulationTier": 2,
"mode": "netem-tcp-shim",
"profile": "$profile",
"interface": "$iface",
"listenPort": $listen_port,
"targetHost": "$target_host",
"targetPort": $target_port,
"netem": {
"delayMs": $delay_ms,
"jitterMs": $jitter_ms,
"lossPercent": $loss_percent,
"bandwidthMbit": $bandwidth_mbit,
"mtu": $mtu
},
"ipAddr": [
$ip_addr
],
"ipRoute": [
$ip_route
],
"tcQdisc": [
$tc_qdisc
]
}
EOF
cat "$out_file"
echo "[netem-shim] forwarding 0.0.0.0:${listen_port} to ${target_host}:${target_port}"
echo "[netem-shim] result file: $out_file"
exec socat "TCP-LISTEN:${listen_port},fork,reuseaddr" "TCP:${target_host}:${target_port}"
+19
View File
@@ -0,0 +1,19 @@
#!/usr/bin/env sh
set -eu
case "${BENCH_COMMAND:-cases}" in
cases)
exec deno task bench:cases
;;
latency-sweep)
exec deno task bench:latency-sweep
;;
p2p-split-node)
exec deno task bench:p2p-split-node
;;
*)
echo "Unknown BENCH_COMMAND: ${BENCH_COMMAND}" >&2
echo "Expected one of: cases, latency-sweep, p2p-split-node" >&2
exit 2
;;
esac
-4
View File
@@ -1,4 +0,0 @@
#!/bin/bash
set -e
docker inspect webdav-test >/dev/null
-15
View File
@@ -1,15 +0,0 @@
#!/bin/bash
set -e
webdavUsername=${webdavUsername:-webdav}
webdavPassword=${webdavPassword:-webdav}
webdavPort=${webdavPort:-8088}
docker run -d \
--name webdav-test \
-p "$webdavPort:8080" \
rclone/rclone serve webdav /data \
--addr :8080 \
--baseurl /dav \
--user "$webdavUsername" \
--pass "$webdavPassword"
-3
View File
@@ -1,3 +0,0 @@
#!/bin/bash
docker stop webdav-test
docker rm webdav-test
+13 -40
View File
@@ -3,12 +3,21 @@ Since 19th July, 2025 (beta1 in 0.25.0-beta1, 13th July, 2025)
The head note of 0.25 is now in [updates_old.md](https://github.com/vrtmrz/obsidian-livesync/blob/main/updates_old.md). Because 0.25 got a lot of updates, thankfully, compatibility is kept and we do not need breaking changes! In other words, when get enough stabled. The next version will be v1.0.0. Even though it my hope.
## Unreleased
## 0.25.80
### Experimental
7th July, 2026
- Added experimental WebDAV support for Journal synchronisation storage, including setup dialogue integration. This is currently a proof of concept for injecting Journal storage backends through the replication layer, and should be treated as an implementation experiment rather than a fully supported setup flow.
- Hardened the WebDAV Journal storage proof of concept by centralising Journal storage adapter selection, improving connection checks, masking remote configuration descriptions, and verifying the Docker WebDAV integration path.
### Fixed
- Improved Markdown conflict auto-merge so that non-overlapping edits are merged while overlapping delete-and-edit cases remain visible for manual resolution (#993).
- Behaviour change:
- When one side deletes an unchanged line and the other side edits a different region, the deleted line is no longer reintroduced into the merged result.
- When one side deletes a line and the other side modifies that same line, the conflict is preserved instead of silently choosing one side.
- Fixed an issue where applying a newer database entry to storage could incorrectly preserve an older local file as a conflict (#994).
- Behaviour change:
- Local storage is preserved as a conflict when it may contain unsynchronised changes that are not represented in the revision history. A newer incoming text entry is applied without creating a conflict only when it clearly extends the existing local text.
- Fixed an issue where choosing Disable and then Overwrite in Hidden File Sync could silently skip hidden files, because the overwrite setup ran while hidden file synchronisation was still disabled (#989, PR #992).
- Hidden File Sync is now re-enabled before the Fetch, Overwrite, or Merge initialisation runs, instead of after it completes. If that initialisation fails, the setting may remain enabled.
## 0.25.79
@@ -103,41 +112,5 @@ Also, this update is a very large one, even if we had a lot of time, and we had
- Some dependencies have been updated.
- Now we check the compatibility with iOS 15 in the CI tests to ensure the plugin continues to work on older iOS versions even after we upgrade some dependencies.
## 0.25.74
8th June, 2026
### Fixed
- Fixed an issue where disabling hidden file synchronisation did not take effect, allowing non-target hidden files to continue to be processed and synchronised by replication or boot-sequence scan (#941).
- Prevented the automatic merging of conflicted revisions when one of the revisions has been deleted, which was causing deleted files to reappear (#911).
- The startup sequence now saves the state more effectively (Thank you so much for @bmcyver)!
## Only CLI
8th June, 2026
I should also consider the version numbering for the CLI...
### Improved
- Added new remote database management commands: `remote-status`, `unlock-remote`, `lock-remote`, and `mark-resolved`.
- --vault option is now available for daemon and mirror commands! (Thank you so much for @starskyzheng)!
- Decoupled the database directory path from the actual vault directory path using the `--vault` (or `-V`) option.
### Fixed (preventive)
- Validated that the specified vault path exists and is indeed a directory before starting the CLI.
- Integrated path resolution and validations for one-off commands (such as `'push'`, `'pull'`, `'cat'`, `'rm'`, `'info'`, and `'resolve'`) against the decoupled vault path instead of the database path.
## 0.25.73
4th June, 2026
### Fixed
- Adjust CouchDB's database name checking to its specification (#926).
- `Reset Syncronisation on This Device` for minio and P2P is now working properly.
Full notes are in
[updates_old.md](https://github.com/vrtmrz/obsidian-livesync/blob/main/updates_old.md).
+58
View File
@@ -4,6 +4,64 @@ Since 19th July, 2025 (beta1 in 0.25.0-beta1, 13th July, 2025)
The head note of 0.25 is now in [updates_old.md](https://github.com/vrtmrz/obsidian-livesync/blob/main/updates_old.md). Because 0.25 got a lot of updates, thankfully, compatibility is kept and we do not need breaking changes! In other words, when get enough stabled. The next version will be v1.0.0. Even though it my hope.
## 0.25.80
7th July, 2026
### Fixed
- Improved Markdown conflict auto-merge so that non-overlapping edits are merged while overlapping delete-and-edit cases remain visible for manual resolution (#993).
- Behaviour change:
- When one side deletes an unchanged line and the other side edits a different region, the deleted line is no longer reintroduced into the merged result.
- When one side deletes a line and the other side modifies that same line, the conflict is preserved instead of silently choosing one side.
- Fixed an issue where applying a newer database entry to storage could incorrectly preserve an older local file as a conflict (#994).
- Behaviour change:
- Local storage is preserved as a conflict when it may contain unsynchronised changes that are not represented in the revision history. A newer incoming text entry is applied without creating a conflict only when it clearly extends the existing local text.
- Fixed an issue where choosing Disable and then Overwrite in Hidden File Sync could silently skip hidden files, because the overwrite setup ran while hidden file synchronisation was still disabled (#989, PR #992).
- Hidden File Sync is now re-enabled before the Fetch, Overwrite, or Merge initialisation runs, instead of after it completes. If that initialisation fails, the setting may remain enabled.
## 0.25.79
29th June, 2026
### Fixed
- Fast Fetch now retries transient stream interruptions and resumes from the latest persisted checkpoint, instead of starting over after ordinary network or platform interruptions (#977, PR #978; commonlib PR #59). Thank you so much for @apple-ouyang for the fix!
- Simple Fetch now remembers the selected setup choices while an interrupted Fetch All operation is still pending, so users are not asked the same questions again on retry (#977, PR #978). Thank you so much for @apple-ouyang for the fix!
- No longer hidden storage events, such as `.git` paths, reach the normal target-file filter when internal file synchronisation is disabled. This avoids noisy non-target logs before those files are skipped (commonlib PR #60). Thank you so much for @apple-ouyang for the fix!
- Fixed an issue where a file deleted from storage could be resurrected by the offline scanner because the database tombstone was not written when the storage file was already gone (commonlib PR #56). Thank you so much for @cosmic-fire-eng for the fix!
### Improved
- Local database maintenance commands now ask before applying the required chunk settings, and can apply those prerequisites before continuing (#980, PR #981). Thank you so much for @apple-ouyang for the improvement!
- Improved CouchDB replication event handling by using the new `StreamInbox` helper from `octagonal-wheels` (commonlib PR #62).
### Documentation
- Added `nginx` to the setup documentation table of contents (PR #976). Thank you so much for @kiraventom for the improvement!
### Miscellaneous
- Updated `octagonal-wheels` to `0.1.47` across the plug-in and workspace packages to use the newly published helper modules.
## 0.25.78
23rd June, 2026
### Fixed
- No longer fast synchronisation (a.k.a. Fast Fetch) causes a rewind and re-fetch of the entire database when some errors occur during the process (#972, PR #973). Thank you so much for @apple-ouyang for the fix!
### Improved
- Overhauled the Object Storage (e.g., MinIO and S3) replication engine ('Journal Replicator 2nd Edition').
- It now leverages the standard Web Streams API for a resilient, backpressure-aware architecture, reducing memory footprints/temporary storage usage on large vaults.
- Decoupled the physical storage logic to make it easier to add new storage backends in the future.
- Stricter compliance with CouchDB's replication protocol (proper `_revisions` transfers with `new_edits: false`) when using Object Storage.
### Testing
- Added comprehensive unit tests for the new `JournalSyncCore` engine, covering streams, backpressure, and `new_edits: false` validation.
- Improved integration test workflows in the CI pipeline to run MinIO tests automatically using standard environment variables.
## 0.25.77
19th June, 2026