Re-evaluate optional features for 1.0

This commit is contained in:
vorotamoroz
2026-07-21 01:53:13 +00:00
parent ed7471fba5
commit 3eda3316c5
22 changed files with 1253 additions and 32 deletions
@@ -0,0 +1,49 @@
# Feature maturity for 1.0
## Status
Proposed for the 1.0 integration branch.
## Context
Self-hosted LiveSync accumulated several labels such as 'experimental', 'Beta', 'obsolete', and 'sunset' over the 0.x line. Those labels did not consistently describe the current implementation. Some formerly experimental features now have maintained unit, Compose, and real-Obsidian coverage, while some old database-format options remain executable only because existing data must still be opened safely.
Version 1.0 needs to distinguish supported opt-in features from previews and from compatibility paths. Removing a label does not make a network environment reliable, and retaining a setting key does not make it a recommendation.
## Decision
### Supported opt-in features
- Peer-to-Peer Synchronisation is supported. Commonlib owns the transport lifecycle, the Compose suite covers transfer, replacement, disconnect, and reconnect behaviour, and the Obsidian host verifies its current pane boundary. Documentation must describe environmental WebRTC limitations without calling the feature experimental.
- Hidden File Sync is supported as an advanced, separately initialised feature. It remains disabled during ordinary Setup URI initialisation and has its own two-Vault, conflict, filtering, and notification acceptance workflow.
- Customisation Sync is supported as an advanced opt-in feature. Its maintained two-Vault real-Obsidian workflow covers snippets, configuration files, and plug-in files, including updates and deletion of source synchronisation data. It remains separate from Hidden File Sync and must not manage the same files concurrently.
- Data Compression is maintained as an advanced opt-in storage and bandwidth trade-off. It remains disabled by default. The three-repeat CLI and CouchDB benchmark reduced stored chunk data and upload bodies by about 9% for the mixed fixture, but processing and worker-memory costs remained substantial. The [Data Compression specification](../specs_data_compression.md) records the contract, measurements, compatibility behaviour, execution model, and reproduction path for future default-setting decisions.
- The real-Obsidian E2E runner is maintained release infrastructure rather than an experimental Harness.
### Retained previews
- JWT authentication remains experimental because it depends on specialised CouchDB server configuration and does not yet have a maintained server-backed authentication matrix. The current implementation, Setup URI transport, focused unit coverage, and reported ES512 use justify retaining it.
- Ignore files remain Beta. They have focused target-filter tests, but nested rules, hidden-file expectations, and open user reports still require review.
- Automatic newer-file conflict resolution remains Beta and disabled by default because it can deliberately overwrite one side of a conflict.
- Garbage Collection V3 remains Beta and explicitly initiated. Its algorithm and safety protocol are outside this classification change and require a separate decision.
### Compatibility-only and sunset paths
- E2EE V1 and its dynamic iteration-count setting remain for existing encrypted databases. E2EE V2 is the new-Vault contract.
- The old IndexedDB adapter remains only with its migration path back to IDB.
- `xxhash64` is the current hash contract. Other hash algorithms remain available for existing databases and edge-case recovery, not as experimental alternatives for new Vaults.
- Eden chunks remain accepted at runtime and in transported settings, but are not offered in the settings interface.
- `doNotUseFixedRevisionForChunks` remains an inert compatibility input. Chunk revisions are always content-derived.
- The deprecated cleaned-database reconciliation callback remains internal while an old IndexedDB client may still encounter that remote state. It is not a user-selectable maintenance action and is omitted from the settings reference.
### Already removed
The obsolete mocked browser Harness, automatic bulk chunk pre-send, legacy trash toggle, and fixed-revision control have no supported 1.0 UI path. Their compatibility data, where required, remains accepted independently of their removed controls.
## Consequences
- Supported opt-in features retain focused release gates and user documentation.
- Preview features remain off by default and keep explicit maturity labels.
- Compatibility-only settings must not silently change existing data formats. New configuration should not expose retired formats merely because their decoders remain available.
- Commonlib setting types and Setup URI decoding remain broad enough to read existing configurations. Removing those package contracts requires a separately versioned compatibility decision.
- Deprecated host accessors and Community directory API warnings are a separate refactoring boundary. This decision does not authorise removing broadly used internal access paths.
+25 -12
View File
@@ -4,6 +4,17 @@ NOTE: This document not completed. I'll improve this doc in a while. but your co
There are many settings in Self-hosted LiveSync. This document describes each setting in detail (not how-to). Configuration and settings are divided into several categories and indicated by icons. The icon is as follows:
## Feature maturity for 1.0
The following status applies to optional and compatibility features in the 1.0 line:
| Status | Features | 1.0 policy |
| ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Supported, opt-in | Peer-to-Peer Synchronisation, Hidden File Sync, and Customisation Sync | Maintained and covered by focused real-runtime tests. Enable them only where their separate setup and operational constraints are acceptable. |
| Maintained, advanced | Data Compression | Available as an explicit storage and bandwidth trade-off. It remains disabled by default because the measured processing and memory costs outweigh the mixed-dataset saving. |
| Beta or experimental | JWT authentication, ignore files, automatic newer-file conflict resolution, and Garbage Collection V3 | Retained for explicit testing and specialised use. They remain disabled by default and are not part of the minimum supported setup. |
| Compatibility only | V1 dynamic iteration counts, the old IndexedDB adapter, non-current hash algorithms, Eden chunks, and the stored `doNotUseFixedRevisionForChunks` key | Existing settings and data remain readable. New Vaults use the current defaults, and compatibility controls are shown only where a migration or recovery path still needs them. |
| Icon | Description |
| :--: | ------------------------------------------------------------------ |
| 💬 | [0. Change Log](#0-change-log) |
@@ -213,11 +224,11 @@ The encryption algorithm version used for end-to-end encryption.
- `v2` (V2: AES-256-GCM With HKDF): Recommended and default version.
- `forceV1` or `""` (V1: Legacy): Older legacy encryption. Only use this if you have an existing vault encrypted in the legacy format.
#### Use dynamic iteration count (Experimental)
#### Use dynamic iteration count (legacy V1 compatibility)
Setting key: useDynamicIterationCount
This is an experimental feature and not recommended. If you enable this, the iteration count of the encryption will be dynamically determined. This is useful when you want to improve the performance.
This setting applies only to legacy V1 encryption data. Keep the saved value when opening an existing V1 database. New Vaults use E2EE V2 and do not use this setting.
---
@@ -634,6 +645,8 @@ Comma separated `.gitignore, .dockerignore`
## 6. Customisation sync (Advanced)
Customisation Sync is a supported, advanced opt-in feature. Its current per-file implementation is covered by a two-Vault real-Obsidian workflow for snippets, configuration files, and plug-in files. Hidden File Sync is a separate feature with different setup, selection, and conflict behaviour; do not use both features to manage the same files.
### 1. Customisation Sync
#### Device name
@@ -778,10 +791,10 @@ Limit the maximum size of chunks sent in one request by the explicit **Resend al
### 1. Remote Database Tweak
#### Incubate Chunks in Document (Beta)
#### Incubate Chunks in Document (sunset compatibility)
Setting key: useEden
If enabled, newly created chunks are temporarily kept within the document, and graduated to become independent chunks once stabilised.
This setting is no longer offered for new configuration. Existing saved values remain accepted so that established databases can be opened and migrated without silently changing their structure.
#### Maximum Incubating Chunks
@@ -798,10 +811,14 @@ The maximum total size of chunks that can be incubated within the document. Chun
Setting key: maxAgeInEden
The maximum duration for which chunks can be incubated within the document. Chunks exceeding this period will graduate to independent chunks.
#### Data Compression (Experimental)
#### Data Compression (advanced opt-in)
Setting key: enableCompression
Data Compression applies fflate level 8 to each chunk before E2EE. A chunk is left uncompressed when compression would not make it smaller, and readers continue to accept both representations. Changing the setting does not require a rebuild for compatibility; existing and new representations can coexist.
The setting remains disabled by default because the measured storage and transfer benefit comes with workload-dependent processing and memory costs. See the [Data Compression specification](specs_data_compression.md) for the contract, evidence, execution model, and reproduction command.
### 2. CouchDB Connection Tweak
#### Batch size
@@ -897,10 +914,12 @@ Do not use internal API
Setting key: additionalSuffixOfDatabaseName
LiveSync could not handle multiple vaults which have same name without different prefix, This should be automatically configured.
#### The Hash algorithm for chunk IDs (Experimental)
#### The Hash algorithm for chunk IDs (compatibility)
Setting key: hashAlg
`xxhash64` is the supported current value. Older algorithms remain selectable only as an edge-case compatibility path for existing databases. Changing the algorithm can reduce chunk reuse between devices and requires the normal tweak review.
### 6. Edge case addressing (Behaviour)
#### Fetch database with previous behaviour
@@ -1004,12 +1023,6 @@ Purge all download/upload cache.
Delete all data on the remote server.
### 6. Deprecated
#### Run database cleanup
Attempt to shrink the database by deleting unused chunks. This may not work consistently. Use the 'Overwrite Server Data with This Device's Files' under Reset Synchronisation information.
### 7. Reset
#### Delete local database to reset or uninstall Self-hosted LiveSync
+116
View File
@@ -0,0 +1,116 @@
# Data Compression specification
## Status and decision for 1.0
Data Compression is a maintained, advanced opt-in feature for CouchDB-compatible remote databases. It remains disabled by default in the 1.0 line.
The feature provides a modest, measurable reduction in mixed-workload storage and transfer volume. Its value depends heavily on the chunk contents, while the current implementation adds substantial processing and worker-memory costs. Users with slow, metered, or storage-constrained connections may still find that trade-off worthwhile.
## Stored-data behaviour
The `enableCompression` setting affects chunk documents written through the CouchDB remote connection. It does not compress the local database, and it is separate from the journal format used by Object Storage synchronisation.
For each document containing string `data`, the writer:
1. detects canonical Base64 and decodes it to bytes, or encodes text as UTF-8;
2. applies raw DEFLATE through fflate at level 8;
3. encodes the result as Base64 and adds the LiveSync compressed-data marker; and
4. stores the compressed representation only when its string representation is shorter than the original.
Already compressed data and data which does not become smaller are therefore retained unchanged. Readers always recognise and expand the compressed-data marker, including when their own `enableCompression` setting is off. Compressed and uncompressed chunks can coexist in the same remote database.
When E2EE is enabled, compression is applied before E2EE V2 encryption. The encrypted remote representation does not expose the compression marker.
Changing the setting does not require a rebuild for compatibility: new writes adopt the selected policy, while existing chunks remain readable. All devices are still asked to agree on the remote tweak value so that future writes use one consistent policy. A deliberate remote rebuild can normalise existing storage, but it is optional rather than a prerequisite for synchronisation.
## Execution model
The compression and decompression algorithms are not executed synchronously on Obsidian's UI thread. Browser builds call fflate's asynchronous `deflate` and `inflate` APIs, which create a Web Worker for an operation and terminate it after the callback. The CLI uses the corresponding Node worker-thread path.
The PouchDB transform hook, Base64 detection and conversion, UTF-8 conversion, worker creation, result conversion, and document mutation still run in the calling JavaScript context. The current implementation also creates a separate fflate worker for each attempted chunk rather than reusing LiveSync's persistent splitting and encryption worker pool. `transform-pouch` applies the incoming transform to a bulk batch with `Promise.all`, so a batch can start many of these workers concurrently. It can therefore consume significant total CPU and memory, and worker churn may still affect responsiveness or mobile process limits even though the DEFLATE calculation itself is off the UI thread.
The current benchmark measures the Node CLI process. It does not establish Obsidian UI event-loop latency, Electron renderer responsiveness, mobile WebView memory behaviour, battery use, thermal throttling, or platform watchdog thresholds. Those remain real-runtime validation gaps.
## Reproducible benchmark
The benchmark is implemented under `src/apps/cli/testdeno` and packaged by `test/bench-network`. It uses the real CLI mirror and synchronisation path, Commonlib 0.1.0-rc.4, CouchDB 3.5.0, the V3 RabinKarp splitter, optional Data Compression, and E2EE V2. It runs each of these conditions three times in rotating order:
- E2EE off, compression off;
- E2EE off, compression on;
- E2EE on, compression off; and
- E2EE on, compression on.
The 623,553-byte deterministic fixture contains three Markdown files, two generated JPEG files, two repository PNG files, two JSON files, two TypeScript files, one gzip file, and one high-entropy binary file. Every run materialises and byte-compares all 13 files after synchronisation.
Run it from the repository root:
```bash
BENCH_COMMAND=compression \
BENCH_COMPRESSION_REPEAT_COUNT=3 \
BENCH_COUCHDB_RTT_MS=1 \
docker compose -f test/bench-network/compose.yml run --build --rm bench-runner
```
The latest local three-repeat result was generated on 21st July, 2026. The percentages below compare medians with compression off and on.
| Measurement | E2EE off | E2EE on |
| ------------------------------------------------ | -------: | ------: |
| Stored chunk-data reduction | 9.12% | 9.01% |
| CouchDB external-size reduction | 9.03% | 8.92% |
| CouchDB file-size reduction | 2.57% | 7.62% |
| Upload request-body reduction | 8.58% | 8.61% |
| Complete materialisation response-body reduction | 6.56% | 4.84% |
| Upload wall-time increase | 197.10% | 199.29% |
| Upload CPU-time increase | 650.43% | 581.54% |
| Complete materialisation wall-time increase | 19.58% | 19.75% |
| Complete materialisation CPU-time increase | 45.11% | 43.98% |
With E2EE on, median upload wall time rose from 1.49 seconds to 4.45 seconds, CPU time rose from 1.30 seconds to 8.86 seconds, and upload maximum resident memory rose from about 160 MiB to 403 MiB. The full materialisation workflow starts a CLI process for each file and produced a much higher compressed-path peak of about 983 MiB; treat that figure as evidence about the current CLI workflow rather than a browser decompression lower bound.
The E2EE upload saved 99,034 decoded HTTP body bytes while adding about 2.96 seconds of local processing wall time. A simple serial-transfer estimate puts the wall-time break-even near 0.27 Mbit/s. This estimate excludes headers, contention, request overlap, radio energy, data charges, and server behaviour. The benchmark does not emulate a bandwidth-limited mobile link, so transfer-volume reduction may still be valuable where elapsed time is not the only cost.
## Results by file kind
The E2EE stored chunk-data reductions were:
| Fixture kind | Reduction |
| ------------------- | --------: |
| Markdown | 16.30% |
| JPEG | 4.72% |
| PNG | 6.16% |
| JSON | 72.80% |
| TypeScript | 74.11% |
| gzip | 0% |
| high-entropy binary | 0% |
These results describe payload and chunk characteristics, not a reliable file-extension policy. The JSON and TypeScript fixtures were repetitive and mapped to relatively large chunks. The Markdown files were split into 151 referenced chunks, so small-chunk overhead and limited repetition windows reduced their benefit. JPEG, PNG, and gzip inputs had already undergone format-level compression, while the deterministic binary input was intentionally difficult to compress.
The remote transform sees a content-addressed chunk, not a trustworthy original file type. A chunk can also be deduplicated across files with different extensions. Enabling compression only for selected extensions would therefore either require carrying new provenance into the chunk format or make the representation depend on whichever file first produced a shared chunk. Neither is suitable for the 1.0 format.
## Follow-up optimisation candidates
The first optimisation should remove unbounded per-chunk worker creation:
1. add compression and decompression tasks to a reusable worker pool;
2. run synchronous fflate inside those workers so that it does not create a nested worker for each task;
3. put a bounded scheduler or semaphore before dispatch, rather than submitting the complete `Promise.all` batch at once;
4. transfer derived input and output `ArrayBuffer` objects explicitly in each `postMessage` transfer list; and
5. decide that a result is not smaller inside the worker, so an unhelpful compressed buffer does not need to be returned.
LiveSync's current `bgWorker` is a useful starting point because it creates a fixed pool of approximately half the reported hardware concurrency and selects workers in round-robin order. It is not sufficient unchanged: its `processing` count does not control selection, and it does not limit the number of tasks posted to each worker. Its foreground modules also form a circular dependency: `bgWorker.ts` imports the splitting and encryption adapters, while those adapters import task dispatch and removal from `bgWorker.ts`. Compression must not add another branch to that cycle.
Before adding compression, the Worker code should be separated into a dependency-bottom pool, task registry, scheduler, and transfer transport, with splitting, encryption, and compression implemented as task adapters above it. The shared scheduler should own concurrency limits, cancellation, crash propagation, fairness, and task clean-up. Compression then needs either a separate bounded lane or scheduling which prevents a long level-8 task from starving splitting and encryption work.
The `TransformStream` currently used by `bgWorker.splitting` is local to the calling context; it is not transferred to the Worker. Moving a stream endpoint across contexts could provide real end-to-end back-pressure, but it is not the first memory optimisation for Data Compression. The [Streams Standard transfer algorithm](https://streams.spec.whatwg.org/#transferable-streams) transfers the stream endpoint and posts each chunk through an internal `MessagePort` with an empty transfer list. Binary chunks can therefore still be cloned. Explicitly transferring each `ArrayBuffer` is the clearer zero-copy boundary for independent chunk tasks.
Cross-context transferable streams also cannot yet be required by the supported mobile baseline. WebKit lists `ReadableStream`, `WritableStream`, and `TransformStream` transfer via `postMessage()` as a [Safari 27 beta addition](https://webkit.org/blog/17967/news-from-wwdc26-webkit-in-safari-27-beta/#readablestream-improvements). A future general `bgWorker` redesign may use transferable streams behind capability detection, particularly where back-pressure matters more than copying, but it needs a message-based fallback.
After bounding worker use, adaptive per-chunk compression can reduce unnecessary work without depending on the source filename:
1. skip very small chunks where the marker and worker start-up dominate;
2. sample decoded bytes and estimate entropy or repetition before starting level-8 DEFLATE;
3. require both a minimum byte saving and a minimum percentage saving;
4. compare lower DEFLATE levels against level 8; and
5. replace one-worker-per-chunk operation with a bounded, persistent compression worker pool and back-pressure.
Any optimisation must preserve the current wire contract: content may remain uncompressed, the compressed marker must stay readable, compression must precede E2EE, and mixed representations must interoperate. It should be evaluated with the same four-condition benchmark, a bandwidth-shaped case, UI event-loop latency, and at least one real mobile Obsidian run before reconsidering the default.
+1 -1
View File
@@ -11,7 +11,7 @@ authors:
# Peer-to-Peer Synchronisation Tips
> [!IMPORTANT]
> Peer-to-peer synchronisation is still an experimental feature. Although we have made every effort to ensure its reliability, it may not function correctly in all environments.
> Peer-to-peer synchronisation is a supported opt-in feature. WebRTC connectivity still depends on the networks, relays, and optional TURN servers available to every device, so a working connection cannot be guaranteed in every environment.
## Difficulties with Peer-to-Peer Synchronisation
+603
View File
@@ -0,0 +1,603 @@
import { join } from "@std/path";
import { TempDir } from "./helpers/temp.ts";
import { applyRemoteSyncSettings, initSettingsFile } from "./helpers/settings.ts";
import { assertFilesEqual } from "./helpers/cli.ts";
import { runMeasuredCliOrFail, type CliProcessMeasurement } from "./helpers/measuredCli.ts";
import { createCouchdbDatabase, startCouchdb, stopCouchdb } from "./helpers/docker.ts";
import {
createCompressionBenchmarkDataset,
type CompressionDataset,
type CompressionDatasetEntry,
} from "./helpers/compressionDataset.ts";
import { computeDatasetDigestSha256 } from "./helpers/benchmarkVerification.ts";
import { startCouchdbProxy, type CouchdbProxyCounters } from "./bench-couchdb.ts";
import type { DatasetKind } from "./helpers/dataset.ts";
type CompressionCondition = {
name: string;
encrypt: boolean;
enableCompression: boolean;
};
type CouchDbSizes = {
file: number;
external: number;
active: number;
};
type PerKindRemoteMeasurement = {
sourceFiles: number;
sourceBytes: number;
mappedFiles: number;
uniqueReferencedChunks: number;
storedChunkDataBytes: number;
storedChunkJsonBytes: number;
};
type RemoteMeasurement = {
couchdbSizes: CouchDbSizes;
documentCount: number;
chunkDocumentCount: number;
metadataDocumentCount: number;
compressedMarkerCount: number;
encryptedChunkCount: number;
storedChunkDataBytes: number;
storedChunkJsonBytes: number;
perKind: Record<DatasetKind, PerKindRemoteMeasurement>;
};
type RunResult = {
condition: CompressionCondition;
repeatIndex: number;
executionOrder: number;
databaseName: string;
datasetDigestSha256: string;
dataset: {
totalFiles: number;
totalBytes: number;
filesByKind: CompressionDataset["filesByKind"];
bytesByKind: CompressionDataset["bytesByKind"];
jpegGenerator: string;
};
effectiveSettings: Record<string, unknown>;
mirror: CliProcessMeasurement;
upload: CliProcessMeasurement & { http: CouchdbProxyCounters };
download: CliProcessMeasurement & { http: CouchdbProxyCounters };
materialisation: CliProcessMeasurement & { http: CouchdbProxyCounters };
verification: {
verifiedFiles: number;
complete: boolean;
};
remote: RemoteMeasurement;
};
const CONDITIONS: CompressionCondition[] = [
{ name: "plain", encrypt: false, enableCompression: false },
{ name: "plain-compressed", encrypt: false, enableCompression: true },
{ name: "e2ee", encrypt: true, enableCompression: false },
{ name: "e2ee-compressed", encrypt: true, enableCompression: true },
];
const DATASET_KINDS: DatasetKind[] = ["md", "jpg", "png", "json", "ts", "gz", "bin"];
const COMPRESSED_MARKER = "\u000eLZ\u001d";
function readEnvString(name: string, fallback: string): string {
const value = Deno.env.get(name)?.trim();
return value ? value : fallback;
}
function readEnvPositiveInteger(name: string, fallback: number): number {
const raw = Deno.env.get(name)?.trim();
if (!raw) return fallback;
const value = Number(raw);
if (!Number.isInteger(value) || value <= 0) {
throw new Error(`${name} must be a positive integer, got '${raw}'`);
}
return value;
}
function readEnvPositiveNumber(name: string, fallback: number): number {
const raw = Deno.env.get(name)?.trim();
if (!raw) return fallback;
const value = Number(raw);
if (!Number.isFinite(value) || value <= 0) {
throw new Error(`${name} must be positive, got '${raw}'`);
}
return value;
}
function readEnvBoolean(name: string, fallback: boolean): boolean {
const raw = Deno.env.get(name)?.trim();
if (!raw) return fallback;
return /^(1|true|yes|on)$/i.test(raw);
}
function byteLength(value: string): number {
return new TextEncoder().encode(value).byteLength;
}
function median(values: number[]): number {
if (values.length === 0) return 0;
const sorted = [...values].sort((a, b) => a - b);
const middle = Math.floor(sorted.length / 2);
if (sorted.length % 2 === 1) return sorted[middle];
return (sorted[middle - 1] + sorted[middle]) / 2;
}
function rounded(value: number): number {
return Number(value.toFixed(4));
}
function deltaPercent(enabled: number, disabled: number): number | null {
if (disabled === 0) return null;
return rounded(((enabled - disabled) / disabled) * 100);
}
function reductionPercent(enabled: number, disabled: number): number | null {
if (disabled === 0) return null;
return rounded((1 - enabled / disabled) * 100);
}
function basicAuth(user: string, password: string): string {
return `Basic ${btoa(`${user}:${password}`)}`;
}
async function couchRequest(
baseUri: string,
user: string,
password: string,
path: string,
init: RequestInit = {},
allowedStatuses: number[] = []
): Promise<Response> {
const response = await fetch(`${baseUri.replace(/\/$/, "")}${path}`, {
...init,
headers: {
Authorization: basicAuth(user, password),
...(init.method === "POST" || init.body ? { "Content-Type": "application/json" } : {}),
...init.headers,
},
});
if (!response.ok && !allowedStatuses.includes(response.status)) {
throw new Error(`${init.method ?? "GET"} ${path}: ${response.status} ${await response.text()}`);
}
return response;
}
async function deleteDatabase(baseUri: string, user: string, password: string, databaseName: string): Promise<void> {
const response = await couchRequest(
baseUri,
user,
password,
`/${encodeURIComponent(databaseName)}`,
{ method: "DELETE" },
[404]
);
await response.body?.cancel().catch(() => {});
}
function blankPerKind(dataset: CompressionDataset): Record<DatasetKind, PerKindRemoteMeasurement> {
return Object.fromEntries(
DATASET_KINDS.map((kind) => [
kind,
{
sourceFiles: dataset.filesByKind[kind],
sourceBytes: dataset.bytesByKind[kind],
mappedFiles: 0,
uniqueReferencedChunks: 0,
storedChunkDataBytes: 0,
storedChunkJsonBytes: 0,
},
])
) as Record<DatasetKind, PerKindRemoteMeasurement>;
}
function findDatasetEntry(path: unknown, entries: CompressionDatasetEntry[]): CompressionDatasetEntry | undefined {
if (typeof path !== "string") return undefined;
return entries.find((entry) => path === entry.relativePath || path.endsWith(`/${entry.relativePath}`));
}
async function inspectRemoteDatabase(options: {
baseUri: string;
user: string;
password: string;
databaseName: string;
dataset: CompressionDataset;
}): Promise<RemoteMeasurement> {
const dbPath = `/${encodeURIComponent(options.databaseName)}`;
await couchRequest(options.baseUri, options.user, options.password, `${dbPath}/_ensure_full_commit`, {
method: "POST",
body: "{}",
});
const [info, allDocs] = await Promise.all([
couchRequest(options.baseUri, options.user, options.password, dbPath).then((response) => response.json()),
couchRequest(options.baseUri, options.user, options.password, `${dbPath}/_all_docs?include_docs=true`).then(
(response) => response.json()
),
]);
const rows = (allDocs as { rows?: Array<{ doc?: Record<string, unknown> }> }).rows ?? [];
const docs = rows.flatMap((row) => (row.doc ? [row.doc] : []));
const docsById = new Map(docs.flatMap((doc) => (typeof doc._id === "string" ? [[doc._id, doc] as const] : [])));
const metadataDocs = docs.filter((doc) => Array.isArray(doc.children) && typeof doc.path === "string");
const chunkDocs = docs.filter(
(doc) =>
(doc.type === "leaf" || doc.type === "chunkpack") &&
typeof doc.data === "string" &&
typeof doc._id === "string"
);
const perKind = blankPerKind(options.dataset);
const chunkIdsByKind = new Map(DATASET_KINDS.map((kind) => [kind, new Set<string>()] as const));
for (const doc of metadataDocs) {
const entry = findDatasetEntry(doc.path, options.dataset.entries);
if (!entry) continue;
perKind[entry.kind].mappedFiles += 1;
for (const child of doc.children as unknown[]) {
if (typeof child === "string") chunkIdsByKind.get(entry.kind)!.add(child);
}
}
for (const kind of DATASET_KINDS) {
const chunkIds = chunkIdsByKind.get(kind)!;
perKind[kind].uniqueReferencedChunks = chunkIds.size;
for (const chunkId of chunkIds) {
const chunk = docsById.get(chunkId);
if (!chunk || typeof chunk.data !== "string") continue;
perKind[kind].storedChunkDataBytes += byteLength(chunk.data);
perKind[kind].storedChunkJsonBytes += byteLength(JSON.stringify(chunk));
}
}
const sizeInfo = (info as { sizes?: Partial<CouchDbSizes>; doc_count?: number }).sizes ?? {};
return {
couchdbSizes: {
file: sizeInfo.file ?? 0,
external: sizeInfo.external ?? 0,
active: sizeInfo.active ?? 0,
},
documentCount: (info as { doc_count?: number }).doc_count ?? docs.length,
chunkDocumentCount: chunkDocs.length,
metadataDocumentCount: metadataDocs.length,
compressedMarkerCount: chunkDocs.filter(
(doc) => typeof doc.data === "string" && doc.data.startsWith(COMPRESSED_MARKER)
).length,
encryptedChunkCount: chunkDocs.filter((doc) => doc.e_ === true).length,
storedChunkDataBytes: chunkDocs.reduce(
(sum, doc) => sum + (typeof doc.data === "string" ? byteLength(doc.data) : 0),
0
),
storedChunkJsonBytes: chunkDocs.reduce((sum, doc) => sum + byteLength(JSON.stringify(doc)), 0),
perKind,
};
}
async function verifyDataset(
workDir: TempDir,
vaultB: string,
settingsB: string,
entries: CompressionDatasetEntry[]
): Promise<CliProcessMeasurement> {
const started = performance.now();
const measurements: CliProcessMeasurement[] = [];
for (const entry of entries) {
const pulledPath = workDir.join(`verify-${entry.kind}-${entry.relativePath.split("/").at(-1)}`);
measurements.push(
await runMeasuredCliOrFail(vaultB, "--settings", settingsB, "pull", entry.relativePath, pulledPath)
);
await assertFilesEqual(entry.absolutePath, pulledPath, `compression benchmark mismatch: ${entry.relativePath}`);
}
const elapsedMs = performance.now() - started;
const userCpuMs = measurements.reduce((sum, measurement) => sum + measurement.userCpuMs, 0);
const systemCpuMs = measurements.reduce((sum, measurement) => sum + measurement.systemCpuMs, 0);
const totalCpuMs = userCpuMs + systemCpuMs;
return {
elapsedMs: rounded(elapsedMs),
userCpuMs: rounded(userCpuMs),
systemCpuMs: rounded(systemCpuMs),
totalCpuMs: rounded(totalCpuMs),
cpuToWallRatio: rounded(totalCpuMs / elapsedMs),
maxResidentSetKiB: Math.max(...measurements.map((measurement) => measurement.maxResidentSetKiB)),
};
}
function summariseResults(results: RunResult[]) {
const byCondition = Object.fromEntries(
CONDITIONS.map((condition) => {
const runs = results.filter((result) => result.condition.name === condition.name);
return [
condition.name,
{
repeats: runs.length,
remoteStoredChunkDataBytesMedian: median(runs.map((run) => run.remote.storedChunkDataBytes)),
couchdbExternalBytesMedian: median(runs.map((run) => run.remote.couchdbSizes.external)),
couchdbFileBytesMedian: median(runs.map((run) => run.remote.couchdbSizes.file)),
uploadRequestBodyBytesMedian: median(runs.map((run) => run.upload.http.requestBodyBytes)),
uploadResponseBodyBytesMedian: median(runs.map((run) => run.upload.http.responseBodyBytes)),
downloadRequestBodyBytesMedian: median(runs.map((run) => run.download.http.requestBodyBytes)),
downloadResponseBodyBytesMedian: median(runs.map((run) => run.download.http.responseBodyBytes)),
uploadElapsedMsMedian: median(runs.map((run) => run.upload.elapsedMs)),
uploadCpuMsMedian: median(runs.map((run) => run.upload.totalCpuMs)),
downloadElapsedMsMedian: median(runs.map((run) => run.download.elapsedMs)),
downloadCpuMsMedian: median(runs.map((run) => run.download.totalCpuMs)),
materialisationElapsedMsMedian: median(runs.map((run) => run.materialisation.elapsedMs)),
materialisationCpuMsMedian: median(runs.map((run) => run.materialisation.totalCpuMs)),
materialisationResponseBodyBytesMedian: median(
runs.map((run) => run.materialisation.http.responseBodyBytes)
),
completeDownloadResponseBodyBytesMedian: median(
runs.map(
(run) => run.download.http.responseBodyBytes + run.materialisation.http.responseBodyBytes
)
),
completeDownloadElapsedMsMedian: median(
runs.map((run) => run.download.elapsedMs + run.materialisation.elapsedMs)
),
completeDownloadCpuMsMedian: median(
runs.map((run) => run.download.totalCpuMs + run.materialisation.totalCpuMs)
),
maxResidentSetKiBMedian: median(
runs.map((run) =>
Math.max(
run.upload.maxResidentSetKiB,
run.download.maxResidentSetKiB,
run.materialisation.maxResidentSetKiB
)
)
),
perKindStoredChunkDataBytesMedian: Object.fromEntries(
DATASET_KINDS.map((kind) => [
kind,
median(runs.map((run) => run.remote.perKind[kind].storedChunkDataBytes)),
])
),
},
];
})
);
const comparisons = [false, true].map((encrypt) => {
const disabledName = encrypt ? "e2ee" : "plain";
const enabledName = encrypt ? "e2ee-compressed" : "plain-compressed";
const disabled = byCondition[disabledName] as Record<string, unknown>;
const enabled = byCondition[enabledName] as Record<string, unknown>;
const disabledPerKind = disabled.perKindStoredChunkDataBytesMedian as Record<DatasetKind, number>;
const enabledPerKind = enabled.perKindStoredChunkDataBytesMedian as Record<DatasetKind, number>;
return {
encrypt,
disabledCondition: disabledName,
enabledCondition: enabledName,
storedChunkDataReductionPercent: reductionPercent(
enabled.remoteStoredChunkDataBytesMedian as number,
disabled.remoteStoredChunkDataBytesMedian as number
),
couchdbExternalReductionPercent: reductionPercent(
enabled.couchdbExternalBytesMedian as number,
disabled.couchdbExternalBytesMedian as number
),
couchdbFileReductionPercent: reductionPercent(
enabled.couchdbFileBytesMedian as number,
disabled.couchdbFileBytesMedian as number
),
uploadRequestBodyReductionPercent: reductionPercent(
enabled.uploadRequestBodyBytesMedian as number,
disabled.uploadRequestBodyBytesMedian as number
),
completeDownloadResponseBodyReductionPercent: reductionPercent(
enabled.completeDownloadResponseBodyBytesMedian as number,
disabled.completeDownloadResponseBodyBytesMedian as number
),
uploadElapsedDeltaPercent: deltaPercent(
enabled.uploadElapsedMsMedian as number,
disabled.uploadElapsedMsMedian as number
),
uploadCpuDeltaPercent: deltaPercent(
enabled.uploadCpuMsMedian as number,
disabled.uploadCpuMsMedian as number
),
downloadElapsedDeltaPercent: deltaPercent(
enabled.downloadElapsedMsMedian as number,
disabled.downloadElapsedMsMedian as number
),
downloadCpuDeltaPercent: deltaPercent(
enabled.downloadCpuMsMedian as number,
disabled.downloadCpuMsMedian as number
),
completeDownloadElapsedDeltaPercent: deltaPercent(
enabled.completeDownloadElapsedMsMedian as number,
disabled.completeDownloadElapsedMsMedian as number
),
completeDownloadCpuDeltaPercent: deltaPercent(
enabled.completeDownloadCpuMsMedian as number,
disabled.completeDownloadCpuMsMedian as number
),
perKindStoredChunkDataReductionPercent: Object.fromEntries(
DATASET_KINDS.map((kind) => [kind, reductionPercent(enabledPerKind[kind], disabledPerKind[kind])])
),
};
});
return { byCondition, comparisons };
}
async function main(): Promise<void> {
const backendUri = readEnvString("BENCH_COUCHDB_BACKEND_URI", "http://127.0.0.1:5989");
const proxyUri = readEnvString("BENCH_COUCHDB_URI", "http://127.0.0.1:15989");
const user = readEnvString("BENCH_COUCHDB_USER", readEnvString("username", "admin"));
const password = readEnvString("BENCH_COUCHDB_PASSWORD", readEnvString("password", "password"));
const databasePrefix = readEnvString("BENCH_COUCHDB_DBNAME", `compression-bench-${Date.now()}`);
const repeatCount = readEnvPositiveInteger("BENCH_COMPRESSION_REPEAT_COUNT", 1);
const requestedRttMs = readEnvPositiveNumber("BENCH_COUCHDB_RTT_MS", 1);
const managedCouchdb = readEnvBoolean("BENCH_COUCHDB_MANAGED", true);
const passphrase = readEnvString("BENCH_PASSPHRASE", "compression-benchmark-passphrase");
const resultRoot = readEnvString("BENCH_COMPRESSION_RESULT_ROOT", "bench-results");
const resultPath =
Deno.env.get("BENCH_COMPRESSION_RESULT_JSON")?.trim() ||
join(resultRoot, `compression-${new Date().toISOString().replaceAll(":", "-")}.json`);
const createdDatabases = new Set<string>();
const results: RunResult[] = [];
let managedStarted = false;
await Deno.mkdir(resultRoot, { recursive: true });
const proxy = startCouchdbProxy({ backendUri, proxyUri, requestedRttMs });
try {
for (let repeatIndex = 1; repeatIndex <= repeatCount; repeatIndex++) {
const rotation = (repeatIndex - 1) % CONDITIONS.length;
const orderedConditions = [...CONDITIONS.slice(rotation), ...CONDITIONS.slice(0, rotation)];
for (const [executionOffset, condition] of orderedConditions.entries()) {
const databaseName = `${databasePrefix}-${repeatIndex}-${condition.name}`.toLowerCase();
if (managedCouchdb && !managedStarted) {
await startCouchdb(backendUri, user, password, databaseName);
managedStarted = true;
} else {
await createCouchdbDatabase(backendUri, user, password, databaseName);
}
createdDatabases.add(databaseName);
await using workDir = await TempDir.create(`livesync-compression-${condition.name}`);
const vaultA = workDir.join("vault-a");
const vaultB = workDir.join("vault-b");
const settingsA = workDir.join("settings-a.json");
const settingsB = workDir.join("settings-b.json");
await Deno.mkdir(vaultA, { recursive: true });
await Deno.mkdir(vaultB, { recursive: true });
await initSettingsFile(settingsA);
await initSettingsFile(settingsB);
await Promise.all(
[settingsA, settingsB].map((settingsFile) =>
applyRemoteSyncSettings(settingsFile, {
remoteType: "COUCHDB",
couchdbUri: proxyUri,
couchdbUser: user,
couchdbPassword: password,
couchdbDbname: databaseName,
encrypt: condition.encrypt,
passphrase,
enableCompression: condition.enableCompression,
usePathObfuscation: false,
})
)
);
const dataset = await createCompressionBenchmarkDataset({ rootDir: vaultA });
const datasetDigestSha256 = await computeDatasetDigestSha256(dataset.entries);
const mirror = await runMeasuredCliOrFail(vaultA, "--settings", settingsA, "mirror");
proxy.resetCounters();
const uploadMeasurement = await runMeasuredCliOrFail(vaultA, "--settings", settingsA, "sync");
const upload = { ...uploadMeasurement, http: proxy.snapshotCounters() };
proxy.resetCounters();
const downloadMeasurement = await runMeasuredCliOrFail(vaultB, "--settings", settingsB, "sync");
const download = { ...downloadMeasurement, http: proxy.snapshotCounters() };
proxy.resetCounters();
const materialisationMeasurement = await verifyDataset(workDir, vaultB, settingsB, dataset.entries);
const materialisation = {
...materialisationMeasurement,
http: proxy.snapshotCounters(),
};
const remote = await inspectRemoteDatabase({
baseUri: backendUri,
user,
password,
databaseName,
dataset,
});
const settings = JSON.parse(await Deno.readTextFile(settingsA)) as Record<string, unknown>;
const effectiveSettings = Object.fromEntries(
[
"encrypt",
"enableCompression",
"E2EEAlgorithm",
"usePathObfuscation",
"chunkSplitterVersion",
"customChunkSize",
"minimumChunkSize",
"hashAlg",
].map((key) => [key, settings[key]])
);
results.push({
condition,
repeatIndex,
executionOrder: executionOffset + 1,
databaseName,
datasetDigestSha256,
dataset: {
totalFiles: dataset.totalFiles,
totalBytes: dataset.totalBytes,
filesByKind: dataset.filesByKind,
bytesByKind: dataset.bytesByKind,
jpegGenerator: dataset.jpegGenerator,
},
effectiveSettings,
mirror,
upload,
download,
materialisation,
verification: { verifiedFiles: dataset.entries.length, complete: true },
remote,
});
await deleteDatabase(backendUri, user, password, databaseName);
createdDatabases.delete(databaseName);
console.error(
`[Compression benchmark] repeat ${repeatIndex}/${repeatCount} ${condition.name}: ` +
`${dataset.totalFiles} files, ${remote.chunkDocumentCount} chunks, ` +
`${remote.storedChunkDataBytes} stored chunk-data bytes`
);
}
}
const output = {
schemaVersion: 1,
mode: "couchdb-cli-compression-benchmark",
generatedAt: new Date().toISOString(),
commonlibVersion: (
JSON.parse(
await Deno.readTextFile(
join(import.meta.dirname!, "../../../../node_modules/@vrtmrz/livesync-commonlib/package.json")
)
) as { version: string }
).version,
couchdbVersion: (
(await couchRequest(backendUri, user, password, "/").then((response) => response.json())) as {
version?: string;
}
).version,
compressionImplementation: "Commonlib replicationFilter using fflate level 8 before E2EE V2",
chunkingImplementation: "LiveSync CLI mirror using the effective settings recorded for each run",
requestedRttMs,
httpByteScope:
"Decoded HTTP request and response body bytes observed by the local proxy; headers are excluded.",
limitations: [
"Synthetic JPEGs exercise a deterministic image-like fixture but are not a photographic corpus.",
"PNG, Markdown, JSON, and TypeScript inputs are current repository files and therefore change with the source tree.",
"Wall and CPU times include CLI process start-up; compare repeated medians rather than treating one run as a universal result.",
"Full materialisation starts one CLI process per file and can repeat lazy chunk fetches; treat it as a CLI workflow measurement rather than a raw download lower bound.",
"The benchmark uses a local CouchDB and a fixed latency proxy, not a contended production server or a real WAN.",
"Path obfuscation is explicitly disabled so raw metadata can be mapped back to file kinds.",
],
repeatCount,
conditions: CONDITIONS,
summary: summariseResults(results),
runs: results,
};
await Deno.writeTextFile(resultPath, JSON.stringify(output, null, 2));
console.log(JSON.stringify(output, null, 2));
console.error(`[Compression benchmark] wrote ${resultPath}`);
} finally {
await proxy.stop();
for (const databaseName of createdDatabases) {
await deleteDatabase(backendUri, user, password, databaseName).catch((error) => console.error(error));
}
if (managedCouchdb && managedStarted) {
await stopCouchdb().catch(() => {});
}
}
}
if (import.meta.main) {
main().catch((error) => {
console.error("[Compression benchmark fatal]", error);
Deno.exit(1);
});
}
+22
View File
@@ -188,11 +188,19 @@ function readOptionalResultPath(): string | undefined {
export type CouchdbProxyHandle = {
stop: () => Promise<void>;
resetCounters: () => void;
snapshotCounters: () => CouchdbProxyCounters;
applied: boolean;
note: string;
directionalDelayMs: number;
};
export type CouchdbProxyCounters = {
requestCount: number;
requestBodyBytes: number;
responseBodyBytes: number;
};
export function startCouchdbProxy(
options: {
backendUri: string;
@@ -208,6 +216,11 @@ export function startCouchdbProxy(
((milliseconds: number) =>
new Promise<void>((resolve) => setTimeout(resolve, milliseconds)));
const controller = new AbortController();
const counters: CouchdbProxyCounters = {
requestCount: 0,
requestBodyBytes: 0,
responseBodyBytes: 0,
};
const listener = Deno.serve(
{
@@ -238,6 +251,8 @@ export function startCouchdbProxy(
requestBody = undefined;
}
}
counters.requestCount += 1;
counters.requestBodyBytes += requestBody?.byteLength ?? 0;
const upstream = await fetch(targetUrl, {
method: request.method,
@@ -249,6 +264,7 @@ export function startCouchdbProxy(
const responseHeaders = new Headers(upstream.headers);
responseHeaders.delete("content-length");
const responseBody = await upstream.arrayBuffer();
counters.responseBodyBytes += responseBody.byteLength;
await delay(halfDelayMs);
return new Response(responseBody, {
@@ -264,6 +280,12 @@ export function startCouchdbProxy(
directionalDelayMs: halfDelayMs,
note:
`local reverse proxy on ${proxy.origin} with ${halfDelayMs}ms request-path and ${halfDelayMs}ms response-path delay`,
resetCounters: () => {
counters.requestCount = 0;
counters.requestBodyBytes = 0;
counters.responseBodyBytes = 0;
},
snapshotCounters: () => ({ ...counters }),
stop: async () => {
controller.abort();
await listener.finished.catch(() => {});
+1
View File
@@ -23,6 +23,7 @@
"test:benchmark-contract": "deno test --env-file=.test.env -A --no-check test-benchmark-contract.ts",
"bench:p2p": "deno run --env-file=.test.env -A --no-check bench-p2p.ts",
"bench:couchdb": "deno run --env-file=.test.env -A --no-check bench-couchdb.ts",
"bench:compression": "deno run --env-file=.test.env -A --no-check bench-compression.ts",
"bench:cases": "deno run --env-file=.test.env -A --no-check bench-network-cases.ts",
"bench:latency-sweep": "deno run --env-file=.test.env -A --no-check bench-latency-sweep.ts",
"bench:p2p-split-node": "deno run --env-file=.test.env -A --no-check bench-p2p-split-node.ts",
+1 -1
View File
@@ -7,7 +7,7 @@ import { join } from "@std/path";
// CLI root (src/apps/cli/) is two levels up.
// import.meta.dirname is available in Deno 1.40+ as an OS-native path string.
export const CLI_DIR: string = join(import.meta.dirname!, "..", "..");
const CLI_DIST = join(CLI_DIR, "dist", "index.cjs");
export const CLI_DIST = join(CLI_DIR, "dist", "index.cjs");
// ---------------------------------------------------------------------------
// Result type
@@ -0,0 +1,188 @@
import { join } from "@std/path";
import type { DatasetEntry, DatasetKind } from "./dataset.ts";
export type CompressionDatasetEntry = DatasetEntry & {
source: string;
};
export type CompressionDataset = {
entries: CompressionDatasetEntry[];
totalFiles: number;
totalBytes: number;
bytesByKind: Record<DatasetKind, number>;
filesByKind: Record<DatasetKind, number>;
jpegGenerator: string;
};
export type JpegEncoder = (inputPpm: string, outputJpeg: string) => Promise<string>;
const ALL_KINDS: DatasetKind[] = ["md", "jpg", "png", "json", "ts", "gz", "bin"];
const REPOSITORY_ROOT = join(import.meta.dirname!, "..", "..", "..", "..", "..");
function fnv1a32(input: string): number {
let hash = 0x811c9dc5;
for (let i = 0; i < input.length; i++) {
hash ^= input.charCodeAt(i) & 0xff;
hash = Math.imul(hash, 0x01000193);
}
return hash >>> 0;
}
function createXorshift32(seed: number): () => number {
let state = seed || 0x9e3779b9;
return () => {
state ^= state << 13;
state ^= state >>> 17;
state ^= state << 5;
return state >>> 0;
};
}
function createSyntheticPpm(width: number, height: number, seed: string, textured: boolean): Uint8Array {
const header = new TextEncoder().encode(`P6\n${width} ${height}\n255\n`);
const pixels = new Uint8Array(width * height * 3);
const nextRandom = createXorshift32(fnv1a32(seed));
for (let y = 0; y < height; y++) {
for (let x = 0; x < width; x++) {
const offset = (y * width + x) * 3;
const noise = textured ? (nextRandom() & 0x3f) - 32 : 0;
pixels[offset] = Math.max(0, Math.min(255, Math.floor((x * 255) / (width - 1)) + noise));
pixels[offset + 1] = Math.max(0, Math.min(255, Math.floor((y * 255) / (height - 1)) + noise));
pixels[offset + 2] = Math.max(0, Math.min(255, Math.floor(((x + y) * 255) / (width + height - 2)) - noise));
}
}
const result = new Uint8Array(header.length + pixels.length);
result.set(header);
result.set(pixels, header.length);
return result;
}
async function gzip(input: Uint8Array): Promise<Uint8Array> {
const copied = new Uint8Array(input.byteLength);
copied.set(input);
const stream = new Blob([copied.buffer]).stream().pipeThrough(new CompressionStream("gzip"));
return new Uint8Array(await new Response(stream).arrayBuffer());
}
export async function encodeJpegWithCjpeg(inputPpm: string, outputJpeg: string): Promise<string> {
const command = new Deno.Command("cjpeg", {
args: ["-quality", "85", "-optimize", "-outfile", outputJpeg, inputPpm],
stdin: "null",
stdout: "piped",
stderr: "piped",
});
let result: Deno.CommandOutput;
try {
result = await command.output();
} catch (error) {
if (error instanceof Deno.errors.NotFound) {
throw new Error(
"cjpeg is required for the compression benchmark. Use the Compose runner or install libjpeg tools."
);
}
throw error;
}
if (!result.success) {
throw new Error(`cjpeg failed: ${new TextDecoder().decode(result.stderr)}`);
}
return "cjpeg quality=85, optimise=true, synthetic PPM 640x480";
}
async function writeRandomBinary(path: string, size: number, seed: string): Promise<void> {
const bytes = new Uint8Array(size);
const nextRandom = createXorshift32(fnv1a32(seed));
for (let index = 0; index < bytes.length; index++) {
bytes[index] = nextRandom() & 0xff;
}
await Deno.writeFile(path, bytes);
}
export async function createCompressionBenchmarkDataset(options: {
rootDir: string;
datasetDirName?: string;
repositoryRoot?: string;
seed?: string;
jpegEncoder?: JpegEncoder;
}): Promise<CompressionDataset> {
const datasetDirName = options.datasetDirName ?? "compression-benchmark";
const repositoryRoot = options.repositoryRoot ?? REPOSITORY_ROOT;
const seed = options.seed ?? "livesync-compression-benchmark";
const jpegEncoder = options.jpegEncoder ?? encodeJpegWithCjpeg;
const datasetRoot = join(options.rootDir, datasetDirName);
const entries: CompressionDatasetEntry[] = [];
let jpegGenerator = "";
for (const kind of ALL_KINDS) {
await Deno.mkdir(join(datasetRoot, kind), { recursive: true });
}
const addFile = async (kind: DatasetKind, absolutePath: string, source: string) => {
const relativePath = absolutePath
.slice(options.rootDir.length + 1)
.split("\\")
.join("/");
const size = (await Deno.stat(absolutePath)).size;
entries.push({ kind, relativePath, absolutePath, size, source });
};
const copyRepositoryFile = async (kind: DatasetKind, sourcePath: string, targetName: string) => {
const destination = join(datasetRoot, kind, targetName);
await Deno.copyFile(join(repositoryRoot, sourcePath), destination);
await addFile(kind, destination, sourcePath);
};
await copyRepositoryFile("md", "docs/settings.md", "settings.md");
await copyRepositoryFile("md", "docs/quick_setup.md", "quick-setup.md");
await copyRepositoryFile("md", "updates.md", "updates.md");
await copyRepositoryFile("png", "instruction_images/cloudant_1.png", "cloudant-1.png");
await copyRepositoryFile(
"png",
"images/quick-setup/guide-quick-setup-first-setup-uri.png",
"quick-setup-first-setup-uri.png"
);
await copyRepositoryFile("json", "package.json", "package.json");
await copyRepositoryFile("json", "manifest.json", "manifest.json");
await copyRepositoryFile("ts", "src/modules/core/ModuleReplicator.ts", "ModuleReplicator.ts");
await copyRepositoryFile("ts", "src/modules/core/ReplicateResultProcessor.ts", "ReplicateResultProcessor.ts");
const markdownBytes = await Deno.readFile(join(repositoryRoot, "docs/settings.md"));
const gzipPath = join(datasetRoot, "gz", "settings.md.gz");
await Deno.writeFile(gzipPath, await gzip(markdownBytes));
await addFile("gz", gzipPath, "generated gzip of docs/settings.md");
const randomPath = join(datasetRoot, "bin", "deterministic-random.bin");
await writeRandomBinary(randomPath, 256 * 1024, seed);
await addFile("bin", randomPath, `deterministic xorshift32 seed=${seed}`);
for (const [name, textured] of [
["smooth-gradient.jpg", false],
["textured-gradient.jpg", true],
] as const) {
const ppmPath = await Deno.makeTempFile({ dir: options.rootDir, prefix: "compression-jpeg-", suffix: ".ppm" });
const jpegPath = join(datasetRoot, "jpg", name);
try {
await Deno.writeFile(ppmPath, createSyntheticPpm(640, 480, `${seed}-${name}`, textured));
jpegGenerator = await jpegEncoder(ppmPath, jpegPath);
} finally {
await Deno.remove(ppmPath).catch(() => {});
}
await addFile("jpg", jpegPath, `${jpegGenerator}; ${textured ? "textured" : "smooth"}`);
}
const bytesByKind = Object.fromEntries(ALL_KINDS.map((kind) => [kind, 0])) as Record<DatasetKind, number>;
const filesByKind = Object.fromEntries(ALL_KINDS.map((kind) => [kind, 0])) as Record<DatasetKind, number>;
for (const entry of entries) {
bytesByKind[entry.kind] += entry.size;
filesByKind[entry.kind] += 1;
}
return {
entries,
totalFiles: entries.length,
totalBytes: entries.reduce((sum, entry) => sum + entry.size, 0),
bytesByKind,
filesByKind,
jpegGenerator,
};
}
+3 -1
View File
@@ -9,8 +9,10 @@ export type DeterministicDatasetConfig = {
binSizeBytes: number;
};
export type DatasetKind = "md" | "jpg" | "png" | "json" | "ts" | "gz" | "bin";
export type DatasetEntry = {
kind: "md" | "bin";
kind: DatasetKind;
relativePath: string;
absolutePath: string;
size: number;
@@ -0,0 +1,61 @@
import { CLI_DIR, CLI_DIST } from "./cli.ts";
export type CliProcessMeasurement = {
elapsedMs: number;
userCpuMs: number;
systemCpuMs: number;
totalCpuMs: number;
cpuToWallRatio: number;
maxResidentSetKiB: number;
};
const MARKER = "__LIVESYNC_GNU_TIME__";
export async function runMeasuredCliOrFail(...args: string[]): Promise<CliProcessMeasurement> {
const started = performance.now();
let output: Deno.CommandOutput;
try {
output = await new Deno.Command("/usr/bin/time", {
args: ["-f", `${MARKER}%U\t%S\t%M`, "node", CLI_DIST, ...args],
cwd: CLI_DIR,
stdin: "null",
stdout: "piped",
stderr: "piped",
}).output();
} catch (error) {
if (error instanceof Deno.errors.NotFound) {
throw new Error(
"GNU /usr/bin/time is required for the compression benchmark. Use the Compose runner or install GNU time."
);
}
throw error;
}
const elapsedMs = performance.now() - started;
const stderr = new TextDecoder().decode(output.stderr);
const stdout = new TextDecoder().decode(output.stdout);
const measurementLine = stderr.split(/\r?\n/).find((line) => line.startsWith(MARKER));
if (!output.success) {
throw new Error(`CLI exited with code ${output.code}\nstdout: ${stdout}\nstderr: ${stderr}`);
}
if (!measurementLine) {
throw new Error(`GNU time did not emit the expected measurement marker\nstderr: ${stderr}`);
}
const [userSeconds, systemSeconds, maxResidentSetKiB] = measurementLine
.slice(MARKER.length)
.split("\t")
.map(Number);
if (![userSeconds, systemSeconds, maxResidentSetKiB].every(Number.isFinite)) {
throw new Error(`Could not parse GNU time measurement: ${measurementLine}`);
}
const userCpuMs = userSeconds * 1000;
const systemCpuMs = systemSeconds * 1000;
const totalCpuMs = userCpuMs + systemCpuMs;
return {
elapsedMs: Number(elapsedMs.toFixed(1)),
userCpuMs: Number(userCpuMs.toFixed(1)),
systemCpuMs: Number(systemCpuMs.toFixed(1)),
totalCpuMs: Number(totalCpuMs.toFixed(1)),
cpuToWallRatio: Number((totalCpuMs / elapsedMs).toFixed(4)),
maxResidentSetKiB,
};
}
@@ -123,6 +123,8 @@ export async function applyRemoteSyncSettings(
minioSecretKey?: string;
encrypt?: boolean;
passphrase?: string;
enableCompression?: boolean;
usePathObfuscation?: boolean;
}
): Promise<void> {
const data = JSON.parse(await Deno.readTextFile(settingsFile));
@@ -149,6 +151,12 @@ export async function applyRemoteSyncSettings(
data.usePluginSync = false;
data.encrypt = options.encrypt === true;
data.passphrase = options.encrypt ? (options.passphrase ?? "") : "";
if (options.enableCompression !== undefined) {
data.enableCompression = options.enableCompression;
}
if (options.usePathObfuscation !== undefined) {
data.usePathObfuscation = options.usePathObfuscation;
}
data.isConfigured = true;
await Deno.writeTextFile(settingsFile, JSON.stringify(data, null, 2));
}
@@ -6,6 +6,7 @@ import {
selectVerificationEntries,
} from "./helpers/benchmarkVerification.ts";
import type { DatasetEntry } from "./helpers/dataset.ts";
import { createCompressionBenchmarkDataset } from "./helpers/compressionDataset.ts";
function getFreePort(): number {
const listener = Deno.listen({ hostname: "127.0.0.1", port: 0 });
@@ -106,6 +107,22 @@ Deno.test("CouchDB latency proxy applies half the requested RTT in each directio
assertEquals(await response.text(), "ok");
assertEquals(proxy.directionalDelayMs, 10);
assertEquals(delays, [10, 10]);
assertEquals(proxy.snapshotCounters(), {
requestCount: 1,
requestBodyBytes: 0,
responseBodyBytes: 2,
});
proxy.resetCounters();
const posted = await fetch(`http://127.0.0.1:${proxyPort}/probe`, {
method: "POST",
body: "abc",
});
assertEquals(await posted.text(), "ok");
assertEquals(proxy.snapshotCounters(), {
requestCount: 1,
requestBodyBytes: 3,
responseBodyBytes: 2,
});
} finally {
await proxy.stop();
await backend.shutdown();
@@ -124,6 +141,100 @@ Deno.test("CouchDB latency proxy applies half the requested RTT in each directio
}
});
Deno.test("compression benchmark dataset covers representative file kinds deterministically", async () => {
const fixtureRoot = await Deno.makeTempDir({
prefix: "livesync-compression-contract-",
});
const repositoryRoot = `${fixtureRoot}/repository`;
const vaultA = `${fixtureRoot}/vault-a`;
const vaultB = `${fixtureRoot}/vault-b`;
const repositoryFiles = [
"docs/settings.md",
"docs/quick_setup.md",
"updates.md",
"instruction_images/cloudant_1.png",
"images/quick-setup/guide-quick-setup-first-setup-uri.png",
"package.json",
"manifest.json",
"src/modules/core/ModuleReplicator.ts",
"src/modules/core/ReplicateResultProcessor.ts",
];
try {
for (const [index, relativePath] of repositoryFiles.entries()) {
const absolutePath = `${repositoryRoot}/${relativePath}`;
await Deno.mkdir(
absolutePath.slice(0, absolutePath.lastIndexOf("/")),
{ recursive: true },
);
await Deno.writeFile(
absolutePath,
new TextEncoder().encode(
`fixture-${index}-${relativePath}\n`.repeat(20),
),
);
}
await Deno.mkdir(vaultA, { recursive: true });
await Deno.mkdir(vaultB, { recursive: true });
const jpegEncoder = async (_input: string, output: string) => {
await Deno.writeFile(
output,
new Uint8Array([
0xff,
0xd8,
0xff,
0xdb,
0,
1,
2,
3,
0xff,
0xd9,
]),
);
return "contract JPEG encoder";
};
const first = await createCompressionBenchmarkDataset({
rootDir: vaultA,
repositoryRoot,
seed: "contract-seed",
jpegEncoder,
});
const second = await createCompressionBenchmarkDataset({
rootDir: vaultB,
repositoryRoot,
seed: "contract-seed",
jpegEncoder,
});
assertEquals(first.filesByKind, {
md: 3,
jpg: 2,
png: 2,
json: 2,
ts: 2,
gz: 1,
bin: 1,
});
assertEquals(first.totalFiles, 13);
assertEquals(first.jpegGenerator, "contract JPEG encoder");
assertEquals(
first.entries.map((entry) => [
entry.kind,
entry.relativePath,
entry.size,
]),
second.entries.map((entry) => [
entry.kind,
entry.relativePath,
entry.size,
]),
);
assert(first.entries.every((entry) => entry.size > 0));
} finally {
await Deno.remove(fixtureRoot, { recursive: true }).catch(() => {});
}
});
Deno.test("benchmark verification mode selects either all files or a labelled sample", () => {
const entries: DatasetEntry[] = [
{ kind: "md", relativePath: "a.md", absolutePath: "/a", size: 1 },
@@ -45,4 +45,7 @@ export function paneAdvanced(this: ObsidianLiveSyncSettingTab, paneEl: HTMLEleme
// clampMax: 100, clampMin: 1, onUpdate: onlyOnCouchDB
// })
});
void addPanel(paneEl, "Remote Database Tweak").then((paneEl) => {
new Setting(paneEl).setClass("wizardHidden").autoWireToggle("enableCompression");
});
}
@@ -231,15 +231,4 @@ export function panePatches(this: ObsidianLiveSyncSettingTab, paneEl: HTMLElemen
}
});
});
void addPanel(paneEl, "Remote Database Tweak (In sunset)").then((paneEl) => {
// new Setting(paneEl).autoWireToggle("useEden").setClass("wizardHidden");
// const onlyUsingEden = visibleOnly(() => this.isConfiguredAs("useEden", true));
// new Setting(paneEl).autoWireNumeric("maxChunksInEden", { onUpdate: onlyUsingEden }).setClass("wizardHidden");
// new Setting(paneEl)
// .autoWireNumeric("maxTotalLengthInEden", { onUpdate: onlyUsingEden })
// .setClass("wizardHidden");
// new Setting(paneEl).autoWireNumeric("maxAgeInEden", { onUpdate: onlyUsingEden }).setClass("wizardHidden");
new Setting(paneEl).autoWireToggle("enableCompression").setClass("wizardHidden");
});
}
+2 -1
View File
@@ -3,7 +3,7 @@
FROM node:24-slim
RUN apt-get update \
&& apt-get install -y --no-install-recommends ca-certificates curl unzip python3 make g++ iproute2 \
&& apt-get install -y --no-install-recommends ca-certificates curl unzip python3 make g++ iproute2 libjpeg-turbo-progs time \
&& rm -rf /var/lib/apt/lists/*
ENV DENO_INSTALL=/usr/local
@@ -28,6 +28,7 @@ RUN deno cache --lock=deno.lock \
bench-p2p-split-node.ts \
bench-p2p.ts \
bench-couchdb.ts \
bench-compression.ts \
test-p2p-sync.ts \
test-p2p-replicator-replacement.ts \
test-p2p-relay-disconnect.ts
+43
View File
@@ -127,6 +127,49 @@ This sweep is useful for finding where the remote CouchDB path falls behind the
local direct P2P path in the current HTTP-proxy latency model. It should not be
presented as a full smartphone/VPN model.
## Data Compression benchmark
The [Data Compression specification](../../docs/specs_data_compression.md) records the current storage contract, 1.0 decision, measured result, and follow-up optimisation candidates. This section is the benchmark runbook.
The compression benchmark compares the exact CLI and Commonlib CouchDB path in
four configurations: E2EE off or on, each with Data Compression off or on. It
uses the normal CLI `mirror` and `sync` commands, so the recorded CouchDB
documents have passed through the current RabinKarp chunk splitter, optional
fflate compression, and E2EE V2 rather than a synthetic document transform.
```bash
BENCH_COMMAND=compression \
BENCH_COMPRESSION_REPEAT_COUNT=3 \
BENCH_COUCHDB_RTT_MS=1 \
docker compose -f test/bench-network/compose.yml run --build --rm bench-runner
```
The fixture contains current repository Markdown, PNG, JSON, and TypeScript
files; two deterministic JPEGs generated with `cjpeg`; a gzip-compressed
Markdown file; and deterministic high-entropy binary data. Every run verifies
all files after the second client has synchronised them. The JSON result under
`test/bench-network/bench-results/` records:
- source bytes and stored chunk bytes by file kind;
- raw CouchDB external, active, and file sizes;
- request and response body bytes observed by the local HTTP proxy, including
the combined initial sync and full materialisation download;
- upload and download wall time, user and system CPU time, and maximum resident
memory for each CLI process; and
- percentage changes caused by enabling compression with E2EE both off and on.
Use at least three repeats when making a default-setting decision. A `1 ms`
requested RTT keeps the local run focused on transform and storage costs. Run a
separate representative RTT when evaluating whether reduced request bodies
outweigh compression CPU on the intended network. HTTP byte counters cover
decoded bodies and exclude headers, while process timings include CLI start-up.
Full materialisation starts one CLI process per file and can repeat lazy chunk
fetches, so treat that phase as a CLI workflow measurement rather than a raw
download lower bound. Stored chunk size and upload request size are the more
direct transform comparisons.
The generated JPEGs are deterministic image-like fixtures, not a photographic
corpus, so broader media conclusions require a separately reviewed corpus.
## Network emulation smoke
The optional `netem` profile checks whether a Linux runner can apply traffic
+3
View File
@@ -75,6 +75,8 @@ services:
BENCH_CASES: ${BENCH_CASES:-couchdb-baseline,p2p-direct-local}
BENCH_REPEAT_COUNT: ${BENCH_REPEAT_COUNT:-1}
BENCH_CASES_ROOT: /workspace/src/apps/cli/testdeno/bench-results
BENCH_COMPRESSION_RESULT_ROOT: /workspace/src/apps/cli/testdeno/bench-results
BENCH_COMPRESSION_REPEAT_COUNT: ${BENCH_COMPRESSION_REPEAT_COUNT:-1}
BENCH_SWEEP_ROOT: /workspace/src/apps/cli/testdeno/bench-results
BENCH_SWEEP_RTT_MS: ${BENCH_SWEEP_RTT_MS:-20,50,100,150,300}
BENCH_SWEEP_INCLUDE_P2P: ${BENCH_SWEEP_INCLUDE_P2P:-true}
@@ -97,6 +99,7 @@ services:
BENCH_PEERS_TIMEOUT: ${BENCH_PEERS_TIMEOUT:-60}
LIVESYNC_P2P_RELAY_READY_TIMEOUT_MS: ${LIVESYNC_P2P_RELAY_READY_TIMEOUT_MS:-60000}
BENCH_LIVESYNC_TEST_TEE: ${BENCH_LIVESYNC_TEST_TEE:-0}
LIVESYNC_TEST_TEE: ${BENCH_LIVESYNC_TEST_TEE:-0}
LIVESYNC_CLI_DEBUG: ${LIVESYNC_CLI_DEBUG:-0}
LIVESYNC_CLI_VERBOSE: ${LIVESYNC_CLI_VERBOSE:-0}
CLI_E2E_TASK: ${CLI_E2E_TASK:-test:p2p:ci}
+4 -1
View File
@@ -8,12 +8,15 @@ case "${BENCH_COMMAND:-cases}" in
latency-sweep)
exec deno task bench:latency-sweep
;;
compression)
exec deno task bench:compression
;;
p2p-split-node)
exec deno task bench:p2p-split-node
;;
*)
echo "Unknown BENCH_COMMAND: ${BENCH_COMMAND}" >&2
echo "Expected one of: cases, latency-sweep, p2p-split-node" >&2
echo "Expected one of: cases, latency-sweep, compression, p2p-split-node" >&2
exit 2
;;
esac
+1 -1
View File
@@ -1,6 +1,6 @@
# Real Obsidian E2E Runner
This directory contains the experimental real Obsidian end-to-end runner.
This directory contains the maintained real Obsidian end-to-end runner.
The generic application discovery, isolated-vault, plug-in installation, process lifecycle, CLI, CDP, and readiness implementation comes from `@vrtmrz/obsidian-test-session`. The small modules under `runner/` preserve LiveSync's existing imports and supply its plug-in ID and artefact location. LiveSync-specific fixtures, services, settings, workflows, and assertions remain in this repository.
+5 -3
View File
@@ -166,7 +166,7 @@ async function verifyCompatibilityReview(): Promise<void> {
);
}
async function verifyDeletionSettings(): Promise<void> {
async function verifyEffectiveSettings(): Promise<void> {
await withObsidianPage(obsidianRemoteDebuggingPort(), async (page) => {
await page.evaluate(() => {
const setting = (globalThis as ObsidianTestGlobal).app?.setting;
@@ -230,13 +230,15 @@ async function main(): Promise<void> {
syncOnFileOpen: false,
syncAfterMerge: false,
periodicReplication: false,
handleFilenameCaseSensitive: false,
useAdvancedMode: true,
useEdgeCaseMode: true,
},
});
await waitForLiveSyncCoreReady(cli.binary, session.cliEnv);
await verifyCompatibilityReview();
await verifyDeletionSettings();
console.log("Compatibility review and deletion settings expose only effective user controls.");
await verifyEffectiveSettings();
console.log("Compatibility review and settings expose only effective user controls.");
} finally {
if (session) {
await session.app.stop();
+3
View File
@@ -14,6 +14,9 @@ Earlier releases remain available in the [0.25 release history](docs/releases/0.
### Improved
- Clarified the 1.0 maturity of optional features. P2P and Hidden File Sync are supported opt-in features; JWT, ignore files, automatic newer-file conflict resolution, and Garbage Collection V3 remain previews; and legacy database-format settings remain compatibility paths rather than recommendations.
- Re-evaluated Data Compression. It provides a modest but measurable reduction in stored and transferred chunk data, with workload-dependent benefits and costs. See the [Data Compression specification](docs/specs_data_compression.md) for its behaviour, measurements, compatibility, and 1.0 default-setting decision.
- Classified Customisation Sync as a supported advanced workflow, backed by its maintained two-Vault real-Obsidian regression, without changing its synchronisation behaviour.
- Wizard-driven new-device and existing-device setup now reserves Rebuild or Fetch before enabling imported settings, preventing ordinary start-up work from running ahead of the selected initialisation.
- Manual onboarding now creates and selects CouchDB, Object Storage, and P2P remote profiles directly while preserving existing profiles. Current Setup URIs retain profile names and selections, while older flat settings remain supported through compatibility migration.
- P2P panes and explicit rebuild actions now use the current transport after settings or database replacement. Disconnecting leaves the LiveSync room and closes signalling relay sockets without destroying Trystero-owned shared peers.