mirror of
https://github.com/vrtmrz/obsidian-livesync.git
synced 2026-10-25 10:33:21 +00:00
Prepare consistent Chunk sizes and fresh Fast Fetch integration
This commit is contained in:
vorotamoroz committed
1 parent
ce0524d49f
commit
d1e4c34054
24 files changed
+2042
-14
No files matched your search
@@ -0,0 +1,72 @@
|
||||
# Chunk-size research harness
|
||||
|
||||
These scripts support the October 2026 Chunk-size investigation. The decision,
|
||||
measurements, and their limits are recorded in Commonlib's
|
||||
`docs/adr/2026_10_chunking_and_transport_batching.md` and its evidence companion.
|
||||
They are research tools with experiment-specific assertions, separate from
|
||||
the maintained unit and integration suites.
|
||||
|
||||
The original experiments used Commonlib 0.1.44 and unpublished candidates with
|
||||
the same version label. `local.mjs` checks that version explicitly. Record the
|
||||
installed package's source revision and tarball digest when comparing results;
|
||||
the version label alone does not identify the tested implementation. Scripts
|
||||
which import internal `dist` paths also depend on that package layout.
|
||||
|
||||
## Local measurements
|
||||
|
||||
Install the exact Commonlib candidate to assess, then run from the repository
|
||||
root with Node.js 24:
|
||||
|
||||
```sh
|
||||
NODE_OPTIONS=--max-old-space-size=3072 node utils/chunk-size-benchmark/local.mjs --output .bench/chunk-size --sizes 0,1,4 --repeats 1
|
||||
node --test utils/chunk-size-benchmark/rabin-window.test.mjs
|
||||
```
|
||||
|
||||
The local run generates a deterministic corpus from repository Markdown,
|
||||
images, and seeded binary data. It records initial, overwrite, and insertion
|
||||
stages, Chunk identifiers, reconstructed content checks, and timings. Keep
|
||||
the generated corpus when comparing revisions: source Markdown may change.
|
||||
Generated Vaults, logs, and results belong under `.bench/` and are not source
|
||||
files for the normal test runner.
|
||||
|
||||
## Transport measurements
|
||||
|
||||
Build the CLI and start the disposable fixtures required for the chosen
|
||||
transport. These use loopback ports and public benchmark-only credentials.
|
||||
The scripts create and delete their own test databases and buckets.
|
||||
GNU `time`, Docker Compose, and a Linux CLI runtime are required.
|
||||
|
||||
```sh
|
||||
NODE_OPTIONS=--max-old-space-size=3072 npm run build --workspace self-hosted-livesync-cli
|
||||
docker compose -p livesync-chunk-benchmark -f utils/chunk-size-benchmark/compose.yml up -d couchdb object-storage
|
||||
node utils/chunk-size-benchmark/remote.mjs --output .bench/chunk-size --sizes 0,1,4 --repeats 1 --rtt-ms 0
|
||||
docker compose -p livesync-chunk-benchmark -f utils/chunk-size-benchmark/compose.yml down --volumes
|
||||
```
|
||||
|
||||
`remote.mjs` accepts `--providers couchdb`, `--providers s3`, or both, plus
|
||||
`--corpus`, `--source-cli`, and `--target-cli`. Its HTTP proxy adds the requested
|
||||
RTT across the request and response legs. This models per-request delay rather
|
||||
than network bandwidth, packet loss, or a production service.
|
||||
|
||||
For P2P, start the `relay` fixture instead and run `p2p.mjs` with the same
|
||||
output, corpus, size, repeat, and CLI options, omitting `--rtt-ms`. Its
|
||||
DataChannel counters measure application payloads and record the selected
|
||||
connection path. An archived CLI entry point needs all of its generated
|
||||
CommonJS chunks beside it. Run transports sequentially and stop the named
|
||||
Compose project afterwards.
|
||||
|
||||
## Historical analysis helpers
|
||||
|
||||
- `summarise.mjs`, `report.mjs`, and `verify-results.mjs` consume the original
|
||||
result layouts. The verifier requires that experiment's exact variants,
|
||||
configuration counts, and files; it is not a general mixed-version verifier.
|
||||
- `cloudant-limit.mjs` temporarily changes the disposable CouchDB document
|
||||
limit and restores it afterwards. `cloudant-json-boundary.mjs` probes final
|
||||
serialised document sizes against the same fixture. These scripts model
|
||||
published Cloudant constraints; they do not connect to IBM Cloudant or
|
||||
certify its acceptance of a Vault.
|
||||
|
||||
Some helpers retain historical output-directory and size defaults. Supply
|
||||
explicit paths and inspect their assertions before reusing them for a new
|
||||
experiment. The recorded implementation matrix checks file content in both
|
||||
writer directions; revision-conflict retention has separate Commonlib tests.
|
||||
@@ -0,0 +1,108 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import { createHash } from 'node:crypto';
|
||||
import { promises as fs } from 'node:fs';
|
||||
import { dirname, join, resolve } from 'node:path';
|
||||
import { fileURLToPath, pathToFileURL } from 'node:url';
|
||||
|
||||
const packageRoot = dirname(fileURLToPath(import.meta.resolve('@vrtmrz/livesync-commonlib/package.json')));
|
||||
const { splitPiecesRabinKarp } = await import(pathToFileURL(join(packageRoot, 'dist/string_and_binary/chunks.js')).href);
|
||||
const { getConfiguredFunctionsForEncryption } = await import(pathToFileURL(join(packageRoot, 'dist/pouchdb/encryption.js')).href);
|
||||
const { compressDoc } = await import(pathToFileURL(join(packageRoot, 'dist/pouchdb/compress.js')).href);
|
||||
const output = resolve(process.argv[2] || '.bench/rabin-window-fixed/cloudant-json-boundary');
|
||||
await fs.mkdir(output, { recursive: true });
|
||||
const uri = 'http://127.0.0.1:15994';
|
||||
const headers = { Authorization: `Basic ${Buffer.from('bench:benchmark-only-password').toString('base64')}`, 'Content-Type': 'application/json' };
|
||||
const configPath = '/_node/_local/_config/couchdb/max_document_size';
|
||||
const previousResponse = await fetch(uri + configPath, { headers });
|
||||
assert(previousResponse.ok || previousResponse.status === 404);
|
||||
const previous = previousResponse.ok ? await previousResponse.json() : undefined;
|
||||
const requestConfigPath = '/_node/_local/_config/chttpd/max_http_request_size';
|
||||
const previousRequestResponse = await fetch(uri + requestConfigPath, { headers });
|
||||
assert(previousRequestResponse.ok || previousRequestResponse.status === 404);
|
||||
const previousRequestLimit = previousRequestResponse.ok ? await previousRequestResponse.json() : undefined;
|
||||
const database = `chunkbench-json-budget-${Date.now()}`;
|
||||
const inputs = {
|
||||
binary: new Blob([new Uint8Array(4 * 1024 * 1024)], { type: 'application/octet-stream' }),
|
||||
nulText: new Blob(['\0'.repeat(4 * 1024 * 1024)], { type: 'text/plain' }),
|
||||
escapedText: new Blob(['\\'.repeat(4 * 1024 * 1024)], { type: 'text/plain' }),
|
||||
};
|
||||
const cases = [
|
||||
{ kind: 'binary', customChunkSize: 4, encryption: 'v1', compression: false, expectedAccepted: true },
|
||||
{ kind: 'binary', customChunkSize: 4, encryption: 'v2', compression: false, expectedAccepted: true },
|
||||
{ kind: 'binary', customChunkSize: 5, encryption: 'v2', compression: false, expectedAccepted: false },
|
||||
{ kind: 'nulText', customChunkSize: 0, encryption: 'none', compression: false, expectedAccepted: true },
|
||||
{ kind: 'nulText', customChunkSize: 4, encryption: 'none', compression: false },
|
||||
{ kind: 'escapedText', customChunkSize: 4, encryption: 'none', compression: false },
|
||||
{ kind: 'nulText', customChunkSize: 4, encryption: 'v2', compression: false, expectedAccepted: true },
|
||||
{ kind: 'nulText', customChunkSize: 4, encryption: 'none', compression: true, expectedAccepted: true },
|
||||
];
|
||||
const records = [];
|
||||
const record = { scope: 'Synthetic leaf-document probes using the installed splitter, compression, encryption, and JSON serialisation, sent to the owned CouchDB fixture with a 1,000,000-byte document limit and 64,000,000-byte request limit. This is not an IBM Cloudant service measurement or a whole-file synchronisation test. CouchDB 3.5.0 estimates document size from parsed strings and does not count JSON escapes; Cloudant internal size accounting has not been verified.', source: 'https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-limits', sizeCalculationSource: 'https://github.com/apache/couchdb/blob/3.5.0/src/couch/src/couch_ejson_size.erl', documentLimitBytes: 1000000, requestLimitBytes: 64000000, previousDocumentLimit: previous ?? null, previousRequestLimit: previousRequestLimit ?? null, cases: records, batches: [] };
|
||||
try {
|
||||
assert((await fetch(uri + configPath, { method: 'PUT', headers, body: JSON.stringify('1000000') })).ok);
|
||||
assert((await fetch(uri + requestConfigPath, { method: 'PUT', headers, body: JSON.stringify('64000000') })).ok);
|
||||
assert((await fetch(`${uri}/${database}`, { method: 'PUT', headers })).ok);
|
||||
for (const condition of cases) {
|
||||
const input = inputs[condition.kind];
|
||||
const chunks = await splitPiecesRabinKarp(input, 102400 * (condition.customChunkSize + 1), condition.kind !== 'binary', 20);
|
||||
let largest = '';
|
||||
for await (const chunk of chunks()) if (chunk.length > largest.length) largest = chunk;
|
||||
const id = `h:+${createHash('sha256').update(JSON.stringify(condition)).digest('hex')}`;
|
||||
let doc = { _id: id, type: 'leaf', data: largest };
|
||||
if (condition.compression) doc = await compressDoc(doc);
|
||||
if (condition.encryption !== 'none') {
|
||||
const { incoming, outgoing } = getConfiguredFunctionsForEncryption('public-json-budget-probe-passphrase', false, false, async () => new Uint8Array(16), condition.encryption);
|
||||
const encrypted = await incoming(doc);
|
||||
assert.equal(encrypted.e_, true);
|
||||
const decoded = await outgoing(structuredClone(encrypted));
|
||||
assert.equal(decoded.data, doc.data);
|
||||
doc = encrypted;
|
||||
}
|
||||
doc = { ...doc, _rev: `1-${'0'.repeat(32)}`, _revisions: { start: 1, ids: ['0'.repeat(32)] } };
|
||||
const jsonBytes = Buffer.byteLength(JSON.stringify(doc));
|
||||
const body = JSON.stringify({ docs: [doc], new_edits: false });
|
||||
const response = await fetch(`${uri}/${database}/_bulk_docs`, { method: 'POST', headers, body });
|
||||
const result = await response.json();
|
||||
const accepted = response.ok && Array.isArray(result) && result.every(entry => !entry.error);
|
||||
const measurement = { ...condition, largestPieceUtf8Bytes: Buffer.byteLength(largest), jsonBytes, jsonBytesUnderLimit: jsonBytes < record.documentLimitBytes, requestBodyBytes: Buffer.byteLength(body), status: response.status, accepted, response: result };
|
||||
records.push(measurement);
|
||||
await fs.writeFile(join(output, 'results.json'), JSON.stringify(record, null, 2));
|
||||
if (condition.expectedAccepted !== undefined) assert.equal(accepted, condition.expectedAccepted, JSON.stringify(measurement));
|
||||
assert(Buffer.byteLength(body) < record.requestLimitBytes);
|
||||
console.log(JSON.stringify({ ...measurement, response: undefined }));
|
||||
}
|
||||
const batchInput = new Uint8Array(512000 * 25);
|
||||
for (let index = 0; index < 25; index++) batchInput[(index + 1) * 512000 - 1] = index + 1;
|
||||
const splitBatch = await splitPiecesRabinKarp(new Blob([batchInput], { type: 'text/plain' }), 512000, true, 20);
|
||||
const batchPieces = [];
|
||||
for await (const piece of splitBatch()) batchPieces.push(piece);
|
||||
assert.equal(batchPieces.length, 25);
|
||||
assert(batchPieces.every(piece => Buffer.byteLength(piece) === 512000));
|
||||
assert.equal(new Set(batchPieces).size, 25);
|
||||
for (const count of [25, 20]) {
|
||||
const docs = batchPieces.slice(0, count).map((piece, index) => ({
|
||||
_id: `h:+${createHash('sha256').update(`escaped-batch-${count}-${index}`).digest('hex')}`,
|
||||
type: 'leaf', data: piece,
|
||||
_rev: `1-${'0'.repeat(32)}`, _revisions: { start: 1, ids: ['0'.repeat(32)] },
|
||||
}));
|
||||
const body = JSON.stringify({ docs, new_edits: false });
|
||||
const response = await fetch(`${uri}/${database}/_bulk_docs`, { method: 'POST', headers, body });
|
||||
const result = await response.json();
|
||||
const accepted = response.ok && Array.isArray(result) && result.every(entry => !entry.error);
|
||||
const measurement = { count, rawBytesPerLeaf: 512000, requestBodyBytes: Buffer.byteLength(body), largestDocumentJsonBytes: Math.max(...docs.map(doc => Buffer.byteLength(JSON.stringify(doc)))), status: response.status, accepted, response: result };
|
||||
record.batches.push(measurement);
|
||||
await fs.writeFile(join(output, 'results.json'), JSON.stringify(record, null, 2));
|
||||
assert.equal(accepted, count === 20, JSON.stringify(measurement));
|
||||
if (count === 25) assert.equal(response.status, 413);
|
||||
console.log(JSON.stringify({ ...measurement, response: undefined }));
|
||||
}
|
||||
} finally {
|
||||
await fetch(`${uri}/${database}`, { method: 'DELETE', headers });
|
||||
const restored = await fetch(uri + configPath, previous === undefined ? { method: 'DELETE', headers } : { method: 'PUT', headers, body: JSON.stringify(previous) });
|
||||
record.configurationRestored = restored.ok;
|
||||
const requestRestored = await fetch(uri + requestConfigPath, previousRequestLimit === undefined ? { method: 'DELETE', headers } : { method: 'PUT', headers, body: JSON.stringify(previousRequestLimit) });
|
||||
record.requestConfigurationRestored = requestRestored.ok;
|
||||
await fs.writeFile(join(output, 'results.json'), JSON.stringify(record, null, 2));
|
||||
assert(record.configurationRestored);
|
||||
assert(record.requestConfigurationRestored);
|
||||
}
|
||||
@@ -0,0 +1,50 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import { spawn } from 'node:child_process';
|
||||
import { promises as fs } from 'node:fs';
|
||||
import { dirname, join, resolve } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
|
||||
const root = resolve(dirname(fileURLToPath(import.meta.url)), '../..');
|
||||
const options = new Map();
|
||||
for (let index = 2; index < process.argv.length; index += 2) {
|
||||
assert(['--output', '--corpus'].includes(process.argv[index]), `Unknown option: ${process.argv[index]}`);
|
||||
assert(process.argv[index + 1], `Missing value for ${process.argv[index]}`);
|
||||
options.set(process.argv[index], process.argv[index + 1]);
|
||||
}
|
||||
const output = resolve(options.get('--output') || '.bench/chunk-size-cloudant-model');
|
||||
const corpusPath = resolve(options.get('--corpus') || '.bench/chunk-size/corpus/manifest.json');
|
||||
const configUri = 'http://127.0.0.1:15994/_node/_local/_config/couchdb/max_document_size';
|
||||
const headers = { Authorization: `Basic ${Buffer.from('bench:benchmark-only-password').toString('base64')}`, 'Content-Type': 'application/json' };
|
||||
const previousResponse = await fetch(configUri, { headers });
|
||||
assert(previousResponse.ok || previousResponse.status === 404);
|
||||
const previous = previousResponse.ok ? await previousResponse.json() : undefined;
|
||||
const manifest = JSON.parse(await fs.readFile(corpusPath, 'utf8'));
|
||||
manifest.stages = manifest.stages.filter((stage) => stage.name === 'initial');
|
||||
for (const stage of manifest.stages) stage.directory = resolve(dirname(corpusPath), stage.directory);
|
||||
await fs.mkdir(output, { recursive: true });
|
||||
const initialManifest = join(output, 'initial-only-manifest.json');
|
||||
await fs.writeFile(initialManifest, JSON.stringify(manifest, null, 2));
|
||||
const record = { scope: 'CouchDB with a 1,000,000-byte document limit, modelling the published Cloudant document-size constraint. The IBM Cloudant service itself is not measured.', source: 'https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-limits', limitBytes: 1000000, publishedRequestLimitBytes: 64000000, variant: process.env.LIVESYNC_BENCH_VARIANT || 'unlabelled', previousConfiguredValue: previous ?? null };
|
||||
try {
|
||||
const configured = await fetch(configUri, { method: 'PUT', headers, body: JSON.stringify(String(record.limitBytes)) });
|
||||
assert(configured.ok, `Could not set fixture document limit: ${configured.status}`);
|
||||
const child = spawn(process.execPath, [join(root, 'utils/chunk-size-benchmark/remote.mjs'), '--output', output, '--corpus', initialManifest, '--providers', 'couchdb', '--sizes', '4,10', '--repeats', '1'], {
|
||||
cwd: root, stdio: ['ignore', 'pipe', 'pipe'], env: { ...process.env, NODE_OPTIONS: '--max-old-space-size=3072' },
|
||||
});
|
||||
let log = ''; child.stdout.on('data', (data) => { log += data; process.stdout.write(data); }); child.stderr.on('data', (data) => { log += data; });
|
||||
record.exitCode = await new Promise((accept, reject) => { child.once('error', reject); child.once('close', accept); });
|
||||
await fs.writeFile(join(output, 'run.log'), log);
|
||||
const measured = JSON.parse(await fs.readFile(join(output, 'remote/results-rtt0.json'), 'utf8'));
|
||||
record.cases = measured.results.map((entry) => ({ name: entry.name, customChunkSize: entry.customChunkSize, complete: entry.complete, error: entry.error, stages: entry.stages.map((stage) => ({ name: stage.name, largestDocumentJsonBytes: stage.remote.largestDocumentJsonBytes, maxUploadRequestBodyBytes: stage.upload.http.maxRequestBodyBytes, verifiedFiles: stage.verifiedFiles })) }));
|
||||
assert(record.cases.some((entry) => entry.customChunkSize === 4 && entry.complete), 'The 500 KiB candidate must complete under the modelled limit');
|
||||
assert(record.cases.some((entry) => entry.customChunkSize === 10 && !entry.complete), 'The 1,100 KiB candidate must hit the modelled document limit');
|
||||
const rejectionLog = await fs.readFile(join(output, 'remote/couchdb-c10-r1/logs/initial-upload.txt'), 'utf8');
|
||||
record.sizeRejectionObserved = /413|document_too_large|document.*too large|too_large|request entity too large/iu.test(rejectionLog);
|
||||
assert(record.sizeRejectionObserved, 'The rejection log must identify the document-size limit');
|
||||
} finally {
|
||||
const restored = await fetch(configUri, previous === undefined ? { method: 'DELETE', headers } : { method: 'PUT', headers, body: JSON.stringify(previous) });
|
||||
record.configurationRestored = restored.ok;
|
||||
await fs.writeFile(join(output, 'constraint-result.json'), JSON.stringify(record, null, 2));
|
||||
assert(record.configurationRestored, 'Fixture document limit restoration failed');
|
||||
}
|
||||
console.log(JSON.stringify(record));
|
||||
@@ -0,0 +1,30 @@
|
||||
services:
|
||||
relay:
|
||||
image: ghcr.io/hoytech/strfry@sha256:36f1886d185a88ca57c66ebe52e6e9e8428dac2486eea0a5d50ff934f18b60c3
|
||||
command: ["--config", "/etc/strfry.conf", "relay"]
|
||||
volumes:
|
||||
- ../../test/fixtures/p2p-relay/strfry.conf:/etc/strfry.conf:ro
|
||||
tmpfs:
|
||||
- /app/strfry-db:rw,size=128m,mode=1777
|
||||
ports:
|
||||
- "127.0.0.1:14094:7777"
|
||||
mem_limit: 256m
|
||||
couchdb:
|
||||
image: couchdb:3.5.0@sha256:1c0a30a54535377bc9ae23f42efb1ca4e92abe796f78ec98b768ac1d5874cc1b
|
||||
environment:
|
||||
COUCHDB_USER: bench
|
||||
COUCHDB_PASSWORD: benchmark-only-password
|
||||
COUCHDB_SINGLE_NODE: "true"
|
||||
ports:
|
||||
- "127.0.0.1:15994:5984"
|
||||
mem_limit: 768m
|
||||
object-storage:
|
||||
image: rustfs/rustfs:1.0.0-rc.6@sha256:97171b3d72cd47dc81000f92ea84de25608bfc35a94c965501afaeb5d99f6035
|
||||
environment:
|
||||
RUSTFS_ACCESS_KEY: bench-access-key
|
||||
RUSTFS_SECRET_KEY: benchmark-only-secret-key
|
||||
RUSTFS_CONSOLE_ENABLE: "false"
|
||||
command: ["/data"]
|
||||
ports:
|
||||
- "127.0.0.1:19094:9000"
|
||||
mem_limit: 1g
|
||||
@@ -0,0 +1,310 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { copyFile, mkdir, readFile, readdir, stat, utimes, writeFile } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const MODULE_DIR = path.dirname(fileURLToPath(import.meta.url));
|
||||
const DEFAULT_REPO_ROOT = path.resolve(MODULE_DIR, "../..");
|
||||
const SEED = 0x4c53594e;
|
||||
const KIB = 1024;
|
||||
const MIB = 1024 * KIB;
|
||||
const BINARY_BYTES = 8 * MIB;
|
||||
const SMALL_MARKDOWN_BYTES = 512 * KIB;
|
||||
const LARGE_MARKDOWN_BYTES = 5 * MIB;
|
||||
const BINARY_EDIT_BYTES = 4 * KIB;
|
||||
const INITIAL_MTIME = new Date("2020-01-01T00:00:00.000Z");
|
||||
const STAGE_MTIMES = {
|
||||
overwrite: new Date("2020-01-02T00:00:00.000Z"),
|
||||
insert: new Date("2020-01-03T00:00:00.000Z"),
|
||||
};
|
||||
|
||||
const MARKDOWN_FILES = [
|
||||
"README.md",
|
||||
"devs.md",
|
||||
"docs/settings.md",
|
||||
"docs/troubleshooting.md",
|
||||
"docs/quick_setup.md",
|
||||
"docs/datastructure.md",
|
||||
"docs/p2p.md",
|
||||
"docs/tips/fast-setup.md",
|
||||
"docs/design_docs/intention_of_chunks.md",
|
||||
"docs/specs_data_compression.md",
|
||||
];
|
||||
|
||||
const SNIPPET_FILES = [
|
||||
"docs/terms.md",
|
||||
"docs/glossary.md",
|
||||
"docs/quick_setup_ja.md",
|
||||
"docs/design_docs/unicode_path_identity.md",
|
||||
"docs/settings.md",
|
||||
];
|
||||
|
||||
const CORPUS_PATHS = {
|
||||
markdownSmall: "benchmark/generated-small.md",
|
||||
markdownLarge: "benchmark/generated-large.md",
|
||||
binary: "benchmark/random-8m.bin",
|
||||
cloudantImage: "assets/cloudant-setup.png",
|
||||
setupImage: "assets/quick-setup-first-setup-uri.png",
|
||||
};
|
||||
|
||||
function sha256(bytes) {
|
||||
return createHash("sha256").update(bytes).digest("hex");
|
||||
}
|
||||
|
||||
function createPrng(seed) {
|
||||
let state = seed >>> 0;
|
||||
return () => {
|
||||
state ^= state << 13;
|
||||
state ^= state >>> 17;
|
||||
state ^= state << 5;
|
||||
return state >>> 0;
|
||||
};
|
||||
}
|
||||
|
||||
function makeRandomBytes(length, seed) {
|
||||
const next = createPrng(seed);
|
||||
const output = Buffer.allocUnsafe(length);
|
||||
for (let offset = 0; offset < length; offset += 4) {
|
||||
const value = next();
|
||||
output[offset] = value & 0xff;
|
||||
if (offset + 1 < length) output[offset + 1] = (value >>> 8) & 0xff;
|
||||
if (offset + 2 < length) output[offset + 2] = (value >>> 16) & 0xff;
|
||||
if (offset + 3 < length) output[offset + 3] = (value >>> 24) & 0xff;
|
||||
}
|
||||
return output;
|
||||
}
|
||||
|
||||
function extractParagraphs(source) {
|
||||
const withoutFrontmatter = source.replace(/^---\r?\n[\s\S]*?\r?\n---\r?\n/u, "");
|
||||
return withoutFrontmatter
|
||||
.replace(/\r\n/gu, "\n")
|
||||
.split(/\n{2,}/u)
|
||||
.map((paragraph) => paragraph.trim())
|
||||
.filter((paragraph) => {
|
||||
if (paragraph.length < 80) return false;
|
||||
if (/^(?:#{1,6}\s|!\[|\|)/u.test(paragraph)) return false;
|
||||
return true;
|
||||
});
|
||||
}
|
||||
|
||||
async function readSnippetParagraphs(repoRoot) {
|
||||
const excerpts = [];
|
||||
for (const file of SNIPPET_FILES) {
|
||||
const source = await readFile(path.join(repoRoot, file), "utf8");
|
||||
for (const paragraph of extractParagraphs(source)) {
|
||||
const excerpt = Array.from(paragraph).slice(0, 1000).join("");
|
||||
excerpts.push({ source: file, text: excerpt });
|
||||
}
|
||||
}
|
||||
if (excerpts.length === 0) throw new Error("No repository Markdown snippets were available.");
|
||||
return excerpts;
|
||||
}
|
||||
|
||||
function buildGeneratedMarkdown({ targetBytes, title, variant, excerpts }) {
|
||||
const lines = [
|
||||
`# ${title}`,
|
||||
"",
|
||||
"A deterministic UTF-8 fixture contains varied passages from the project documentation.",
|
||||
"",
|
||||
];
|
||||
let byteLength = Buffer.byteLength(lines.join("\n"), "utf8");
|
||||
const profiles = ["replication", "metadata", "recovery", "integrity", "configuration", "history", "incremental", "transport"];
|
||||
const unicodeLabels = ["résumé", "café", "日本語", "中文", "Δοκιμή", "ångström", "naïve", "雪", "λ", "mañana"];
|
||||
let record = 0;
|
||||
while (byteLength < targetBytes) {
|
||||
const excerpt = excerpts[(record * 7 + variant.length) % excerpts.length];
|
||||
const profile = profiles[(record * 3 + variant.length) % profiles.length];
|
||||
const unicode = unicodeLabels[(record * 5 + variant.length) % unicodeLabels.length];
|
||||
const tag = ((SEED ^ (record * 0x9e3779b1)) >>> 0).toString(16).padStart(8, "0");
|
||||
const paragraph = [
|
||||
`## Record ${String(record + 1).padStart(6, "0")} · ${profile} · ${variant}`,
|
||||
"",
|
||||
excerpt.text,
|
||||
"",
|
||||
`Fixture label ${tag} uses ${unicode}; source excerpt: ${excerpt.source}; revision ${record % 97}.`,
|
||||
"",
|
||||
].join("\n");
|
||||
lines.push(paragraph);
|
||||
byteLength += Buffer.byteLength(paragraph, "utf8");
|
||||
record += 1;
|
||||
}
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
function insertNearStart(markdown, paragraph) {
|
||||
const firstGap = markdown.indexOf("\n\n");
|
||||
const introGap = markdown.indexOf("\n\n", firstGap + 2);
|
||||
if (firstGap < 0 || introGap < 0) throw new Error("Generated Markdown fixture has no introductory paragraph.");
|
||||
const insertionPoint = introGap + 2;
|
||||
return `${markdown.slice(0, insertionPoint)}${paragraph}\n\n${markdown.slice(insertionPoint)}`;
|
||||
}
|
||||
|
||||
async function writeFileWithMtime(directory, relativePath, contents, mtime) {
|
||||
const outputPath = path.join(directory, relativePath);
|
||||
await mkdir(path.dirname(outputPath), { recursive: true });
|
||||
await writeFile(outputPath, contents);
|
||||
await utimes(outputPath, mtime, mtime);
|
||||
}
|
||||
|
||||
async function copyStage(sourceDirectory, targetDirectory, relativePaths) {
|
||||
for (const relativePath of relativePaths) {
|
||||
const sourcePath = path.join(sourceDirectory, relativePath);
|
||||
const targetPath = path.join(targetDirectory, relativePath);
|
||||
await mkdir(path.dirname(targetPath), { recursive: true });
|
||||
const sourceStat = await stat(sourcePath);
|
||||
await copyFile(sourcePath, targetPath);
|
||||
await utimes(targetPath, sourceStat.atime, sourceStat.mtime);
|
||||
}
|
||||
}
|
||||
|
||||
async function setMtime(directory, relativePath, mtime) {
|
||||
const outputPath = path.join(directory, relativePath);
|
||||
await utimes(outputPath, mtime, mtime);
|
||||
}
|
||||
|
||||
async function listFiles(directory) {
|
||||
const files = [];
|
||||
async function visit(currentDirectory) {
|
||||
const entries = await readdir(currentDirectory, { withFileTypes: true });
|
||||
entries.sort((left, right) => left.name.localeCompare(right.name));
|
||||
for (const entry of entries) {
|
||||
const fullPath = path.join(currentDirectory, entry.name);
|
||||
if (entry.isDirectory()) {
|
||||
await visit(fullPath);
|
||||
} else if (entry.isFile()) {
|
||||
files.push(path.relative(directory, fullPath).split(path.sep).join("/"));
|
||||
}
|
||||
}
|
||||
}
|
||||
await visit(directory);
|
||||
return files.sort();
|
||||
}
|
||||
|
||||
async function describeStage(stageName, stageDirectory, paths) {
|
||||
const files = [];
|
||||
for (const relativePath of paths) {
|
||||
const bytes = await readFile(path.join(stageDirectory, relativePath));
|
||||
files.push({ path: relativePath, bytes: bytes.byteLength, sha256: sha256(bytes) });
|
||||
}
|
||||
return { name: stageName, directory: stageName, files };
|
||||
}
|
||||
|
||||
export async function createCorpus(outDir, repoRoot = DEFAULT_REPO_ROOT) {
|
||||
const outputDirectory = path.resolve(outDir);
|
||||
const sourceRoot = path.resolve(repoRoot);
|
||||
const initialDirectory = path.join(outputDirectory, "initial");
|
||||
const overwriteDirectory = path.join(outputDirectory, "overwrite");
|
||||
const insertDirectory = path.join(outputDirectory, "insert");
|
||||
|
||||
await mkdir(outputDirectory, { recursive: true });
|
||||
|
||||
const initialContents = new Map();
|
||||
for (const relativePath of MARKDOWN_FILES) {
|
||||
initialContents.set(relativePath, await readFile(path.join(sourceRoot, relativePath)));
|
||||
}
|
||||
|
||||
const excerpts = await readSnippetParagraphs(sourceRoot);
|
||||
initialContents.set(
|
||||
CORPUS_PATHS.markdownSmall,
|
||||
Buffer.from(
|
||||
buildGeneratedMarkdown({
|
||||
targetBytes: SMALL_MARKDOWN_BYTES,
|
||||
title: "Generated Markdown · 512 KiB",
|
||||
variant: "small",
|
||||
excerpts,
|
||||
}),
|
||||
"utf8",
|
||||
),
|
||||
);
|
||||
initialContents.set(
|
||||
CORPUS_PATHS.markdownLarge,
|
||||
Buffer.from(
|
||||
buildGeneratedMarkdown({
|
||||
targetBytes: LARGE_MARKDOWN_BYTES,
|
||||
title: "Generated Markdown · 5 MiB",
|
||||
variant: "large",
|
||||
excerpts,
|
||||
}),
|
||||
"utf8",
|
||||
),
|
||||
);
|
||||
initialContents.set(CORPUS_PATHS.binary, makeRandomBytes(BINARY_BYTES, SEED));
|
||||
initialContents.set(CORPUS_PATHS.cloudantImage, await readFile(path.join(sourceRoot, "instruction_images/cloudant_1.png")));
|
||||
initialContents.set(CORPUS_PATHS.setupImage, await readFile(path.join(sourceRoot, "images/quick-setup/guide-quick-setup-first-setup-uri.png")));
|
||||
|
||||
const relativePaths = [...initialContents.keys()].sort();
|
||||
for (const [relativePath, contents] of initialContents) {
|
||||
await writeFileWithMtime(initialDirectory, relativePath, contents, INITIAL_MTIME);
|
||||
}
|
||||
|
||||
await copyStage(initialDirectory, overwriteDirectory, relativePaths);
|
||||
const smallMarkdownPath = path.join(overwriteDirectory, CORPUS_PATHS.markdownSmall);
|
||||
const smallMarkdownBefore = await readFile(smallMarkdownPath, "utf8");
|
||||
const originalIntro = "A deterministic UTF-8 fixture contains varied passages from the project documentation.";
|
||||
const changedIntro = "A deterministic UTF-8 fixture records a short content update using varied project passages.";
|
||||
if (!smallMarkdownBefore.includes(originalIntro)) throw new Error("Could not locate the generated Markdown paragraph to overwrite.");
|
||||
await writeFile(smallMarkdownPath, smallMarkdownBefore.replace(originalIntro, changedIntro));
|
||||
const binaryPath = path.join(overwriteDirectory, CORPUS_PATHS.binary);
|
||||
const initialBinary = await readFile(binaryPath);
|
||||
const binaryOverwriteOffset = Math.floor(initialBinary.byteLength * 0.25);
|
||||
const overwrittenBinary = Buffer.from(initialBinary);
|
||||
makeRandomBytes(BINARY_EDIT_BYTES, (SEED ^ 0x00a55a5a) >>> 0).copy(overwrittenBinary, binaryOverwriteOffset);
|
||||
await writeFile(binaryPath, overwrittenBinary);
|
||||
const alternatePng = await readFile(path.join(sourceRoot, "instruction_images/cloudant_2.png"));
|
||||
await writeFile(path.join(overwriteDirectory, CORPUS_PATHS.cloudantImage), alternatePng);
|
||||
for (const relativePath of [CORPUS_PATHS.markdownSmall, CORPUS_PATHS.binary, CORPUS_PATHS.cloudantImage]) {
|
||||
await setMtime(overwriteDirectory, relativePath, STAGE_MTIMES.overwrite);
|
||||
}
|
||||
|
||||
await copyStage(overwriteDirectory, insertDirectory, relativePaths);
|
||||
const insertedBinaryPath = path.join(insertDirectory, CORPUS_PATHS.binary);
|
||||
const overwrittenBinaryForInsert = await readFile(insertedBinaryPath);
|
||||
const binaryInsertOffset = Math.floor(overwrittenBinaryForInsert.byteLength * 0.25);
|
||||
const insertedBinary = Buffer.concat([
|
||||
overwrittenBinaryForInsert.subarray(0, binaryInsertOffset),
|
||||
makeRandomBytes(BINARY_EDIT_BYTES, (SEED ^ 0x005aa5a5) >>> 0),
|
||||
overwrittenBinaryForInsert.subarray(binaryInsertOffset),
|
||||
]);
|
||||
await writeFile(insertedBinaryPath, insertedBinary);
|
||||
|
||||
const insertedParagraphs = [
|
||||
[CORPUS_PATHS.markdownSmall, "The insert stage adds record 0047, profile delta-small, while retaining Unicode labels such as 日本語 and café."],
|
||||
[CORPUS_PATHS.markdownLarge, "The insert stage adds record 0913, profile delta-large, while retaining Unicode labels such as 中文 and Δοκιμή."],
|
||||
];
|
||||
for (const [relativePath, paragraph] of insertedParagraphs) {
|
||||
const markdownPath = path.join(insertDirectory, relativePath);
|
||||
const markdown = await readFile(markdownPath, "utf8");
|
||||
await writeFile(markdownPath, insertNearStart(markdown, paragraph));
|
||||
}
|
||||
for (const relativePath of [CORPUS_PATHS.binary, ...insertedParagraphs.map(([relativePath]) => relativePath)]) {
|
||||
await setMtime(insertDirectory, relativePath, STAGE_MTIMES.insert);
|
||||
}
|
||||
|
||||
const actualPaths = await listFiles(initialDirectory);
|
||||
for (const directory of [overwriteDirectory, insertDirectory]) {
|
||||
const stagePaths = await listFiles(directory);
|
||||
if (JSON.stringify(stagePaths) !== JSON.stringify(actualPaths)) {
|
||||
throw new Error(`Corpus stage ${path.basename(directory)} has different relative paths.`);
|
||||
}
|
||||
}
|
||||
|
||||
const sourceForPath = new Map(MARKDOWN_FILES.map((file) => [file, `repo:${file}`]));
|
||||
sourceForPath.set(CORPUS_PATHS.markdownSmall, "generated:utf8-markdown-512KiB-from-repository-snippets");
|
||||
sourceForPath.set(CORPUS_PATHS.markdownLarge, "generated:utf8-markdown-5MiB-from-repository-snippets");
|
||||
sourceForPath.set(CORPUS_PATHS.binary, `generated:seeded-high-entropy-binary-seed-${SEED.toString(16)}`);
|
||||
sourceForPath.set(CORPUS_PATHS.cloudantImage, "repo:instruction_images/cloudant_1.png");
|
||||
sourceForPath.set(CORPUS_PATHS.setupImage, "repo:images/quick-setup/guide-quick-setup-first-setup-uri.png");
|
||||
|
||||
const files = actualPaths.map((relativePath) => {
|
||||
const kind = relativePath.endsWith(".md") ? "markdown" : "binary";
|
||||
return { path: relativePath, kind, source: sourceForPath.get(relativePath) };
|
||||
});
|
||||
const stages = [
|
||||
await describeStage("initial", initialDirectory, actualPaths),
|
||||
await describeStage("overwrite", overwriteDirectory, actualPaths),
|
||||
await describeStage("insert", insertDirectory, actualPaths),
|
||||
];
|
||||
const manifest = { seed: SEED, files, stages };
|
||||
await writeFile(path.join(outputDirectory, "manifest.json"), `${JSON.stringify(manifest, null, 2)}\n`);
|
||||
return manifest;
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
const { writeFileSync } = require('node:fs');
|
||||
const { RTCDataChannel } = require('werift');
|
||||
|
||||
const output = process.env.LIVESYNC_BENCH_DATACHANNEL_JSON;
|
||||
if (!output) throw new Error('LIVESYNC_BENCH_DATACHANNEL_JSON is required');
|
||||
const channels = new Set();
|
||||
const originalSend = RTCDataChannel.prototype.send;
|
||||
RTCDataChannel.prototype.send = function (...args) {
|
||||
channels.add(this);
|
||||
return originalSend.apply(this, args);
|
||||
};
|
||||
process.once('exit', () => {
|
||||
const rows = [...channels].map((channel) => ({ label: channel.label, payloadBytesSent: channel.bytesSent, messagesSent: channel.messagesSent }));
|
||||
const cpu = process.cpuUsage();
|
||||
writeFileSync(output, JSON.stringify({
|
||||
source: 'werift 0.24.4 RTCDataChannel counters; observed channels through a benchmark-only preload',
|
||||
scope: 'Application DataChannel payload; includes application retransmissions, excludes SCTP/DTLS/UDP/IP overhead, transport retransmissions, and relay signalling',
|
||||
payloadBytesSent: rows.reduce((sum, row) => sum + row.payloadBytesSent, 0),
|
||||
messagesSent: rows.reduce((sum, row) => sum + row.messagesSent, 0),
|
||||
processUsage: { userCpuMs: cpu.user / 1000, systemCpuMs: cpu.system / 1000, maxResidentSetKiB: process.resourceUsage().maxRSS },
|
||||
channels: rows,
|
||||
}, null, 2));
|
||||
});
|
||||
@@ -0,0 +1,596 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { readFile, mkdir, writeFile } from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath, pathToFileURL } from "node:url";
|
||||
|
||||
import { HashManager } from "@vrtmrz/livesync-commonlib/hashing";
|
||||
import { createCorpus } from "./corpus.mjs";
|
||||
|
||||
const MODULE_DIR = path.dirname(fileURLToPath(import.meta.url));
|
||||
const PROJECT_ROOT = path.resolve(MODULE_DIR, "../..");
|
||||
const DEFAULT_SIZES = [0, 1, 4, 10, 20, 40, 60];
|
||||
const DEFAULT_REPEATS = 3;
|
||||
const ID_DERIVATION_KEY = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef";
|
||||
const LEGACY_PASSPHRASE = "public-local-chunk-size-benchmark-fixture";
|
||||
const CHUNK_MANAGER_WITHOUT_CACHE = {
|
||||
getCachedChunk: () => undefined,
|
||||
getChunkIDFromCache: () => false,
|
||||
};
|
||||
|
||||
const COMMONLIB_PACKAGE_JSON = fileURLToPath(import.meta.resolve("@vrtmrz/livesync-commonlib/package.json"));
|
||||
const COMMONLIB_ROOT = path.dirname(COMMONLIB_PACKAGE_JSON);
|
||||
const COMMONLIB_PACKAGE = JSON.parse(await readFile(COMMONLIB_PACKAGE_JSON, "utf8"));
|
||||
const { ContentSplitterRabinKarp } = await import(
|
||||
pathToFileURL(path.join(COMMONLIB_ROOT, "dist/ContentSplitter/ContentSplitterRabinKarp.js")).href
|
||||
);
|
||||
const { prepareChunk } = await import(
|
||||
pathToFileURL(path.join(COMMONLIB_ROOT, "dist/managers/EntryManager/EntryManagerImpls.js")).href
|
||||
);
|
||||
|
||||
function sha256(bytes) {
|
||||
return createHash("sha256").update(bytes).digest("hex");
|
||||
}
|
||||
|
||||
function parseArguments(args) {
|
||||
const parsed = { output: undefined, sizes: [...DEFAULT_SIZES], repeats: DEFAULT_REPEATS, help: false };
|
||||
for (let index = 0; index < args.length; index += 1) {
|
||||
const argument = args[index];
|
||||
if (argument === "--help" || argument === "-h") {
|
||||
parsed.help = true;
|
||||
continue;
|
||||
}
|
||||
const value = args[index + 1];
|
||||
if (!value || value.startsWith("--")) throw new Error(`Missing value for ${argument}.`);
|
||||
if (argument === "--output") {
|
||||
parsed.output = value;
|
||||
} else if (argument === "--sizes") {
|
||||
parsed.sizes = value.split(",").map((item) => {
|
||||
const trimmed = item.trim();
|
||||
if (!/^\d+$/u.test(trimmed)) throw new Error(`Invalid customChunkSize value: ${item}`);
|
||||
const size = Number(trimmed);
|
||||
if (!Number.isSafeInteger(size) || size > 10000) throw new Error(`customChunkSize is out of range: ${item}`);
|
||||
return size;
|
||||
});
|
||||
if (parsed.sizes.length === 0 || new Set(parsed.sizes).size !== parsed.sizes.length) {
|
||||
throw new Error("--sizes must contain one or more distinct integers.");
|
||||
}
|
||||
} else if (argument === "--repeats") {
|
||||
if (!/^\d+$/u.test(value)) throw new Error(`Invalid repeat count: ${value}`);
|
||||
parsed.repeats = Number(value);
|
||||
if (!Number.isSafeInteger(parsed.repeats) || parsed.repeats < 1 || parsed.repeats > 100) {
|
||||
throw new Error("--repeats must be an integer from 1 to 100.");
|
||||
}
|
||||
} else {
|
||||
throw new Error(`Unknown argument: ${argument}`);
|
||||
}
|
||||
index += 1;
|
||||
}
|
||||
if (!parsed.help && !parsed.output) throw new Error("--output <dir> is required.");
|
||||
return parsed;
|
||||
}
|
||||
|
||||
function usage() {
|
||||
return "Usage: node utils/chunk-size-benchmark/local.mjs --output <dir> [--sizes 0,1,4,10,20,40,60] [--repeats 3]";
|
||||
}
|
||||
|
||||
function settingsFor(customChunkSize, mode) {
|
||||
return {
|
||||
chunkSplitterVersion: "v3-rabin-karp",
|
||||
customChunkSize,
|
||||
minimumChunkSize: 20,
|
||||
disableWorkerForGeneratingChunks: true,
|
||||
processSmallFilesInUIThread: false,
|
||||
encrypt: true,
|
||||
E2EEAlgorithm: "v2",
|
||||
hashAlg: "xxhash64",
|
||||
passphrase: mode === "legacy" ? LEGACY_PASSPHRASE : "",
|
||||
idDerivationVersion: mode === "primary" ? 1 : 0,
|
||||
idDerivationKey: mode === "primary" ? ID_DERIVATION_KEY : "",
|
||||
enableCompression: false,
|
||||
useEden: false,
|
||||
};
|
||||
}
|
||||
|
||||
async function createCondition(customChunkSize) {
|
||||
const primarySettings = settingsFor(customChunkSize, "primary");
|
||||
const legacySettings = settingsFor(customChunkSize, "legacy");
|
||||
const primarySettingService = { currentSettings: () => primarySettings };
|
||||
const legacySettingService = { currentSettings: () => legacySettings };
|
||||
const splitter = new ContentSplitterRabinKarp({ settingService: primarySettingService });
|
||||
const primaryHashManager = new HashManager({ settingService: primarySettingService });
|
||||
const legacyHashManager = new HashManager({ settingService: legacySettingService });
|
||||
await Promise.all([splitter.initialised, primaryHashManager.initialise(), legacyHashManager.initialise()]);
|
||||
return { customChunkSize, splitter, primaryHashManager, legacyHashManager };
|
||||
}
|
||||
|
||||
async function warmCondition(condition, file) {
|
||||
await splitAndHashFile(condition, file, new Map(), new Map());
|
||||
}
|
||||
|
||||
async function loadCorpus(manifest, corpusDirectory) {
|
||||
const filesByPath = new Map(manifest.files.map((file) => [file.path, file]));
|
||||
const loadedStages = [];
|
||||
for (const stage of manifest.stages) {
|
||||
const stageDirectory = path.join(corpusDirectory, stage.directory);
|
||||
const loadedFiles = [];
|
||||
for (const expected of stage.files) {
|
||||
const metadata = filesByPath.get(expected.path);
|
||||
if (!metadata) throw new Error(`Corpus manifest has no file metadata for ${expected.path}.`);
|
||||
const data = await readFile(path.join(stageDirectory, expected.path));
|
||||
if (data.byteLength !== expected.bytes || sha256(data) !== expected.sha256) {
|
||||
throw new Error(`Corpus file does not match its manifest: ${stage.name}/${expected.path}`);
|
||||
}
|
||||
loadedFiles.push({ ...metadata, expectedSha256: expected.sha256, data });
|
||||
}
|
||||
loadedStages.push({ name: stage.name, files: loadedFiles });
|
||||
}
|
||||
return loadedStages;
|
||||
}
|
||||
|
||||
function decodeChunk(kind, piece) {
|
||||
return kind === "markdown" ? Buffer.from(piece, "utf8") : Buffer.from(piece, "base64");
|
||||
}
|
||||
|
||||
function assertSameContentForId(idsToPieces, id, piece, label) {
|
||||
const existing = idsToPieces.get(id);
|
||||
if (existing !== undefined && existing !== piece) {
|
||||
throw new Error(`${label} collision: chunk ID ${id} was produced for different content.`);
|
||||
}
|
||||
if (existing === undefined) idsToPieces.set(id, piece);
|
||||
}
|
||||
|
||||
async function splitAndHashFile(condition, file, primaryIdContents, legacyIdContents) {
|
||||
const blobType = file.kind === "markdown" ? "text/plain" : "application/octet-stream";
|
||||
const blob = new Blob([file.data], { type: blobType });
|
||||
const entry = { path: file.path, data: blob };
|
||||
const primaryChunks = [];
|
||||
const primaryCpuStart = process.cpuUsage();
|
||||
const primaryStart = process.hrtime.bigint();
|
||||
const generator = await condition.splitter.splitContent(entry);
|
||||
for await (const piece of generator) {
|
||||
const prepared = await prepareChunk(
|
||||
{ hashManager: condition.primaryHashManager, chunkManager: CHUNK_MANAGER_WITHOUT_CACHE },
|
||||
piece,
|
||||
);
|
||||
assertSameContentForId(primaryIdContents, prepared.id, piece, "Primary keyed hash");
|
||||
primaryChunks.push({ piece, id: prepared.id });
|
||||
}
|
||||
const primaryElapsedMs = Number(process.hrtime.bigint() - primaryStart) / 1e6;
|
||||
const primaryCpu = process.cpuUsage(primaryCpuStart);
|
||||
const primaryCpuMs = (primaryCpu.user + primaryCpu.system) / 1000;
|
||||
|
||||
const legacyCpuStart = process.cpuUsage();
|
||||
const legacyStart = process.hrtime.bigint();
|
||||
for (const chunk of primaryChunks) {
|
||||
const prepared = await prepareChunk(
|
||||
{ hashManager: condition.legacyHashManager, chunkManager: CHUNK_MANAGER_WITHOUT_CACHE },
|
||||
chunk.piece,
|
||||
);
|
||||
assertSameContentForId(legacyIdContents, prepared.id, chunk.piece, "Legacy encrypted xxhash64");
|
||||
chunk.legacyId = prepared.id;
|
||||
}
|
||||
const legacyIdElapsedMs = Number(process.hrtime.bigint() - legacyStart) / 1e6;
|
||||
const legacyCpu = process.cpuUsage(legacyCpuStart);
|
||||
const legacyIdCpuMs = (legacyCpu.user + legacyCpu.system) / 1000;
|
||||
|
||||
const decoded = primaryChunks.map((chunk) => decodeChunk(file.kind, chunk.piece));
|
||||
const reconstructed = Buffer.concat(decoded);
|
||||
if (!reconstructed.equals(file.data) || sha256(reconstructed) !== file.expectedSha256) {
|
||||
throw new Error(`Splitter output did not reconstruct ${file.path} in the ${file.kind} encoding.`);
|
||||
}
|
||||
|
||||
const chunks = primaryChunks.map((chunk, index) => {
|
||||
const raw = decoded[index];
|
||||
return {
|
||||
id: chunk.id,
|
||||
legacyId: chunk.legacyId,
|
||||
rawBytes: raw.byteLength,
|
||||
storedUtf8Bytes: Buffer.byteLength(chunk.piece, "utf8"),
|
||||
dataSha256: sha256(raw),
|
||||
idUtf8Bytes: Buffer.byteLength(chunk.id, "utf8"),
|
||||
legacyIdUtf8Bytes: Buffer.byteLength(chunk.legacyId, "utf8"),
|
||||
piece: chunk.piece,
|
||||
};
|
||||
});
|
||||
const fingerprintInput = chunks.map(({ id, legacyId, rawBytes, dataSha256 }) => [id, legacyId, rawBytes, dataSha256]);
|
||||
const chunkFingerprintSha256 = sha256(Buffer.from(JSON.stringify(fingerprintInput), "utf8"));
|
||||
return {
|
||||
chunks,
|
||||
primaryElapsedMs,
|
||||
primaryCpuMs,
|
||||
legacyIdElapsedMs,
|
||||
legacyIdCpuMs,
|
||||
chunkFingerprintSha256,
|
||||
};
|
||||
}
|
||||
|
||||
function percentile(sortedValues, fraction) {
|
||||
if (sortedValues.length === 0) return 0;
|
||||
return sortedValues[Math.max(0, Math.ceil(sortedValues.length * fraction) - 1)];
|
||||
}
|
||||
|
||||
function summarizeRawChunkBytes(values) {
|
||||
const sorted = [...values].sort((left, right) => left - right);
|
||||
return {
|
||||
count: sorted.length,
|
||||
p50: percentile(sorted, 0.5),
|
||||
p90: percentile(sorted, 0.9),
|
||||
p95: percentile(sorted, 0.95),
|
||||
p99: percentile(sorted, 0.99),
|
||||
max: sorted.at(-1) ?? 0,
|
||||
};
|
||||
}
|
||||
|
||||
function summariseIdLengths(chunks, key) {
|
||||
const lengths = chunks.map((chunk) => Buffer.byteLength(chunk[key], "utf8")).sort((left, right) => left - right);
|
||||
const uniqueLengths = new Map();
|
||||
for (const chunk of chunks) uniqueLengths.set(chunk[key], Buffer.byteLength(chunk[key], "utf8"));
|
||||
const uniqueValues = [...uniqueLengths.values()].sort((left, right) => left - right);
|
||||
return {
|
||||
occurrences: lengths.length,
|
||||
totalUtf8Bytes: lengths.reduce((sum, value) => sum + value, 0),
|
||||
minLengthBytes: lengths[0] ?? 0,
|
||||
medianLengthBytes: percentile(lengths, 0.5),
|
||||
maxLengthBytes: lengths.at(-1) ?? 0,
|
||||
uniqueIds: uniqueValues.length,
|
||||
uniqueTotalUtf8Bytes: uniqueValues.reduce((sum, value) => sum + value, 0),
|
||||
};
|
||||
}
|
||||
|
||||
function uniqueById(chunks, idKey = "id") {
|
||||
const unique = new Map();
|
||||
for (const chunk of chunks) {
|
||||
if (!unique.has(chunk[idKey])) unique.set(chunk[idKey], chunk);
|
||||
}
|
||||
return unique;
|
||||
}
|
||||
|
||||
function dataBytes(chunks, key) {
|
||||
return chunks.reduce((sum, chunk) => sum + chunk[key], 0);
|
||||
}
|
||||
|
||||
function summarizeFile(file, measured, includeChunkArray) {
|
||||
const { chunks } = measured;
|
||||
const unique = uniqueById(chunks);
|
||||
const fileMetric = {
|
||||
path: file.path,
|
||||
kind: file.kind,
|
||||
sourceBytes: file.data.byteLength,
|
||||
chunkCount: chunks.length,
|
||||
uniqueChunkCount: unique.size,
|
||||
storedChunkDataUtf8Bytes: dataBytes(chunks, "storedUtf8Bytes"),
|
||||
uniqueChunkDataUtf8Bytes: dataBytes([...unique.values()], "storedUtf8Bytes"),
|
||||
uniqueChunkRawBytes: dataBytes([...unique.values()], "rawBytes"),
|
||||
rawChunkBytes: summarizeRawChunkBytes(chunks.map((chunk) => chunk.rawBytes)),
|
||||
primaryId: summariseIdLengths(chunks, "id"),
|
||||
legacyId: summariseIdLengths(chunks, "legacyId"),
|
||||
chunkFingerprintSha256: measured.chunkFingerprintSha256,
|
||||
timing: {
|
||||
primarySplitAndIdElapsedMs: measured.primaryElapsedMs,
|
||||
primarySplitAndIdCpuMs: measured.primaryCpuMs,
|
||||
legacyIdElapsedMs: measured.legacyIdElapsedMs,
|
||||
legacyIdCpuMs: measured.legacyIdCpuMs,
|
||||
},
|
||||
};
|
||||
if (includeChunkArray) {
|
||||
fileMetric.chunks = chunks.map(({ id, legacyId, rawBytes, storedUtf8Bytes, dataSha256, idUtf8Bytes, legacyIdUtf8Bytes }) => ({
|
||||
id,
|
||||
legacyId,
|
||||
rawBytes,
|
||||
storedUtf8Bytes,
|
||||
dataSha256,
|
||||
idUtf8Bytes,
|
||||
legacyIdUtf8Bytes,
|
||||
}));
|
||||
}
|
||||
return fileMetric;
|
||||
}
|
||||
|
||||
function aggregateKind(stage, kind, fileResults, historyIds, previousIds) {
|
||||
const files = fileResults.filter((item) => item.file.kind === kind);
|
||||
const chunks = files.flatMap((item) => item.measured.chunks);
|
||||
const unique = uniqueById(chunks);
|
||||
const uniqueChunks = [...unique.values()];
|
||||
const newVsHistory = uniqueChunks.filter((chunk) => !historyIds.has(chunk.id));
|
||||
const newVsPrevious = previousIds === null ? null : uniqueChunks.filter((chunk) => !previousIds.has(chunk.id));
|
||||
const storedBytes = dataBytes(chunks, "storedUtf8Bytes");
|
||||
const primaryIds = summariseIdLengths(chunks, "id");
|
||||
const legacyIds = summariseIdLengths(chunks, "legacyId");
|
||||
|
||||
return {
|
||||
kind,
|
||||
fileCount: files.length,
|
||||
sourceBytes: files.reduce((sum, item) => sum + item.file.data.byteLength, 0),
|
||||
chunkCount: chunks.length,
|
||||
uniqueChunkCount: unique.size,
|
||||
storedChunkDataUtf8Bytes: storedBytes,
|
||||
uniqueChunkDataUtf8Bytes: dataBytes(uniqueChunks, "storedUtf8Bytes"),
|
||||
uniqueChunkRawBytes: dataBytes(uniqueChunks, "rawBytes"),
|
||||
rawChunkBytes: summarizeRawChunkBytes(chunks.map((chunk) => chunk.rawBytes)),
|
||||
primaryId: primaryIds,
|
||||
legacyId: legacyIds,
|
||||
newUniqueChunksVsHistory: newVsHistory.length,
|
||||
newUniqueChunkDataUtf8BytesVsHistory: dataBytes(newVsHistory, "storedUtf8Bytes"),
|
||||
newUniqueChunkRawBytesVsHistory: dataBytes(newVsHistory, "rawBytes"),
|
||||
newUniqueChunksVsPreviousStage: newVsPrevious === null ? null : newVsPrevious.length,
|
||||
newUniqueChunkDataUtf8BytesVsPreviousStage: newVsPrevious === null ? null : dataBytes(newVsPrevious, "storedUtf8Bytes"),
|
||||
newUniqueChunkRawBytesVsPreviousStage: newVsPrevious === null ? null : dataBytes(newVsPrevious, "rawBytes"),
|
||||
primaryElapsedMs: files.reduce((sum, item) => sum + item.measured.primaryElapsedMs, 0),
|
||||
primaryCpuMs: files.reduce((sum, item) => sum + item.measured.primaryCpuMs, 0),
|
||||
legacyIdElapsedMs: files.reduce((sum, item) => sum + item.measured.legacyIdElapsedMs, 0),
|
||||
legacyIdCpuMs: files.reduce((sum, item) => sum + item.measured.legacyIdCpuMs, 0),
|
||||
};
|
||||
}
|
||||
|
||||
async function runCondition(condition, stages, repeat, order, includeChunkArray) {
|
||||
const primaryIdContents = new Map();
|
||||
const legacyIdContents = new Map();
|
||||
const historyIds = new Set();
|
||||
let previousIds = null;
|
||||
const stageResults = [];
|
||||
|
||||
for (const stage of stages) {
|
||||
const fileResults = [];
|
||||
for (const file of stage.files) {
|
||||
const measured = await splitAndHashFile(condition, file, primaryIdContents, legacyIdContents);
|
||||
fileResults.push({ file, measured });
|
||||
}
|
||||
|
||||
const currentIds = new Set();
|
||||
for (const item of fileResults) {
|
||||
for (const chunk of item.measured.chunks) currentIds.add(chunk.id);
|
||||
}
|
||||
const kinds = [...new Set(fileResults.map((item) => item.file.kind))].sort();
|
||||
const byKind = kinds.map((kind) => aggregateKind(stage, kind, fileResults, historyIds, previousIds));
|
||||
const stageFingerprint = fileResults
|
||||
.map((item) => `${item.file.path}\0${item.measured.chunkFingerprintSha256}`)
|
||||
.sort()
|
||||
.join("\n");
|
||||
stageResults.push({
|
||||
name: stage.name,
|
||||
fingerprintSha256: sha256(Buffer.from(stageFingerprint, "utf8")),
|
||||
byKind,
|
||||
files: fileResults.map((item) => summarizeFile(item.file, item.measured, includeChunkArray)),
|
||||
});
|
||||
|
||||
for (const id of currentIds) historyIds.add(id);
|
||||
previousIds = currentIds;
|
||||
}
|
||||
|
||||
return { repeat, conditionOrder: order, stages: stageResults };
|
||||
}
|
||||
|
||||
function median(values) {
|
||||
const sorted = [...values].sort((left, right) => left - right);
|
||||
if (sorted.length === 0) return 0;
|
||||
const middle = Math.floor(sorted.length / 2);
|
||||
return sorted.length % 2 === 0 ? (sorted[middle - 1] + sorted[middle]) / 2 : sorted[middle];
|
||||
}
|
||||
|
||||
function orderForRepeat(sizes, repeat) {
|
||||
const offset = repeat % sizes.length;
|
||||
return [...sizes.slice(offset), ...sizes.slice(0, offset)];
|
||||
}
|
||||
|
||||
function timingsForRuns(runs, stageName, kind) {
|
||||
const samples = runs.map((run) => {
|
||||
const stage = run.stages.find((item) => item.name === stageName);
|
||||
const group = stage.byKind.find((item) => item.kind === kind);
|
||||
return {
|
||||
repeat: run.repeat,
|
||||
primaryElapsedMs: group.primaryElapsedMs,
|
||||
primaryCpuMs: group.primaryCpuMs,
|
||||
legacyIdElapsedMs: group.legacyIdElapsedMs,
|
||||
legacyIdCpuMs: group.legacyIdCpuMs,
|
||||
};
|
||||
});
|
||||
return {
|
||||
primarySplitAndIdElapsedMsMedian: median(samples.map((sample) => sample.primaryElapsedMs)),
|
||||
primarySplitAndIdCpuMsMedian: median(samples.map((sample) => sample.primaryCpuMs)),
|
||||
legacyIdElapsedMsMedian: median(samples.map((sample) => sample.legacyIdElapsedMs)),
|
||||
legacyIdCpuMsMedian: median(samples.map((sample) => sample.legacyIdCpuMs)),
|
||||
repeats: samples,
|
||||
};
|
||||
}
|
||||
|
||||
function timingsForFile(runs, stageName, filePath) {
|
||||
const samples = runs.map((run) => {
|
||||
const stage = run.stages.find((item) => item.name === stageName);
|
||||
const file = stage.files.find((item) => item.path === filePath);
|
||||
return { repeat: run.repeat, ...file.timing };
|
||||
});
|
||||
return {
|
||||
primarySplitAndIdElapsedMsMedian: median(samples.map((sample) => sample.primarySplitAndIdElapsedMs)),
|
||||
primarySplitAndIdCpuMsMedian: median(samples.map((sample) => sample.primarySplitAndIdCpuMs)),
|
||||
legacyIdElapsedMsMedian: median(samples.map((sample) => sample.legacyIdElapsedMs)),
|
||||
legacyIdCpuMsMedian: median(samples.map((sample) => sample.legacyIdCpuMs)),
|
||||
repeats: samples,
|
||||
};
|
||||
}
|
||||
|
||||
function assertRepeatStability(runs, customChunkSize) {
|
||||
const first = runs[0];
|
||||
for (const run of runs.slice(1)) {
|
||||
for (const stage of first.stages) {
|
||||
const currentStage = run.stages.find((item) => item.name === stage.name);
|
||||
if (currentStage.fingerprintSha256 !== stage.fingerprintSha256) {
|
||||
throw new Error(`Chunk boundaries or IDs changed across repeats for customChunkSize=${customChunkSize}, stage=${stage.name}.`);
|
||||
}
|
||||
for (const file of stage.files) {
|
||||
const currentFile = currentStage.files.find((item) => item.path === file.path);
|
||||
if (currentFile.chunkFingerprintSha256 !== file.chunkFingerprintSha256) {
|
||||
throw new Error(`Chunk IDs changed across repeats for ${stage.name}/${file.path} at customChunkSize=${customChunkSize}.`);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function compareCandidateBoundaries(candidates, sizes) {
|
||||
const candidate40 = candidates.get(40);
|
||||
const candidate60 = candidates.get(60);
|
||||
if (!sizes.includes(40) || !sizes.includes(60)) return null;
|
||||
for (const stage of candidate40.stages) {
|
||||
const otherStage = candidate60.stages.find((item) => item.name === stage.name);
|
||||
for (const file of stage.files) {
|
||||
const other = otherStage.files.find((item) => item.path === file.path);
|
||||
if (file.chunkFingerprintSha256 !== other.chunkFingerprintSha256) {
|
||||
throw new Error(`customChunkSize=40 and 60 differ for ${stage.name}/${file.path}.`);
|
||||
}
|
||||
}
|
||||
}
|
||||
return {
|
||||
customChunkSize40And60SamePerFileBoundariesAndIds: true,
|
||||
comparedStages: candidate40.stages.length,
|
||||
comparedFilesPerStage: candidate40.stages[0]?.files.length ?? 0,
|
||||
};
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const args = parseArguments(process.argv.slice(2));
|
||||
if (args.help) {
|
||||
process.stdout.write(`${usage()}\n`);
|
||||
return;
|
||||
}
|
||||
|
||||
const outputDirectory = path.resolve(args.output);
|
||||
const corpusDirectory = path.join(outputDirectory, "corpus");
|
||||
const resultPath = path.join(outputDirectory, "results.json");
|
||||
await mkdir(outputDirectory, { recursive: true });
|
||||
const startedAt = new Date().toISOString();
|
||||
if (COMMONLIB_PACKAGE.version !== "0.1.44") {
|
||||
throw new Error(`Expected @vrtmrz/livesync-commonlib 0.1.44, found ${COMMONLIB_PACKAGE.version}.`);
|
||||
}
|
||||
const manifest = await createCorpus(corpusDirectory, PROJECT_ROOT);
|
||||
const manifestJson = JSON.stringify(manifest);
|
||||
const corpusManifestSha256 = sha256(Buffer.from(manifestJson, "utf8"));
|
||||
const stages = await loadCorpus(manifest, corpusDirectory);
|
||||
const packageJson = JSON.parse(await readFile(path.join(PROJECT_ROOT, "package.json"), "utf8"));
|
||||
|
||||
if (ID_DERIVATION_KEY.length !== 64 || !/^[0-9a-f]{64}$/u.test(ID_DERIVATION_KEY)) {
|
||||
throw new Error("The benchmark ID derivation fixture must be 64 lower-case hex characters.");
|
||||
}
|
||||
|
||||
const conditions = new Map();
|
||||
for (const size of args.sizes) {
|
||||
const condition = await createCondition(size);
|
||||
const warmupFile = stages[0].files.find((file) => file.kind === "markdown" && file.source.startsWith("repo:"));
|
||||
if (!warmupFile) throw new Error("Corpus has no repository Markdown file for warm-up.");
|
||||
await warmCondition(condition, warmupFile);
|
||||
conditions.set(size, condition);
|
||||
}
|
||||
|
||||
const runsBySize = new Map(args.sizes.map((size) => [size, []]));
|
||||
for (let repeat = 0; repeat < args.repeats; repeat += 1) {
|
||||
const order = orderForRepeat(args.sizes, repeat);
|
||||
for (const size of order) {
|
||||
const includeChunkArray = repeat === 0 && (size === 40 || size === 60);
|
||||
const run = await runCondition(conditions.get(size), stages, repeat + 1, order, includeChunkArray);
|
||||
runsBySize.get(size).push(run);
|
||||
}
|
||||
}
|
||||
|
||||
const candidates = new Map();
|
||||
for (const size of args.sizes) {
|
||||
const runs = runsBySize.get(size);
|
||||
assertRepeatStability(runs, size);
|
||||
const firstRun = runs[0];
|
||||
const detailedStages = firstRun.stages.map((stage) => ({
|
||||
name: stage.name,
|
||||
fingerprintSha256: stage.fingerprintSha256,
|
||||
byKind: stage.byKind.map((group) => {
|
||||
const { primaryElapsedMs, primaryCpuMs, legacyIdElapsedMs, legacyIdCpuMs, ...metrics } = group;
|
||||
return {
|
||||
...metrics,
|
||||
timing: timingsForRuns(runs, stage.name, group.kind),
|
||||
};
|
||||
}),
|
||||
files: stage.files.map(({ timing, ...file }) => ({
|
||||
...file,
|
||||
timing: timingsForFile(runs, stage.name, file.path),
|
||||
})),
|
||||
}));
|
||||
const candidate = {
|
||||
customChunkSize: size,
|
||||
chunkSizeCeilingBytes: Math.floor(102400 * (size + 1)),
|
||||
stages: detailedStages,
|
||||
runs: runs.map((run) => ({
|
||||
repeat: run.repeat,
|
||||
conditionOrder: run.conditionOrder,
|
||||
stages: run.stages.map((stage) => ({ name: stage.name, fingerprintSha256: stage.fingerprintSha256 })),
|
||||
})),
|
||||
};
|
||||
candidates.set(size, candidate);
|
||||
}
|
||||
|
||||
const comparisons = compareCandidateBoundaries(candidates, args.sizes);
|
||||
const results = {
|
||||
schemaVersion: 1,
|
||||
benchmark: "local Rabin-Karp splitting and chunk-ID measurement",
|
||||
startedAt,
|
||||
completedAt: new Date().toISOString(),
|
||||
runtime: {
|
||||
node: process.version,
|
||||
v8: process.versions.v8,
|
||||
platform: process.platform,
|
||||
architecture: process.arch,
|
||||
osRelease: os.release(),
|
||||
cpuModel: os.cpus()[0]?.model ?? "unknown",
|
||||
logicalCpuCount: os.cpus().length,
|
||||
projectVersion: packageJson.version,
|
||||
commonlibVersion: COMMONLIB_PACKAGE.version,
|
||||
variant: process.env.LIVESYNC_BENCH_VARIANT || "unlabelled",
|
||||
commonlibChunksSha256: sha256(await readFile(path.join(COMMONLIB_ROOT, "dist/string_and_binary/chunks.js"))),
|
||||
},
|
||||
settings: {
|
||||
splitter: "v3-rabin-karp",
|
||||
minimumChunkSize: 20,
|
||||
workerDisabled: true,
|
||||
E2EEAlgorithm: "v2",
|
||||
encrypt: true,
|
||||
hashAlg: "xxhash64",
|
||||
primaryIdDerivation: {
|
||||
version: 1,
|
||||
keyFixture: "fixed public benchmark fixture; 64 lower-case hex characters omitted",
|
||||
},
|
||||
legacyIdComparison: {
|
||||
idDerivationVersion: 0,
|
||||
algorithm: "encrypted xxhash64 with the fixed public legacy passphrase fixture",
|
||||
passphrase: "omitted",
|
||||
},
|
||||
enableCompression: false,
|
||||
useEden: false,
|
||||
customChunkSizeValues: args.sizes,
|
||||
warmup: "one actual repository Markdown file per condition; excluded from timings",
|
||||
timings: "primary includes splitting and actual EntryManager prepareChunk hashing; legacy timing hashes the same emitted chunks separately",
|
||||
},
|
||||
corpus: {
|
||||
seed: manifest.seed,
|
||||
manifestSha256: corpusManifestSha256,
|
||||
fileCount: manifest.files.length,
|
||||
stages: manifest.stages.map((stage) => ({
|
||||
name: stage.name,
|
||||
directory: stage.directory,
|
||||
fileCount: stage.files.length,
|
||||
sourceBytes: stage.files.reduce((sum, file) => sum + file.bytes, 0),
|
||||
})),
|
||||
},
|
||||
assertions: {
|
||||
everyFileReconstructedFromSplitterOutput: true,
|
||||
duplicateIdsHadIdenticalChunkContent: true,
|
||||
sameBoundariesAndIdsAcrossRepeats: true,
|
||||
...(comparisons ?? {}),
|
||||
},
|
||||
candidates: [...candidates.values()],
|
||||
scope: "Local splitting and hashing only; no network or request timing is measured.",
|
||||
};
|
||||
|
||||
await writeFile(resultPath, `${JSON.stringify(results, null, 2)}\n`);
|
||||
process.stdout.write(`${JSON.stringify({ resultPath, corpusManifestSha256, candidateCount: args.sizes.length, repeats: args.repeats })}\n`);
|
||||
}
|
||||
|
||||
main().catch((error) => {
|
||||
process.stderr.write(`${error instanceof Error ? error.stack ?? error.message : String(error)}\n`);
|
||||
process.exitCode = 1;
|
||||
});
|
||||
@@ -0,0 +1,159 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import { createHash } from 'node:crypto';
|
||||
import { spawn } from 'node:child_process';
|
||||
import { promises as fs } from 'node:fs';
|
||||
import { dirname, join, resolve } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { setTimeout as delay } from 'node:timers/promises';
|
||||
import { createNewVaultSettings, REMOTE_P2P, SALT_OF_PASSPHRASE } from '@vrtmrz/livesync-commonlib/compat/common/types';
|
||||
import { encryptString } from '@vrtmrz/livesync-commonlib/compat/encryption/stringEncryption';
|
||||
|
||||
const root = resolve(dirname(fileURLToPath(import.meta.url)), '../..');
|
||||
const cli = join(root, 'src/apps/cli/dist/index.cjs');
|
||||
const preload = join(root, 'utils/chunk-size-benchmark/datachannel-metrics.cjs');
|
||||
const args = process.argv.slice(2);
|
||||
const option = (key, fallback) => { const at = args.indexOf(`--${key}`); return at < 0 ? fallback : args[at + 1]; };
|
||||
const sourceCli = resolve(option('source-cli', cli));
|
||||
const targetCli = resolve(option('target-cli', cli));
|
||||
const output = resolve(option('output', join(root, '.bench/chunk-size')));
|
||||
const corpusPath = resolve(option('corpus', join(output, 'corpus/manifest.json')));
|
||||
const sizes = option('sizes', '0,4,10,40').split(',').map(Number);
|
||||
const repeats = Number(option('repeats', '3'));
|
||||
const sha256 = (bytes) => createHash('sha256').update(bytes).digest('hex');
|
||||
const cliSha256 = sha256(await fs.readFile(cli));
|
||||
const sourceCliSha256 = sha256(await fs.readFile(sourceCli));
|
||||
const targetCliSha256 = sha256(await fs.readFile(targetCli));
|
||||
const environment = { ...process.env, NODE_OPTIONS: '--max-old-space-size=2048' };
|
||||
|
||||
async function runCli(vault, settings, command, commandArgs, logPath, extraEnvironment = {}, entrypoint = cli) {
|
||||
const marker = '__BENCH_TIME__';
|
||||
const started = performance.now();
|
||||
const nodeArgs = extraEnvironment.LIVESYNC_BENCH_DATACHANNEL_JSON ? ['--require', preload] : [];
|
||||
const child = spawn('/usr/bin/time', ['-f', `${marker}%U\t%S\t%M`, process.execPath, ...nodeArgs, entrypoint, vault, '--settings', settings, command, ...commandArgs], {
|
||||
cwd: root, detached: true, stdio: ['ignore', 'pipe', 'pipe'], env: { ...environment, ...extraEnvironment },
|
||||
});
|
||||
let stdout = ''; let stderr = '';
|
||||
child.stdout.on('data', (data) => { stdout += data; });
|
||||
child.stderr.on('data', (data) => { stderr += data; });
|
||||
const timeout = setTimeout(() => { try { process.kill(-child.pid, 'SIGTERM'); } catch (error) { if (error.code !== 'ESRCH') throw error; } }, 240000);
|
||||
const code = await new Promise((accept, reject) => { child.once('error', reject); child.once('close', accept); });
|
||||
clearTimeout(timeout);
|
||||
await fs.writeFile(logPath, `${stdout}\n${stderr}`);
|
||||
if (code !== 0) throw new Error(`CLI ${command} exited ${code}; ${logPath}\n${stderr.slice(-2000)}`);
|
||||
const line = stderr.split(/\r?\n/).find((row) => row.startsWith(marker));
|
||||
assert(line, 'GNU time output is missing');
|
||||
const [userSeconds, systemSeconds, maxResidentSetKiB] = line.slice(marker.length).split('\t').map(Number);
|
||||
return { elapsedMs: performance.now() - started, userCpuMs: userSeconds * 1000, systemCpuMs: systemSeconds * 1000, maxResidentSetKiB };
|
||||
}
|
||||
|
||||
async function startHost(vault, settings, logPath, metricsPath) {
|
||||
const child = spawn(process.execPath, ['--require', preload, sourceCli, vault, '--settings', settings, 'p2p-host'], { cwd: root, stdio: ['ignore', 'pipe', 'pipe'], env: { ...environment, LIVESYNC_BENCH_DATACHANNEL_JSON: metricsPath } });
|
||||
let combined = ''; let ended = false; let failure;
|
||||
const closed = new Promise((accept) => { child.once('error', (error) => { failure = error; ended = true; accept(); }); child.once('close', () => { ended = true; accept(); }); });
|
||||
child.stdout.on('data', (data) => { combined += data; }); child.stderr.on('data', (data) => { combined += data; });
|
||||
const stop = async () => {
|
||||
if (!ended) child.kill('SIGTERM');
|
||||
const timeout = setTimeout(() => { if (!ended) child.kill('SIGKILL'); }, 10000);
|
||||
await closed; clearTimeout(timeout); await fs.writeFile(logPath, combined);
|
||||
};
|
||||
const started = performance.now();
|
||||
try {
|
||||
while (!combined.includes('P2P host is running')) {
|
||||
if (ended || performance.now() - started > 25000) throw failure || new Error(`P2P host did not start: ${combined.slice(-2500)}`);
|
||||
await delay(100);
|
||||
}
|
||||
} catch (error) { await stop(); throw error; }
|
||||
return { stop, readyMs: performance.now() - started };
|
||||
}
|
||||
|
||||
async function settingsFor(size, room, name) {
|
||||
return {
|
||||
...createNewVaultSettings(), isConfigured: true, remoteType: REMOTE_P2P,
|
||||
encrypt: true, passphrase: 'chunk-size-benchmark-only-passphrase', usePathObfuscation: true, encryptInternalMetadata: true,
|
||||
idDerivationVersion: 1, idDerivationKey: '', encryptedIdDerivationKey: await encryptString('0123456789abcdef'.repeat(4), '*' + SALT_OF_PASSPHRASE),
|
||||
E2EEAlgorithm: 'v2', hashAlg: 'xxhash64', chunkSplitterVersion: 'v3-rabin-karp', minimumChunkSize: 20, customChunkSize: size,
|
||||
enableCompression: false, useEden: false, useIndexedDBAdapter: false, handleFilenameCaseSensitive: false, pathNormalizationVersion: 0,
|
||||
autoAcceptCompatibleTweak: false, readChunksOnline: false, liveSync: false, syncOnStart: false, syncOnSave: false, periodicReplication: false,
|
||||
syncOnFileOpen: false, syncMaxSizeInMB: 50,
|
||||
P2P_Enabled: true, P2P_AutoStart: false, P2P_AutoBroadcast: false, P2P_IsHeadless: true,
|
||||
P2P_AppID: 'livesync-chunk-size-benchmark', P2P_roomID: room, P2P_passphrase: 'chunk-size-local-relay-fixture',
|
||||
P2P_relays: 'ws://127.0.0.1:14094/', P2P_AutoAcceptingPeers: '~.*', P2P_AutoDenyingPeers: '',
|
||||
P2P_turnServers: '', P2P_turnUsername: '', P2P_turnCredential: '', P2P_DevicePeerName: name,
|
||||
};
|
||||
}
|
||||
|
||||
async function copyStage(stage, vault, previous) {
|
||||
const old = new Map((previous?.files || []).map((file) => [file.path, file.sha256]));
|
||||
const changed = [];
|
||||
for (const file of stage.files) {
|
||||
if (old.get(file.path) === file.sha256) continue;
|
||||
const from = join(stage.directory, file.path); const to = join(vault, file.path);
|
||||
await fs.mkdir(dirname(to), { recursive: true }); await fs.copyFile(from, to);
|
||||
const stat = await fs.stat(from); await fs.utimes(to, stat.atime, stat.mtime); changed.push(file.path);
|
||||
}
|
||||
return changed;
|
||||
}
|
||||
|
||||
await fs.mkdir(join(output, 'p2p'), { recursive: true });
|
||||
const manifest = JSON.parse(await fs.readFile(corpusPath, 'utf8'));
|
||||
for (const stage of manifest.stages) stage.directory = resolve(dirname(corpusPath), stage.directory);
|
||||
const corpusDigest = sha256(JSON.stringify(manifest.stages.map((stage) => ({ name: stage.name, files: stage.files }))));
|
||||
const resultPath = join(output, 'p2p/results.json');
|
||||
let results = [];
|
||||
try { const saved = JSON.parse(await fs.readFile(resultPath, 'utf8')); assert.equal(saved.corpusDigest, corpusDigest); assert.equal(saved.sourceCliSha256 ?? saved.cliSha256, sourceCliSha256); assert.equal(saved.targetCliSha256 ?? saved.cliSha256, targetCliSha256); results = saved.results; } catch (error) { if (error.code !== 'ENOENT') throw error; }
|
||||
const save = () => fs.writeFile(resultPath, JSON.stringify({ version: '1.0.39', commonlib: '0.1.44', variant: process.env.LIVESYNC_BENCH_VARIANT || 'unlabelled', cliSha256, sourceCliSha256, targetCliSha256, corpusDigest,
|
||||
scope: 'Local WebRTC direct path; timed target command includes process start-up, peer discovery, connection establishment, pull and reverse synchronisation. DataChannel payload bytes are measured using werift counters observed through a preload on both endpoints; they exclude transport overhead, transport retransmissions, and relay signalling. ICE byte counters are retained as diagnostics and report zero in this runtime. GNU time covers the target process; the preload also records CPU/RSS on each endpoint. Source mirroring and host start-up are recorded separately. E2EE V2 settings with independent IDs; compression disabled.', results }, null, 2));
|
||||
for (let repeat = 0; repeat < repeats; repeat++) {
|
||||
const rotation = repeat % sizes.length; const order = [...sizes.slice(rotation), ...sizes.slice(0, rotation)];
|
||||
for (const size of order) {
|
||||
const name = `p2p-c${size}-r${repeat + 1}`;
|
||||
if (results.some((result) => result.name === name && result.complete)) continue;
|
||||
const work = join(output, 'p2p', name); await fs.mkdir(join(work, 'logs'), { recursive: true });
|
||||
const a = join(work, 'source'); const b = join(work, 'target');
|
||||
await fs.rm(a, { recursive: true, force: true }); await fs.rm(b, { recursive: true, force: true });
|
||||
await fs.mkdir(a, { recursive: true }); await fs.mkdir(b, { recursive: true });
|
||||
const room = `chunk-size-${Date.now().toString(36)}-${name}`;
|
||||
const hostName = `host-${name}`;
|
||||
const settingsA = join(work, 'source-settings.json'); const settingsB = join(work, 'target-settings.json');
|
||||
await fs.writeFile(settingsA, JSON.stringify(await settingsFor(size, room, hostName)));
|
||||
await fs.writeFile(settingsB, JSON.stringify(await settingsFor(size, room, `target-${name}`)));
|
||||
const result = { name, customChunkSize: size, repeat: repeat + 1, stages: [], complete: false };
|
||||
results.push(result); await save();
|
||||
let previous;
|
||||
try {
|
||||
for (const stage of manifest.stages) {
|
||||
const log = (label) => join(work, 'logs', `${stage.name}-${label}.txt`);
|
||||
const changedFiles = await copyStage(stage, a, previous);
|
||||
const mirror = await runCli(a, settingsA, 'mirror', [], log('mirror'), {}, sourceCli);
|
||||
const hostMetricsPath = join(work, `${stage.name}-host-datachannel.json`);
|
||||
const targetMetricsPath = join(work, `${stage.name}-target-datachannel.json`);
|
||||
const host = await startHost(a, settingsA, log('host'), hostMetricsPath);
|
||||
const statsPath = join(work, `${stage.name}-stats.jsonl`); await fs.rm(statsPath, { force: true });
|
||||
let sync;
|
||||
try { sync = await runCli(b, settingsB, 'p2p-sync', [hostName, '120'], log('sync'), { LIVESYNC_P2P_STATS_JSONL: statsPath, LIVESYNC_BENCH_DATACHANNEL_JSON: targetMetricsPath }, targetCli); }
|
||||
finally { await host.stop(); }
|
||||
const rows = (await fs.readFile(statsPath, 'utf8')).trim().split(/\r?\n/);
|
||||
const stats = JSON.parse(rows.at(-1));
|
||||
assert(stats.candidatePathCollected, 'ICE statistics were not collected');
|
||||
assert(!stats.selectedPath.includes('relay'), `Unexpected TURN path: ${stats.selectedPath}`);
|
||||
const dataChannel = { host: JSON.parse(await fs.readFile(hostMetricsPath, 'utf8')), target: JSON.parse(await fs.readFile(targetMetricsPath, 'utf8')) };
|
||||
assert(dataChannel.host.payloadBytesSent > 0 && dataChannel.target.payloadBytesSent > 0, 'DataChannel counters must record both directions');
|
||||
const materialisedByPull = [];
|
||||
for (const file of stage.files) {
|
||||
let bytes; try { bytes = await fs.readFile(join(b, file.path)); } catch (error) { if (error.code !== 'ENOENT') throw error; }
|
||||
if (!bytes || sha256(bytes) !== file.sha256) {
|
||||
const to = join(work, 'verification', file.path); await fs.mkdir(dirname(to), { recursive: true });
|
||||
const measurement = await runCli(b, settingsB, 'pull', [file.path, to], log(`verify-${file.path.replace(/[^a-zA-Z0-9]/g, '-')}`), {}, targetCli);
|
||||
bytes = await fs.readFile(to); materialisedByPull.push({ path: file.path, measurement });
|
||||
}
|
||||
assert.equal(sha256(bytes), file.sha256, `${name} ${stage.name} ${file.path}`);
|
||||
}
|
||||
result.stages.push({ name: stage.name, changedFiles, mirror, hostReadyMs: host.readyMs, sync, stats, dataChannel, verifiedFiles: stage.files.length, materialisedByPull });
|
||||
await save(); console.log(JSON.stringify({ case: name, stage: stage.name, syncMs: Math.round(sync.elapsedMs), selectedPath: stats.selectedPath, hostPayloadBytesSent: dataChannel.host.payloadBytesSent, targetPayloadBytesSent: dataChannel.target.payloadBytesSent }));
|
||||
previous = stage;
|
||||
}
|
||||
result.complete = true; await save();
|
||||
} catch (error) { result.error = String(error); await save(); throw error; }
|
||||
}
|
||||
}
|
||||
console.log(`Completed P2P measurements: ${resultPath}`);
|
||||
@@ -0,0 +1,20 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import { dirname, join } from 'node:path';
|
||||
import { fileURLToPath, pathToFileURL } from 'node:url';
|
||||
import test from 'node:test';
|
||||
|
||||
const packageRoot = dirname(fileURLToPath(import.meta.resolve('@vrtmrz/livesync-commonlib/package.json')));
|
||||
const { splitPiecesRabinKarp } = await import(pathToFileURL(join(packageRoot, 'dist/string_and_binary/chunks.js')).href);
|
||||
|
||||
test('zero windows lose the contribution of a marker in the preceding chunk', async () => {
|
||||
const blockBytes = 300 * 1024;
|
||||
const input = new Uint8Array(blockBytes * 3);
|
||||
for (let end = blockBytes; end <= input.length; end += blockBytes) input[end - 1] = 1;
|
||||
const pieces = await splitPiecesRabinKarp(new Blob([input], { type: 'application/octet-stream' }), 1100 * 1024, false, 20, 'markers.bin', false);
|
||||
const chunks = [];
|
||||
for await (const encoded of pieces()) chunks.push(Buffer.from(encoded, 'base64'));
|
||||
assert.deepEqual(Buffer.concat(chunks), Buffer.from(input), 'all input bytes must survive splitting');
|
||||
// A 48-byte zero window hashes to zero; each isolated marker hashes to one.
|
||||
// Markers are further apart than the 256 KiB minimum and below the maximum.
|
||||
assert.deepEqual(chunks.map((chunk) => chunk.length), [blockBytes, blockBytes, blockBytes]);
|
||||
});
|
||||
@@ -0,0 +1,210 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import { createHash } from 'node:crypto';
|
||||
import { spawn } from 'node:child_process';
|
||||
import { promises as fs } from 'node:fs';
|
||||
import http from 'node:http';
|
||||
import { dirname, join, resolve } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { setTimeout as delay } from 'node:timers/promises';
|
||||
import { createNewVaultSettings, REMOTE_COUCHDB, REMOTE_MINIO, SALT_OF_PASSPHRASE } from '@vrtmrz/livesync-commonlib/compat/common/types';
|
||||
import { encryptString } from '@vrtmrz/livesync-commonlib/compat/encryption/stringEncryption';
|
||||
import { S3Client, CreateBucketCommand, ListObjectsV2Command, DeleteObjectsCommand, DeleteBucketCommand } from '@aws-sdk/client-s3';
|
||||
|
||||
const here = dirname(fileURLToPath(import.meta.url));
|
||||
const root = resolve(here, '../..');
|
||||
const cli = join(root, 'src/apps/cli/dist/index.cjs');
|
||||
const args = process.argv.slice(2);
|
||||
const option = (key, fallback) => { const at = args.indexOf(`--${key}`); return at < 0 ? fallback : args[at + 1]; };
|
||||
const sourceCli = resolve(option('source-cli', cli));
|
||||
const targetCli = resolve(option('target-cli', cli));
|
||||
const output = resolve(option('output', join(root, '.bench/chunk-size')));
|
||||
const corpusPath = resolve(option('corpus', join(output, 'corpus/manifest.json')));
|
||||
const sizes = option('sizes', '0,4,10,20,40').split(',').map(Number);
|
||||
const providers = option('providers', 'couchdb,s3').split(',');
|
||||
const repeats = Number(option('repeats', '3'));
|
||||
const rttMs = Number(option('rtt-ms', '0'));
|
||||
const user = 'bench';
|
||||
const password = 'benchmark-only-password';
|
||||
const accessKeyId = 'bench-access-key';
|
||||
const secretAccessKey = 'benchmark-only-secret-key';
|
||||
const couchUri = 'http://127.0.0.1:15994';
|
||||
const s3Uri = 'http://127.0.0.1:19094';
|
||||
const auth = `Basic ${Buffer.from(`${user}:${password}`).toString('base64')}`;
|
||||
const key = '0123456789abcdef'.repeat(4);
|
||||
const sha256 = (value) => createHash('sha256').update(value).digest('hex');
|
||||
const sourceCliSha256 = sha256(await fs.readFile(sourceCli));
|
||||
const targetCliSha256 = sha256(await fs.readFile(targetCli));
|
||||
const credentials = { accessKeyId, secretAccessKey };
|
||||
const s3 = new S3Client({ endpoint: s3Uri, region: 'us-east-1', forcePathStyle: true, credentials });
|
||||
|
||||
function emptyCounters() { return { requests: 0, requestBodyBytes: 0, responseBodyBytes: 0, maxRequestBodyBytes: 0, byMethod: {}, byOperation: {} }; }
|
||||
|
||||
async function startProxy(backend) {
|
||||
let counters = emptyCounters();
|
||||
const server = http.createServer(async (request, response) => {
|
||||
counters.requests++;
|
||||
counters.byMethod[request.method] = (counters.byMethod[request.method] || 0) + 1;
|
||||
const url = new URL(request.url, backend);
|
||||
const operation = url.pathname.split('/').find((part) => part.startsWith('_')) || (url.searchParams.has('list-type') ? 'list' : 'object');
|
||||
counters.byOperation[operation] = (counters.byOperation[operation] || 0) + 1;
|
||||
if (rttMs) await delay(rttMs / 2);
|
||||
const target = new URL(request.url, backend);
|
||||
const upstream = http.request(target, { method: request.method, headers: request.headers }, async (incoming) => {
|
||||
if (rttMs) await delay(rttMs / 2);
|
||||
response.writeHead(incoming.statusCode, incoming.headers);
|
||||
incoming.on('data', (chunk) => { counters.responseBodyBytes += chunk.length; });
|
||||
incoming.pipe(response);
|
||||
});
|
||||
upstream.on('error', (error) => { response.writeHead(502); response.end(String(error)); });
|
||||
let requestBytes = 0;
|
||||
request.on('data', (chunk) => { counters.requestBodyBytes += chunk.length; requestBytes += chunk.length; });
|
||||
request.on('end', () => { counters.maxRequestBodyBytes = Math.max(counters.maxRequestBodyBytes, requestBytes); });
|
||||
request.pipe(upstream);
|
||||
});
|
||||
await new Promise((accept, reject) => { server.once('error', reject); server.listen(0, '127.0.0.1', accept); });
|
||||
return { uri: `http://127.0.0.1:${server.address().port}`, reset: () => { counters = emptyCounters(); }, snapshot: () => structuredClone(counters), close: () => new Promise((accept) => server.close(accept)) };
|
||||
}
|
||||
|
||||
async function couch(path, method = 'GET', body) {
|
||||
const response = await fetch(`${couchUri}${path}`, { method, headers: { Authorization: auth, 'Content-Type': 'application/json' }, body: body === undefined ? undefined : JSON.stringify(body) });
|
||||
if (!response.ok) throw new Error(`CouchDB ${method} ${path}: ${response.status} ${await response.text()}`);
|
||||
return response.json();
|
||||
}
|
||||
|
||||
async function ready() {
|
||||
for (let attempt = 0; attempt < 30; attempt++) {
|
||||
try { await couch('/_up'); await s3.send(new ListObjectsV2Command({ Bucket: 'missing-benchmark-bucket' })).catch((error) => { if (error.$metadata?.httpStatusCode !== 404) throw error; }); return; }
|
||||
catch (error) { if (attempt === 29) throw error; await delay(1000); }
|
||||
}
|
||||
}
|
||||
|
||||
async function runCli(directory, settings, command, commandArgs, logPath, entrypoint = cli) {
|
||||
const marker = '__BENCH_TIME__';
|
||||
const started = performance.now();
|
||||
const child = spawn('/usr/bin/time', ['-f', `${marker}%U\t%S\t%M`, process.execPath, entrypoint, directory, '--settings', settings, command, ...commandArgs], { cwd: root, stdio: ['ignore', 'pipe', 'pipe'], env: { ...process.env, NODE_OPTIONS: '--max-old-space-size=2048' } });
|
||||
let stdout = ''; let stderr = '';
|
||||
child.stdout.on('data', (data) => { stdout += data; });
|
||||
child.stderr.on('data', (data) => { stderr += data; });
|
||||
const timeout = setTimeout(() => child.kill('SIGTERM'), 180000);
|
||||
const code = await new Promise((accept, reject) => { child.once('error', reject); child.once('exit', accept); });
|
||||
clearTimeout(timeout);
|
||||
const elapsedMs = performance.now() - started;
|
||||
await fs.writeFile(logPath, `${stdout}\n${stderr}`);
|
||||
if (code !== 0) throw new Error(`CLI ${command} exited ${code}; ${logPath}\n${stderr.slice(-2200)}`);
|
||||
const line = stderr.split(/\r?\n/).find((row) => row.startsWith(marker));
|
||||
assert(line, 'GNU time output is missing');
|
||||
const [userSeconds, systemSeconds, maxResidentSetKiB] = line.slice(marker.length).split('\t').map(Number);
|
||||
return { elapsedMs, userCpuMs: userSeconds * 1000, systemCpuMs: systemSeconds * 1000, maxResidentSetKiB };
|
||||
}
|
||||
|
||||
async function settingsFor(provider, endpoint, resource, size) {
|
||||
return {
|
||||
...createNewVaultSettings(), isConfigured: true, remoteType: provider === 'couchdb' ? REMOTE_COUCHDB : REMOTE_MINIO,
|
||||
couchDB_URI: endpoint, couchDB_DBNAME: resource, couchDB_USER: user, couchDB_PASSWORD: password,
|
||||
endpoint, bucket: resource, accessKey: accessKeyId, secretKey: secretAccessKey, region: 'us-east-1', forcePathStyle: true,
|
||||
encrypt: true, passphrase: 'chunk-size-benchmark-only-passphrase', usePathObfuscation: true, encryptInternalMetadata: true,
|
||||
idDerivationVersion: 1, idDerivationKey: '', encryptedIdDerivationKey: await encryptString(key, '*' + SALT_OF_PASSPHRASE), E2EEAlgorithm: 'v2', hashAlg: 'xxhash64',
|
||||
chunkSplitterVersion: 'v3-rabin-karp', minimumChunkSize: 20, customChunkSize: size, autoAcceptCompatibleTweak: false,
|
||||
enableCompression: false, useEden: false, useIndexedDBAdapter: false, handleFilenameCaseSensitive: false, pathNormalizationVersion: 0,
|
||||
readChunksOnline: false, liveSync: false, syncOnStart: false, syncOnSave: false, periodicReplication: false,
|
||||
syncOnFileOpen: false, syncMaxSizeInMB: 50, P2P_Enabled: false,
|
||||
};
|
||||
}
|
||||
|
||||
async function copyStage(stage, vault, previous) {
|
||||
const old = new Map((previous?.files || []).map((file) => [file.path, file.sha256]));
|
||||
const changed = [];
|
||||
for (const file of stage.files) {
|
||||
if (old.get(file.path) === file.sha256) continue;
|
||||
const from = join(stage.directory, file.path); const to = join(vault, file.path);
|
||||
await fs.mkdir(dirname(to), { recursive: true }); await fs.copyFile(from, to);
|
||||
const stat = await fs.stat(from); await fs.utimes(to, stat.atime, stat.mtime); changed.push(file.path);
|
||||
}
|
||||
return changed;
|
||||
}
|
||||
|
||||
async function inspect(provider, resource, previousIds) {
|
||||
if (provider === 'couchdb') {
|
||||
const info = await couch(`/${resource}`);
|
||||
const all = await couch(`/${resource}/_all_docs?include_docs=true`);
|
||||
const chunks = all.rows.filter(({ doc }) => doc?.type === 'leaf');
|
||||
const added = chunks.filter(({ id }) => !previousIds.has(id));
|
||||
const ids = chunks.map(({ id }) => id);
|
||||
const jsonBytes = ({ doc }) => Buffer.byteLength(JSON.stringify(doc));
|
||||
return { ids, metrics: { documentCount: info.doc_count, chunkCount: chunks.length, newChunkCount: added.length, chunkJsonBytes: chunks.reduce((sum, row) => sum + jsonBytes(row), 0), newChunkJsonBytes: added.reduce((sum, row) => sum + jsonBytes(row), 0), largestChunkJsonBytes: Math.max(0, ...chunks.map(jsonBytes)), largestDocumentJsonBytes: Math.max(0, ...all.rows.map(jsonBytes)), storedChunkDataBytes: chunks.reduce((sum, { doc }) => sum + Buffer.byteLength(doc.data || ''), 0), sizes: info.sizes, idLengthMin: Math.min(...ids.map((id) => id.length)), idLengthMax: Math.max(...ids.map((id) => id.length)) } };
|
||||
}
|
||||
const objects = []; let token;
|
||||
do { const result = await s3.send(new ListObjectsV2Command({ Bucket: resource, ContinuationToken: token })); objects.push(...result.Contents || []); token = result.NextContinuationToken; } while (token);
|
||||
return { ids: objects.map(({ Key }) => Key), metrics: { objectCount: objects.length, storedObjectBytes: objects.reduce((sum, { Size }) => sum + Size, 0), newObjectCount: objects.filter(({ Key }) => !previousIds.has(Key)).length } };
|
||||
}
|
||||
|
||||
async function removeResource(provider, resource) {
|
||||
if (provider === 'couchdb') { await couch(`/${resource}`, 'DELETE'); return; }
|
||||
let token;
|
||||
do {
|
||||
const listing = await s3.send(new ListObjectsV2Command({ Bucket: resource, ContinuationToken: token }));
|
||||
if (listing.Contents?.length) await s3.send(new DeleteObjectsCommand({ Bucket: resource, Delete: { Objects: listing.Contents.map(({ Key }) => ({ Key })) } }));
|
||||
token = listing.NextContinuationToken;
|
||||
} while (token);
|
||||
await s3.send(new DeleteBucketCommand({ Bucket: resource }));
|
||||
}
|
||||
|
||||
await fs.mkdir(join(output, 'remote'), { recursive: true });
|
||||
const manifest = JSON.parse(await fs.readFile(corpusPath, 'utf8'));
|
||||
for (const stage of manifest.stages) stage.directory = resolve(dirname(corpusPath), stage.directory);
|
||||
const digest = sha256(JSON.stringify(manifest.stages.map((stage) => ({ name: stage.name, files: stage.files }))));
|
||||
const resultPath = join(output, 'remote', `results-rtt${rttMs}.json`);
|
||||
let results = [];
|
||||
try { const saved = JSON.parse(await fs.readFile(resultPath, 'utf8')); assert.equal(saved.corpusDigest, digest); assert.equal(saved.sourceCliSha256, sourceCliSha256); assert.equal(saved.targetCliSha256, targetCliSha256); results = saved.results; } catch (error) { if (error.code !== 'ENOENT') throw error; }
|
||||
const proxies = { couchdb: await startProxy(couchUri), s3: await startProxy(s3Uri) };
|
||||
const save = async () => fs.writeFile(resultPath, JSON.stringify({ version: '1.0.39', commonlib: '0.1.44', variant: process.env.LIVESYNC_BENCH_VARIANT || 'unlabelled', sourceCliSha256, targetCliSha256, corpusDigest: digest, rttMs, mode: 'local services; HTTP body bytes exclude headers; full chunk download; E2EE V2 with independent IDs; compression disabled', results }, null, 2));
|
||||
try {
|
||||
await ready();
|
||||
for (let repeat = 0; repeat < repeats; repeat++) {
|
||||
const rotation = repeat % sizes.length; const order = [...sizes.slice(rotation), ...sizes.slice(0, rotation)];
|
||||
for (const provider of (repeat % 2 ? [...providers].reverse() : providers)) for (const size of order) {
|
||||
const name = `${provider}-c${size}-r${repeat + 1}`;
|
||||
if (results.some((result) => result.name === name && result.complete)) continue;
|
||||
const resource = `chunks-${Date.now().toString(36)}-${name}`;
|
||||
const work = join(output, 'remote', name); await fs.mkdir(join(work, 'logs'), { recursive: true });
|
||||
const a = join(work, 'source'); const b = join(work, 'target');
|
||||
await fs.rm(a, { recursive: true, force: true }); await fs.rm(b, { recursive: true, force: true });
|
||||
await fs.mkdir(a, { recursive: true }); await fs.mkdir(b, { recursive: true });
|
||||
const settingsA = join(work, 'source-settings.json'); const settingsB = join(work, 'target-settings.json');
|
||||
const settings = await settingsFor(provider, proxies[provider].uri, resource, size);
|
||||
await fs.writeFile(settingsA, JSON.stringify(settings)); await fs.writeFile(settingsB, JSON.stringify(settings));
|
||||
if (provider === 'couchdb') await couch(`/${resource}`, 'PUT'); else await s3.send(new CreateBucketCommand({ Bucket: resource }));
|
||||
const result = { name, provider, customChunkSize: size, repeat: repeat + 1, stages: [], complete: false };
|
||||
results.push(result); await save();
|
||||
try {
|
||||
let previous; let ids = new Set();
|
||||
for (const stage of manifest.stages) {
|
||||
const changedFiles = await copyStage(stage, a, previous);
|
||||
const log = (label) => join(work, 'logs', `${stage.name}-${label}.txt`);
|
||||
const mirror = await runCli(a, settingsA, 'mirror', [], log('mirror'), sourceCli);
|
||||
proxies[provider].reset();
|
||||
const upload = await runCli(a, settingsA, 'sync', [], log('upload'), sourceCli); upload.http = proxies[provider].snapshot();
|
||||
const remote = await inspect(provider, resource, ids); ids = new Set(remote.ids);
|
||||
proxies[provider].reset();
|
||||
const download = await runCli(b, settingsB, 'sync', [], log('download'), targetCli); download.http = proxies[provider].snapshot();
|
||||
const verified = []; const materialisedByPull = [];
|
||||
for (const file of stage.files) {
|
||||
let bytes; try { bytes = await fs.readFile(join(b, file.path)); } catch (error) { if (error.code !== 'ENOENT') throw error; }
|
||||
if (!bytes || sha256(bytes) !== file.sha256) {
|
||||
const target = join(work, 'verification', file.path); await fs.mkdir(dirname(target), { recursive: true });
|
||||
const measurement = await runCli(b, settingsB, 'pull', [file.path, target], log(`verify-${file.path.replace(/[^a-zA-Z0-9]/g, '-')}`), targetCli);
|
||||
bytes = await fs.readFile(target); materialisedByPull.push({ path: file.path, measurement });
|
||||
}
|
||||
assert.equal(sha256(bytes), file.sha256, `${name} ${stage.name} ${file.path}`); verified.push(file.path);
|
||||
}
|
||||
result.stages.push({ name: stage.name, changedFiles, mirror, upload, download, remote: remote.metrics, verifiedFiles: verified.length, materialisedByPull });
|
||||
await save(); console.log(JSON.stringify({ case: name, stage: stage.name, uploadMs: Math.round(upload.elapsedMs), downloadMs: Math.round(download.elapsedMs), requestBytes: upload.http.requestBodyBytes, responseBytes: download.http.responseBodyBytes, remote: remote.metrics }));
|
||||
previous = stage;
|
||||
}
|
||||
result.complete = true; await save();
|
||||
} catch (error) { result.error = String(error); await save(); throw error; }
|
||||
finally { await removeResource(provider, resource); }
|
||||
}
|
||||
}
|
||||
} finally { await Promise.all(Object.values(proxies).map((proxy) => proxy.close())); s3.destroy(); }
|
||||
console.log(`Completed remote measurements: ${resultPath}`);
|
||||
@@ -0,0 +1,85 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import { promises as fs } from 'node:fs';
|
||||
import { join, resolve } from 'node:path';
|
||||
|
||||
const directory = resolve(process.argv[2] || '.bench/rabin-window-fixed');
|
||||
const baselineDirectory = resolve(process.argv[3] || '.bench/chunk-size');
|
||||
const load = async (file, base = directory) => JSON.parse(await fs.readFile(join(base, file), 'utf8'));
|
||||
const summary = await load('summary.json');
|
||||
const baseline = await load('summary.json', baselineDirectory);
|
||||
const comparison = await load('comparison.json');
|
||||
const validation = await load('validation.json');
|
||||
const boundary = await load('cloudant-json-boundary/results.json');
|
||||
const consumer = await load('context/validated-consumer.json');
|
||||
const recovery = await load('context/cli-bundle-recovery.json');
|
||||
assert.equal(validation.completedRemoteCases, 30);
|
||||
assert.equal(validation.completedP2PCases, 12);
|
||||
assert.equal(summary.corpus.manifestSha256, baseline.corpus.manifestSha256);
|
||||
const mib = bytes => (bytes / 1048576).toFixed(3);
|
||||
const second = milliseconds => (milliseconds / 1000).toFixed(2);
|
||||
const getRemote = (data, provider, size, stage) => data.remote.find(row => row.provider === provider && row.customChunkSize === size && row.stage === stage);
|
||||
const getP2P = (data, size, stage) => data.p2p.find(row => row.customChunkSize === size && row.stage === stage);
|
||||
const getLocal = (data, size, stage) => data.local.find(row => row.customChunkSize === size && row.stage === stage);
|
||||
const decrease = (before, after) => ((1 - after / before) * 100).toFixed(1);
|
||||
const lines = [];
|
||||
const paragraph = text => lines.push(text, '');
|
||||
paragraph('Rabin–Karp窓処理修正後の実測（Self-hosted LiveSync 1.0.39 / Commonlib 0.1.44の未公開修正パッケージ）');
|
||||
paragraph('窓処理は、今回の挿入後のチャンク再利用を妨げていた原因の一つだった。最小限の修正をCommonlibへ加え、前回と完全に同じ入力・設定・候補を再測定した。今回の混在コーパスでは、通常の新規推奨値を4（上限500 KiB）へ揃える案が有力になった。Cloudant対応は接続先ごとの推奨値ではなく、将来の同期先も含むvault共通のデータ制約として扱う必要がある。ただし数値4だけで全設定のCloudant互換性を保証する結論ではない。初期値、推奨値、UI、および移行処理はまだ変更していない。');
|
||||
paragraph(`LiveSyncの基点は ${consumer.liveSyncBaseRevision}、Commonlibの基点は ${consumer.commonlibBaseRevision}。Commonlib所有リポジトリーの作業ツリーで修正し、単体・境界・リリースツール・パッケージ検証を通したtarballをLiveSyncへfile依存として入れた。公開版0.1.44を置換して公開したものではない。候補の実体は ${consumer.packageIntegrity}、インストール済みchunks.jsのSHA-256は ${consumer.installedChunksSha256}。`);
|
||||
paragraph('修正はsrc/string_and_binary/chunks.tsの窓更新条件を pos >= start + windowSize から pos >= windowSize へ変える1行と、その目的を示すコメントである。境界でstartが進んでも、hashの48バイト窓を連続して更新する。形式、ID導出方法、設定キー、最小・平均・最大チャンク目標は変更していない。');
|
||||
paragraph('先に3件の振る舞いの回帰テストを追加し、未修正の公開0.1.44相当ソースで3件とも期待した境界assertionが失敗することを確認した。バイナリ300/300/300 KiBの目印列は従来300/256/256/88 KiBとなり、挿入後の同一末尾チャンクを再利用できなかった。テキストの目印列でも過去のチャンクの影響が残った。修正後は3件とも成功し、再構成した全バイトも一致した。Commonlib全123ファイル・2,416テスト、型、boundary、release tools、packed consumerが成功した。LiveSyncはnpm 10クリーンインストール、通常npm ci、check、単体1,297成功・1 skipped、およびObsidian bundle/CLI/WebApp/WebPeerビルドが成功した。Community lintの既存51 warningsは残っている。');
|
||||
paragraph(`入力は前回と同じ15ファイル、初期合計 ${mib(summary.corpus.stages[0].sourceBytes)} MiB。通常のプロジェクトMarkdown10本、約512 KiBと約5 MiBの生成Markdown、PNG2枚、および固定seedの8 MiB高エントロピーバイナリである。manifest SHA-256は ${summary.corpus.manifestSha256}。段階は初回、4 KiBのバイナリ上書き等、続いてバイナリ途中4 KiB挿入と生成Markdownへの段落挿入。設定はRabin–Karp、minimumChunkSize=20、E2EE V2、独立ID鍵version 1、パス・内部Metadata暗号化、チャンク圧縮なし、Edenなし、全chunk取得で固定した。`);
|
||||
paragraph('ローカル分割は0,1,4,10,20,40,60を各3回。HTTPはCouchDB 3.5.0とRustFSを0,4,10,20,40各3回、P2Pは0,4,10,40各3回。同じホスト上の生成テストvaultとタスク専用コンテナを使用し、WAN遅延やパケットロスは加えていない。CPU負荷をかける検証・測定は同時に実行していない。Node v24.18.0 / Linux arm64 / Neoverse-N1 / 論理CPU3である。');
|
||||
paragraph('以下はHTTPの実送信本文の3回中央値。ヘッダー・TLSは含めない。同期秒数はCLI起動を含み、事前のmirrorと転送後のファイル復元・SHA照合を含まない。OSキャッシュやサービスは各回に再起動していない。S3のJournalパック圧縮はenableCompressionによるチャンク圧縮とは別である。');
|
||||
lines.push('remote | value | 初期送信MiB | 上書き送信MiB | 挿入送信MiB | 初期送信秒 | 初期受信秒 | 初期HTTP要求数');
|
||||
for (const provider of ['couchdb', 's3']) for (const size of [0, 4, 10, 20, 40]) {
|
||||
const initial = getRemote(summary, provider, size, 'initial');
|
||||
const overwrite = getRemote(summary, provider, size, 'overwrite');
|
||||
const insert = getRemote(summary, provider, size, 'insert');
|
||||
lines.push([provider, size, mib(initial.upload_requestBodyBytes.median), mib(overwrite.upload_requestBodyBytes.median), mib(insert.upload_requestBodyBytes.median), second(initial.upload_elapsedMs.median), second(initial.download_elapsedMs.median), initial.upload_requests.median].join(' | '));
|
||||
}
|
||||
lines.push('');
|
||||
paragraph('P2Pは全回host<->host経路。werift 0.24.4のcandidate-pair bytesカウンターは0のため採用せず、RTCDataChannelの実送信payloadを両端で観測した。下表は送信元host側で、RPC制御とアプリ層再送を含み、SCTP/DTLS/UDP/IPのヘッダー、下位層再送、Nostr signallingは含まない。HTTP本文量と完全に同じ層ではない。同期時間は探索・接続・双方向同期を含む。');
|
||||
lines.push('value | 初期送信MiB | 上書き送信MiB | 挿入送信MiB 中央値 [min..max] | 初期同期秒 | 上書き同期秒 | 挿入同期秒');
|
||||
for (const size of [0, 4, 10, 40]) {
|
||||
const initial = getP2P(summary, size, 'initial'); const overwrite = getP2P(summary, size, 'overwrite'); const insert = getP2P(summary, size, 'insert');
|
||||
lines.push([size, mib(initial.host_payloadBytesSent.median), mib(overwrite.host_payloadBytesSent.median), `${mib(insert.host_payloadBytesSent.median)} [${mib(insert.host_payloadBytesSent.min)}..${mib(insert.host_payloadBytesSent.max)}]`, second(initial.elapsedMs.median), second(overwrite.elapsedMs.median), second(insert.elapsedMs.median)].join(' | '));
|
||||
}
|
||||
lines.push('');
|
||||
paragraph('値4の途中挿入における修正前→修正後の比較は次のとおり。');
|
||||
lines.push('対象 | 修正前MiB | 修正後MiB | 減少率');
|
||||
const oldRaw = getLocal(baseline, 4, 'insert').binaryNewRawBytes; const newRaw = getLocal(summary, 4, 'insert').binaryNewRawBytes;
|
||||
lines.push(`新しく必要なバイナリの生データ | ${mib(oldRaw)} | ${mib(newRaw)} | ${decrease(oldRaw, newRaw)}%`);
|
||||
for (const provider of ['couchdb', 's3']) {
|
||||
const oldValue = getRemote(baseline, provider, 4, 'insert').upload_requestBodyBytes.median; const newValue = getRemote(summary, provider, 4, 'insert').upload_requestBodyBytes.median;
|
||||
lines.push(`${provider}送信本文 | ${mib(oldValue)} | ${mib(newValue)} | ${decrease(oldValue, newValue)}%`);
|
||||
}
|
||||
const oldPeer = getP2P(baseline, 4, 'insert').host_payloadBytesSent.median; const newPeer = getP2P(summary, 4, 'insert').host_payloadBytesSent.median;
|
||||
lines.push(`P2P host payload | ${mib(oldPeer)} | ${mib(newPeer)} | ${decrease(oldPeer, newPeer)}%`, '');
|
||||
paragraph('通常の4 MiB未満のMarkdown11本×3段階は、修正後も7候補の全てで分割境界とIDが一致した。customChunkSizeを上げてもこの領域のID数は減らない。新しい独立ID鍵方式のIDは67文字で、旧暗号化xxhash64比較は14〜16文字、中央値16文字だった。ID命名の変更がIDを短くしたという前提は置けない。');
|
||||
paragraph('0/1ではバイナリ・大きいテキストの最小値が最大値へクランプされ、100/200 KiBの固定分割になるため、窓修正後も挿入時の再同期の利点は出ない。小さい局所上書きだけなら0が有利だが、今回の挿入を含む編集では4の方が大幅に少なく済む。40と60は実装上の4 MiB上限に達し、全15ファイル×3段階で分割境界とIDが完全一致した。60の重複するネットワーク測定は省いた。');
|
||||
for (const provider of ['couchdb', 's3']) {
|
||||
const four = getRemote(summary, provider, 4, 'initial').upload_requestBodyBytes.median;
|
||||
const forty = getRemote(summary, provider, 40, 'initial').upload_requestBodyBytes.median;
|
||||
paragraph(`${provider}で4から40へ増やした初回送信量の削減は ${((1 - forty / four) * 100).toFixed(3)}%程度だった。上書き・挿入では40の送信量が大きく、今回の条件から大きいチャンクを維持する性能上の根拠は得られなかった。`);
|
||||
}
|
||||
paragraph('S3ではどの候補も初回7オブジェクト・14 HTTP要求だった。4と40は編集時もそれぞれ1オブジェクト追加・6 HTTP要求で、chunk数の削減がそのままPUT削減になる構成ではない。AWS等の実サービスの課金やWAN性能を測った値ではない。P2Pでは回ごとの再送等の変動があるため、中央値だけでなくsummary.jsonのmin/maxも参照する。150msの断片再要求との因果は計測しておらず、この処理は変更していない。');
|
||||
paragraph('Cloudantの現行IBM公式limits（2026-10-23確認）は、単一JSON文書1 MB、リクエスト全体64MB、添付10 MB。https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-limits 。chunksは添付ではなくJSON文書として送る。モデルには保守的に1,000,000 / 64,000,000 bytesを使った。通常の暗号化条件ではCouchDBに文書上限だけをかけ、4は15ファイルの復元まで成功、10は413となった。4の最大保存文書は910,461 bytes、通常HTTP測定の最大リクエストは約10.27 MBで、両方の上限内だった。');
|
||||
paragraph('別の境界probeでは、実際の分割器・圧縮・暗号化関数を用い、レプリケーションのrevisionフィールドを含む合成leafを送信した。結果は次のとおり。');
|
||||
lines.push('入力 | value | 暗号化 | 圧縮 | JSON文書bytes | CouchDB HTTP status');
|
||||
for (const row of boundary.cases) lines.push([row.kind, row.customChunkSize, row.encryption, row.compression, row.jsonBytes, row.status].join(' | '));
|
||||
lines.push('');
|
||||
paragraph('値4のバイナリはraw 512,000 bytes → base64 682,668 bytes → E2EE V1/V2で約910.5 KBとなった。値5のE2EE V2は1,092,573 bytesとなり、文書上限で413が返った。したがって暗号化バイナリに対して4は整数の候補として上限に近く、余裕を残す候補である。');
|
||||
paragraph('ただし、CouchDB 3.5.0のmax_document_sizeはparsed document bodyをcouch_ejson_size:encoded_sizeで概算し、文字列はUTF-8長に2を足すだけでJSON escapeを数えない。このためNULの未暗号化テキストはwire上3,072,213 bytesでも、モデルの1 MB文書制限では受理された。Cloudant内部の算定が完全に同一であるとは確認していない。単純にwire JSONが1 MB超なら必ずCloudantの文書制限で失敗する、とこの結果から断定してはいけない。公式CouchDBソースはcontext/couch_doc-3.5.0.erlとcontext/couch_ejson_size-3.5.0.erlに保存した。');
|
||||
paragraph('一括リクエストは別の制限である。分割器が作った500 KiBの異なる制御文字テキスト25個をまとめると76,805,358 bytesとなり、64,000,000-byte request制限で413になった。20個なら61,444,288 bytesで受理された。これは合成bulk requestの境界probeであり、アプリケーションの通常同期全体で同じbulkが組まれると実証したテストではない。現在の通常Couch同期のbatch_size=25では、暗号化文書が1 MB以内なら25 MB程度だが、暗号化しないテキストのescape膨張には件数制限だけでは十分でない。optional bulk chunk sendには独自のbyte制限もあるので、その経路と通常Pouch replicationの経路を区別する必要がある。');
|
||||
paragraph('CloudantモデルはローカルCouchDBで制限を模したものであり、IBM Cloudantサービス自体の同期・性能は測定していない。全probe後に両制限設定を復元し、合成文書のDBも削除した。Cloudantの互換性を約束するなら、leafだけでなくMetadataを含む最終文書とバッチの実サイズも方針の対象にするのが妥当である。');
|
||||
paragraph(`旧版と修正版の混在は、同じ旧依存リンクからビルドを復元し、旧entryが前回の保存物と完全一致することを確認して実測した（${recovery.originalEntryBitIdenticalToBaselineArchive}）。旧→新、新→旧の各3段階×15ファイル、計${validation.mixedVersionVerifiedFileStages}件でSHA-256が一致した。チャンクの保存形式とID方式は引き続き読める。再利用効率を揃えるには書込み端末の分割実装も揃える方がよい。最初の互換性試行は旧bundleの付随CJS不足で実行前に失敗し、そのログと復元記録を保存した。復元後の全試行は成功した。`);
|
||||
const cutover = comparison.cutover.find(row => row.customChunkSize === 40);
|
||||
paragraph(`窓の修正で分割境界が変わるため、既存ファイルを再保存する初回には追加チャンクが多く必要になり得る。ローカルの同じ初期データを値40で比較すると、新方式のunique raw ${mib(cutover.totalNewAlgorithmUniqueRawBytes)} MiB中、旧初期chunkに存在しない部分は ${mib(cutover.bytesNotPresentInOldAlgorithmInitialChunks)} MiBだった。これは内容/ID照合による算定で、実際のアップグレード時の転送量を測定した値ではない。以後の同じ方式での編集に対する改善とは分けて扱う。`);
|
||||
paragraph('設定整理への提案は、通常の新規作成・推奨値をremote種別によらず4へ揃える方向で進め、旧設定の省略値や既存remoteで合意済みのtweakを一律に書き換えないことである。60と10を標準候補として分ける根拠は今回弱い。一方、Cloudantへ将来同期する意図を問うなら、現在の接続先ではなくvault全体の互換方針として扱い、全書込み端末で守る必要がある。暗号化・JSON化・バッチ化後の上限を内部で導出し、ユーザーへ組合せの数字を選ばせることは避けたい。');
|
||||
paragraph('現在のremote profile URIは接続欄だけを含み、customChunkSizeは共通設定である。通常のprofile切替でその値を変えないが、起動時のCloudant接続先補正やpreferred tweak採用では変わる経路がある。保存済みleafは転送時に再分割されない。Cloudant対応を後から有効化して値だけ下げても、既存の大きいleafがそのまま残るため、互換方針を変える移行では既存DBの検査・再構成も別途設計する必要がある。これは今回未実装の設計案である。');
|
||||
paragraph(`検証結果は、主測定42ケース・126段階×15ファイル=${validation.verifiedFileStages}件のSHA-256一致、保存後設定${validation.checkedPersistedConfigurations}件の条件一致、混在版${validation.mixedVersionVerifiedFileStages}件のSHA-256一致。通常Markdown、40/60同一、全分割の再構成、反復でのID・境界再現性、および同じIDの内容一致も成功した。`);
|
||||
paragraph('保存物は生JSON、summary/CSV、comparison.json、validation.json、全入力corpus、各caseのログ・設定・P2Pカウンター、scripts、contextの完全な旧/新CLI bundleとlockfile、commonlibの修正差分・回帰テスト・検証ログ・tarball、およびartifacts.sha256である。巨大な作業用DB複製は含めず、再生成できる入力と証拠を保存する。旧baselineは親ディレクトリーにそのまま残している。');
|
||||
paragraph('再現時はLiveSync固定コミットとCommonlib固定コミットの作業ツリーを用意し、Commonlibのproduction.patchとsource内の回帰テストを反映する。Commonlibの元リポジトリーで検証し、保存済みtarballと同じ実体をLiveSyncにfile依存として入れる。scriptsをLiveSyncのutils/chunk-size-benchmarkへ置き、npm 10 ci、通常npm ci、check、unit、全app buildの順に実行する。旧版の再構築には公開0.1.44パッケージを使い、実行本体だけでなくdist全体を別ディレクトリーへ残す。新しい出力先とLIVESYNC_BENCH_VARIANT=rolling-window-fixedを使ってlocal/remote/cloudant probes/compatibility/p2pを逐次実行し、summarise、compare、verify-results、reportの順に集計する。ローカル専用fixtureの起動・停止にもcomposeのproject名を指定する。');
|
||||
await fs.writeFile(join(directory, 'report.txt'), lines.join('\n'));
|
||||
console.log(JSON.stringify({ report: join(directory, 'report.txt'), bytes: Buffer.byteLength(lines.join('\n')) }));
|
||||
@@ -0,0 +1,83 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import { promises as fs } from 'node:fs';
|
||||
import { join, resolve } from 'node:path';
|
||||
|
||||
const directory = resolve(process.argv[2] || '.bench/chunk-size');
|
||||
const local = JSON.parse(await fs.readFile(join(directory, 'results.json'), 'utf8'));
|
||||
const optionalJson = async (path) => { try { return JSON.parse(await fs.readFile(path, 'utf8')); } catch (error) { if (error.code === 'ENOENT') return undefined; throw error; } };
|
||||
const remote = await optionalJson(join(directory, 'remote/results-rtt0.json'));
|
||||
const p2p = await optionalJson(join(directory, 'p2p/results.json'));
|
||||
const median = (numbers) => { const sorted = [...numbers].sort((a, b) => a - b); const mid = Math.floor(sorted.length / 2); return sorted.length % 2 ? sorted[mid] : (sorted[mid - 1] + sorted[mid]) / 2; };
|
||||
const stats = (numbers) => ({ n: numbers.length, median: median(numbers), min: Math.min(...numbers), max: Math.max(...numbers) });
|
||||
|
||||
let normalMarkdownFileStages = 0;
|
||||
for (const stage of local.candidates[0].stages) {
|
||||
for (const file of stage.files.filter((file) => file.kind === 'markdown' && file.sourceBytes < 4 * 1024 * 1024)) {
|
||||
const fingerprints = local.candidates.map((candidate) => candidate.stages.find((entry) => entry.name === stage.name).files.find((entry) => entry.path === file.path).chunkFingerprintSha256);
|
||||
assert.equal(new Set(fingerprints).size, 1, `${stage.name}/${file.path}`); normalMarkdownFileStages++;
|
||||
}
|
||||
}
|
||||
|
||||
const localRows = local.candidates.flatMap((candidate) => candidate.stages.map((stage) => {
|
||||
const binary = stage.byKind.find((group) => group.kind === 'binary');
|
||||
const markdown = stage.byKind.find((group) => group.kind === 'markdown');
|
||||
const random = stage.files.find((file) => file.path === 'benchmark/random-8m.bin');
|
||||
return {
|
||||
customChunkSize: candidate.customChunkSize, stage: stage.name, rawCeilingBytes: candidate.chunkSizeCeilingBytes,
|
||||
allChunks: stage.files.reduce((sum, file) => sum + file.chunkCount, 0), binaryChunks: binary.chunkCount,
|
||||
randomBinaryChunks: random.chunkCount, randomBinaryCpuMs: random.timing.primarySplitAndIdCpuMsMedian,
|
||||
binaryNewRawBytes: binary.newUniqueChunkRawBytesVsHistory, markdownNewRawBytes: markdown.newUniqueChunkRawBytesVsHistory,
|
||||
splitAndIdCpuMs: binary.timing.primarySplitAndIdCpuMsMedian + markdown.timing.primarySplitAndIdCpuMsMedian,
|
||||
};
|
||||
}));
|
||||
|
||||
function grouped(entries, makeRow) {
|
||||
const groups = new Map();
|
||||
for (const entry of entries) for (const stage of entry.stages) {
|
||||
const key = `${entry.provider || 'p2p'}/${entry.customChunkSize}/${stage.name}`;
|
||||
const group = groups.get(key) || { provider: entry.provider || 'p2p', customChunkSize: entry.customChunkSize, stage: stage.name, samples: [] };
|
||||
group.samples.push(stage); groups.set(key, group);
|
||||
}
|
||||
return [...groups.values()].map(makeRow);
|
||||
}
|
||||
|
||||
const remoteRows = grouped((remote?.results || []).filter((entry) => entry.complete), (group) => {
|
||||
const result = { provider: group.provider, customChunkSize: group.customChunkSize, stage: group.stage, repeats: group.samples.length };
|
||||
for (const direction of ['mirror', 'upload', 'download']) {
|
||||
for (const metric of ['elapsedMs', 'userCpuMs', 'systemCpuMs', 'maxResidentSetKiB']) result[`${direction}_${metric}`] = stats(group.samples.map((stage) => stage[direction][metric]));
|
||||
if (direction !== 'mirror') for (const metric of ['requests', 'requestBodyBytes', 'responseBodyBytes', 'maxRequestBodyBytes']) {
|
||||
const values = group.samples.map((stage) => stage[direction].http[metric]).filter(Number.isFinite);
|
||||
if (values.length) result[`${direction}_${metric}`] = stats(values);
|
||||
}
|
||||
}
|
||||
for (const metric of ['objectCount', 'storedObjectBytes', 'chunkCount', 'newChunkCount', 'chunkJsonBytes', 'largestChunkJsonBytes', 'largestDocumentJsonBytes']) {
|
||||
const values = group.samples.map((stage) => stage.remote[metric]).filter(Number.isFinite);
|
||||
if (values.length) result[metric] = stats(values);
|
||||
}
|
||||
result.verifiedFilesPerSample = group.samples.map((stage) => stage.verifiedFiles);
|
||||
return result;
|
||||
});
|
||||
|
||||
const p2pRows = grouped((p2p?.results || []).filter((entry) => entry.complete), (group) => {
|
||||
const result = { provider: 'p2p', customChunkSize: group.customChunkSize, stage: group.stage, repeats: group.samples.length };
|
||||
for (const metric of ['elapsedMs', 'userCpuMs', 'systemCpuMs', 'maxResidentSetKiB']) result[metric] = stats(group.samples.map((stage) => stage.sync[metric]));
|
||||
result.currentRoundTripTime = stats(group.samples.map((stage) => stage.stats.selectedPair.currentRoundTripTime));
|
||||
for (const side of ['host', 'target']) for (const metric of ['payloadBytesSent', 'messagesSent']) result[`${side}_${metric}`] = stats(group.samples.map((stage) => stage.dataChannel[side][metric]));
|
||||
for (const side of ['host', 'target']) for (const metric of ['userCpuMs', 'systemCpuMs', 'maxResidentSetKiB']) result[`${side}_${metric}`] = stats(group.samples.map((stage) => stage.dataChannel[side].processUsage[metric]));
|
||||
result.paths = [...new Set(group.samples.map((stage) => stage.stats.selectedPath))];
|
||||
result.verifiedFilesPerSample = group.samples.map((stage) => stage.verifiedFiles);
|
||||
return result;
|
||||
});
|
||||
|
||||
const summary = { version: local.runtime.projectVersion, commonlib: local.runtime.commonlibVersion, variant: local.runtime.variant, commonlibChunksSha256: local.runtime.commonlibChunksSha256, corpus: local.corpus,
|
||||
assertions: { ...local.assertions, normalMarkdownIdenticalAcrossAllSevenValues: true, normalMarkdownFileStages }, local: localRows, remote: remoteRows, p2p: p2pRows };
|
||||
await fs.writeFile(join(directory, 'summary.json'), JSON.stringify(summary, null, 2));
|
||||
|
||||
function csv(rows, fields) {
|
||||
const cell = (value) => JSON.stringify(typeof value === 'object' && value !== null ? value.median : value ?? '');
|
||||
return `${fields.join(',')}\n${rows.map((row) => fields.map((field) => cell(row[field])).join(',')).join('\n')}\n`;
|
||||
}
|
||||
await fs.writeFile(join(directory, 'local-summary.csv'), csv(localRows, Object.keys(localRows[0])));
|
||||
if (remoteRows.length) await fs.writeFile(join(directory, 'remote-summary.csv'), csv(remoteRows, Object.keys(remoteRows[0]).filter((key) => key !== 'verifiedFilesPerSample')));
|
||||
if (p2pRows.length) await fs.writeFile(join(directory, 'p2p-summary.csv'), csv(p2pRows, Object.keys(p2pRows[0]).filter((key) => !['paths', 'verifiedFilesPerSample'].includes(key))));
|
||||
console.log(JSON.stringify({ directory, completeRemoteCases: remote?.results.filter((entry) => entry.complete).length || 0, completeP2PCases: p2p?.results.filter((entry) => entry.complete).length || 0, assertions: summary.assertions }));
|
||||
@@ -0,0 +1,82 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import { createHash } from 'node:crypto';
|
||||
import { promises as fs } from 'node:fs';
|
||||
import { join, resolve } from 'node:path';
|
||||
|
||||
const directory = resolve(process.argv[2] || '.bench/rabin-window-fixed');
|
||||
const load = async name => JSON.parse(await fs.readFile(join(directory, name), 'utf8'));
|
||||
const local = await load('results.json');
|
||||
const summary = await load('summary.json');
|
||||
const remote = await load('remote/results-rtt0.json');
|
||||
const p2p = await load('p2p/results.json');
|
||||
const manifest = await load('corpus/manifest.json');
|
||||
const sha256 = value => createHash('sha256').update(value).digest('hex');
|
||||
assert.equal(remote.corpusDigest, p2p.corpusDigest);
|
||||
assert.equal(remote.corpusDigest, sha256(JSON.stringify(manifest.stages.map(stage => ({ name: stage.name, files: stage.files })))));
|
||||
assert.equal(local.corpus.manifestSha256, sha256(JSON.stringify(manifest)));
|
||||
assert.equal(remote.results.length, 30);
|
||||
assert.equal(p2p.results.length, 12);
|
||||
let verified = 0;
|
||||
let configurations = 0;
|
||||
for (const [subdirectory, result] of [['remote', remote], ['p2p', p2p]]) {
|
||||
assert(result.variant === 'rolling-window-fixed');
|
||||
const groups = new Map();
|
||||
for (const entry of result.results) {
|
||||
assert(entry.complete && !entry.error, entry.name);
|
||||
assert.equal(entry.stages.length, 3);
|
||||
const key = `${entry.provider || 'p2p'}/${entry.customChunkSize}`;
|
||||
groups.set(key, (groups.get(key) || 0) + 1);
|
||||
for (const stage of entry.stages) { assert.equal(stage.verifiedFiles, 15); verified += stage.verifiedFiles; }
|
||||
for (const side of ['source', 'target']) {
|
||||
const settings = await load(join(subdirectory, entry.name, `${side}-settings.json`));
|
||||
assert.equal(settings.customChunkSize, entry.customChunkSize);
|
||||
assert.equal(settings.chunkSplitterVersion, 'v3-rabin-karp');
|
||||
assert.equal(settings.minimumChunkSize, 20);
|
||||
assert.equal(settings.idDerivationVersion, 1);
|
||||
assert.equal(settings.encrypt, true);
|
||||
assert.equal(settings.E2EEAlgorithm, 'v2');
|
||||
assert.equal(settings.useEden, false);
|
||||
assert.equal(settings.enableCompression, false);
|
||||
assert.equal(settings.autoAcceptCompatibleTweak, false);
|
||||
assert.equal(settings.batch_size, 25);
|
||||
configurations++;
|
||||
}
|
||||
}
|
||||
assert([...groups.values()].every(count => count === 3));
|
||||
}
|
||||
assert.equal(verified, 1890);
|
||||
assert.equal(configurations, 84);
|
||||
let mixedVerified = 0;
|
||||
const compatibility = [];
|
||||
for (const name of ['compatibility-old-to-new', 'compatibility-new-to-old']) {
|
||||
const result = await load(`${name}/remote/results-rtt0.json`);
|
||||
const completed = result.results.filter(entry => entry.complete);
|
||||
assert.equal(completed.length, 1);
|
||||
assert.notEqual(result.sourceCliSha256, result.targetCliSha256);
|
||||
const entry = completed[0];
|
||||
assert(entry.complete && !entry.error);
|
||||
assert.equal(entry.stages.length, 3);
|
||||
for (const stage of entry.stages) { assert.equal(stage.verifiedFiles, 15); mixedVerified += stage.verifiedFiles; }
|
||||
compatibility.push({ name, sourceCliSha256: result.sourceCliSha256, targetCliSha256: result.targetCliSha256, verifiedFiles: 45, retainedIncompleteSetupAttempts: result.results.filter(entry => !entry.complete).length });
|
||||
}
|
||||
const model = await load('cloudant-model/constraint-result.json');
|
||||
assert(model.configurationRestored && model.sizeRejectionObserved);
|
||||
assert(model.cases.some(entry => entry.customChunkSize === 4 && entry.complete));
|
||||
assert(model.cases.some(entry => entry.customChunkSize === 10 && !entry.complete));
|
||||
const boundary = await load('cloudant-json-boundary/results.json');
|
||||
assert(boundary.configurationRestored && boundary.requestConfigurationRestored);
|
||||
assert.equal(boundary.cases.length, 8);
|
||||
assert.equal(boundary.batches.length, 2);
|
||||
assert(boundary.batches.some(entry => entry.count === 25 && entry.status === 413 && !entry.accepted));
|
||||
assert(boundary.batches.some(entry => entry.count === 20 && entry.accepted));
|
||||
assert(Object.values(local.assertions).every(value => value === true || typeof value === 'number'));
|
||||
const result = { completedRemoteCases: 30, completedP2PCases: 12, repeatsPerCondition: 3,
|
||||
verifiedFileStages: verified, checkedPersistedConfigurations: configurations,
|
||||
sameCorpusDigestForRemoteAndP2P: true, localAssertions: summary.assertions,
|
||||
compatibility, mixedVersionVerifiedFileStages: mixedVerified,
|
||||
cloudantConstraintModel: { candidate4Completed: true, candidate10RejectedWith413: true, fixtureConfigurationRestored: true },
|
||||
cloudantJSONBoundary: { cases: boundary.cases.length, batches: boundary.batches.length, request25RejectedWith413: true, request20Accepted: true, fixtureConfigurationRestored: true, cloudantServiceMeasured: false },
|
||||
rabinWindowDiagnostic: { command: 'node --test utils/chunk-size-benchmark/rabin-window.test.mjs', exitCode: 0, productionFixApplied: true },
|
||||
};
|
||||
await fs.writeFile(join(directory, 'validation.json'), JSON.stringify(result, null, 2));
|
||||
console.log(JSON.stringify(result));
|
||||
Reference in new issue
Block a user