public/obsidian-livesync
1
0
Fork 0
mirror of https://github.com/vrtmrz/obsidian-livesync.git synced 2026-07-23 04:52:58 +00:00
Code Issues Packages Projects Releases Wiki Activity

Re-evaluate optional features for 1.0

Browse Source
This commit is contained in:
vorotamoroz
2026-07-21 01:53:13 +00:00
parent ed7471fba5
commit 3eda3316c5
22 changed files with 1253 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/adr/2026_07_feature_maturity_for_1_0.md
+49
View File
@@ -0,0 +1,49 @@
# Feature maturity for 1.0
## Status
Proposed for the 1.0 integration branch.
## Context
Self-hosted LiveSync accumulated several labels such as 'experimental', 'Beta', 'obsolete', and 'sunset' over the 0.x line. Those labels did not consistently describe the current implementation. Some formerly experimental features now have maintained unit, Compose, and real-Obsidian coverage, while some old database-format options remain executable only because existing data must still be opened safely.
Version 1.0 needs to distinguish supported opt-in features from previews and from compatibility paths. Removing a label does not make a network environment reliable, and retaining a setting key does not make it a recommendation.
## Decision
### Supported opt-in features
- Peer-to-Peer Synchronisation is supported. Commonlib owns the transport lifecycle, the Compose suite covers transfer, replacement, disconnect, and reconnect behaviour, and the Obsidian host verifies its current pane boundary. Documentation must describe environmental WebRTC limitations without calling the feature experimental.
- Hidden File Sync is supported as an advanced, separately initialised feature. It remains disabled during ordinary Setup URI initialisation and has its own two-Vault, conflict, filtering, and notification acceptance workflow.
- Customisation Sync is supported as an advanced opt-in feature. Its maintained two-Vault real-Obsidian workflow covers snippets, configuration files, and plug-in files, including updates and deletion of source synchronisation data. It remains separate from Hidden File Sync and must not manage the same files concurrently.
- Data Compression is maintained as an advanced opt-in storage and bandwidth trade-off. It remains disabled by default. The three-repeat CLI and CouchDB benchmark reduced stored chunk data and upload bodies by about 9% for the mixed fixture, but processing and worker-memory costs remained substantial. The [Data Compression specification](../specs_data_compression.md) records the contract, measurements, compatibility behaviour, execution model, and reproduction path for future default-setting decisions.
- The real-Obsidian E2E runner is maintained release infrastructure rather than an experimental Harness.
### Retained previews
- JWT authentication remains experimental because it depends on specialised CouchDB server configuration and does not yet have a maintained server-backed authentication matrix. The current implementation, Setup URI transport, focused unit coverage, and reported ES512 use justify retaining it.
- Ignore files remain Beta. They have focused target-filter tests, but nested rules, hidden-file expectations, and open user reports still require review.
- Automatic newer-file conflict resolution remains Beta and disabled by default because it can deliberately overwrite one side of a conflict.
- Garbage Collection V3 remains Beta and explicitly initiated. Its algorithm and safety protocol are outside this classification change and require a separate decision.
### Compatibility-only and sunset paths
- E2EE V1 and its dynamic iteration-count setting remain for existing encrypted databases. E2EE V2 is the new-Vault contract.
- The old IndexedDB adapter remains only with its migration path back to IDB.
- `xxhash64` is the current hash contract. Other hash algorithms remain available for existing databases and edge-case recovery, not as experimental alternatives for new Vaults.
- Eden chunks remain accepted at runtime and in transported settings, but are not offered in the settings interface.
- `doNotUseFixedRevisionForChunks` remains an inert compatibility input. Chunk revisions are always content-derived.
- The deprecated cleaned-database reconciliation callback remains internal while an old IndexedDB client may still encounter that remote state. It is not a user-selectable maintenance action and is omitted from the settings reference.
### Already removed
The obsolete mocked browser Harness, automatic bulk chunk pre-send, legacy trash toggle, and fixed-revision control have no supported 1.0 UI path. Their compatibility data, where required, remains accepted independently of their removed controls.
## Consequences
- Supported opt-in features retain focused release gates and user documentation.
- Preview features remain off by default and keep explicit maturity labels.
- Compatibility-only settings must not silently change existing data formats. New configuration should not expose retired formats merely because their decoders remain available.
- Commonlib setting types and Setup URI decoding remain broad enough to read existing configurations. Removing those package contracts requires a separately versioned compatibility decision.
- Deprecated host accessors and Community directory API warnings are a separate refactoring boundary. This decision does not authorise removing broadly used internal access paths.
docs/settings.md
+25 -12
View File
@@ -4,6 +4,17 @@ NOTE: This document not completed. I'll improve this doc in a while. but your co
There are many settings in Self-hosted LiveSync. This document describes each setting in detail (not how-to). Configuration and settings are divided into several categories and indicated by icons. The icon is as follows:
## Feature maturity for 1.0
The following status applies to optional and compatibility features in the 1.0 line:
| Status | Features | 1.0 policy |
| ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Supported, opt-in | Peer-to-Peer Synchronisation, Hidden File Sync, and Customisation Sync | Maintained and covered by focused real-runtime tests. Enable them only where their separate setup and operational constraints are acceptable. |
| Maintained, advanced | Data Compression | Available as an explicit storage and bandwidth trade-off. It remains disabled by default because the measured processing and memory costs outweigh the mixed-dataset saving. |
| Beta or experimental | JWT authentication, ignore files, automatic newer-file conflict resolution, and Garbage Collection V3 | Retained for explicit testing and specialised use. They remain disabled by default and are not part of the minimum supported setup. |
| Compatibility only | V1 dynamic iteration counts, the old IndexedDB adapter, non-current hash algorithms, Eden chunks, and the stored `doNotUseFixedRevisionForChunks` key | Existing settings and data remain readable. New Vaults use the current defaults, and compatibility controls are shown only where a migration or recovery path still needs them. |
| Icon | Description |
| :--: | ------------------------------------------------------------------ |
| 💬 | [0. Change Log](#0-change-log) |
@@ -213,11 +224,11 @@ The encryption algorithm version used for end-to-end encryption.
- `v2` (V2: AES-256-GCM With HKDF): Recommended and default version.
- `forceV1` or `""` (V1: Legacy): Older legacy encryption. Only use this if you have an existing vault encrypted in the legacy format.
#### Use dynamic iteration count (Experimental)
#### Use dynamic iteration count (legacy V1 compatibility)
Setting key: useDynamicIterationCount
This is an experimental feature and not recommended. If you enable this, the iteration count of the encryption will be dynamically determined. This is useful when you want to improve the performance.
This setting applies only to legacy V1 encryption data. Keep the saved value when opening an existing V1 database. New Vaults use E2EE V2 and do not use this setting.
---
@@ -634,6 +645,8 @@ Comma separated `.gitignore, .dockerignore`
## 6. Customisation sync (Advanced)
Customisation Sync is a supported, advanced opt-in feature. Its current per-file implementation is covered by a two-Vault real-Obsidian workflow for snippets, configuration files, and plug-in files. Hidden File Sync is a separate feature with different setup, selection, and conflict behaviour; do not use both features to manage the same files.
### 1. Customisation Sync
#### Device name
@@ -778,10 +791,10 @@ Limit the maximum size of chunks sent in one request by the explicit **Resend al
### 1. Remote Database Tweak
#### Incubate Chunks in Document (Beta)
#### Incubate Chunks in Document (sunset compatibility)
Setting key: useEden
If enabled, newly created chunks are temporarily kept within the document, and graduated to become independent chunks once stabilised.
This setting is no longer offered for new configuration. Existing saved values remain accepted so that established databases can be opened and migrated without silently changing their structure.
#### Maximum Incubating Chunks
@@ -798,10 +811,14 @@ The maximum total size of chunks that can be incubated within the document. Chun
Setting key: maxAgeInEden
The maximum duration for which chunks can be incubated within the document. Chunks exceeding this period will graduate to independent chunks.
#### Data Compression (Experimental)
#### Data Compression (advanced opt-in)
Setting key: enableCompression
Data Compression applies fflate level 8 to each chunk before E2EE. A chunk is left uncompressed when compression would not make it smaller, and readers continue to accept both representations. Changing the setting does not require a rebuild for compatibility; existing and new representations can coexist.
The setting remains disabled by default because the measured storage and transfer benefit comes with workload-dependent processing and memory costs. See the [Data Compression specification](specs_data_compression.md) for the contract, evidence, execution model, and reproduction command.
### 2. CouchDB Connection Tweak
#### Batch size
@@ -897,10 +914,12 @@ Do not use internal API
Setting key: additionalSuffixOfDatabaseName
LiveSync could not handle multiple vaults which have same name without different prefix, This should be automatically configured.
#### The Hash algorithm for chunk IDs (Experimental)
#### The Hash algorithm for chunk IDs (compatibility)
Setting key: hashAlg
`xxhash64` is the supported current value. Older algorithms remain selectable only as an edge-case compatibility path for existing databases. Changing the algorithm can reduce chunk reuse between devices and requires the normal tweak review.
### 6. Edge case addressing (Behaviour)
#### Fetch database with previous behaviour
@@ -1004,12 +1023,6 @@ Purge all download/upload cache.
Delete all data on the remote server.
### 6. Deprecated
#### Run database cleanup
Attempt to shrink the database by deleting unused chunks. This may not work consistently. Use the 'Overwrite Server Data with This Device's Files' under Reset Synchronisation information.
### 7. Reset
#### Delete local database to reset or uninstall Self-hosted LiveSync
docs/specs_data_compression.md
+116
View File
@@ -0,0 +1,116 @@
# Data Compression specification
## Status and decision for 1.0
Data Compression is a maintained, advanced opt-in feature for CouchDB-compatible remote databases. It remains disabled by default in the 1.0 line.
The feature provides a modest, measurable reduction in mixed-workload storage and transfer volume. Its value depends heavily on the chunk contents, while the current implementation adds substantial processing and worker-memory costs. Users with slow, metered, or storage-constrained connections may still find that trade-off worthwhile.
## Stored-data behaviour
The `enableCompression` setting affects chunk documents written through the CouchDB remote connection. It does not compress the local database, and it is separate from the journal format used by Object Storage synchronisation.
For each document containing string `data`, the writer:
1. detects canonical Base64 and decodes it to bytes, or encodes text as UTF-8;
2. applies raw DEFLATE through fflate at level 8;
3. encodes the result as Base64 and adds the LiveSync compressed-data marker; and
4. stores the compressed representation only when its string representation is shorter than the original.
Already compressed data and data which does not become smaller are therefore retained unchanged. Readers always recognise and expand the compressed-data marker, including when their own `enableCompression` setting is off. Compressed and uncompressed chunks can coexist in the same remote database.
When E2EE is enabled, compression is applied before E2EE V2 encryption. The encrypted remote representation does not expose the compression marker.
Changing the setting does not require a rebuild for compatibility: new writes adopt the selected policy, while existing chunks remain readable. All devices are still asked to agree on the remote tweak value so that future writes use one consistent policy. A deliberate remote rebuild can normalise existing storage, but it is optional rather than a prerequisite for synchronisation.
## Execution model
The compression and decompression algorithms are not executed synchronously on Obsidian's UI thread. Browser builds call fflate's asynchronous `deflate` and `inflate` APIs, which create a Web Worker for an operation and terminate it after the callback. The CLI uses the corresponding Node worker-thread path.
The PouchDB transform hook, Base64 detection and conversion, UTF-8 conversion, worker creation, result conversion, and document mutation still run in the calling JavaScript context. The current implementation also creates a separate fflate worker for each attempted chunk rather than reusing LiveSync's persistent splitting and encryption worker pool. `transform-pouch` applies the incoming transform to a bulk batch with `Promise.all`, so a batch can start many of these workers concurrently. It can therefore consume significant total CPU and memory, and worker churn may still affect responsiveness or mobile process limits even though the DEFLATE calculation itself is off the UI thread.
The current benchmark measures the Node CLI process. It does not establish Obsidian UI event-loop latency, Electron renderer responsiveness, mobile WebView memory behaviour, battery use, thermal throttling, or platform watchdog thresholds. Those remain real-runtime validation gaps.
## Reproducible benchmark
The benchmark is implemented under `src/apps/cli/testdeno` and packaged by `test/bench-network`. It uses the real CLI mirror and synchronisation path, Commonlib 0.1.0-rc.4, CouchDB 3.5.0, the V3 Rabin–Karp splitter, optional Data Compression, and E2EE V2. It runs each of these conditions three times in rotating order:
- E2EE off, compression off;
- E2EE off, compression on;
- E2EE on, compression off; and
- E2EE on, compression on.
The 623,553-byte deterministic fixture contains three Markdown files, two generated JPEG files, two repository PNG files, two JSON files, two TypeScript files, one gzip file, and one high-entropy binary file. Every run materialises and byte-compares all 13 files after synchronisation.
Run it from the repository root:
```bash
BENCH_COMMAND=compression \
BENCH_COMPRESSION_REPEAT_COUNT=3 \
BENCH_COUCHDB_RTT_MS=1 \
docker compose -f test/bench-network/compose.yml run --build --rm bench-runner
```
The latest local three-repeat result was generated on 21st July, 2026. The percentages below compare medians with compression off and on.
| Measurement | E2EE off | E2EE on |
| ------------------------------------------------ | -------: | ------: |
| Stored chunk-data reduction | 9.12% | 9.01% |
| CouchDB external-size reduction | 9.03% | 8.92% |
| CouchDB file-size reduction | 2.57% | 7.62% |
| Upload request-body reduction | 8.58% | 8.61% |
| Complete materialisation response-body reduction | 6.56% | 4.84% |
| Upload wall-time increase | 197.10% | 199.29% |
| Upload CPU-time increase | 650.43% | 581.54% |
| Complete materialisation wall-time increase | 19.58% | 19.75% |
| Complete materialisation CPU-time increase | 45.11% | 43.98% |
With E2EE on, median upload wall time rose from 1.49 seconds to 4.45 seconds, CPU time rose from 1.30 seconds to 8.86 seconds, and upload maximum resident memory rose from about 160 MiB to 403 MiB. The full materialisation workflow starts a CLI process for each file and produced a much higher compressed-path peak of about 983 MiB; treat that figure as evidence about the current CLI workflow rather than a browser decompression lower bound.
The E2EE upload saved 99,034 decoded HTTP body bytes while adding about 2.96 seconds of local processing wall time. A simple serial-transfer estimate puts the wall-time break-even near 0.27 Mbit/s. This estimate excludes headers, contention, request overlap, radio energy, data charges, and server behaviour. The benchmark does not emulate a bandwidth-limited mobile link, so transfer-volume reduction may still be valuable where elapsed time is not the only cost.
## Results by file kind
The E2EE stored chunk-data reductions were:
| Fixture kind | Reduction |
| ------------------- | --------: |
| Markdown | 16.30% |
| JPEG | 4.72% |
| PNG | 6.16% |
| JSON | 72.80% |
| TypeScript | 74.11% |
| gzip | 0% |
| high-entropy binary | 0% |
These results describe payload and chunk characteristics, not a reliable file-extension policy. The JSON and TypeScript fixtures were repetitive and mapped to relatively large chunks. The Markdown files were split into 151 referenced chunks, so small-chunk overhead and limited repetition windows reduced their benefit. JPEG, PNG, and gzip inputs had already undergone format-level compression, while the deterministic binary input was intentionally difficult to compress.
The remote transform sees a content-addressed chunk, not a trustworthy original file type. A chunk can also be deduplicated across files with different extensions. Enabling compression only for selected extensions would therefore either require carrying new provenance into the chunk format or make the representation depend on whichever file first produced a shared chunk. Neither is suitable for the 1.0 format.
## Follow-up optimisation candidates
The first optimisation should remove unbounded per-chunk worker creation:
1. add compression and decompression tasks to a reusable worker pool;
2. run synchronous fflate inside those workers so that it does not create a nested worker for each task;
3. put a bounded scheduler or semaphore before dispatch, rather than submitting the complete `Promise.all` batch at once;
4. transfer derived input and output `ArrayBuffer` objects explicitly in each `postMessage` transfer list; and
5. decide that a result is not smaller inside the worker, so an unhelpful compressed buffer does not need to be returned.
LiveSync's current `bgWorker` is a useful starting point because it creates a fixed pool of approximately half the reported hardware concurrency and selects workers in round-robin order. It is not sufficient unchanged: its `processing` count does not control selection, and it does not limit the number of tasks posted to each worker. Its foreground modules also form a circular dependency: `bgWorker.ts` imports the splitting and encryption adapters, while those adapters import task dispatch and removal from `bgWorker.ts`. Compression must not add another branch to that cycle.
Before adding compression, the Worker code should be separated into a dependency-bottom pool, task registry, scheduler, and transfer transport, with splitting, encryption, and compression implemented as task adapters above it. The shared scheduler should own concurrency limits, cancellation, crash propagation, fairness, and task clean-up. Compression then needs either a separate bounded lane or scheduling which prevents a long level-8 task from starving splitting and encryption work.
The `TransformStream` currently used by `bgWorker.splitting` is local to the calling context; it is not transferred to the Worker. Moving a stream endpoint across contexts could provide real end-to-end back-pressure, but it is not the first memory optimisation for Data Compression. The [Streams Standard transfer algorithm](https://streams.spec.whatwg.org/#transferable-streams) transfers the stream endpoint and posts each chunk through an internal `MessagePort` with an empty transfer list. Binary chunks can therefore still be cloned. Explicitly transferring each `ArrayBuffer` is the clearer zero-copy boundary for independent chunk tasks.
Cross-context transferable streams also cannot yet be required by the supported mobile baseline. WebKit lists `ReadableStream`, `WritableStream`, and `TransformStream` transfer via `postMessage()` as a [Safari 27 beta addition](https://webkit.org/blog/17967/news-from-wwdc26-webkit-in-safari-27-beta/#readablestream-improvements). A future general `bgWorker` redesign may use transferable streams behind capability detection, particularly where back-pressure matters more than copying, but it needs a message-based fallback.
After bounding worker use, adaptive per-chunk compression can reduce unnecessary work without depending on the source filename:
1. skip very small chunks where the marker and worker start-up dominate;
2. sample decoded bytes and estimate entropy or repetition before starting level-8 DEFLATE;
3. require both a minimum byte saving and a minimum percentage saving;
4. compare lower DEFLATE levels against level 8; and
5. replace one-worker-per-chunk operation with a bounded, persistent compression worker pool and back-pressure.
Any optimisation must preserve the current wire contract: content may remain uncompressed, the compressed marker must stay readable, compression must precede E2EE, and mixed representations must interoperate. It should be evaluated with the same four-condition benchmark, a bandwidth-shaped case, UI event-loop latency, and at least one real mobile Obsidian run before reconsidering the default.
docs/tips/p2p-sync-tips.md
+1 -1
View File
@@ -11,7 +11,7 @@ authors:
# Peer-to-Peer Synchronisation Tips
> [!IMPORTANT]
> Peer-to-peer synchronisation is still an experimental feature. Although we have made every effort to ensure its reliability, it may not function correctly in all environments.
> Peer-to-peer synchronisation is a supported opt-in feature. WebRTC connectivity still depends on the networks, relays, and optional TURN servers available to every device, so a working connection cannot be guaranteed in every environment.
## Difficulties with Peer-to-Peer Synchronisation