Tracked to v1.0.46; mentioning provenance improvements

This commit is contained in:
vorotamoroz committed 2026-11-09 19:41:01 +09:00
1 parent c2e4e9198b
commit ad8655a08b
4 files changed
+114 -67

No files matched your search

+31 -28
View File
@@ -15,83 +15,86 @@ authors:
affiliations:
- name: 'Independent Researcher'
index: 1
date: 5 September 2026
date: 7 November 2026
bibliography: paper.bib
---
# Summary
Self-hosted LiveSync is an open-source synchronisation plug-in for Obsidian [@obsidian], a note-taking application that stores documents as local Markdown files. It replicates a user's vault—a directory containing notes and attachments—across desktop and mobile devices using user-controlled storage or direct peer-to-peer connections.
Self-hosted LiveSync is an open-source synchronisation plug-in for Obsidian [@obsidian], which stores notes as local Markdown files. It replicates a user's vault—a directory containing notes and attachments—across desktop and mobile devices via self-hosted storage or direct peer-to-peer connections.
The plug-in allows users to continue editing offline and synchronise upon reconnection, even when conflicting edits arise—such as when two disconnected devices modify the same note concurrently. Rather than forcing immediate reconciliation or unconditionally overwriting competing changes, the system supports automatic merging of non-overlapping edits and preserves competing versions for deferred review, depending on file formats and configured policies. Built-in inspection tools help users investigate conflicts or missing content and recover files when surviving copies exist.
Users can edit offline and synchronise upon reconnection, even after concurrent edits to a note. Rather than forcing immediate reconciliation or unconditionally overwriting changes, the plug-in merges non-overlapping edits automatically or retains competing versions for deferred review by format and policy, with inspection tools for conflict analysis and recovery when surviving copies exist.
The software serves researchers, engineers, and practitioners who require continued note-taking across multiple devices while controlling their data storage.
The software serves researchers, engineers, and practitioners requiring multi-device note-taking under self-managed storage.
# Statement of Need
Research and engineering workflows depend on long-lived notes, observations, design decisions, and supporting files. The author's work required managing the software installed on each device, keeping files on infrastructure under personal control, and using server software with an established operational record. These constraints motivated a synchronisation engine running directly inside Obsidian across desktop and mobile platforms without external client daemons.
Research and engineering workflows rely on long-lived notes, observations, design records, and attachments. The author's work required managing the software installed on each device, keeping files on self-managed infrastructure, and using server software with an established operational record. These constraints motivated an in-app synchronisation engine running directly inside Obsidian across desktop and mobile platforms without external client daemons.
Conflicts arising from concurrent edits on disconnected devices are recognised only after devices exchange updates. An edit or deletion may be unintended, concurrent modifications may diverge, or external tools may update files independently of database events. Overwriting with a single version without retaining competing revisions risks irreversibly discarding information before users can evaluate the divergence.
Concurrent edits on disconnected devices surface only when updates are exchanged. Because unintended edits or deletions, divergent changes, or external modifications can occur outside database events, overwriting with one version without retaining competing revisions risks discarding information before review.
During fieldwork and mobile operations, researchers and practitioners often need to continue recording observations and transferring notes between devices before reviewing competing edits. Self-hosted LiveSync supports this separation of recording and reconciliation: replication proceeds while concurrent branches remain unresolved, protecting competing edits until they can be reviewed or resolved according to configured policies.
During fieldwork and mobile use, users often record and transfer notes before reviewing conflicts. Self-hosted LiveSync separates recording from reconciliation: replication continues while concurrent branches remain unresolved until reviewed or resolved by policy.
# State of the Field
Local-first software combines local availability with multi-device synchronisation and collaboration while avoiding dependence on a hosted service as the sole owner of user data [@kleppmann2019localfirst]. Within the Obsidian ecosystem, Obsidian Sync provides an integrated hosted service [@obsidiansync]; Obsidian Git provides version-control-oriented push and pull workflows [@obsidiangit]; Syncthing operates at the filesystem layer [@syncthing]; and Remotely Save connects Obsidian to several cloud and self-hosted storage APIs [@remotelysave].
Local-first software combines local availability with multi-device synchronisation without a hosted service as sole data owner [@kleppmann2019localfirst]. For Obsidian, Obsidian Sync offers a hosted service [@obsidiansync], Obsidian Git provides Git workflows [@obsidiangit], Syncthing synchronises filesystem trees [@syncthing], and Remotely Save connects to cloud and self-hosted storage APIs [@remotelysave].
These approaches differ in how they represent concurrent changes. Syncthing propagates conflict copies as ordinary files [@syncthingsync], while Git can fetch divergent histories into separate tracking branches before merging them [@gitfetch], an approach automated on desktop and mobile by Obsidian Git [@obsidiangit]. Conflict-free replicated data types (CRDTs) can also expose alternatives: Automerge retains concurrent assignments to an object property for inspection [@automergeconflicts]. Self-hosted LiveSync retains competing file versions as leaves—the current versions of divergent branches—in the metadata document's revision tree until resolved by configured policies or explicit user action.
These tools represent concurrency differently: Syncthing propagates conflict copies as ordinary files [@syncthingsync], Git fetches divergent histories into tracking branches before merging [@gitfetch] (automated by Obsidian Git [@obsidiangit]), and CRDTs such as Automerge retain concurrent property assignments [@automergeconflicts]. Self-hosted LiveSync retains competing file versions as leaves—current versions of divergent branches—in the metadata revision tree until resolved.
These existing tools address distinct operational needs: hosted services prioritise turnkey convenience, external file synchronisers manage arbitrary filesystem trees, and version-control tools introduce explicit commit workflows. In the author's environment, however, managed devices prohibited background client daemons, while retaining competing revisions alongside their associated file updates required integrating replication directly with local file operations. Self-hosted LiveSync was therefore implemented as an Obsidian plug-in running entirely within the application runtime, backed by a decoupled, platform-independent engine to maintain uniform revision semantics across backends.
Hosted services prioritise convenience, external synchronisers manage raw filesystems, and version-control tools use explicit commits. In the author's environment, however, managed devices prohibited background client daemons, while retaining competing revisions alongside their associated file updates required coupling replication directly with local file operations. Self-hosted LiveSync was therefore implemented to run in-app inside Obsidian over a decoupled, platform-independent engine with uniform revision semantics.
Self-hosted LiveSync does not introduce a new database replication algorithm; rather, its contribution lies in applying revision-aware database semantics to an externally editable file vault. Content-addressed chunks, device-local branch provenance, and built-in recovery tools support continued editing while conflict review is deferred, protecting divergent work without altering standard note-taking workflows.
Rather than proposing a new replication algorithm, Self-hosted LiveSync applies revision-aware database semantics to an externally editable vault, using content-addressed chunks, device-local provenance, and inspection tools to support editing while conflict review is deferred.
# Software Design
The shared replication and conflict-handling services are published as `@vrtmrz/livesync-commonlib` [@commonlib021] and used by the Obsidian plug-in, command-line interface (CLI), web application, and web peer.
Shared replication, conflict-handling, and logical path-identity services are published as `@vrtmrz/livesync-commonlib` [@commonlib040] for the Obsidian plug-in, command-line interface (CLI), web application, and web peer, separating shared logical namespaces and path comparisons from host-owned physical file operations.
## Revision-aware Vault representation
Each Vault file is represented in local PouchDB [@pouchdb] by a metadata document containing its path, size, modification time, and references to separate chunk documents. Chunks are content-addressed and can therefore be reused across revisions and files with identical content regions. When concurrent updates occur across devices, they form multiple competing leaves around the metadata document. PouchDB selects a deterministic winner for default retrieval, but this choice is an internal tie-breaker rather than evidence that the winner is newer, safer, or the version represented by a particular device's Vault.
Each Vault file is stored in local PouchDB [@pouchdb] as a metadata document recording its path, size, modification time, and references to content-derived chunks from the V3 Rabin–Karp rolling-hash splitter by default, reusing unchanged chunks across revisions and files. Concurrent updates form competing leaves around the metadata document; PouchDB's deterministic default winner does not prove that branch is newer, safer, or displayed in the Vault.
When concurrent updates diverge into branches $\alpha$ and $\beta$, both leaves replicate to other devices before resolution. Automatic three-way merging applies to Markdown (`.md`), Canvas (`.canvas`), and JSON (`.json`) files when both leaves and their nearest available shared ancestor are readable; other formats are excluded. In Markdown, concurrent insertions at the same offset concatenate sequentially by modification time. Because CouchDB replication transfers ancestry identifiers without ancestor content [@couchdbreplication], missing ancestral history or conflicting edits defer automatic merging. When automatic merging is disabled or inapplicable and both versions are readable, JSON and differing binary files resolve by modification time as a compatibility fallback, even when 'Always overwrite with a newer file' is disabled; enabling that option extends modification-time resolution to text conflicts. Otherwise, competing text versions remain for manual resolution, and can be compared via a two-way diff when both leaves are readable.
When concurrent updates diverge into branches $\alpha$ and $\beta$, both leaves replicate before resolution. Automatic three-way merging applies to Markdown (`.md`), Canvas (`.canvas`), and JSON (`.json`) files when both leaves and their nearest available shared ancestor are readable (same-offset Markdown insertions concatenate by modification time). Because CouchDB replication transfers ancestry identifiers without ancestor bodies [@couchdbreplication], missing ancestor content or overlapping edits defer automatic merging. Without automatic merging, readable JSON and differing binary conflicts resolve by modification time even with 'Always overwrite with a newer file' disabled; that option extends modification-time resolution to text conflicts, which otherwise await manual two-way diff resolution.
## Device-local branch provenance
While the database can retain competing branches concurrently, a local Vault can instantiate only a single concrete file at any given path. To resolve which branch a local file represents, the plug-in stores an exact database revision and observed local modification time in a device-local key-value store. This revision serves as the path's **branch anchor**, updated after a successful database-to-Vault reflection or Vault-to-database write.
While the database retains competing branches, a Vault instantiates one file per path. After database-to-Vault reflection or Vault-to-database writes, the plug-in records the database revision, local modification time, SHA-256 hash, and byte length in a device-local store.
During active conflicts, local edits or logical deletions become children of the anchored revision, advancing that branch while keeping competing leaves intact. Cross-path renames store the target before logically deleting only the anchored source branch. If provenance is unavailable, the plug-in binds a file to an existing revision only when its bytes match exactly one available revision body; otherwise, it retains the conflict for manual resolution rather than guessing from paths or timestamps. Without active conflicts, ordinary writes simply advance the database revision.
This **branch anchor** tracks the displayed branch across saves and conflicts. Matching the readable recorded revision or saved hash and size shows a Vault file is unchanged—even after pruning that revision body—so unedited local files do not overwrite newer database state.
The built-in conflict inspector examines the current winner, every conflict leaf, and the nearest available shared ancestor. It reports missing chunks and file/database differences, permitting operations on an explicitly selected current leaf. Mutating operations recheck the revision beforehand, preventing a stale inspection from deleting or extending a superseded branch.
When local bytes differ from the readable recorded revision, edits extend that anchored branch while keeping competing leaves intact. Without readable provenance or matching content evidence, saves and reflection check current non-deleted leaves rather than ancestors: a unique match reconstructs provenance, multiple matches avoid duplicate writes without identifying a displayed branch, and no match preserves an independent conflict branch rather than guessing from paths or timestamps. During conflicts, deletions logically delete the proven displayed branch (or retain all branches if unproven), case-only renames resolving to the same document ID extend the displayed branch in place, and cross-path renames write the target before logically deleting only the proven source branch. Provenance is device-local evidence, not an atomic transaction or missing-ancestor substitute; after an interrupted save, start-up replay converges duplicates or defers concurrent updates to merge or review in tested scenarios.
The conflict inspector examines the winner, every conflict leaf, and the nearest available shared ancestor, rechecking revisions before mutating actions so stale views cannot delete or extend superseded branches. A read-only integrity diagnostic compares metadata and active conflicts across the Vault, local database, and CouchDB, distinguishing uninspected from verified content with optional pseudonymised reporting.
## Transport-independent replication
Using CouchDB's revision model as a baseline, Self-hosted LiveSync decouples database representation from network transport, replicating file-metadata revision identifiers and competing leaves intact across backends. CouchDB [@couchdb] provides native revision-aware replication. S3-compatible storage journals metadata leaf revisions and ancestry identifiers without synthesising new revisions, while chunk documents are stored as new local revisions with content-derived identifiers. The WebRTC peer-to-peer (P2P) adapter [@webrtc] uses Trystero [@trystero] DataChannels and an RPC-based replication shim to batch document requests while preserving identical revision semantics directly between peers. In CouchDB and journal transfers, Web Streams pipeline data to limit the amount of data buffered in memory during transfer.
Using CouchDB's revision model, Self-hosted LiveSync decouples database representation from transport, replicating metadata revision IDs and competing leaves intact. CouchDB [@couchdb] provides native revision-aware replication; Fast Setup verifies and retains all competing leaves—including logically deleted branches—and hands eligible fresh fetches to ordinary replication. S3-compatible storage journals metadata leaf revisions and ancestry identifiers without synthesising new revisions, storing content-addressed chunks. The WebRTC peer-to-peer (P2P) adapter [@webrtc] batches requests over Trystero [@trystero] DataChannels and an RPC shim while preserving revision semantics, and Web Streams pipeline CouchDB and journal transfers.
Transports combine flexibly: while P2P requires concurrent online presence, pairing it with CouchDB or object storage bridges offline intervals. All transports support end-to-end content encryption and path obfuscation. P2P encrypts session descriptions during connection negotiation, though signalling relays and network services can still observe connection timing and network addresses.
Transports combine when synchronising clients share a path-identity mode, pairing P2P with CouchDB or object storage across offline intervals. On supported filesystem hosts (Obsidian and the CLI), optional Unicode NFC path identity equates composed (NFC) and decomposed (NFD) path spellings into one logical path identity across CouchDB, object storage, and P2P without renaming physical files or changing chunk IDs, while keeping letter-case handling separate [@livesyncnfc1046]. Existing setups retain their configured mode by default, and migrating a legacy database to NFC requires resolving collisions before an explicit rebuild and peer fetch [@livesyncrecovery1046].
All transports support end-to-end encryption, path obfuscation, and an optional independent ID key; on CouchDB with required encryption settings, new writes encrypt internal synchronisation metadata properties, while document IDs, revision structure, document counts, and P2P relay metadata remain observable.
## Retention and recovery
Because alternative branches share unchanged chunks, retaining competing leaves incurs storage and transfer costs primarily for new chunks and revision metadata. Remote database rebuilds reclaim space, while an explicitly initiated beta garbage-collection workflow provides in-place CouchDB cleanup by protecting chunks reachable from the current winner, every conflict leaf, and the available ancestry needed to inspect active conflicts. Because superseded chunks may be collected, historical revisions are not an unconditional backup.
Because divergent branches share unchanged chunks, retaining competing leaves adds storage and transfer costs mainly for new chunks and revision metadata. Remote rebuilds reclaim space, while an explicit beta garbage-collection workflow cleans CouchDB in place by protecting chunks reachable from the winner, every conflict leaf, and available conflict ancestry; because superseded chunks may be collected, historical revisions are not an unconditional backup.
When chunks are missing, unreadable current revisions remain in the tree rather than being automatically discarded during conflict processing. Because missing chunks may still exist on other devices, users can defer recovery until they reconnect and synchronise. The conflict inspector identifies affected revisions and facilitates retrieval retries and explicit recovery actions. Recovery ultimately requires surviving content on a device, in remote storage, or in a backup.
When chunks are missing, unreadable current revisions remain in the tree during conflict processing, deferring recovery until surviving copies reconnect. Likewise, during S3-compatible journal reception, decode, processing, or local-save failures halt replication before advancing the checkpoint so that retrying can re-receive the failed journal within the retained remote journals [@livesyncrecovery1046]. Recovery workflows and the conflict inspector assist retries and recovery when surviving copies exist.
# Research Impact Statement
Self-hosted LiveSync originated in the author's multi-platform software engineering workflows, capturing screenshots and recording observations across multiple devices, including when a primary device was unavailable. Today, the same workflow supports the author's patent prior-art investigations, synchronising notes and reflections made while reading prior patent literature. Retaining competing revisions allows divergent observations to be compared after the fact rather than overwritten immediately.
Self-hosted LiveSync originated in the author's multi-platform software engineering workflows, capturing screenshots and recording observations across devices even when a primary device was unavailable. Today, the same workflow also supports the author's patent prior-art investigations, synchronising notes and reflections while reviewing patent literature and retaining competing revisions for later comparison rather than overwriting them immediately.
Unit and integration tests cover revision ancestry, chunk reachability, unavailable content, and host composition. CLI and real-Obsidian scenarios exercise conflict propagation and resolution, including edits, logical deletions, and renames while competing leaves remain active. A three-node P2P scenario verifies that unresolved leaves move between devices before resolution. Reusable headless test infrastructure is archived independently [@fancykit]. Deterministic fixtures compare P2P and CouchDB paths over identical generated data; controlled local measurements do not establish universal performance.
Test procedures in 1.0.46 [@livesynctest1046] cover revision ancestry, provenance reconstruction, chunk reachability, unavailable content, CLI and real-Obsidian conflict propagation and resolution (edits, logical deletions, renames, interrupted saves, and start-up reconciliation), and three-node P2P leaf transfer, alongside the Unicode NFC acceptance matrix and dated Linux milestone E2E verification across CouchDB, object storage, and P2P [@livesyncnfc1046]. Public CI for 1.0.46 [@livesyncci1046] confirms automated unit, contract, and CLI/Docker suites with Commonlib 0.4.0 [@commonlib040], without claiming real-Obsidian milestone E2E reruns on 1.0.46 or native iOS/macOS filesystem validation. Archived headless infrastructure [@fancykit] and deterministic fixtures compare P2P, CouchDB, and chunking paths without claiming universal performance.
As of 2 September 2026, the Obsidian plug-in directory reported more than 900,000 downloads, desktop and mobile support, and placement in its Research category [@obsidianplugin]. The GitHub repository recorded over 12,200 stars, 440 forks, and contributions from a broad user community [@selfhostedlivesyncrepo]. These figures demonstrate community adoption rather than direct research impact, but they provide evidence that the software operates beyond a single private workflow.
As of 2 September 2026, the Obsidian plug-in directory reported over 900,000 downloads, desktop and mobile support, and placement in its Research category [@obsidianplugin], with over 12,200 GitHub stars, 440 forks, and community contributions [@selfhostedlivesyncrepo]. These figures indicate community adoption rather than direct research impact, providing evidence that the software operates beyond a single private workflow.
The software described here is Self-hosted LiveSync 1.0.23 [@selfhostedlivesync], released under the MIT licence and pinned to Commonlib 0.1.21 [@commonlib021]. Zenodo archives the plug-in, the reusable test harness [@fancykit], and an earlier Commonlib 0.1.19 snapshot [@commonlib]. Platform-independent logic supports independent testing and reuse.
The software described here is Self-hosted LiveSync 1.0.46 [@selfhostedlivesync1046], released under the MIT licence and pinned to Commonlib 0.4.0 [@commonlib040]. Zenodo archives earlier snapshots of the plug-in [@selfhostedlivesync] and Commonlib [@commonlib], alongside the test harness [@fancykit].
# AI Usage Disclosure
OpenAI Codex using GPT-5 was used from July to September 2026 for code navigation, test and benchmark scaffolding, CI and documentation edits, manuscript editing, proofreading, review, citation verification, and result summarisation. GPT-6 assisted with September manuscript review and revision through Codex. No other generative AI tools prepared the manuscript. GitHub Copilot assisted with commits in this release, and Google Gemini (Flash versions 3.5 to 3.8) supported codebase verification. The human author validated all assisted outputs, made core design decisions, ran verification commands, and remains responsible for the accuracy, originality, licensing, and ethical compliance of the submitted materials.
OpenAI Codex (GPT-5 and GPT-6) was used from July to November 2026 for code navigation, test and benchmark scaffolding, CI and documentation edits, and manuscript drafting and review. GitHub Copilot assisted with commits, and Google Gemini (Flash 3.5–3.8 and Gemini 4) supported codebase verification and manuscript revision. The author validated all outputs, made core design decisions, ran verification, and remains responsible for the accuracy, originality, licensing, and ethical compliance of the submission.
# Acknowledgements
The author acknowledges project contributors, users, and upstream maintainers of PouchDB, CouchDB, and Trystero. The project has received community support through GitHub Sponsors, development-tool licensing from JetBrains, and support through OpenAI's Codex for Open Source programme.
The author thanks contributors, users, and maintainers of PouchDB, CouchDB, and Trystero, alongside GitHub Sponsors, JetBrains tool licensing, and OpenAI's Codex for Open Source programme.
# References