obsidian-livesync/docs/datastructure.md

date, commonlib-version, self-hosted-livesync-version, status

date	commonlib-version	self-hosted-livesync-version	status
2026-09-03	0.1.21	1.0.24	accepted

Database Data Structures

Scope and Authority

This document is a developer overview of the database structures used by the current Self-hosted LiveSync 1.0 series. It is not a stable, forward-compatible API for constructing CouchDB documents by hand.

The executable authority for document types, path and identifier encoding, chunk splitting and hashing, encryption, compression, and content reconstruction is the exact @vrtmrz/livesync-commonlib version recorded in the repository lockfile. Commonlib owns this domain under the package-boundary decision. When this overview and that installed package differ, correct this document and treat the package behaviour as authoritative for the affected release.

Three representations must be distinguished:

1. the decoded application representation used by Commonlib services;
2. the local PouchDB representation, including CouchDB revision metadata; and
3. the raw remote representation after any configured compression, E2EE, or path-obfuscation transform.

The examples below describe the first two representations unless a section explicitly discusses the raw remote representation. The exact raw remote shape depends on the configured transforms and protocol version and cannot be inferred from the decoded or local examples alone.

Principal Document Families

• file Metadata, including compatibility-only legacy Metadata;
• Chunks and compatibility transport structures such as Chunk Packs;
• database version, synchronisation, Milestone, and Node information; and
• CouchDB revision and deletion records.

Commonlib's EntryDoc is a union across several of these families. It is not synonymous with file Metadata.

Common CouchDB Fields

Database documents share this base shape:

export interface DatabaseEntry {
    _id: DocumentID;
    _rev?: string;
    _deleted?: boolean;
    _conflicts?: string[];
}

• _id identifies one CouchDB document.
• _rev identifies one revision of that document.
• _conflicts is returned when conflict information is requested. It is CouchDB revision metadata, not part of the persisted application document.
• _deleted: true creates a CouchDB tombstone. It is distinct from the logical file-deletion field deleted: true described below.

File Metadata

Current files are stored as chunked Metadata. The following is a simplified shape; the exported Commonlib declarations remain authoritative:

type ChunkedMetadata = DatabaseEntry & {
    ctime: number;
    mtime: number;
    size: number;
    deleted?: boolean;
    eden: Record<string, { data: string; epoch: number }>;
    path: FilePathWithPrefix;
    children: string[];
    type: "plain" | "newnote";
};

children contains Chunk document IDs in reconstruction order. A normal save persists every referenced Chunk before it persists the Metadata which names those Chunks. The writes are separate database operations rather than one atomic transaction, so another client may still observe the Metadata first. The resulting retrieval contract is documented in Chunk Retrieval and Waiting.

The current persisted file types are:

• plain, for text content represented by literal text Chunks; and
• newnote, for binary content represented by Base64 Chunks.

The compatibility-only notes type stores content directly in its data field rather than in children. Existing data may be read through selected legacy paths, but current writers do not create notes documents, and not every current replication path accepts newly created legacy documents.

datatype appears on Commonlib's loaded and saving representations. The current Metadata writer does not persist it, so it is absent from ordinary current CouchDB Metadata.

eden remains in the shared type for existing data compatibility. New configuration does not enable Eden, and current writers do not create incubated Eden Chunks for a new configuration.

Times and Size

ctime and mtime are Unix epoch times in milliseconds. size is the byte size of the decoded file content supplied by the storage boundary. Current storage adapters and generated Blob paths obtain it from filesystem metadata or Blob.size; JavaScript String.length is a UTF-16 code-unit count and is not a valid substitute for non-ASCII content.

Paths, Identifiers, and Namespaces

At the decoded boundary, path records the logical path, including any feature namespace prefix. _id is derived from that path by Commonlib's path service:

• a path beginning with _ receives a leading / in its document ID so that CouchDB does not interpret it as a reserved identifier;
• the path is folded to lower case only when handleFilenameCaseSensitive is disabled; and
• when path obfuscation is enabled, the body of the document ID is replaced by an f: SHA-256-derived value. A feature prefix is retained, so an obfuscated Hidden File Sync ID can begin with i:f:.

The main namespaces are:

Prefix	Meaning
none	An ordinary Vault file
i:	Hidden File Sync Metadata
ix:	Customisation Sync Metadata
ps:	Compatibility namespace for plug-in storage data
f:	Obfuscated document-ID body
h:	Chunk document
h:+	Chunk whose identifier incorporates encryption material

Namespaces identify storage and path handling; they do not by themselves select an Entry type. Current Hidden File Sync and Customisation Sync writers store chunked plain or newnote documents under i: and ix:. The application-local type: "plugin" interface is not a Commonlib Entry type and is not the current Customisation Sync storage format. Although Commonlib retains an internalfile constant for compatibility, current Hidden File Sync producers do not use it as their persisted type.

The decoded path does not become f:{obfuscated path}. Compression, E2EE, and path obfuscation can change raw remote identifiers and properties. Commonlib owns those transforms, and their exact representation depends on the selected settings.

The validation and explicit repair contract for ordinary-file Metadata whose stored _id does not agree with its decoded path is defined in Normal-file Metadata Document ID Validation and Repair.

Chunk Documents

export type EntryLeaf = DatabaseEntry & {
    type: "leaf";
    data: string;
    isCorrupted?: boolean;
};

A leaf stores one content-addressed piece. For plain Metadata, data is literal text. For newnote Metadata, data is Base64 text representing binary bytes. Concatenating and decoding the children in order reconstructs the decoded file content.

Commonlib's configured HashManager produces content-derived Chunk identifiers. Their representation can vary with compatibility and encryption settings. Historical hash algorithms remain readable only as compatibility settings.

Chunk revisions are content-derived irrespective of the obsolete stored doNotUseFixedRevisionForChunks setting. Compression and E2EE may transform a Chunk's raw remote data and add representation markers, so the remote value can differ from the decoded Chunk data.

File Deletion

LiveSync distinguishes two operations:

• deleted: true is a logical deletion of a file. With Metadata retention enabled, an ordinary current-file deletion preserves the existing Metadata fields and Chunk references, updates mtime, and creates a new Metadata revision. This permits the deletion to participate in synchronisation and conflict history.
• _deleted: true is a CouchDB tombstone for one document revision. That revision does not retain the application body. Tombstones are used by explicit compatibility and clean-up paths.

A logical deletion does not clear children or set size to zero. The deleteMetadataOfDeletedFiles setting can request an immediate tombstone instead. Whether retained, logically deleted Metadata is later tombstoned depends on the configured deletion-retention settings.

Branch-specific conflict operations and their ancestry requirements are defined in the Conflict Resolution specification.

Control Documents

The principal control documents are:

Document	Identifier	Purpose
Version information	obsydian_livesync_version	Records the internal database version. The historical spelling is retained for compatibility.
Synchronisation information	syncinfo	Stores rebuild-related synchronisation information for CouchDB-based operation.
CouchDB synchronisation parameters	_local/obsidian_livesync_sync_parameters	Stores the protocol version and PBKDF2 salt on the remote.
Journal synchronisation parameters	_obsidian_livesync_journal_sync_parameters.json	Journal counterpart of the synchronisation-parameter record.
Milestone information	_local/obsidian_livesync_milestone	Records accepted Nodes, locking, clean-up state, Chunk version ranges, and synchronisation tweak values.
Node information	_local/obsidian_livesync_nodeinfo	Records the local Node identifier and compatibility markers.

The _local/ records are CouchDB-local documents and do not replicate like ordinary Metadata and Chunks. Synchronisation parameters are checked before connecting; an incompatible protocol or encryption configuration stops synchronisation rather than being ignored.