Compare commits

...

5 Commits

Author SHA1 Message Date
vorotamoroz 0890a97222 test(obsidian): cover adaptive S3 journal upload 2026-08-01 03:52:57 +00:00
vorotamoroz b7da78ce45 feat: expose adaptive journal options for S3 2026-07-31 19:24:40 +00:00
vorotamoroz 53d4bd527e docs: split adaptive journal decisions by provider 2026-07-31 19:24:40 +00:00
vorotamoroz 64870a32ea docs: specify adaptive journal sync design 2026-07-31 19:24:40 +00:00
vorotamoroz 5ede11d032 docs: plan adaptive journal synchronisation 2026-07-31 19:24:40 +00:00
21 changed files with 2190 additions and 23 deletions
@@ -0,0 +1,90 @@
# Architectural Decision Record: Use Native Batch CAS for Adaptive PostgREST
## Status
Proposed as the final experimental provider, after the S3 and WebDAV delivery sequences have established the common
protocol, CLI, host, and end-to-end boundaries.
## Context
PostgREST can expose PostgreSQL uniqueness, row-level security, bounded set operations, and transactions. Treating it
as another opaque object store works, but leaves Metadata and Chunks combined and cannot use a multi-key Chunk query.
Using one HTTP request per logical Chunk would make latency dominate synchronisation.
The common protocol decision is recorded in
[Adaptive Journal as an explicit protocol](2026_07_adaptive_journal_protocol.md). The tables, binary RPC framing,
transaction rules, and privacy model are specified in the
[Adaptive Journal Sync design](../design_docs/adaptive_journal_sync.md).
## Decision
Adaptive PostgREST separates Metadata publication from immutable Chunk storage physically as well as logically.
- Vault- and repository-scoped Chunk rows use the Remote Chunk key as an insert-only unique address.
- Bounded binary RPCs implement `hasMany`, `getMany`, and `putMany` while preserving input order and per-entry status.
- Writer descriptors, Metadata batches, and commits remain small append-only records.
- A transactional commit verifies the sorted required-Chunk-key digest and the existence of every referenced Chunk
before making Metadata visible.
- Concurrent insertion of a different encrypted frame for the same logical Chunk reads, validates, and accepts the
winning plaintext only when it represents the same logical value.
- Row-level security scopes every operation to the configured Vault. Server-visible Remote Chunk keys reveal equality
within that repository but do not reveal plaintext Chunk IDs.
PostgREST does not use object packs, catalogue deltas, or Range retrieval for native Chunk rows. The shared binary Chunk
record remains independently verifiable, but PostgreSQL supplies the batch index and uniqueness boundary.
The SQL schema and RPC contract are versioned together with the Adaptive format. A missing or incompatible schema is
detected before publication and requires applying the reviewed schema or rebuilding the remote. The client does not
perform an implicit data-format migration.
## Staged acceptance
### Adapter and Commonlib integration
Unit tests own binary-envelope limits, ordering, status decoding, insert conflicts, rollback, error classification,
Vault isolation, and malformed responses. Disposable PostgreSQL and PostgREST integration tests own the real SQL
schema, row-level security, RPC transactions, and two-client Metadata and Chunk synchronisation.
### CLI end-to-end acceptance
The built CLI applies an Adaptive PostgREST Setup URI to a second independent database and synchronises text and binary
Chunk-backed files through disposable PostgreSQL and PostgREST services. The scenario proves bounded native batching at
the headless product boundary without repeating the SQL failure matrix.
### Host settings and UI
The PostgREST dialogue persists the endpoint, Vault identifier, authentication configuration, Adaptive format, and
expected repository ID. Focused tests own connection-string and Setup URI preservation, validation, and format-mismatch
guidance; they do not execute SQL.
### Real-host end-to-end acceptance
One real-Obsidian workflow applies the reviewed schema and performs a representative Chunk-backed transfer. The
disposable Commonlib integration remains authoritative for RLS, transaction rollback, and the complete RPC matrix.
## Alternatives rejected
### Reuse immutable object packs in PostgreSQL
This would preserve portability at the cost of native multi-key lookup and transaction guarantees. The repository
contract already permits a different physical representation.
### Store raw file content with Metadata
Metadata must remain small and must continue to refer to immutable logical Chunks. Combining content with Metadata
would break the maintained PouchDB model and duplicate unchanged content across revisions.
### Encode Chunk bodies as JSON values
Base64 and large JSON arrays add framing and memory overhead and make limits harder to enforce. Bounded binary `bytea`
RPC envelopes provide deterministic lengths and status ordering.
## Consequences
- PostgREST has the largest provider-specific implementation because it includes reviewed SQL, RLS, RPC framing, and
transaction behaviour.
- Placing it last lets the common protocol, CLI, and host boundaries stabilise before introducing that larger surface.
- Native batching can reduce request count substantially relative to object-per-Chunk storage, but actual speed still
depends on database, proxy, network, and workload measurements.
- PostgREST and object stores share logical records and correctness rules without pretending to share a physical
layout.
@@ -0,0 +1,98 @@
# Architectural Decision Record: Introduce Adaptive Journal as an Explicit Protocol
## Status
Proposed for staged implementation. This decision does not change the current default Journal format or promise a
compatible in-place migration.
## Context
The existing Journal protocol publishes PouchDB Metadata and Chunk documents together in opaque immutable objects.
That representation is portable and remains the compatibility baseline, but it prevents a remote from answering a
bounded multi-key Chunk query or amortising object-store requests independently from Metadata publication.
S3-compatible Object Storage, WebDAV, and PostgREST have materially different physical capabilities. Making each one
implement a separate replication algorithm would duplicate ordering, encryption, recovery, and Chunk-delivery rules.
Making every one use an identical physical layout would discard useful native batching and transactions.
The detailed binary formats, state machines, privacy properties, and recovery rules are specified in the
[Adaptive Journal Sync design](../design_docs/adaptive_journal_sync.md).
## Decision
Adaptive Journal is a second, explicit Journal protocol version owned by Commonlib. It keeps one logical repository
contract while allowing provider-specific physical storage.
- Metadata events and raw Chunk records remain separate throughout publication.
- Logical Chunks are immutable and content-addressable. A changed value has a new logical Chunk ID.
- A repository-scoped Remote Chunk key is derived locally from the exact logical Chunk ID after accepting the
repository manifest.
- Chunks become durable before a commit makes Metadata references visible.
- Ordinary publication creates immutable records. Catalogue snapshots, caches, and indexes are derived and
reconstructible.
- Each writer publishes a dense sequence identified by a stable host ID and a persisted random writer epoch. A reader
tracks one frontier per writer stream; it does not depend on a remote `startAfter` ordering contract.
- Remote operations return typed outcomes which distinguish absence, conflict, permanent rejection, retryable failure,
and an ambiguous mutation which must be verified before retrying.
The adaptive repository exposes batched Chunk availability, publication, and retrieval even when its implementation
uses immutable packs internally. The caller does not issue one remote request per Chunk.
### Repository identity and compatibility
A new repository has one immutable, conditionally created manifest. The first device generates its candidate locally,
and the winning manifest fixes the repository ID, Security Seed, protocol parameters, and required capabilities.
Every client pins the accepted repository ID in local repository state. A Setup URI exported from an accepted binding
should carry that non-secret expected ID.
`opaque-v1` and `adaptive-v1` are separate remote formats. Format selection is explicit in the remote configuration,
and a mismatch fails before publication. The implementation detects incompatible remote data, but it does not migrate
it. Changing format requires an explicit remote rebuild or a new remote namespace.
Deprecated data formats do not enlarge the Adaptive protocol. Compatibility remains at the existing decoding
boundaries, and rebuilding the remote is the recovery path for an unsupported Adaptive layout.
### Provider delivery sequence
Each provider is delivered and reviewed through four boundaries, in order:
1. **Adapter and Commonlib integration.** Implement semantic capabilities, typed failure handling, format detection,
unit tests, and disposable real-service integration tests.
2. **CLI end-to-end acceptance.** Use the built CLI and a real disposable service to apply a Setup URI, synchronise two
independent local databases, and restore text and binary Chunk content. This proves the headless product boundary
without Obsidian or Svelte.
3. **Host settings and UI.** Add only provider-specific controls, validation, profile persistence, Setup URI transport,
and focused host tests.
4. **Real-host end-to-end acceptance.** Exercise one representative setup and synchronisation path in a real Obsidian
instance. This test verifies host composition and does not repeat the adapter capability matrix or the complete CLI
conflict suite.
The initial order is S3-compatible Object Storage, WebDAV, then PostgREST. A provider completes these boundaries before
the next provider is presented for integration review. The sequence keeps each review independently attributable and
keeps the real-host tests small.
## Alternatives rejected
### One cross-provider implementation change
This hides which provider requires a shared-core change, makes failures difficult to attribute, and forces reviewers to
understand SQL RPCs, object packs, WebDAV behaviour, CLI composition, and Obsidian UI in one change.
### Add the Host UI before headless acceptance
This makes an application-level failure ambiguous between the protocol, adapter, CLI-independent host composition, and
presentation. The CLI provides the smaller executable boundary first.
### Use one physical representation for every provider
One-object-per-Chunk storage causes excessive object requests, while forcing packs into PostgreSQL discards bounded
multi-key RPCs and transactions. Common semantics do not require common physical storage.
## Consequences
- Commonlib owns protocol correctness and provider adapters; Self-hosted LiveSync owns CLI composition, Obsidian host
integration, and presentation.
- Provider reviews can stop at the first failed boundary without involving later UI or real-host tests.
- The final end-to-end suite remains an acceptance layer rather than a duplicate protocol test suite.
- Adding another provider requires the same semantic contract and staged evidence, not another replication algorithm.
- Remote rebuild remains an explicit operational requirement while Adaptive Journal is evolving.
+96
View File
@@ -0,0 +1,96 @@
# Architectural Decision Record: Use S3 as the Reference Adaptive Object Store
## Status
Proposed as an improvement to the maintained S3-compatible Journal path. Adaptive Journal remains explicit and
opt-in; the existing opaque format remains the compatibility default.
## Context
S3-compatible Object Storage already supplies the maintained Journal object model and is the smallest provider on
which to prove Adaptive immutable packs. It has a standard conditional-create request, paginated listing, binary
objects, and optional byte-range retrieval, but compatible endpoints can still differ in consistency, proxy behaviour,
and Range handling.
The common protocol decision is recorded in
[Adaptive Journal as an explicit protocol](2026_07_adaptive_journal_protocol.md). The complete object-pack and catalogue
formats are specified in the [Adaptive Journal Sync design](../design_docs/adaptive_journal_sync.md).
## Decision
S3-compatible storage is the reference implementation of the Adaptive object-store strategy.
- It stores the manifest, writer control records, Metadata batches, commits, Chunk packs, indexes, and catalogue records
as immutable objects under the configured bucket namespace.
- Manifest and every immutable publication use conditional create. A successful create response is sufficient; the
client does not add a confirmation request solely for caution.
- A lost or ambiguous mutation response returns `verify-first`. The caller reads the exact key before retrying.
- A conditional-request conflict which does not establish an existing immutable value remains retryable and is not
reported as a successful create.
- Listing follows every continuation token and proves complete prefix enumeration. Folder marker objects and stale
capability-probe objects are not repository data.
- Format inspection distinguishes empty, `opaque-v1`, `adaptive-v1`, and mixed repositories. Mixed or mismatched data
fails closed, and reset remains an explicit batched operation.
The portable retrieval policy is `whole-pack`. A user may select `range` when the endpoint capability probe has
confirmed exact byte-range behaviour. Range responses must use `206`, carry a matching `Content-Range`, and return the
requested number of bytes. Loss of optional Range support falls back to whole-pack retrieval after reporting the
capability change; it does not make valid packs unreadable.
Capability results which prove support or lack of support may be cached for the endpoint identity. A transient probe
failure is not cached as a permanent result. Probe objects use a reserved random prefix and are removed when possible;
incomplete cleanup is reported and ignored by format detection.
## Staged acceptance
### Adapter and Commonlib integration
Focused tests cover conditional creation, ambiguous responses, binary fidelity, read-after-write visibility, paginated
listing, deletion visibility, Range validation, format detection, reset batching, and probe cleanup. A disposable
MinIO integration proves two-client Adaptive Metadata and Chunk synchronisation and both retrieval paths.
### CLI end-to-end acceptance
One real-MinIO scenario uses two independent CLI databases. Device A uses whole-pack retrieval. Device B receives an
Adaptive S3 Setup URI selecting Range retrieval. The scenario verifies that the URI preserves the format and policy,
that both devices can publish and receive more than one synchronisation round, and that text and binary content are
reconstructed from Chunks.
This test owns the built CLI, settings persistence, Setup URI decoding, and headless composition. It does not repeat
every S3 error classification already owned by Commonlib.
### Host settings and UI
The Object Storage dialogue exposes `opaque-v1` and `adaptive-v1`, with whole-pack as the Adaptive default and Range as
an explicit preference. It preserves the expected repository ID and read policy through saved profiles and Setup URI
handling. Focused host tests own normalisation, validation, persistence, and warnings; they do not contact S3.
### Real-host end-to-end acceptance
One real-Obsidian workflow applies an Adaptive S3 configuration and proves a representative Chunk-backed file transfer
against disposable MinIO. It verifies the Obsidian composition boundary only. The CLI suite remains the broader
headless synchronisation acceptance test.
## Alternatives rejected
### Re-upload a mutable pack when one Chunk changes
Packs are immutable publication units. A changed logical Chunk is added to a new pack, and a new catalogue delta makes
that location discoverable. Existing packs are replaced only by a separately protected compaction process.
### Require Range for Adaptive S3
Whole-pack reads often provide better throughput and work on more endpoints. Range is a deployment preference, not a
correctness requirement.
### Confirm every successful mutation with another request
This doubles request count in the ordinary success path. Exact-key verification is reserved for ambiguous outcomes and
explicit diagnostics.
## Consequences
- S3 establishes the object-store contract before the more variable WebDAV implementation.
- The two retrieval policies are exercised without duplicating the complete CLI scenario.
- Existing opaque S3 repositories remain readable and never become Adaptive implicitly.
- Endpoint capability evidence, rather than the S3-compatible label alone, controls optional behaviour.
@@ -0,0 +1,96 @@
# Architectural Decision Record: Gate Adaptive WebDAV with an Endpoint Safety Check
## Status
Proposed as an experimental provider after the S3 Adaptive path has completed adapter, CLI, host, and real-host
acceptance.
## Context
WebDAV exposes an object-shaped interface suitable for immutable packs, but method support and conditional semantics
vary across servers, gateways, reverse proxies, and authentication layers. A server can advertise WebDAV while failing
binary fidelity, replacing an object despite `If-None-Match: *`, returning incomplete listings, or ignoring Range.
The common protocol decision is recorded in
[Adaptive Journal as an explicit protocol](2026_07_adaptive_journal_protocol.md). The pack format, flat object mapping,
and detailed safety-check sequence are specified in the
[Adaptive Journal Sync design](../design_docs/adaptive_journal_sync.md).
## Decision
Adaptive WebDAV uses the same immutable pack and catalogue semantics proven by S3, with a WebDAV-specific flat object
mapping inside the configured collection. Logical writer ordering comes from the host ID, writer epoch, and dense
sequence embedded in authenticated records. Correctness does not depend on a server-provided `startAfter` order.
Before a new Adaptive repository becomes writable, a non-destructive endpoint safety checker exercises random reserved
probe keys and reports observed semantic capabilities:
- binary write and exact read-back;
- read-after-write visibility;
- complete collection listing for the probe keys;
- `If-None-Match: *` preventing replacement;
- delete visibility; and
- exact byte-range behaviour, reported separately as optional.
The checker is an implementation acceptance target for compatible servers, not a claim that every WebDAV server must
support Adaptive Journal. It touches no repository objects, attempts to remove every probe, reports incomplete cleanup,
and never interprets authentication, permission, timeout, malformed response, or server failure as absence.
Conditional create, binary fidelity, complete listing, read-after-write visibility, and delete visibility are required.
Range remains optional. The user selects whole-pack or Range retrieval based on their endpoint and own latency and
throughput preference; the checker does not benchmark or recommend a policy.
Successful mutations do not receive unconditional confirmation requests. An ambiguous response is classified as
`verify-first`, and the exact immutable key is read before retrying. The existing opaque WebDAV layout and Adaptive
layout are detected separately; a mismatch requires a remote rebuild or another namespace.
## Staged acceptance
### Adapter and Commonlib integration
Unit tests own HTTP status classification, conditional-create verification, flat-name round trips, complete listing,
probe isolation, cleanup reporting, Range validation, and whole-pack fallback. A disposable WebDAV integration runs
the safety checker before exercising Adaptive Metadata and Chunk synchronisation.
### CLI end-to-end acceptance
The built CLI applies an Adaptive WebDAV Setup URI to a second independent database and synchronises text and binary
Chunk-backed files through a real disposable server. One client uses whole-pack retrieval; Range is added to this layer
only when the selected test server proves it. The CLI test does not repeat the complete endpoint capability matrix.
### Host settings and UI
The WebDAV dialogue exposes Adaptive mode only with clear capability-check results. It persists the expected repository
ID and the selected retrieval policy, defaults to whole-pack, and reports that Range is optional. Focused tests own
profile and Setup URI preservation without contacting a server.
### Real-host end-to-end acceptance
One real-Obsidian workflow uses the same known disposable WebDAV implementation, runs the safety gate, and proves a
representative Chunk-backed transfer. Servers outside that fixture are diagnosed by the checker rather than added to a
large real-host matrix.
## Alternatives rejected
### Trust advertised WebDAV methods
Method advertisement does not prove the conditional, listing, visibility, and byte semantics required for immutable
publication.
### Treat Range as mandatory
Whole-pack retrieval is correct and often competitive for throughput. Requiring Range would exclude otherwise safe
servers for an optional optimisation.
### Add server-specific compatibility branches
The remote may be any implementation or proxy composition. Semantic checks produce a maintainable contract, while a
growing server-name table would remain incomplete and become stale.
## Consequences
- WebDAV remains experimental because suitability is endpoint-specific.
- The safety checker gives a concrete reason when a server cannot host Adaptive Journal.
- Common object-pack behaviour is inherited from the earlier S3 boundary, keeping WebDAV-specific tests focused on HTTP
semantics and name mapping.
- Optional Range support can improve request efficiency without becoming a data-availability requirement.
File diff suppressed because it is too large Load Diff
+12
View File
@@ -301,6 +301,18 @@ Setting key: bucketCustomHeaders
Custom HTTP headers to include in every request sent to the Object Storage bucket. Specify them in the format `Header-Name: Value`, with each header on a new line.
#### Journal data format
Setting key: journalFormat
Existing Object Storage profiles use `opaque-v1` unless Adaptive Journal is selected explicitly. `adaptive-v1` uses an authenticated manifest and immutable Commit Bundles under a separate namespace, with larger Packs stored separately. Changing between formats requires an explicit remote Rebuild; LiveSync does not migrate or read both representations.
#### Pack retrieval
Setting key: packReadPolicy
Adaptive Journal can download a complete immutable Pack (`whole-pack`) or request only the required byte ranges (`range`). Complete Pack retrieval is the portable, throughput-oriented default. Selecting Range requires exact byte-range support from the configured S3-compatible endpoint. The connection test verifies that capability, and synchronisation refuses an unsupported selection before writing.
#### Test Connection
#### Apply Settings
+1
View File
@@ -68,6 +68,7 @@
"test:e2e:obsidian:couchdb-manual-setup-workflow": "tsx test/e2e-obsidian/scripts/couchdb-manual-setup-workflow.ts",
"test:e2e:obsidian:cli-to-obsidian-sync": "tsx test/e2e-obsidian/scripts/cli-to-obsidian-sync.ts",
"test:e2e:obsidian:minio-upload": "tsx test/e2e-obsidian/scripts/minio-upload.ts",
"test:e2e:obsidian:adaptive-s3": "tsx test/e2e-obsidian/scripts/minio-upload.ts --adaptive",
"test:e2e:obsidian:object-storage-setup-uri-workflow": "tsx test/e2e-obsidian/scripts/object-storage-setup-uri-workflow.ts",
"test:e2e:obsidian:p2p-setup-uri-workflow": "tsx test/e2e-obsidian/scripts/p2p-setup-uri-workflow.ts",
"pretest:e2e:obsidian:p2p-connection-check": "npm run build && npm run build --workspace webpeer",
@@ -288,6 +288,13 @@ export const allMessages: Readonly<Record<string, Readonly<Record<string, string
zh: "生效中的远程配置",
"zh-tw": "目前啟用的遠端設定",
},
"Adaptive Journal (experimental)": {
def: "Adaptive Journal (experimental)",
},
"Adaptive Journal uses immutable objects and a separate remote format. Existing Opaque Journal data is not migrated or read. Rebuild the remote when changing formats.":
{
def: "Adaptive Journal uses immutable objects and a separate remote format. Existing Opaque Journal data is not migrated or read. Rebuild the remote when changing formats.",
},
"Add default patterns": {
def: "Add default patterns",
es: "Añadir patrones predeterminados",
@@ -784,6 +791,10 @@ export const allMessages: Readonly<Record<string, Readonly<Record<string, string
zh: "兼容性(问题修复)",
"zh-tw": "相容性(問題修復)",
},
"Complete Pack reads favour throughput. Range reads can reduce transferred bytes. The connection test verifies exact Range support on this endpoint, and synchronisation refuses an unsupported selection before writing.":
{
def: "Complete Pack reads favour throughput. Range reads can reduce transferred bytes. The connection test verifies exact Range support on this endpoint, and synchronisation refuses an unsupported selection before writing.",
},
"Compute revisions for chunks": {
def: "Compute revisions for chunks",
es: "Calcular revisiones para los chunks",
@@ -1585,6 +1596,9 @@ export const allMessages: Readonly<Record<string, Readonly<Record<string, string
ko: "문서 기록",
"zh-tw": "文件歷程",
},
"Download complete Packs": {
def: "Download complete Packs",
},
Duplicate: {
def: "Duplicate",
es: "Duplicar",
@@ -2605,6 +2619,9 @@ export const allMessages: Readonly<Record<string, Readonly<Record<string, string
ru: "Интервал (сек)",
zh: "间隔(秒)",
},
"Invalid Object Storage settings: ${reason}": {
def: "Invalid Object Storage settings: ${reason}",
},
INVERTED: {
def: "INVERTED",
es: "INVERTIDO",
@@ -2617,6 +2634,9 @@ export const allMessages: Readonly<Record<string, Readonly<Record<string, string
def: "It is strongly advised to create a backup before proceeding. Continuing without a backup may lead to data loss.",
es: "Se recomienda encarecidamente crear una copia de seguridad antes de continuar. Continuar sin copia de seguridad puede provocar pérdida de datos.",
},
"Journal data format": {
def: "Journal data format",
},
"Just for a minute, please!": {
def: "Just for a minute, please!",
es: "¡Solo un momento, por favor!",
@@ -6187,6 +6207,9 @@ export const allMessages: Readonly<Record<string, Readonly<Record<string, string
def: "On this device, switch to the camera app or use a QR code scanner to scan the displayed QR code.",
es: "En este dispositivo, cambia a la aplicación de cámara o usa un lector de QR para escanear el código mostrado.",
},
"Opaque Journal (current format)": {
def: "Opaque Journal (current format)",
},
Open: {
def: "Open",
es: "Abrir",
@@ -6447,6 +6470,9 @@ export const allMessages: Readonly<Record<string, Readonly<Record<string, string
ru: "P2P Sync с name начат.",
zh: "P2P Sync with ${name} have been started.",
},
"Pack retrieval": {
def: "Pack retrieval",
},
"paneMaintenance.markDeviceResolvedAfterBackup": {
def: "paneMaintenance.markDeviceResolvedAfterBackup",
es: "Marcar el dispositivo como resuelto después de hacer una copia de seguridad",
@@ -11069,6 +11095,9 @@ export const allMessages: Readonly<Record<string, Readonly<Record<string, string
def: "Use Random Number",
es: "Usar número aleatorio",
},
"Use S3 Range requests": {
def: "Use S3 Range requests",
},
"Use Segmented-splitter": {
def: "Use Segmented-splitter",
es: "Usar divisor segmentado",
+9
View File
@@ -44,6 +44,8 @@
"Action": "Action",
"Activate": "Activate",
"Active Remote Configuration": "Active Remote Configuration",
"Adaptive Journal (experimental)": "Adaptive Journal (experimental)",
"Adaptive Journal uses immutable objects and a separate remote format. Existing Opaque Journal data is not migrated or read. Rebuild the remote when changing formats.": "Adaptive Journal uses immutable objects and a separate remote format. Existing Opaque Journal data is not migrated or read. Rebuild the remote when changing formats.",
"Add default patterns": "Add default patterns",
"Add new connection": "Add new connection",
"AddOn Module (ConfigSync) has not been loaded. This is very unexpected situation. Please report this issue.": "AddOn Module (ConfigSync) has not been loaded. This is very unexpected situation. Please report this issue.",
@@ -112,6 +114,7 @@
"Compatibility (Metadata)": "Compatibility (Metadata)",
"Compatibility (Remote Database)": "Compatibility (Remote Database)",
"Compatibility (Trouble addressed)": "Compatibility (Trouble addressed)",
"Complete Pack reads favour throughput. Range reads can reduce transferred bytes. The connection test verifies exact Range support on this endpoint, and synchronisation refuses an unsupported selection before writing.": "Complete Pack reads favour throughput. Range reads can reduce transferred bytes. The connection test verifies exact Range support on this endpoint, and synchronisation refuses an unsupported selection before writing.",
"Compute revisions for chunks": "Compute revisions for chunks",
"Configuration": "Configuration",
"Configuration Encryption": "Configuration Encryption",
@@ -212,6 +215,7 @@
"Doctor.Message.SomeSkipped": "We left some issues as is. Shall I ask you again on next startup?",
"Doctor.RULES.E2EE_V02500.REASON": "The End-to-End Encryption has got now more robust and faster. Also because, the previous E2EE was found to be compromised in a re-conducted code review. It should be applied as soon as possible. Really apologises for your inconvenience. And, this setting is not forward compatible. All synchronised devices must be updated to v0.25.0 or higher. Rebuilds are not required and will be converted from the new transfer to the new format, However, it is recommended to rebuild whenever possible.",
"Document History": "Document History",
"Download complete Packs": "Download complete Packs",
"Duplicate": "Duplicate",
"Duplicate remote": "Duplicate remote",
"E2EE Configuration": "E2EE Configuration",
@@ -355,9 +359,11 @@
"Initialise journal received history. On the next sync, every item except this device sent will be downloaded again.": "Initialise journal received history. On the next sync, every item except this device sent will be downloaded again.",
"Initialise journal sent history. On the next sync, every item except this device received will be sent again.": "Initialise journal sent history. On the next sync, every item except this device received will be sent again.",
"Interval (sec)": "Interval (sec)",
"Invalid Object Storage settings: ${reason}": "Invalid Object Storage settings: ${reason}",
"INVERTED": "INVERTED",
"Issue detection log:": "Issue detection log:",
"It is strongly advised to create a backup before proceeding. Continuing without a backup may lead to data loss.": "It is strongly advised to create a backup before proceeding. Continuing without a backup may lead to data loss.",
"Journal data format": "Journal data format",
"Just for a minute, please!": "Just for a minute, please!",
"JWT (JSON Web Token) authentication allows you to securely authenticate with the CouchDB server using tokens. Ensure that your CouchDB server is configured to accept JWTs and that the provided key and settings match the server's configuration. Incidentally, I have not verified it very thoroughly.": "JWT (JSON Web Token) authentication allows you to securely authenticate with the CouchDB server using tokens. Ensure that your CouchDB server is configured to accept JWTs and that the provided key and settings match the server's configuration. Incidentally, I have not verified it very thoroughly.",
"JWT Algorithm": "JWT Algorithm",
@@ -738,6 +744,7 @@
"On the source device, open Obsidian.": "On the source device, open Obsidian.",
"On this device, please keep this Vault open.": "On this device, please keep this Vault open.",
"On this device, switch to the camera app or use a QR code scanner to scan the displayed QR code.": "On this device, switch to the camera app or use a QR code scanner to scan the displayed QR code.",
"Opaque Journal (current format)": "Opaque Journal (current format)",
"Open": "Open",
"Open connection": "Open connection",
"Open P2P Setup...": "Open P2P Setup...",
@@ -768,6 +775,7 @@
"P2P.SyncAlreadyRunning": "P2P Sync is already running.",
"P2P.SyncCompleted": "P2P Sync completed.",
"P2P.SyncStartedWith": "P2P Sync with ${name} have been started.",
"Pack retrieval": "Pack retrieval",
"paneMaintenance.markDeviceResolvedAfterBackup": "paneMaintenance.markDeviceResolvedAfterBackup",
"paneMaintenance.remoteLockedAndDeviceNotAccepted": "paneMaintenance.remoteLockedAndDeviceNotAccepted",
"paneMaintenance.remoteLockedResolvedDevice": "paneMaintenance.remoteLockedResolvedDevice",
@@ -1418,6 +1426,7 @@
"Use JWT Authentication": "Use JWT Authentication",
"Use Path-Style Access": "Use Path-Style Access",
"Use Random Number": "Use Random Number",
"Use S3 Range requests": "Use S3 Range requests",
"Use Segmented-splitter": "Use Segmented-splitter",
"Use splitting-limit-capped chunk splitter": "Use splitting-limit-capped chunk splitter",
"Use the trash bin": "Use the trash bin",
+15
View File
@@ -66,6 +66,21 @@ AddOn Module (HiddenFileSync) has not been loaded. This is very unexpected situa
situation. Please report this issue.
Advanced: Advanced
Advanced Settings: Advanced Settings
Adaptive Journal (experimental): Adaptive Journal (experimental)
Adaptive Journal uses immutable objects and a separate remote format. Existing Opaque Journal data is not migrated or read. Rebuild the remote when changing formats.:
Adaptive Journal uses immutable objects and a separate remote format.
Existing Opaque Journal data is not migrated or read. Rebuild the remote when
changing formats.
Complete Pack reads favour throughput. Range reads can reduce transferred bytes. The connection test verifies exact Range support on this endpoint, and synchronisation refuses an unsupported selection before writing.:
Complete Pack reads favour throughput. Range reads can reduce transferred
bytes. The connection test verifies exact Range support on this endpoint, and
synchronisation refuses an unsupported selection before writing.
Download complete Packs: Download complete Packs
"Invalid Object Storage settings: ${reason}": "Invalid Object Storage settings: ${reason}"
Journal data format: Journal data format
Opaque Journal (current format): Opaque Journal (current format)
Pack retrieval: Pack retrieval
Use S3 Range requests: Use S3 Range requests
After restarting, the data on this device will be uploaded to the server as the 'master copy'. Please be aware that any unintended data currently on the server will be completely overwritten.:
After restarting, the data on this device will be uploaded to the server as
the 'master copy'. Please be aware that any unintended data currently on the
@@ -28,6 +28,9 @@ describe("syncActivatedRemoteSettings", () => {
useCustomRequestHandler: false,
forcePathStyle: true,
bucketCustomHeaders: "",
expectedRepositoryId: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
journalFormat: "adaptive-v1" as const,
packReadPolicy: "range" as const,
};
syncActivatedRemoteSettings(target, source);
@@ -40,6 +43,9 @@ describe("syncActivatedRemoteSettings", () => {
expect(target.bucket).toBe("vault");
expect(target.region).toBe("sz-hq");
expect(target.bucketPrefix).toBe("folder/");
expect(target.expectedRepositoryId).toBe("AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA");
expect(target.journalFormat).toBe("adaptive-v1");
expect(target.packReadPolicy).toBe("range");
expect(target.encrypt).toBe(true);
});
@@ -407,6 +407,9 @@ describe("SetupManager", () => {
useCustomRequestHandler: false,
bucketCustomHeaders: "",
forcePathStyle: true,
expectedRepositoryId: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
journalFormat: "adaptive-v1",
packReadPolicy: "range",
})
.mockResolvedValueOnce(true);
@@ -419,6 +422,9 @@ describe("SetupManager", () => {
const activeProfile = current.remoteConfigurations[current.activeConfigurationId];
expect(activeProfile?.name).toBe("S3 notes");
expect(activeProfile?.uri).toContain("sls+s3://key:secret@storage.example");
expect(activeProfile?.uri).toContain("journalFormat=adaptive-v1");
expect(activeProfile?.uri).toContain("packReadPolicy=range");
expect(activeProfile?.uri).toContain("expectedRepositoryId=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA");
});
it("creates and selects a P2P profile during fresh manual onboarding", async () => {
@@ -20,10 +20,16 @@
import { copyTo, pickBucketSyncSettings } from "@vrtmrz/livesync-commonlib/compat/common/utils";
import { TYPE_CANCELLED, type SetupRemoteBucketResultType } from "./setupDialogTypes";
import { $msg as translateMessage } from "@/common/translation";
import { normaliseS3JournalSettings } from "./s3JournalSettings";
const default_setting = pickBucketSyncSettings(DEFAULT_SETTINGS);
let syncSetting = $state<BucketSyncSetting>({ ...default_setting });
let syncSetting = $state<BucketSyncSetting>({
...default_setting,
expectedRepositoryId: default_setting.expectedRepositoryId ?? "",
journalFormat: default_setting.journalFormat ?? "opaque-v1",
packReadPolicy: default_setting.packReadPolicy ?? "whole-pack",
});
type Props = GuestDialogProps<SetupRemoteBucketResultType, BucketSyncSetting>;
@@ -58,11 +64,10 @@
isEndpointSupplied
);
});
const isAdaptive = $derived(syncSetting.journalFormat === "adaptive-v1");
function generateSetting() {
const connSetting: BucketSyncSetting = {
...syncSetting,
};
const connSetting = normaliseS3JournalSettings(syncSetting);
const trialSettings: BucketSyncSetting = {
...connSetting,
};
@@ -115,8 +120,13 @@
}
}
function commit() {
const setting = pickBucketSyncSettings(generateSetting());
setResult(setting);
error = "";
try {
const setting = pickBucketSyncSettings(generateSetting());
setResult(setting);
} catch (e) {
error = translateMessage("Invalid Object Storage settings: ${reason}", { reason: `${e}` });
}
}
function cancel() {
setResult(TYPE_CANCELLED);
@@ -220,6 +230,30 @@
</InfoNote>
<ExtraItems title={translateMessage("Advanced Settings")}>
<InputRow label={translateMessage("Journal data format")}>
<select name="s3-journal-format" bind:value={syncSetting.journalFormat}>
<option value="opaque-v1">{translateMessage("Opaque Journal (current format)")}</option>
<option value="adaptive-v1">{translateMessage("Adaptive Journal (experimental)")}</option>
</select>
</InputRow>
<InfoNote warning visible={isAdaptive}>
{translateMessage(
"Adaptive Journal uses immutable objects and a separate remote format. Existing Opaque Journal data is not migrated or read. Rebuild the remote when changing formats."
)}
</InfoNote>
{#if isAdaptive}
<InputRow label={translateMessage("Pack retrieval")}>
<select name="s3-pack-read-policy" bind:value={syncSetting.packReadPolicy}>
<option value="whole-pack">{translateMessage("Download complete Packs")}</option>
<option value="range">{translateMessage("Use S3 Range requests")}</option>
</select>
</InputRow>
<InfoNote>
{translateMessage(
"Complete Pack reads favour throughput. Range reads can reduce transferred bytes. The connection test verifies exact Range support on this endpoint, and synchronisation refuses an unsupported selection before writing."
)}
</InfoNote>
{/if}
<InputRow label={translateMessage("Custom Headers")}>
<textarea
name="bucket-custom-headers"
@@ -0,0 +1,25 @@
import { DEFAULT_SETTINGS, REMOTE_MINIO, type BucketSyncSetting } from "@vrtmrz/livesync-commonlib/compat/common/types";
import { journalProtocolConfigurationForSettings } from "@vrtmrz/livesync-commonlib/journal-storage";
export function normaliseS3JournalSettings(settings: BucketSyncSetting): BucketSyncSetting {
const journalFormat = settings.journalFormat ?? "opaque-v1";
const candidate: BucketSyncSetting = {
...settings,
bucket: settings.bucket.trim(),
bucketPrefix: settings.bucketPrefix.trim(),
endpoint: settings.endpoint.trim(),
expectedRepositoryId: journalFormat === "adaptive-v1" ? (settings.expectedRepositoryId ?? "").trim() : "",
journalFormat,
packReadPolicy: journalFormat === "adaptive-v1" ? (settings.packReadPolicy ?? "whole-pack") : "whole-pack",
region: settings.region.trim(),
};
const protocol = journalProtocolConfigurationForSettings({
...DEFAULT_SETTINGS,
...candidate,
remoteType: REMOTE_MINIO,
});
return {
...candidate,
...protocol,
};
}
@@ -0,0 +1,79 @@
import { describe, expect, it } from "vitest";
import { DEFAULT_SETTINGS, type BucketSyncSetting } from "@vrtmrz/livesync-commonlib/compat/common/types";
import { pickBucketSyncSettings } from "@vrtmrz/livesync-commonlib/compat/common/utils";
import { normaliseS3JournalSettings } from "./s3JournalSettings";
function settings(overrides: Partial<BucketSyncSetting> = {}): BucketSyncSetting {
return {
...pickBucketSyncSettings(DEFAULT_SETTINGS),
...overrides,
};
}
describe("normaliseS3JournalSettings", () => {
it("retains and validates Adaptive repository options", () => {
const repositoryId = "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA";
const result = normaliseS3JournalSettings(
settings({
bucket: " vault ",
bucketPrefix: " journals/ ",
endpoint: " https://storage.example ",
expectedRepositoryId: ` ${repositoryId} `,
journalFormat: "adaptive-v1",
packReadPolicy: "range",
region: " auto ",
})
);
expect(result).toMatchObject({
bucket: "vault",
bucketPrefix: "journals/",
endpoint: "https://storage.example",
expectedRepositoryId: repositoryId,
journalFormat: "adaptive-v1",
packReadPolicy: "range",
region: "auto",
});
});
it("uses conservative Opaque defaults for older profiles", () => {
const legacy = settings();
delete legacy.expectedRepositoryId;
delete legacy.journalFormat;
delete legacy.packReadPolicy;
expect(normaliseS3JournalSettings(legacy)).toMatchObject({
expectedRepositoryId: "",
journalFormat: "opaque-v1",
packReadPolicy: "whole-pack",
});
});
it("clears Adaptive-only options when Opaque Journal is selected", () => {
expect(
normaliseS3JournalSettings(
settings({
expectedRepositoryId: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
journalFormat: "opaque-v1",
packReadPolicy: "range",
})
)
).toMatchObject({
expectedRepositoryId: "",
journalFormat: "opaque-v1",
packReadPolicy: "whole-pack",
});
});
it("rejects a non-canonical expected repository ID", () => {
expect(() =>
normaliseS3JournalSettings(
settings({
expectedRepositoryId: "not-a-repository-id",
journalFormat: "adaptive-v1",
})
)
).toThrow("expectedRepositoryId must be a canonical base64url-encoded 32-byte value");
});
});
+3 -1
View File
@@ -114,7 +114,7 @@ The mobile pass uses Obsidian's `app.emulateMobile(true)`, a 390 by 844 CSS-pixe
`test:e2e:obsidian:p2p-pane` starts one configured CouchDB-only session with no P2P profile and separate configured P2P sessions for desktop and mobile. It proves that the command remains registered while the retired command, automatic pane, and ribbon entry without a P2P configuration are absent. For the configured P2P profiles, it verifies that the desktop ribbon is available, the current status command reaches the pane without it opening at start-up, checks its connection control and horizontal layout, and captures unobstructed desktop and mobile screenshots. The mobile session uses a fresh Vault, profile, and Obsidian process, enters `app.emulateMobile(true)` through `lifecycle.beforePluginStart`, and requires the P2P view to belong to the right drawer rather than inheriting desktop workspace state. It deliberately uses no relay or peer: replacement of the active replicator is covered by focused unit tests, the Deno and Compose CLI P2P lifecycle suite covers the headless transport, and `p2p-setup-uri-workflow` owns the visible transfer path between two real Obsidian sessions.
`test:e2e:obsidian:local-suite` builds the plug-in and, unless `LIVESYNC_CLI_COMMAND` selects an external CLI, the local LiveSync CLI. It then runs discovery, smoke, the onboarding invitation, Svelte dialogue mounting, revision repair, settings UI, the Review Harness, the P2P status pane, Vault reflection, CouchDB upload and manual setup, CLI-to-Obsidian synchronisation, Object Storage upload and Setup URI round-trip, P2P Setup URI round-trip, startup scan, provisioned CouchDB Setup URI, two-vault synchronisation, Hidden File Sync, Customisation Sync, and setting Markdown export in sequence. Start the local CouchDB, MinIO, and P2P relay fixtures before running it, or use `test:e2e:obsidian:local-suite:services` to let the wrapper stop leftover fixtures, start fresh fixtures, and stop them again after the run.
`test:e2e:obsidian:local-suite` builds the plug-in and, unless `LIVESYNC_CLI_COMMAND` selects an external CLI, the local LiveSync CLI. It then runs discovery, smoke, the onboarding invitation, Svelte dialogue mounting, revision repair, settings UI, the Review Harness, the P2P status pane, Vault reflection, CouchDB upload and manual setup, CLI-to-Obsidian synchronisation, Opaque and Adaptive Object Storage uploads, the Object Storage Setup URI round-trip, the P2P Setup URI round-trip, startup scan, the provisioned CouchDB Setup URI, two-vault synchronisation, Hidden File Sync, Customisation Sync, and setting Markdown export in sequence. Start the local CouchDB, MinIO, and P2P relay fixtures before running it, or use `test:e2e:obsidian:local-suite:services` to let the wrapper stop leftover fixtures, start fresh fixtures, and stop them again after the run.
`test:e2e:obsidian:couchdb-upload` reuses the CouchDB variables from `.test.env` or the process environment. It expects a reachable CouchDB service, creates a unique database, starts from configured plug-in data without the device-local compatibility marker, and verifies the copied-or-restored Vault explanation in the actual compatibility dialogue. It captures the summary and details, resumes explicitly, confirms that the marker was recorded, creates a note in real Obsidian, commits the note into the local database, runs one-shot synchronisation, and verifies that the remote database contains both the metadata document and its chunk documents.
@@ -147,6 +147,8 @@ LIVESYNC_CLI_COMMAND="docker run --rm --network host --user $(id -u):$(id -g) --
`test:e2e:obsidian:minio-upload` reuses the Object Storage variables from `.test.env` or the process environment. It expects a reachable S3-compatible service and starts with isolated Object Storage settings and the device-local compatibility acknowledgement already in place, keeping the scenario focused on upload rather than unconfigured start-up or setup. It confirms those settings through `obsidian-cli eval`, creates a note in real Obsidian, runs one-shot Journal Sync, and verifies through the AWS SDK that objects were written under a unique bucket prefix. Adapter tests separately observe an in-progress SDK command, while this real-runtime workflow verifies the resulting request counters advance and rebalance.
`test:e2e:obsidian:adaptive-s3` reuses that one-device upload workflow with the experimental Adaptive Journal format and Range retrieval selected explicitly. It requires real Obsidian to retain those settings, publish a Chunk-backed note through one-shot synchronisation, and produce an authenticated manifest, a writer record, and an immutable Commit Bundle under a disposable MinIO prefix without writing the Opaque Journal milestone. Commonlib tests own the storage protocol and failure classification, while the CLI E2E owns bidirectional synchronisation, external Packs, and both Pack retrieval policies; this scenario verifies only the Obsidian composition boundary.
`test:e2e:obsidian:object-storage-setup-uri-workflow` uses the public Commonlib-backed tool to generate the initial Setup URI for a unique MinIO prefix, completes visible initialisation on the first device, and then asks that working real Obsidian device to create a new Setup URI through the registered command. A second real Obsidian device imports only the device-generated URI. The workflow verifies A-to-B and B-to-A notes, captures the documented onboarding choices, and removes the Object Storage prefix only after both sessions have stopped.
`test:e2e:obsidian:p2p-setup-uri-workflow` runs two concurrent isolated real Obsidian sessions against the local Compose Nostr relay fixture. The first device imports a generated initial Setup URI and completes its signalling test with zero peers, creates a Setup URI for the second device through the registered command, and remains online while the second device imports it. The second device must select the expected online source before Fetch can rebuild its local database. The workflow accepts each connection request visibly on the receiving device, verifies the initial A-to-B fetch, checks that the menu for the three persistent per-peer actions remains within the viewport, reconnects both P2P sessions in join order, and verifies the B-to-A return journey. Every started session remains tracked until teardown completes.
@@ -19,6 +19,9 @@ export type ConfiguredSettings = {
endpoint?: string;
bucket?: string;
bucketPrefix?: string;
expectedRepositoryId?: string;
journalFormat?: string;
packReadPolicy?: string;
};
export type CoreReadiness = {
@@ -320,6 +323,9 @@ export async function configureObjectStorage(
"endpoint:current.endpoint,",
"bucket:current.bucket,",
"bucketPrefix:current.bucketPrefix,",
"expectedRepositoryId:current.expectedRepositoryId,",
"journalFormat:current.journalFormat,",
"packReadPolicy:current.packReadPolicy,",
"});",
"})()",
].join(""),
+1
View File
@@ -30,6 +30,7 @@ const testSteps: Step[] = [
args: ["run", "test:e2e:obsidian:cli-to-obsidian-sync"],
},
{ name: "Object Storage upload", args: ["run", "test:e2e:obsidian:minio-upload"] },
{ name: "Adaptive S3 upload", args: ["run", "test:e2e:obsidian:adaptive-s3"] },
{
name: "Object Storage Setup URI workflow",
args: ["run", "test:e2e:obsidian:object-storage-setup-uri-workflow"],
+84 -16
View File
@@ -33,6 +33,7 @@ import {
listObjectStorageObjects,
loadObjectStorageConfig,
makeUniqueBucketPrefix,
readObjectStorageJson,
} from "../runner/objectStorage.ts";
import { startObsidianLiveSyncSession, type ObsidianLiveSyncSession } from "../runner/session.ts";
import { createTemporaryVault } from "../runner/vault.ts";
@@ -40,9 +41,21 @@ import { REMOTE_ACTIVITY_EXPECTED_STATE, waitForRemoteActivityState } from "../r
process.env.E2E_OBSIDIAN_CLI_TIMEOUT_MS ??= "30000";
const notePath = "E2E/minio-upload.md";
const adaptive = process.argv.includes("--adaptive");
const unsupportedArguments = process.argv.slice(2).filter((argument) => argument !== "--adaptive");
if (unsupportedArguments.length > 0) {
throw new Error(`Unsupported Object Storage upload argument: ${unsupportedArguments.join(", ")}`);
}
const journalSettings = adaptive
? {
journalFormat: "adaptive-v1",
packReadPolicy: "range",
}
: {};
const scenarioName = adaptive ? "Adaptive S3" : "Object Storage";
const notePath = adaptive ? "E2E/adaptive-s3-upload.md" : "E2E/minio-upload.md";
const noteContent = [
"# Object Storage upload from real Obsidian",
`# ${scenarioName} upload from real Obsidian`,
"",
"This note is created through Obsidian and uploaded by Self-hosted LiveSync to S3-compatible Object Storage.",
"The test is intentionally small, but it crosses the real Obsidian, Journal Sync, and AWS SDK boundary.",
@@ -78,7 +91,7 @@ async function createNoteAndWaitForLocalDb(cliBinary: string, env: NodeJS.Proces
);
}
async function waitForObjectStorageObjects(prefix: string): Promise<string[]> {
async function waitForObjectStorageObjects(prefix: string, requiredKeyPrefix?: string): Promise<string[]> {
const objectStorage = await loadObjectStorageConfig();
const timeoutMs = Number(process.env.E2E_OBSIDIAN_OBJECT_STORAGE_TIMEOUT_MS ?? 20000);
const deadline = Date.now() + timeoutMs;
@@ -86,12 +99,48 @@ async function waitForObjectStorageObjects(prefix: string): Promise<string[]> {
while (Date.now() < deadline) {
const objects = await listObjectStorageObjects(objectStorage, prefix);
keys = objects.flatMap((object) => (object.Key ? [object.Key] : []));
if (keys.length > 0) {
if (keys.length > 0 && (!requiredKeyPrefix || keys.some((key) => key.startsWith(requiredKeyPrefix)))) {
return keys;
}
await new Promise((resolve) => setTimeout(resolve, 500));
}
throw new Error(`Timed out waiting for Object Storage objects under ${prefix}. Last keys: ${keys.join(", ")}`);
throw new Error(
`Timed out waiting for Object Storage objects under ${prefix}${requiredKeyPrefix ? ` with prefix ${requiredKeyPrefix}` : ""}. Last keys: ${keys.join(", ")}`
);
}
async function assertAdaptiveObjects(prefix: string, keys: string[]): Promise<void> {
const objectStorage = await loadObjectStorageConfig();
const manifestKey = `${prefix}a1~manifest.json`;
const requiredPrefixes = [`${prefix}a1~writer~`, `${prefix}a1~commit~`];
if (!keys.includes(manifestKey)) {
throw new Error(`Adaptive Journal manifest is missing. Keys: ${keys.join(", ")}`);
}
for (const requiredPrefix of requiredPrefixes) {
if (!keys.some((key) => key.startsWith(requiredPrefix))) {
throw new Error(`Adaptive Journal object prefix ${requiredPrefix} is missing. Keys: ${keys.join(", ")}`);
}
}
if (keys.includes(`${prefix}_00000000-milestone.json`)) {
throw new Error("Adaptive Journal wrote the legacy Opaque Journal milestone.");
}
const manifest = await readObjectStorageJson<{
format?: unknown;
formatVersion?: unknown;
manifestAuth?: unknown;
objectLayout?: unknown;
repositoryId?: unknown;
}>(objectStorage, manifestKey);
assertEqual(manifest.format, "adaptive-journal", "Unexpected Adaptive Journal manifest format.");
assertEqual(manifest.formatVersion, 1, "Unexpected Adaptive Journal manifest version.");
assertEqual(manifest.objectLayout, "commit-bundle-v1", "Unexpected Adaptive Journal object layout.");
if (typeof manifest.repositoryId !== "string" || manifest.repositoryId.length === 0) {
throw new Error("Adaptive Journal manifest did not contain a repository ID.");
}
if (typeof manifest.manifestAuth !== "string" || manifest.manifestAuth.length === 0) {
throw new Error("Adaptive Journal manifest did not contain its authentication value.");
}
}
async function main(): Promise<void> {
@@ -102,7 +151,7 @@ async function main(): Promise<void> {
}
const objectStorage = await loadObjectStorageConfig();
const bucketPrefix = makeUniqueBucketPrefix("minio-upload");
const bucketPrefix = makeUniqueBucketPrefix(adaptive ? "adaptive-s3-upload" : "minio-upload");
const vault = await createTemporaryVault();
let session: ObsidianLiveSyncSession | undefined;
@@ -119,18 +168,26 @@ async function main(): Promise<void> {
cliBinary: cli.binary,
vault,
startupGraceMs: Number(process.env.E2E_OBSIDIAN_STARTUP_GRACE_MS ?? 1000),
pluginData: createE2eObjectStoragePluginData({
...objectStorage,
bucketPrefix,
}),
pluginData: createE2eObjectStoragePluginData(
{
...objectStorage,
bucketPrefix,
},
journalSettings
),
localStorageEntries: createE2eObsidianDeviceLocalState(vault.name),
});
await waitForLiveSyncCoreReady(cli.binary, session.cliEnv);
const configured = await configureObjectStorage(cli.binary, session.cliEnv, {
...objectStorage,
bucketPrefix,
});
const configured = await configureObjectStorage(
cli.binary,
session.cliEnv,
{
...objectStorage,
bucketPrefix,
},
journalSettings
);
await waitForLiveSyncCoreReady(cli.binary, session.cliEnv);
assertEqual(configured.isConfigured, true, "Self-hosted LiveSync was not marked as configured.");
assertEqual(configured.remoteType, "MINIO", "Remote type was not Object Storage.");
@@ -138,6 +195,11 @@ async function main(): Promise<void> {
assertEqual(configured.bucket, objectStorage.bucket, "Configured Object Storage bucket did not match.");
assertEqual(configured.bucketPrefix, bucketPrefix, "Configured Object Storage bucket prefix did not match.");
assertEqual(configured.liveSync, false, "LiveSync should remain disabled during this one-shot workflow.");
if (adaptive) {
assertEqual(configured.journalFormat, "adaptive-v1", "Adaptive Journal format was not retained.");
assertEqual(configured.packReadPolicy, "range", "Adaptive Journal Pack retrieval was not retained.");
assertEqual(configured.expectedRepositoryId, "", "A new Adaptive repository should not be pre-bound.");
}
await prepareRemote(cli.binary, session.cliEnv);
const activityBeforeUpload = await waitForRemoteActivityState(
@@ -159,10 +221,16 @@ async function main(): Promise<void> {
"Object Storage remote-request counters did not rebalance after synchronisation."
);
const keys = await waitForObjectStorageObjects(bucketPrefix);
const keys = await waitForObjectStorageObjects(
bucketPrefix,
adaptive ? `${bucketPrefix}a1~commit~` : undefined
);
if (adaptive) {
await assertAdaptiveObjects(bucketPrefix, keys);
}
console.log(
`Uploaded ${localEntry.path} through Journal Sync to ${objectStorage.bucket}/${bucketPrefix} (${keys.length} object(s)); tracked requests: ${activityAfterUpload.requestCount - activityBeforeUpload.requestCount}`
`Uploaded ${localEntry.path} through ${scenarioName} Journal Sync to ${objectStorage.bucket}/${bucketPrefix} (${keys.length} object(s)); tracked requests: ${activityAfterUpload.requestCount - activityBeforeUpload.requestCount}`
);
} finally {
if (session) {
+1
View File
@@ -18,6 +18,7 @@ const focusedScenarios = new Set([
"couchdb-manual-setup-workflow",
"cli-to-obsidian-sync",
"minio-upload",
"adaptive-s3",
"object-storage-setup-uri-workflow",
"p2p-setup-uri-workflow",
"startup-scan",
+6
View File
@@ -12,6 +12,12 @@ Earlier releases remain available in the 0.25 release history and the legacy rel
## Unreleased
### Synchronisation and storage
#### Improved
- Object Storage setup can select the experimental Adaptive Journal format and choose complete Pack or verified Range retrieval. Existing Opaque Journal repositories remain the default and require an explicit remote Rebuild before changing formats.
### P2P and experimental browser applications
#### Improved