Merge pull request #1312 from vrtmrz/docs/setup-selection-guide

docs: clarify backend selection, Fly.io costs, and Object Storage request scaling
2026-11-07 00:38:19 +00:00 · 2026-11-06 12:00:39 +09:00
commit 24a1bfc125
@@ -12,7 +12,7 @@ A document's original date and reviewed-version fields describe its baseline. Re
 | Choose a CouchDB setup | [Own server](setup_own_server.md), [Docker stack](../docker/README.md), [Fly.io](setup_flyio.md), or [IBM Cloudant (managed)](setup_cloudant.md) |
 | Configure CouchDB clients | [Quick setup](quick_setup.md) |
 | Find hosting services proposed by the community | [Third-party hosting listings](../README.md#hosting-services-proposed-by-the-community) |
-| Set up S3-compatible storage | [Object Storage Setup](setup_object_storage.md) |
+| Set up S3-compatible storage | [Object Storage Setup](setup_object_storage.md) and [transfers and service costs](setup_object_storage_costs.md) |
 | Set up peer-to-peer synchronisation | [P2P communication model](p2p.md), then [Peer-to-Peer Setup](setup_p2p.md) |
 | Diagnose P2P connectivity | [Browser-assisted check](../src/apps/webpeer/README.md#p2p-connection-check), [network tips](tips/p2p-sync-tips.md), and [self-hosted TURN](../docker/coturn/README.md) |
 | Understand a setting or command | [Settings reference](settings.md) |
@@ -102,6 +102,12 @@ recovery guidance, or diagnostics intended for users.
  CouchDB or Object Storage remote, with the current device first. Records
  describe participation, not current online activity. **Forget device** removes
  an obsolete device record; it does not delete Vault files or revoke credentials.
+- **Journal / Journal Sync:** The synchronisation mechanism used with
+  S3-compatible Object Storage and non-database remotes. Rather than
+  transferring individual files directly, Self-hosted LiveSync groups database
+  changes (Metadata documents and content Chunks) into compressed, time-stamped
+  journal packages. Devices track their synchronisation sequence using markers
+  and exchange only missing packages.
 - **JWT Authentication:** An experimental CouchDB authentication option which
  uses a JSON Web Token instead of standard credentials. It requires a private
  key or secret, algorithm, expiry duration, subject, and key ID.
@@ -11,7 +11,9 @@ Before starting:
 - prepare the remote service and a Setup URI; and
 - keep the Setup URI and its passphrase separate from each other.

-This walkthrough covers the recommended provisioned CouchDB path. Follow [Set up a CouchDB server](./setup_own_server.md) to prepare the server and Setup URI.
+This walkthrough covers the CouchDB synchronisation path (using [your own server](./setup_own_server.md), the [Docker stack](../docker/README.md), or [Fly.io](./setup_flyio.md)).
+- For S3-compatible cloud storage (such as Cloudflare R2 or Amazon S3), follow [Object Storage Setup](./setup_object_storage.md).
+- For direct device-to-device synchronisation without a storage server, follow [Peer-to-Peer Setup](./setup_p2p.md).

 ## What a Setup URI contains

@@ -3,16 +3,32 @@

 This is how to configure fly.io and CouchDB on it for Self-hosted LiveSync.

-The CouchDB instance is yours to manage. You can configure it yourself or set up another instance and switch to it, following the preservation and recovery guidance below.
+The CouchDB instance is yours to manage, including its updates, credentials, and backups.
+
+This guide prepares hosted CouchDB, then leads to [first-device setup](quick_setup.md#configure-couchdb-manually-on-the-first-device), [a Setup URI from that working device](quick_setup.md#create-a-setup-uri-for-another-device), and [additional-device setup with a two-way check](quick_setup.md#add-another-device). Review the costs below before deploying an application.
+
+## Costs before deployment
+
+Fly.io provides cloud infrastructure on which you run your own CouchDB container. A valid credit or debit card is required upon registration. A time-limited trial or sign-up credit does not provide ongoing free hosting.
+
+For an example calculation, assume one Machine and a 1 GB persistent volume in Ashburn (`iad`), running continuously for 30 days. Applying the rates below gives a subtotal of approximately **USD 2.34**, with outbound data transfer and optional resources charged separately:[^fly-pricing]
+
+| Resource | Basis of calculation | Monthly cost in this example |
+| --- | --- | --- |
+| One `shared-cpu-1x` Machine (256 MB RAM) | Running continuously for 30 days (720 hours) | ~USD 2.19 |
+| One 1 GB persistent volume | Provisioned storage (~USD 0.15/GB-month) | ~USD 0.15 |
+| **Calculated subtotal** | Compute and persistent storage | **~USD 2.34 / month** |
+
+*Note: Rates vary depending on the chosen Fly.io region, outbound network bandwidth, and optional dedicated IP addresses. The template configuration enables automatic machine suspension (`auto_stop_machines = true`), which may reduce running hours and compute costs when synchronisation is idle, though provisioned storage volumes continue to be billed. Check the current official pricing and recalculate costs for your chosen configuration before deploying.*
+
+[^fly-pricing]: November 2026 ([Fly.io resource pricing](https://docs.fly.io/about/pricing)); Ashburn (`iad`), with 30 days of continuous Machine operation.

 > [!WARNING]
 > Before replacing or deleting an existing CouchDB instance, back up every available Vault and the remote database, then [choose the authoritative copy](recovery.md#first-choose-the-authoritative-copy). Stop or suspend synchronisation on other devices while reviewing recovery options. Local files alone do not establish that every needed edit or revision has been preserved.

 > [!NOTE] 
 > **What and why is the Fly.io?**  
-> At some point, we started to experience problems related to our IBM Cloudant account. At the same time, Self-hosted LiveSync started to improve its functionality, requiring CouchDB in a more natural state to use all its features.
->
-> Fly.io charges according to current account and resource usage, including Machines, persistent volumes, and network usage. Check [Fly.io pricing](https://fly.io/pricing/) and the [pricing documentation](https://fly.io/docs/about/pricing/) before deploying. A time-limited trial does not provide ongoing free hosting.
+> Fly.io hosts the Machine and persistent volume while you administer CouchDB. For other environments, see [your own server](setup_own_server.md), the [Docker stack](../docker/README.md), or [IBM Cloudant](setup_cloudant.md).

 ## Required materials

@@ -267,8 +283,16 @@ Note: Each of these should also be repeated until finished in 200.

 #### 6. Use it from Self-hosted LiveSync

-Now the CouchDB is ready to use from Self-hosted LiveSync. Use your Fly.io app's hostname in `URI`, and the unique credentials configured above in `Username` and `Password`. The `Database name` could be anything you want.
-Please refer to [Configure CouchDB manually on the first device](./quick_setup.md#configure-couchdb-manually-on-the-first-device).
+Now CouchDB is ready for Self-hosted LiveSync. Use these values during manual setup:
+
+| Prepared value | Setup dialogue field |
+| --- | --- |
+| `https://<your-app>.fly.dev` | `URL` |
+| Your configured `COUCHDB_USER` | `Username` |
+| Your configured `COUCHDB_PASSWORD` secret | `Password` |
+| Your chosen database name, such as `obsidiannotes` | `Database Name` |
+
+Please refer to [Configure CouchDB manually on the first device](./quick_setup.md#configure-couchdb-manually-on-the-first-device). After your first device synchronises a test note, generate a new Setup URI on that device and follow the additional-device steps in Quick setup. Confirm that a note created on each device reaches the other unchanged.

 ## Delete the Instance

@@ -2,18 +2,43 @@

 This guide establishes Object Storage synchronisation on a first device, generates a Setup URI for another device from that working device, and verifies synchronisation in both directions.

-Object Storage uses the S3-compatible API. Prepare the following before starting:
+Object Storage exchanges compressed [journals](glossary.md#user-facing-and-operational-terms) through an S3-compatible service or server (such as Cloudflare R2, Amazon S3, or MinIO). It supports manual, periodic, and event-triggered synchronisation; real-time LiveSync mode is unavailable. A CouchDB server is not required.
+
+Before selecting a commercial storage service or configuring polling intervals, see [Object Storage transfers, request scaling, and service costs](setup_object_storage_costs.md).
+
+## Before starting
+
+Prepare the following:

 - an HTTPS endpoint reachable by every device;
 - an access key and secret key with access to the selected bucket;
 - a bucket name and region;
- a unique bucket prefix when the bucket is shared; and
+- a unique bucket prefix when the bucket is shared or reserved for this Vault;
+- CORS configuration permitting Obsidian's web origins (`app://obsidian.md`, `capacitor://localhost`, `http://localhost`) with `ETag` exposed (or enable `Use internal API` during setup to bypass CORS restrictions); and
 - separate passphrases for Vault encryption and Setup URI encryption.

 Back up every Vault involved, and do not use Obsidian Sync, iCloud synchronisation, or another synchronisation service on the same Vault.

+### Setting fields reference
+
+Match your storage service credentials to the plug-in setup fields:
+
+| Value from your service or server | Setup dialogue field | Setup URI generator variable |
+| --- | --- | --- |
+| S3 API endpoint (not the admin web console) | `Endpoint URL` | `endpoint` |
+| Access key ID | `Access Key ID` | `access_key` |
+| Secret access key | `Secret Access Key` | `secret_key` |
+| Existing bucket name | `Bucket Name` | `bucket` |
+| Region specified by the provider (e.g., `auto` for R2) | `Region` | `region` |
+| Folder prefix reserved for this Vault, such as `my-vault/` | `Folder Prefix` | `bucket_prefix` |
+| Path-style addressing (required by some servers like MinIO) | `Use Path-Style Access` | `force_path_style` |
+| Bypass CORS via Obsidian's native request API | `Use internal API` | `use_custom_request_handler` |
+
 ## Generate the initial Setup URI

+> [!TIP]
+> If you prefer to configure settings directly in Obsidian rather than using the command-line Setup URI generator, see [Configure the first device manually](#configure-the-first-device-manually).
+
 The public generator applies the Object Storage preset and records the connection as the selected remote profile. Run it from a trusted terminal:

 ```sh
@@ -101,6 +126,19 @@ Confirm that the first device's test note appears unchanged. Create a second ord

 ![Second-device note received by the first Object Storage device](../images/object-storage-setup/guide-object-storage-setup-second-to-first.png)

+## Configure the first device manually
+
+If you prefer to configure settings directly in Obsidian rather than using the Deno Setup URI generator, use this manual procedure:
+
+1. Install and enable Self-hosted LiveSync, then open onboarding from the `Welcome to Self-hosted LiveSync` Notice.
+2. Select `I am setting this up for the first time`, then choose `Configure a remote manually` and select `Proceed with manual configuration`.
+3. Configure `End-to-End Encryption` and enter a strong Vault encryption passphrase. Save this passphrase securely.
+4. Select the S3-compatible Object Storage option and proceed to `S3/MinIO/R2 Configuration`.
+5. Enter your bucket settings using the [Setting fields reference](#setting-fields-reference) table above.
+6. Select `Test Settings and Continue`. If the connection test fails, verify the API endpoint, credentials, bucket permissions, and region. If CORS blocks the request or cannot be configured on your bucket, enable `Use internal API` to route requests through Obsidian's native HTTP handler.
+7. Proceed to `Restart and Initialise Server`, then complete the confirmation steps as described in [Set up the first device](#set-up-the-first-device).
+8. Once your test note synchronises, follow [Generate the second-device Setup URI](#generate-the-second-device-setup-uri) to connect subsequent devices easily.
+
 ## Safety notes

 - Treat the endpoint, bucket, prefix, access key, secret key, Vault passphrase, Setup URI, and Setup URI passphrase as sensitive.
@@ -0,0 +1,125 @@
+# Object Storage transfers, request scaling, and service costs
+
+This guide explains how Self-hosted LiveSync transfers data when using S3-compatible Object Storage, why API requests do not scale with the total number of files in your Vault, and how to calculate example costs from request counts before selecting a storage provider.
+
+For setting up the plug-in with an existing bucket, see [Object Storage Setup](setup_object_storage.md).
+
+---
+
+## How Journal Sync transfers data
+
+Self-hosted LiveSync stores file properties in [Metadata](glossary.md#user-facing-and-operational-terms) and file contents in Chunks. When synchronising with Object Storage, it does not manage files as individual remote objects. Instead, it packages database changes into compressed, time-stamped **journals** using the [Journal Replicator](design_docs_of_journalsync_2nd.md) architecture.
+
+### Why API requests do not scale with Vault file count
+
+Traditional file-synchronisation tools often inspect every file in the remote storage to detect changes, issuing `LIST` or `HEAD` operations proportional to the total number of files in the Vault. In that model, a Vault with 10,000 files generates thousands of requests on every synchronisation cycle, quickly exhausting free allowances and incurring substantial API fees.
+
+Self-hosted LiveSync works fundamentally differently:
+
+- **Incremental journal tracking**: Each device tracks its synchronisation progress using sequence markers.
+- **Journal listing**: The journal listing step uses `ListObjectsV2` to inspect journal packages after the last received journal.
+- **Journal transfer**: Metadata and Chunks are packed together in journals. Each new journal package is uploaded with `PutObject`, and each missing package is downloaded with `GetObject`; individual Chunks are not separate remote objects.
+- **Control requests**: Synchronisation also checks bucket availability (`HeadBucket`), reads synchronisation parameters and device participation data (`GetObject`), and can update device participation data (`PutObject`), even when no Vault files have changed.
+
+Consequently, **whether your Vault contains 100 notes or 50,000 notes, the journal listing step of an idle check uses one LIST request**. The full request count also includes control requests, journal transfers, and any retries. Routine checks do not need a separate request for each unchanged Vault file.
+
+---
+
+## Mapping activities to S3-compatible API operations
+
+| Activity | Primary S3 API operation | Provider pricing category | What affects the request count |
+| --- | --- | --- | --- |
+| **Check for changes** | `ListObjectsV2` | Class A / Write-equivalent | Frequency of periodic sync checks and event triggers across active devices. |
+| **Send changes** | `PutObject` | Class A / Write | Number of journal packages uploaded, each containing Metadata and/or Chunks. |
+| **Receive changes** | `GetObject` | Class B / Read | Number of journal packages downloaded from other devices. |
+| **Connection, compatibility, and device checks** | `HeadBucket`, `GetObject`, `PutObject` | Provider-specific read/write categories | Bucket availability checks and reads or updates of synchronisation parameters and device participation records, including during idle cycles. |
+| **First-device initialisation** | `ListObjectsV2`, `PutObject` | Class A / Write | Listing and uploading journal packages containing the initial Metadata and Chunks, in addition to control requests. |
+
+> [!NOTE]
+> In most S3-compatible pricing models (including Amazon S3 and Cloudflare R2), `ListObjectsV2` is classified as a write-equivalent (Class A) operation, even though it reads metadata. Keeping periodic synchronisation intervals reasonable is the most effective way to manage request volume.
+
+---
+
+## Estimating monthly request volume
+
+To estimate your monthly request volume, consider three factors:
+1. **Number of active devices**: Devices running Obsidian concurrently.
+2. **Synchronisation interval**: The periodic polling interval configured in settings.
+3. **Daily editing volume**: Number of modifications and new notes created per day.
+
+### Calculation formula for routine checks
+
+$$\text{Monthly listing requests} = \text{Devices} \times \text{Hours active/day} \times \left(\frac{60}{\text{Interval in minutes}}\right) \times 30\text{ days}$$
+
+### Worked example
+
+This example counts journal listing, uploads, and downloads. Bucket checks, synchronisation parameters, device participation records, and retries add further requests and are excluded from this subtotal.
+
+Suppose you use two devices (for example, a desktop and a laptop):
+
+- Both devices run Obsidian for 8 hours per day.
+- Periodic synchronisation is set to once every **5 minutes** (12 checks per hour).
+- For the journal transfer calculation, assume a combined 50 PUT requests and 100 GET requests per day across both devices. These are illustrative request counts; actual transfers depend on edit activity and journal packing.
+
+```text
+Routine listing checks:
+2 devices × 8 hours/day × 12 checks/hour × 30 days = 5,760 LIST requests/month
+
+Uploads:   50 PUT requests/day × 30 days = 1,500 PUT requests/month
+Downloads: 100 GET requests/day × 30 days = 3,000 GET requests/month
+
+Counted journal requests: 7,260 LIST/PUT and 3,000 GET requests/month
+```
+
+At the paid Amazon S3 Standard rates for US East (N. Virginia) below:[^s3-pricing]
+
+```text
+LIST: 5,760 ÷ 1,000 × USD 0.005  = USD 0.0288
+PUT:  1,500 ÷ 1,000 × USD 0.005  = USD 0.0075
+GET:  3,000 ÷ 1,000 × USD 0.0004 = USD 0.0012
+
+Journal request subtotal in this example: USD 0.0375/month (about USD 0.04)
+```
+
+Add control requests, retries, storage, and data transfer when estimating the full cost. On Cloudflare R2 Standard, the counted journal requests fit the monthly free allowances if the account's other usage leaves enough allowance available.[^r2-pricing]
+
+---
+
+## Initial upload and first-time device download
+
+Initial setup transfers journal packages containing the existing Metadata and Chunks. A joining device downloads the existing packages which it has not yet processed:
+
+1. **Initial Vault upload (first device)**:
+   - The first device splits files into Metadata documents and content Chunks, then groups those records into compressed journal packages.
+   - Each package is uploaded with one `PutObject` request. The number of requests depends on journal packing and the size of the records, rather than a one-to-one mapping to Vault files or Chunks.
+2. **Initial Vault download (adding a second device)**:
+   - The joining device downloads journal packages with `GetObject` and applies the Metadata and Chunks they contain to its local database.
+   - The GET request count follows the number of downloaded packages; outbound data transfer depends on their total size. Control requests and retries add further operations.
+
+### Financial impact of initial transfers
+
+Calculate initial request charges from the number of API operations:
+
+- **GET requests**: At the paid Amazon S3 rate used below, 5,000 GET requests cost USD 0.002, excluding storage and data transfer.[^s3-pricing] Cloudflare R2 Standard includes 10,000,000 free Class B operations every month.[^r2-pricing]
+- **Egress bandwidth**: Cloudflare R2 charges zero egress fees.[^r2-pricing] For paid Amazon S3 usage, calculate outbound data transfer separately using the current rates for your region.[^s3-pricing]
+- **Later synchronisation**: Once caught up, the device checks for new journal packages and downloads those it has not yet processed.
+
+---
+
+## Comparing provider allowances and costs
+
+The table applies published rates checked in November 2026 to the journal requests counted in the worked example above. Prices and free allowances may change. Before choosing a provider or configuring synchronisation, check the linked official pricing pages and recalculate costs for your region, storage usage, request volume, and data transfer.
+
+| Provider | Included monthly free allowances | Paid rates | Calculated journal request subtotal |
+| --- | --- | --- | --- |
+| **Cloudflare R2 Standard**[^r2-pricing] | • **1,000,000 Class A** (LIST, PUT)<br>• **10,000,000 Class B** (GET)<br>• **10 GB-month storage**<br>• **Zero egress fees** | • Class A: USD 4.50 / million<br>• Class B: USD 0.36 / million<br>• Storage: USD 0.015 / GB-month | **USD 0.00 for the counted journal requests**, assuming sufficient free allowance remains. The calculation uses 7,260 Class A and 3,000 Class B operations. |
+| **Amazon S3 Standard (paid usage)**<br>US East (N. Virginia)[^s3-pricing] | — | • PUT/LIST: USD 0.005 / 1,000<br>• GET: USD 0.0004 / 1,000<br>• Storage: USD 0.023 / GB-month (first 50 TB)<br>• Internet data transfer charged separately | **About USD 0.04** for the counted journal requests. Other requests, storage, and data transfer are additional. |
+
+### Practical recommendations
+
+- **Cloudflare R2 may suit individual users** seeking a backend without server maintenance. The counted journal requests fit its monthly free allowances; include control requests, storage, and other account usage when comparing with the published limits.[^r2-pricing]
+- **Avoid excessively short polling intervals** (such as polling every 10–15 seconds) unless necessary. A 1–5 minute interval provides timely synchronisation while preserving low request counts.
+- **Batch saves and event triggers**: Enabling 'On Events' (sync on save or startup) allows you to lengthen the periodic interval without delaying synchronisation when active edits occur.
+
+[^r2-pricing]: November 2026 ([Cloudflare R2 pricing](https://developers.cloudflare.com/r2/pricing/)); Standard storage.
+[^s3-pricing]: November 2026 ([Amazon S3 pricing](https://aws.amazon.com/s3/pricing/)); S3 Standard paid usage in US East (N. Virginia).