diff --git a/README.md b/README.md index bf7bb72a..696d4e34 100644 --- a/README.md +++ b/README.md @@ -38,24 +38,30 @@ Self-hosting lets us choose a server we trust and manage it ourselves. Please se ### 1. Choose a synchronisation backend -Each Vault selects **one** remote synchronisation method: **CouchDB**, **Object Storage**, or **WebRTC Peer-to-Peer (P2P)**. They are distinct alternatives rather than layered dependencies — for example, if you choose Object Storage, you do not need a CouchDB server. +Choose **CouchDB** or **Object Storage** for central storage, or use **WebRTC Peer-to-Peer (P2P)** without a central storage server. CouchDB and Object Storage are alternatives: choosing Object Storage does not require a CouchDB server. P2P can also run alongside either central backend; its connection is selected independently. > [!TIP] > **You can switch backends at any time**: Self-hosted LiveSync supports multiple connection profiles via **Saved connections** in settings. You can begin with a hosted option such as Fly.io or Cloudflare R2, and switch to your own server later without rebuilding your Vault files. #### Backend comparison and decision guide -| Backend / Environment | LiveSync Mode
(Real-time continuous) | Sync while others offline | Mobile support
(iOS and Android) | Maintenance effort | Estimated monthly cost | Recommended for | +| Backend / Environment | LiveSync Mode
(Real-time continuous) | Sync while others offline | Mobile support
(iOS and Android) | Maintenance effort | Monthly cost example | Recommended for | | :--- | :---: | :---: | :---: | :---: | :---: | :--- | -| **CouchDB (Self-hosted)**
*(Own VPS, NAS, Docker, or Raspberry Pi)* | ✅ Full | ✅ Yes | ⚠️ Requires HTTPS or VPN | High (manage OS, Docker, and backups) | Free (on existing hardware) | Users with home servers or private infrastructure | -| **CouchDB (Fly.io)**
*(Cloud container PaaS)* | ✅ Full | ✅ Yes | ✅ Out of the box (HTTPS included) | Low (scripted setup via notebook) | ~USD 2.00–3.00 / mo
*(see [cost details](docs/setup_flyio.md#costs-before-deployment))* | Users wanting real-time sync without managing hardware | -| **Object Storage (S3 / R2)**
*(Cloudflare R2, AWS S3, or MinIO)* | ❌ Periodic or on-save only | ✅ Yes | ✅ Out of the box (HTTPS included) | Minimal (bucket configuration only) | Often USD 0.00 on R2
*(see [cost guide](docs/setup_object_storage_costs.md))* | Users wanting zero server maintenance at near-zero cost | +| **CouchDB (Self-hosted)**
*(Own VPS, NAS, Docker, or Raspberry Pi)* | ✅ Full | ✅ Yes | ⚠️ Requires HTTPS (public or private endpoint)[^private-https] | High (manage OS, Docker, and backups) | Free (on existing hardware) | Users with home servers or private infrastructure | +| **CouchDB (Fly.io)**
*(Cloud container PaaS)* | ✅ Full | ✅ Yes | ✅ Out of the box (HTTPS included) | Low (scripted setup via notebook) | ~USD 2.34 / mo in Ashburn[^fly-pricing]
*(see [cost details](docs/setup_flyio.md#costs-before-deployment))* | Users wanting real-time sync without managing hardware | +| **Object Storage (S3 / R2)**
*(Cloudflare R2, AWS S3, or MinIO)* | ❌ Periodic or on-save only | ✅ Yes | Managed services: HTTPS included
Self-hosted: configure HTTPS | Managed services: bucket configuration
Self-hosted: manage the server and backups | USD 0.00 within R2 free allowances[^r2-pricing]
*(see [cost guide](docs/setup_object_storage_costs.md))* | Users who prefer Object Storage without server maintenance | | **WebRTC Peer-to-Peer**
*(Direct device-to-device)* | ✅ Full *(while connected)* | ❌ Both devices must be online | ⚠️ Limited by OS background tasks | None (free public signalling relay) | Completely Free | Users who prefer no central storage server | > [!NOTE] -> **Why Object Storage does not incur high API costs in LiveSync**: +> **How Object Storage request counts affect costs**: > Unlike traditional file-synchronisation tools that scan every single file on each sync (generating thousands of `LIST` or `HEAD` operations), LiveSync uses **Journal Sync**. It packages only recent changes into compressed delta journals. -> **The number of API requests does not scale with your Vault's file count**. Whether your Vault contains 100 or 50,000 files, an idle check requires only a single listing operation. While initial upload and connecting a new device require a one-off transfer of all existing file Chunks, routine incremental synchronisation consumes very few requests. On services such as Cloudflare R2, typical individual usage comfortably remains within the permanent free allowance (1,000,000 Class A operations per month). See [Object Storage transfers, request scaling, and service costs](docs/setup_object_storage_costs.md) for detailed calculations and initial transfer estimates. +> **Routine checks do not inspect each Vault file**. Whether your Vault contains 100 or 50,000 files, the journal listing step of an idle check uses one LIST request. Synchronisation also checks the bucket and reads or updates synchronisation parameters and device records. Metadata and Chunks travel together in journal packages, so initial and incremental transfer counts depend on those packages. See [Object Storage transfers, request scaling, and service costs](docs/setup_object_storage_costs.md) for a calculation example covering journal listing and transfers, with control requests identified separately.[^r2-pricing] + +The figures above are calculation examples. Prices and free allowances may change. Check the linked official pricing pages and recalculate costs for your chosen configuration before deploying. + +[^fly-pricing]: November 2026 ([Fly.io resource pricing](https://docs.fly.io/about/pricing)); one `shared-cpu-1x` Machine with 256 MB RAM and a 1 GB persistent volume in Ashburn (`iad`), running continuously for 30 days. Outbound data transfer and optional resources are additional. +[^r2-pricing]: November 2026 ([Cloudflare R2 pricing](https://developers.cloudflare.com/r2/pricing/)); Standard storage. Free allowances apply to the account's combined usage. +[^private-https]: A private HTTPS endpoint accessible only to your devices can be provided by services such as [Tailscale Serve](https://tailscale.com/docs/features/tailscale-serve). See the [Docker stack](docker/README.md#profile-details) for setup options. @@ -65,7 +71,7 @@ Follow the setup guide for your selected method: - **CouchDB**: - **Self-hosted**: deploy using the [Docker stack](docker/README.md) or follow [Your own server](docs/setup_own_server.md). - - **Fly.io**: follow [Setup CouchDB on Fly.io](docs/setup_flyio.md) for a scripted cloud deployment (~USD 2–3/month). + - **Fly.io**: follow [Setup CouchDB on Fly.io](docs/setup_flyio.md) for a scripted cloud deployment. - **IBM Cloudant**: follow [IBM Cloudant setup](docs/setup_cloudant.md) for managed CouchDB hosting. - **Object Storage**: - Follow [Object Storage Setup](docs/setup_object_storage.md) using Cloudflare R2, Amazon S3, or a self-hosted MinIO bucket. @@ -85,10 +91,10 @@ If you would like to test WebRTC connectivity between your devices before settin Regardless of your chosen backend, connecting your devices follows four straightforward steps: 1. **Prepare your backend** using the guide selected in Step 2 (not required for P2P with the public relay). -2. **Set up your first device**: install Self-hosted LiveSync, enter your connection settings and End-to-End Encryption passphrase, and initialise the remote database from your Vault: - - For CouchDB: follow [Quick Setup](docs/quick_setup.md). - - For Object Storage: follow [Object Storage Setup](docs/setup_object_storage.md). - - For Peer-to-Peer: follow [Peer-to-Peer Setup](docs/setup_p2p.md). +2. **Set up your first device**: install Self-hosted LiveSync and configure your connection and encryption using the appropriate guide: + - For CouchDB: follow [Quick Setup](docs/quick_setup.md) to initialise the local and remote databases from your Vault. + - For Object Storage: follow [Object Storage Setup](docs/setup_object_storage.md) to initialise the local database and upload its journals to your bucket. + - For Peer-to-Peer: follow [Peer-to-Peer Setup](docs/setup_p2p.md) to initialise the local LiveSync database. P2P has no central remote database. 3. **Connect additional devices using a Setup URI**: on your working first device, run `Copy settings as a new Setup URI` from the command palette. Open that encrypted URI on your second device so it receives the identical encryption and connection profiles. 4. **Verify two-way synchronisation**: create a test note on your first device and confirm that it appears on the second. Then edit the note on the second device and confirm that the change reflects back on the first device. Configure optional features (such as customisation or hidden-file sync) only after this verification succeeds. diff --git a/docs/setup_flyio.md b/docs/setup_flyio.md index 89466096..5ba45b67 100644 --- a/docs/setup_flyio.md +++ b/docs/setup_flyio.md @@ -11,15 +11,17 @@ This guide prepares hosted CouchDB, then leads to [first-device setup](quick_set Fly.io provides cloud infrastructure on which you run your own CouchDB container. A valid credit or debit card is required upon registration. A time-limited trial or sign-up credit does not provide ongoing free hosting. -As a general cost illustration, based on [Fly.io pricing](https://fly.io/pricing/) (reviewed late 2026), a minimal configuration typically costs around **USD 2.00–3.00 per month**: +For an example calculation, assume one Machine and a 1 GB persistent volume in Ashburn (`iad`), running continuously for 30 days. Applying the rates below gives a subtotal of approximately **USD 2.34**, with outbound data transfer and optional resources charged separately:[^fly-pricing] -| Resource | Basis of calculation | Approximate monthly cost | +| Resource | Basis of calculation | Monthly cost in this example | | --- | --- | --- | -| One `shared-cpu-1x` Machine (256 MB RAM) | Running continuously (~730 hours @ ~USD 0.0030/hr) | ~USD 2.19 | +| One `shared-cpu-1x` Machine (256 MB RAM) | Running continuously for 30 days (720 hours) | ~USD 2.19 | | One 1 GB persistent volume | Provisioned storage (~USD 0.15/GB-month) | ~USD 0.15 | -| **Estimated subtotal** | Compute and persistent storage | **~USD 2.34 / month** | +| **Calculated subtotal** | Compute and persistent storage | **~USD 2.34 / month** | -*Note: Rates vary depending on the chosen Fly.io region, outbound network bandwidth, and optional dedicated IP addresses. The template configuration enables automatic machine suspension (`auto_stop_machines = true`), which may reduce running hours and compute costs when synchronisation is idle, though provisioned storage volumes continue to be billed.* +*Note: Rates vary depending on the chosen Fly.io region, outbound network bandwidth, and optional dedicated IP addresses. The template configuration enables automatic machine suspension (`auto_stop_machines = true`), which may reduce running hours and compute costs when synchronisation is idle, though provisioned storage volumes continue to be billed. Check the current official pricing and recalculate costs for your chosen configuration before deploying.* + +[^fly-pricing]: November 2026 ([Fly.io resource pricing](https://docs.fly.io/about/pricing)); Ashburn (`iad`), with 30 days of continuous Machine operation. > [!WARNING] > Before replacing or deleting an existing CouchDB instance, back up every available Vault and the remote database, then [choose the authoritative copy](recovery.md#first-choose-the-authoritative-copy). Stop or suspend synchronisation on other devices while reviewing recovery options. Local files alone do not establish that every needed edit or revision has been preserved. diff --git a/docs/setup_object_storage_costs.md b/docs/setup_object_storage_costs.md index 8c4e1ce9..4c03e1d9 100644 --- a/docs/setup_object_storage_costs.md +++ b/docs/setup_object_storage_costs.md @@ -1,6 +1,6 @@ # Object Storage transfers, request scaling, and service costs -This guide explains how Self-hosted LiveSync transfers data when using S3-compatible Object Storage, why API requests do not scale with the total number of files in your Vault, and how to estimate monthly costs before selecting a storage provider. +This guide explains how Self-hosted LiveSync transfers data when using S3-compatible Object Storage, why API requests do not scale with the total number of files in your Vault, and how to calculate example costs from request counts before selecting a storage provider. For setting up the plug-in with an existing bucket, see [Object Storage Setup](setup_object_storage.md). @@ -17,10 +17,11 @@ Traditional file-synchronisation tools often inspect every file in the remote st Self-hosted LiveSync works fundamentally differently: - **Incremental journal tracking**: Each device tracks its synchronisation progress using sequence markers. -- **Single listing check**: To check for incoming changes, a device performs a single `ListObjectsV2` operation to inspect new journal packages created since its last recorded sequence. -- **Delta transfer**: Only newly created journal packages and newly generated Chunks are uploaded (`PutObject`) or downloaded (`GetObject`). Unmodified files require zero API requests. +- **Journal listing**: The journal listing step uses `ListObjectsV2` to inspect journal packages after the last received journal. +- **Journal transfer**: Metadata and Chunks are packed together in journals. Each new journal package is uploaded with `PutObject`, and each missing package is downloaded with `GetObject`; individual Chunks are not separate remote objects. +- **Control requests**: Synchronisation also checks bucket availability (`HeadBucket`), reads synchronisation parameters and device participation data (`GetObject`), and can update device participation data (`PutObject`), even when no Vault files have changed. -Consequently, **whether your Vault contains 100 notes or 50,000 notes, an idle synchronisation check uses exactly one listing request**. The number of API requests depends on your synchronisation schedule and edit activity, not on your Vault's overall file count. +Consequently, **whether your Vault contains 100 notes or 50,000 notes, the journal listing step of an idle check uses one LIST request**. The full request count also includes control requests, journal transfers, and any retries. Routine checks do not need a separate request for each unchanged Vault file. --- @@ -29,10 +30,10 @@ Consequently, **whether your Vault contains 100 notes or 50,000 notes, an idle s | Activity | Primary S3 API operation | Provider pricing category | What affects the request count | | --- | --- | --- | --- | | **Check for changes** | `ListObjectsV2` | Class A / Write-equivalent | Frequency of periodic sync checks and event triggers across active devices. | -| **Send changes** | `PutObject` | Class A / Write | Number of journal packs and new Chunks produced by local edits. | -| **Receive changes** | `GetObject` | Class B / Read | Number of missing journal packs and Chunks downloaded from other devices. | -| **Device & settings sync** | `GetObject`, `PutObject` | Class A & Class B | Reading or updating remote configuration and device participation records. | -| **First-device initialisation** | `ListObjectsV2`, `PutObject` | Class A / Write | Uploading the initial Vault data and setting files. | +| **Send changes** | `PutObject` | Class A / Write | Number of journal packages uploaded, each containing Metadata and/or Chunks. | +| **Receive changes** | `GetObject` | Class B / Read | Number of journal packages downloaded from other devices. | +| **Connection, compatibility, and device checks** | `HeadBucket`, `GetObject`, `PutObject` | Provider-specific read/write categories | Bucket availability checks and reads or updates of synchronisation parameters and device participation records, including during idle cycles. | +| **First-device initialisation** | `ListObjectsV2`, `PutObject` | Class A / Write | Listing and uploading journal packages containing the initial Metadata and Chunks, in addition to control requests. | > [!NOTE] > In most S3-compatible pricing models (including Amazon S3 and Cloudflare R2), `ListObjectsV2` is classified as a write-equivalent (Class A) operation, even though it reads metadata. Keeping periodic synchronisation intervals reasonable is the most effective way to manage request volume. @@ -50,58 +51,75 @@ To estimate your monthly request volume, consider three factors: $$\text{Monthly listing requests} = \text{Devices} \times \text{Hours active/day} \times \left(\frac{60}{\text{Interval in minutes}}\right) \times 30\text{ days}$$ -### Realistic scenario +### Worked example + +This example counts journal listing, uploads, and downloads. Bucket checks, synchronisation parameters, device participation records, and retries add further requests and are excluded from this subtotal. Suppose you use two devices (for example, a desktop and a laptop): + - Both devices run Obsidian for 8 hours per day. -- Periodic synchronisation is set to the default of once every **5 minutes** (12 checks per hour). -- You create or edit approximately 50 notes per day. +- Periodic synchronisation is set to once every **5 minutes** (12 checks per hour). +- For the journal transfer calculation, assume a combined 50 PUT requests and 100 GET requests per day across both devices. These are illustrative request counts; actual transfers depend on edit activity and journal packing. ```text Routine listing checks: 2 devices × 8 hours/day × 12 checks/hour × 30 days = 5,760 LIST requests/month -Upload and download transfers (estimated): -50 edits/day × 30 days × ~2–3 transfer requests = ~3,000–4,500 GET/PUT requests/month +Uploads: 50 PUT requests/day × 30 days = 1,500 PUT requests/month +Downloads: 100 GET requests/day × 30 days = 3,000 GET requests/month -Total monthly request volume: ~9,000–10,500 requests +Counted journal requests: 7,260 LIST/PUT and 3,000 GET requests/month ``` +At the paid Amazon S3 Standard rates for US East (N. Virginia) below:[^s3-pricing] + +```text +LIST: 5,760 ÷ 1,000 × USD 0.005 = USD 0.0288 +PUT: 1,500 ÷ 1,000 × USD 0.005 = USD 0.0075 +GET: 3,000 ÷ 1,000 × USD 0.0004 = USD 0.0012 + +Journal request subtotal in this example: USD 0.0375/month (about USD 0.04) +``` + +Add control requests, retries, storage, and data transfer when estimating the full cost. On Cloudflare R2 Standard, the counted journal requests fit the monthly free allowances if the account's other usage leaves enough allowance available.[^r2-pricing] + --- ## Initial upload and first-time device download -While day-to-day incremental synchronisation produces very few requests, initial setup and adding subsequent devices require a one-off transfer of all existing Vault data: +Initial setup transfers journal packages containing the existing Metadata and Chunks. A joining device downloads the existing packages which it has not yet processed: 1. **Initial Vault upload (first device)**: - - When the first device initialises the remote bucket, it splits all files into Metadata documents and content Chunks. - - Each Chunk and initial journal package is uploaded via `PutObject`. For a Vault containing 3,000 files, this typically generates several thousand `PutObject` operations (Class A). + - The first device splits files into Metadata documents and content Chunks, then groups those records into compressed journal packages. + - Each package is uploaded with one `PutObject` request. The number of requests depends on journal packing and the size of the records, rather than a one-to-one mapping to Vault files or Chunks. 2. **Initial Vault download (adding a second device)**: - - When another device joins and fetches the Vault for the first time, it downloads all existing Chunks and Metadata from the bucket. - - This results in a temporary burst of `GetObject` operations (Class B) and internet data egress proportional to your Vault's total data size. + - The joining device downloads journal packages with `GetObject` and applies the Metadata and Chunks they contain to its local database. + - The GET request count follows the number of downloaded packages; outbound data transfer depends on their total size. Control requests and retries add further operations. ### Financial impact of initial transfers -Even during this initial population phase, costs remain negligible under standard pricing models: +Calculate initial request charges from the number of API operations: -- **Class B operations (`GetObject`) are exceptionally inexpensive**: Amazon S3 charges approximately USD 0.0004 per 1,000 GET requests (downloading 5,000 chunks costs approximately USD 0.002). Cloudflare R2 includes 10,000,000 free Class B operations every month. -- **Egress bandwidth**: Cloudflare R2 charges zero egress fees. Amazon S3 includes 100 GB of free outbound internet data transfer per month under the AWS Free Tier. -- **One-off activity**: This transfer burst occurs only once per newly connected device. Subsequent synchronisations immediately return to the minimal incremental pattern calculated above. +- **GET requests**: At the paid Amazon S3 rate used below, 5,000 GET requests cost USD 0.002, excluding storage and data transfer.[^s3-pricing] Cloudflare R2 Standard includes 10,000,000 free Class B operations every month.[^r2-pricing] +- **Egress bandwidth**: Cloudflare R2 charges zero egress fees.[^r2-pricing] For paid Amazon S3 usage, calculate outbound data transfer separately using the current rates for your region.[^s3-pricing] +- **Later synchronisation**: Once caught up, the device checks for new journal packages and downloads those it has not yet processed. --- ## Comparing provider allowances and costs -The rates below reflect published pricing as of late 2026. Please verify current rates on the provider's pricing page before opening an account. +The table applies published rates checked in November 2026 to the journal requests counted in the worked example above. Prices and free allowances may change. Before choosing a provider or configuring synchronisation, check the linked official pricing pages and recalculate costs for your region, storage usage, request volume, and data transfer. -| Provider | Included free allowances | Paid rates beyond allowance | Practical outcome for typical use | +| Provider | Included monthly free allowances | Paid rates | Calculated journal request subtotal | | --- | --- | --- | --- | -| **[Cloudflare R2](https://developers.cloudflare.com/r2/pricing/)** (Standard) | • **1,000,000 Class A** (LIST, PUT) / mo
• **10,000,000 Class B** (GET) / mo
• **10 GB storage** / mo
• **Zero egress fees** | • Class A: USD 4.50 / million
• Class B: USD 0.36 / million
• Storage: USD 0.015 / GB-month | **USD 0.00 / month (Free)**.
The scenario above consumes less than 1% of the monthly free Class A allowance. | -| **[Amazon S3](https://aws.amazon.com/s3/pricing/)** (Standard) | • Free tier for first 12 months (2,000 PUT/LIST, 20,000 GET, 5 GB storage) | • PUT/LIST: ~USD 0.005 / 1,000
• GET: ~USD 0.0004 / 1,000
• Storage: ~USD 0.023 / GB-month
• Standard internet egress applies | **Under USD 0.10 / month** after the free tier expires. | -| **[Backblaze B2](https://www.backblaze.com/cloud-storage/pricing)** | • 10 GB free storage
• 2,500 Class A calls / day free | • Calls: USD 0.005 / 10,000
• Storage: USD 0.006 / GB-month | **Under USD 0.10 / month**. | +| **Cloudflare R2 Standard**[^r2-pricing] | • **1,000,000 Class A** (LIST, PUT)
• **10,000,000 Class B** (GET)
• **10 GB-month storage**
• **Zero egress fees** | • Class A: USD 4.50 / million
• Class B: USD 0.36 / million
• Storage: USD 0.015 / GB-month | **USD 0.00 for the counted journal requests**, assuming sufficient free allowance remains. The calculation uses 7,260 Class A and 3,000 Class B operations. | +| **Amazon S3 Standard (paid usage)**
US East (N. Virginia)[^s3-pricing] | — | • PUT/LIST: USD 0.005 / 1,000
• GET: USD 0.0004 / 1,000
• Storage: USD 0.023 / GB-month (first 50 TB)
• Internet data transfer charged separately | **About USD 0.04** for the counted journal requests. Other requests, storage, and data transfer are additional. | ### Practical recommendations -- **Cloudflare R2 is generally recommended** for individual users seeking a zero-maintenance, zero-cost cloud backend, as its generous free tier and free egress ensure typical Vaults incur no charges. +- **Cloudflare R2 may suit individual users** seeking a backend without server maintenance. The counted journal requests fit its monthly free allowances; include control requests, storage, and other account usage when comparing with the published limits.[^r2-pricing] - **Avoid excessively short polling intervals** (such as polling every 10–15 seconds) unless necessary. A 1–5 minute interval provides timely synchronisation while preserving low request counts. - **Batch saves and event triggers**: Enabling 'On Events' (sync on save or startup) allows you to lengthen the periodic interval without delaying synchronisation when active edits occur. + +[^r2-pricing]: November 2026 ([Cloudflare R2 pricing](https://developers.cloudflare.com/r2/pricing/)); Standard storage. +[^s3-pricing]: November 2026 ([Amazon S3 pricing](https://aws.amazon.com/s3/pricing/)); S3 Standard paid usage in US East (N. Virginia).