mirror of
https://github.com/vrtmrz/obsidian-livesync.git
synced 2026-11-05 15:58:19 +00:00
docs: clarify backend selection, Fly.io costs, and Object Storage request scaling
This commit is contained in:
vorotamoroz committed
1 parent
03263a3888
commit
09c595ffe9
6 files changed
+219
-28
No files matched your search
+1
-1
@@ -12,7 +12,7 @@ A document's original date and reviewed-version fields describe its baseline. Re
|
||||
| Choose a CouchDB setup | [Own server](setup_own_server.md), [Docker stack](../docker/README.md), [Fly.io](setup_flyio.md), or [IBM Cloudant (managed)](setup_cloudant.md) |
|
||||
| Configure CouchDB clients | [Quick setup](quick_setup.md) |
|
||||
| Find hosting services proposed by the community | [Third-party hosting listings](../README.md#hosting-services-proposed-by-the-community) |
|
||||
| Set up S3-compatible storage | [Object Storage Setup](setup_object_storage.md) |
|
||||
| Set up S3-compatible storage | [Object Storage Setup](setup_object_storage.md) and [transfers and service costs](setup_object_storage_costs.md) |
|
||||
| Set up peer-to-peer synchronisation | [P2P communication model](p2p.md), then [Peer-to-Peer Setup](setup_p2p.md) |
|
||||
| Diagnose P2P connectivity | [Browser-assisted check](../src/apps/webpeer/README.md#p2p-connection-check), [network tips](tips/p2p-sync-tips.md), and [self-hosted TURN](../docker/coturn/README.md) |
|
||||
| Understand a setting or command | [Settings reference](settings.md) |
|
||||
|
||||
+3
-1
@@ -11,7 +11,9 @@ Before starting:
|
||||
- prepare the remote service and a Setup URI; and
|
||||
- keep the Setup URI and its passphrase separate from each other.
|
||||
|
||||
This walkthrough covers the recommended provisioned CouchDB path. Follow [Set up a CouchDB server](./setup_own_server.md) to prepare the server and Setup URI.
|
||||
This walkthrough covers the CouchDB synchronisation path (using [your own server](./setup_own_server.md), the [Docker stack](../docker/README.md), or [Fly.io](./setup_flyio.md)).
|
||||
- For S3-compatible cloud storage (such as Cloudflare R2 or Amazon S3), follow [Object Storage Setup](./setup_object_storage.md).
|
||||
- For direct device-to-device synchronisation without a storage server, follow [Peer-to-Peer Setup](./setup_p2p.md).
|
||||
|
||||
## What a Setup URI contains
|
||||
|
||||
|
||||
+28
-6
@@ -3,16 +3,30 @@
|
||||
|
||||
This is how to configure fly.io and CouchDB on it for Self-hosted LiveSync.
|
||||
|
||||
The CouchDB instance is yours to manage. You can configure it yourself or set up another instance and switch to it, following the preservation and recovery guidance below.
|
||||
The CouchDB instance is yours to manage, including its updates, credentials, and backups.
|
||||
|
||||
This guide prepares hosted CouchDB, then leads to [first-device setup](quick_setup.md#configure-couchdb-manually-on-the-first-device), [a Setup URI from that working device](quick_setup.md#create-a-setup-uri-for-another-device), and [additional-device setup with a two-way check](quick_setup.md#add-another-device). Review the costs below before deploying an application.
|
||||
|
||||
## Costs before deployment
|
||||
|
||||
Fly.io provides cloud infrastructure on which you run your own CouchDB container. A valid credit or debit card is required upon registration. A time-limited trial or sign-up credit does not provide ongoing free hosting.
|
||||
|
||||
As a general cost illustration, based on [Fly.io pricing](https://fly.io/pricing/) (reviewed late 2026), a minimal configuration typically costs around **USD 2.00–3.00 per month**:
|
||||
|
||||
| Resource | Basis of calculation | Approximate monthly cost |
|
||||
| --- | --- | --- |
|
||||
| One `shared-cpu-1x` Machine (256 MB RAM) | Running continuously (~730 hours @ ~USD 0.0030/hr) | ~USD 2.19 |
|
||||
| One 1 GB persistent volume | Provisioned storage (~USD 0.15/GB-month) | ~USD 0.15 |
|
||||
| **Estimated subtotal** | Compute and persistent storage | **~USD 2.34 / month** |
|
||||
|
||||
*Note: Rates vary depending on the chosen Fly.io region, outbound network bandwidth, and optional dedicated IP addresses. The template configuration enables automatic machine suspension (`auto_stop_machines = true`), which may reduce running hours and compute costs when synchronisation is idle, though provisioned storage volumes continue to be billed.*
|
||||
|
||||
> [!WARNING]
|
||||
> Before replacing or deleting an existing CouchDB instance, back up every available Vault and the remote database, then [choose the authoritative copy](recovery.md#first-choose-the-authoritative-copy). Stop or suspend synchronisation on other devices while reviewing recovery options. Local files alone do not establish that every needed edit or revision has been preserved.
|
||||
|
||||
> [!NOTE]
|
||||
> **What and why is the Fly.io?**
|
||||
> At some point, we started to experience problems related to our IBM Cloudant account. At the same time, Self-hosted LiveSync started to improve its functionality, requiring CouchDB in a more natural state to use all its features.
|
||||
>
|
||||
> Fly.io charges according to current account and resource usage, including Machines, persistent volumes, and network usage. Check [Fly.io pricing](https://fly.io/pricing/) and the [pricing documentation](https://fly.io/docs/about/pricing/) before deploying. A time-limited trial does not provide ongoing free hosting.
|
||||
> Fly.io hosts the Machine and persistent volume while you administer CouchDB. For other environments, see [your own server](setup_own_server.md), the [Docker stack](../docker/README.md), or [IBM Cloudant](setup_cloudant.md).
|
||||
|
||||
## Required materials
|
||||
|
||||
@@ -267,8 +281,16 @@ Note: Each of these should also be repeated until finished in 200.
|
||||
|
||||
#### 6. Use it from Self-hosted LiveSync
|
||||
|
||||
Now the CouchDB is ready to use from Self-hosted LiveSync. Use your Fly.io app's hostname in `URI`, and the unique credentials configured above in `Username` and `Password`. The `Database name` could be anything you want.
|
||||
Please refer to [Configure CouchDB manually on the first device](./quick_setup.md#configure-couchdb-manually-on-the-first-device).
|
||||
Now CouchDB is ready for Self-hosted LiveSync. Use these values during manual setup:
|
||||
|
||||
| Prepared value | Setup dialogue field |
|
||||
| --- | --- |
|
||||
| `https://<your-app>.fly.dev` | `URL` |
|
||||
| Your configured `COUCHDB_USER` | `Username` |
|
||||
| Your configured `COUCHDB_PASSWORD` secret | `Password` |
|
||||
| Your chosen database name, such as `obsidiannotes` | `Database Name` |
|
||||
|
||||
Please refer to [Configure CouchDB manually on the first device](./quick_setup.md#configure-couchdb-manually-on-the-first-device). After your first device synchronises a test note, generate a new Setup URI on that device and follow the additional-device steps in Quick setup. Confirm that a note created on each device reaches the other unchanged.
|
||||
|
||||
## Delete the Instance
|
||||
|
||||
|
||||
@@ -2,16 +2,37 @@
|
||||
|
||||
This guide establishes Object Storage synchronisation on a first device, generates a Setup URI for another device from that working device, and verifies synchronisation in both directions.
|
||||
|
||||
Object Storage uses the S3-compatible API. Prepare the following before starting:
|
||||
Object Storage exchanges compressed [journals](glossary.md#user-facing-and-operational-terms) through an S3-compatible service or server (such as Cloudflare R2, Amazon S3, or MinIO). It supports manual, periodic, and event-triggered synchronisation; real-time LiveSync mode is unavailable. A CouchDB server is not required.
|
||||
|
||||
Before selecting a commercial storage service or configuring polling intervals, see [Object Storage transfers, request scaling, and service costs](setup_object_storage_costs.md).
|
||||
|
||||
## Before starting
|
||||
|
||||
Prepare the following:
|
||||
|
||||
- an HTTPS endpoint reachable by every device;
|
||||
- an access key and secret key with access to the selected bucket;
|
||||
- a bucket name and region;
|
||||
- a unique bucket prefix when the bucket is shared; and
|
||||
- a unique bucket prefix when the bucket is shared or reserved for this Vault;
|
||||
- CORS configuration permitting Obsidian's web origins (`app://obsidian.md`, `capacitor://localhost`, `http://localhost`) with `ETag` exposed; and
|
||||
- separate passphrases for Vault encryption and Setup URI encryption.
|
||||
|
||||
Back up every Vault involved, and do not use Obsidian Sync, iCloud synchronisation, or another synchronisation service on the same Vault.
|
||||
|
||||
### Setting fields reference
|
||||
|
||||
Match your storage service credentials to the plug-in setup fields:
|
||||
|
||||
| Value from your service or server | Setup dialogue field | Setup URI generator variable |
|
||||
| --- | --- | --- |
|
||||
| S3 API endpoint (not the admin web console) | `Endpoint URL` | `endpoint` |
|
||||
| Access key ID | `Access Key ID` | `access_key` |
|
||||
| Secret access key | `Secret Access Key` | `secret_key` |
|
||||
| Existing bucket name | `Bucket Name` | `bucket` |
|
||||
| Region specified by the provider (e.g., `auto` for R2) | `Region` | `region` |
|
||||
| Folder prefix reserved for this Vault, such as `my-vault/` | `Folder Prefix` | `bucket_prefix` |
|
||||
| Path-style addressing (required by some servers like MinIO) | `Use Path-Style Access` | `force_path_style` |
|
||||
|
||||
## Generate the initial Setup URI
|
||||
|
||||
The public generator applies the Object Storage preset and records the connection as the selected remote profile. Run it from a trusted terminal:
|
||||
@@ -101,6 +122,19 @@ Confirm that the first device's test note appears unchanged. Create a second ord
|
||||
|
||||
|
||||
|
||||
## Configure the first device manually
|
||||
|
||||
If you prefer to configure settings directly in Obsidian rather than using the Deno Setup URI generator, use this manual procedure:
|
||||
|
||||
1. Install and enable Self-hosted LiveSync, then open onboarding from the `Welcome to Self-hosted LiveSync` Notice.
|
||||
2. Select `I am setting this up for the first time`, then choose `Configure a remote manually` and select `Proceed with manual configuration`.
|
||||
3. Configure `End-to-End Encryption` and enter a strong Vault encryption passphrase. Save this passphrase securely.
|
||||
4. Select the S3-compatible Object Storage option and proceed to `S3/MinIO/R2 Configuration`.
|
||||
5. Enter your bucket settings using the [Setting fields reference](#setting-fields-reference) table above.
|
||||
6. Select `Test Settings and Continue`. If the connection test fails, verify the API endpoint, credentials, bucket permissions, region, and CORS configuration.
|
||||
7. Proceed to `Restart and Initialise Server`, then complete the confirmation steps as described in [Set up the first device](#set-up-the-first-device).
|
||||
8. Once your test note synchronises, follow [Generate the second-device Setup URI](#generate-the-second-device-setup-uri) to connect subsequent devices easily.
|
||||
|
||||
## Safety notes
|
||||
|
||||
- Treat the endpoint, bucket, prefix, access key, secret key, Vault passphrase, Setup URI, and Setup URI passphrase as sensitive.
|
||||
|
||||
@@ -0,0 +1,107 @@
|
||||
# Object Storage transfers, request scaling, and service costs
|
||||
|
||||
This guide explains how Self-hosted LiveSync transfers data when using S3-compatible Object Storage, why API requests do not scale with the total number of files in your Vault, and how to estimate monthly costs before selecting a storage provider.
|
||||
|
||||
For setting up the plug-in with an existing bucket, see [Object Storage Setup](setup_object_storage.md).
|
||||
|
||||
---
|
||||
|
||||
## How Journal Sync transfers data
|
||||
|
||||
Self-hosted LiveSync stores file properties in [Metadata](glossary.md#user-facing-and-operational-terms) and file contents in Chunks. When synchronising with Object Storage, it does not manage files as individual remote objects. Instead, it packages database changes into compressed, time-stamped **journals** using the [Journal Replicator](design_docs_of_journalsync_2nd.md) architecture.
|
||||
|
||||
### Why API requests do not scale with Vault file count
|
||||
|
||||
Traditional file-synchronisation tools often inspect every file in the remote storage to detect changes, issuing `LIST` or `HEAD` operations proportional to the total number of files in the Vault. In that model, a Vault with 10,000 files generates thousands of requests on every synchronisation cycle, quickly exhausting free allowances and incurring substantial API fees.
|
||||
|
||||
Self-hosted LiveSync works fundamentally differently:
|
||||
|
||||
- **Incremental journal tracking**: Each device tracks its synchronisation progress using sequence markers.
|
||||
- **Single listing check**: To check for incoming changes, a device performs a single `ListObjectsV2` operation to inspect new journal packages created since its last recorded sequence.
|
||||
- **Delta transfer**: Only newly created journal packages and newly generated Chunks are uploaded (`PutObject`) or downloaded (`GetObject`). Unmodified files require zero API requests.
|
||||
|
||||
Consequently, **whether your Vault contains 100 notes or 50,000 notes, an idle synchronisation check uses exactly one listing request**. The number of API requests depends on your synchronisation schedule and edit activity, not on your Vault's overall file count.
|
||||
|
||||
---
|
||||
|
||||
## Mapping activities to S3-compatible API operations
|
||||
|
||||
| Activity | Primary S3 API operation | Provider pricing category | What affects the request count |
|
||||
| --- | --- | --- | --- |
|
||||
| **Check for changes** | `ListObjectsV2` | Class A / Write-equivalent | Frequency of periodic sync checks and event triggers across active devices. |
|
||||
| **Send changes** | `PutObject` | Class A / Write | Number of journal packs and new Chunks produced by local edits. |
|
||||
| **Receive changes** | `GetObject` | Class B / Read | Number of missing journal packs and Chunks downloaded from other devices. |
|
||||
| **Device & settings sync** | `GetObject`, `PutObject` | Class A & Class B | Reading or updating remote configuration and device participation records. |
|
||||
| **First-device initialisation** | `ListObjectsV2`, `PutObject` | Class A / Write | Uploading the initial Vault data and setting files. |
|
||||
|
||||
> [!NOTE]
|
||||
> In most S3-compatible pricing models (including Amazon S3 and Cloudflare R2), `ListObjectsV2` is classified as a write-equivalent (Class A) operation, even though it reads metadata. Keeping periodic synchronisation intervals reasonable is the most effective way to manage request volume.
|
||||
|
||||
---
|
||||
|
||||
## Estimating monthly request volume
|
||||
|
||||
To estimate your monthly request volume, consider three factors:
|
||||
1. **Number of active devices**: Devices running Obsidian concurrently.
|
||||
2. **Synchronisation interval**: The periodic polling interval configured in settings.
|
||||
3. **Daily editing volume**: Number of modifications and new notes created per day.
|
||||
|
||||
### Calculation formula for routine checks
|
||||
|
||||
$$\text{Monthly listing requests} = \text{Devices} \times \text{Hours active/day} \times \left(\frac{60}{\text{Interval in minutes}}\right) \times 30\text{ days}$$
|
||||
|
||||
### Realistic scenario
|
||||
|
||||
Suppose you use two devices (for example, a desktop and a laptop):
|
||||
- Both devices run Obsidian for 8 hours per day.
|
||||
- Periodic synchronisation is set to the default of once every **5 minutes** (12 checks per hour).
|
||||
- You create or edit approximately 50 notes per day.
|
||||
|
||||
```text
|
||||
Routine listing checks:
|
||||
2 devices × 8 hours/day × 12 checks/hour × 30 days = 5,760 LIST requests/month
|
||||
|
||||
Upload and download transfers (estimated):
|
||||
50 edits/day × 30 days × ~2–3 transfer requests = ~3,000–4,500 GET/PUT requests/month
|
||||
|
||||
Total monthly request volume: ~9,000–10,500 requests
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Initial upload and first-time device download
|
||||
|
||||
While day-to-day incremental synchronisation produces very few requests, initial setup and adding subsequent devices require a one-off transfer of all existing Vault data:
|
||||
|
||||
1. **Initial Vault upload (first device)**:
|
||||
- When the first device initialises the remote bucket, it splits all files into Metadata documents and content Chunks.
|
||||
- Each Chunk and initial journal package is uploaded via `PutObject`. For a Vault containing 3,000 files, this typically generates several thousand `PutObject` operations (Class A).
|
||||
2. **Initial Vault download (adding a second device)**:
|
||||
- When another device joins and fetches the Vault for the first time, it downloads all existing Chunks and Metadata from the bucket.
|
||||
- This results in a temporary burst of `GetObject` operations (Class B) and internet data egress proportional to your Vault's total data size.
|
||||
|
||||
### Financial impact of initial transfers
|
||||
|
||||
Even during this initial population phase, costs remain negligible under standard pricing models:
|
||||
|
||||
- **Class B operations (`GetObject`) are exceptionally inexpensive**: Amazon S3 charges approximately USD 0.0004 per 1,000 GET requests (downloading 5,000 chunks costs approximately USD 0.002). Cloudflare R2 includes 10,000,000 free Class B operations every month.
|
||||
- **Egress bandwidth**: Cloudflare R2 charges zero egress fees. Amazon S3 includes 100 GB of free outbound internet data transfer per month under the AWS Free Tier.
|
||||
- **One-off activity**: This transfer burst occurs only once per newly connected device. Subsequent synchronisations immediately return to the minimal incremental pattern calculated above.
|
||||
|
||||
---
|
||||
|
||||
## Comparing provider allowances and costs
|
||||
|
||||
The rates below reflect published pricing as of late 2026. Please verify current rates on the provider's pricing page before opening an account.
|
||||
|
||||
| Provider | Included free allowances | Paid rates beyond allowance | Practical outcome for typical use |
|
||||
| --- | --- | --- | --- |
|
||||
| **[Cloudflare R2](https://developers.cloudflare.com/r2/pricing/)** (Standard) | • **1,000,000 Class A** (LIST, PUT) / mo<br>• **10,000,000 Class B** (GET) / mo<br>• **10 GB storage** / mo<br>• **Zero egress fees** | • Class A: USD 4.50 / million<br>• Class B: USD 0.36 / million<br>• Storage: USD 0.015 / GB-month | **USD 0.00 / month (Free)**.<br>The scenario above consumes less than 1% of the monthly free Class A allowance. |
|
||||
| **[Amazon S3](https://aws.amazon.com/s3/pricing/)** (Standard) | • Free tier for first 12 months (2,000 PUT/LIST, 20,000 GET, 5 GB storage) | • PUT/LIST: ~USD 0.005 / 1,000<br>• GET: ~USD 0.0004 / 1,000<br>• Storage: ~USD 0.023 / GB-month<br>• Standard internet egress applies | **Under USD 0.10 / month** after the free tier expires. |
|
||||
| **[Backblaze B2](https://www.backblaze.com/cloud-storage/pricing)** | • 10 GB free storage<br>• 2,500 Class A calls / day free | • Calls: USD 0.005 / 10,000<br>• Storage: USD 0.006 / GB-month | **Under USD 0.10 / month**. |
|
||||
|
||||
### Practical recommendations
|
||||
|
||||
- **Cloudflare R2 is generally recommended** for individual users seeking a zero-maintenance, zero-cost cloud backend, as its generous free tier and free egress ensure typical Vaults incur no charges.
|
||||
- **Avoid excessively short polling intervals** (such as polling every 10–15 seconds) unless necessary. A 1–5 minute interval provides timely synchronisation while preserving low request counts.
|
||||
- **Batch saves and event triggers**: Enabling 'On Events' (sync on save or startup) allows you to lengthen the periodic interval without delaying synchronisation when active edits occur.
|
||||
Reference in new issue
Block a user