Compare commits

...

11 Commits

Author SHA1 Message Date
vorotamoroz
1179438df8 bump 2025-06-20 12:43:39 +01:00
vorotamoroz
47ea8f6859 ## 0.24.29
### Fixed

- Synchronisation with buckets now works correctly, regardless of whether a prefix is set or the bucket has been (re-) initialised (#664).
- An information message is now displayed again, during any automatic synchronisation is enabled (#662).

### Tidied up

- Importing paths have been tidied up.
2025-06-20 12:43:15 +01:00
vorotamoroz
670fe16486 Merge branch 'main' of https://github.com/vrtmrz/obsidian-livesync 2025-06-16 02:53:30 +01:00
vorotamoroz
3f0093916c Add some note 2025-06-16 02:52:59 +01:00
vorotamoroz
9503474d06 bump 2025-06-15 18:49:31 +09:00
vorotamoroz
ddf7b243e4 ## 0.24.28
### Fixed

- Batch Update is no longer available in LiveSync mode to avoid unexpected behaviour. (#653)
- Now compatible with Cloudflare R2 again for bucket synchronisation.
- Prevention of broken behaviour due to database connection failures added (#649).
2025-06-15 18:49:16 +09:00
vorotamoroz
f37561c3c1 Update Library 2025-06-15 18:24:19 +09:00
vorotamoroz
f01429decc Merge pull request #633 from jmarmstrong1207/patch-1
Add simple docker compose to the setup guide
2025-06-10 11:27:22 +09:00
vorotamoroz
c0fcb66924 bump 2025-06-10 02:52:45 +01:00
vorotamoroz
5f76b9809b ## 0.24.27
### Improved

- We can use prefix for path for the Bucket synchronisation.
- The "Use Request API to avoid `inevitable` CORS problem" option is now promoted to the normal setting, not a niche patch.

### Fixed

- Now switching replicators applied immediately, without the need to restart Obsidian.

### Tidied up

- Some dependencies have been updated to the latest version.
2025-06-10 02:51:47 +01:00
James Armstrong
1f1a39e5a0 Add simple docker compose 2025-04-28 06:10:29 -07:00
13 changed files with 1812 additions and 2510 deletions

View File

@@ -49,6 +49,23 @@ Once CouchDB run, these directories will be owned by uid:`5984`. Please chown it
```
$ docker run --name couchdb-for-ols -d --restart always -e COUCHDB_USER=${username} -e COUCHDB_PASSWORD=${password} -v ${PWD}/couchdb-data:/opt/couchdb/data -v ${PWD}/couchdb-etc:/opt/couchdb/etc/local.d -p 5984:5984 couchdb
```
If you prefer a compose file instead of docker run, here is the equivalent below:
```
services:
couchdb:
image: couchdb:latest
container_name: couchdb-for-ols
user: 1000:1000
environment:
- COUCHDB_USER=${username}
- COUCHDB_PASSWORD=${password}
volumes:
- ./couchdb-data:/opt/couchdb/data
- ./couchdb-etc:/opt/couchdb/etc/local.d
ports:
- 5984:5984
restart: unless-stopped
```
### B. Install CouchDB directly
Please refer to the [official document](https://docs.couchdb.org/en/stable/install/index.html). However, we do not have to configure it fully. Just the administrator needs to be configured.

View File

@@ -2,6 +2,7 @@
- [Tips and Troubleshooting](#tips-and-troubleshooting)
- [Tips](#tips)
- [CORS avoidance](#cors-avoidance)
- [CORS configuration with reverse proxy](#cors-configuration-with-reverse-proxy)
- [Nginx](#nginx)
- [Nginx and subdirectory](#nginx-and-subdirectory)
@@ -21,6 +22,10 @@
- [Why are the logs volatile and ephemeral?](#why-are-the-logs-volatile-and-ephemeral)
- [Some network logs are not written into the file.](#some-network-logs-are-not-written-into-the-file)
- [If a file were deleted or trimmed, the capacity of the database should be reduced, right?](#if-a-file-were-deleted-or-trimmed-the-capacity-of-the-database-should-be-reduced-right)
- [How to launch the DevTools](#how-to-launch-the-devtools)
- [On Desktop Devices](#on-desktop-devices)
- [On Android](#on-android)
- [On iOS, iPadOS devices](#on-ios-ipados-devices)
- [How can I use the DevTools?](#how-can-i-use-the-devtools)
- [Checking the network log](#checking-the-network-log)
- [Troubleshooting](#troubleshooting)
@@ -33,6 +38,11 @@
## Tips
### CORS avoidance
If we are unable to configure CORS properly for any reason (for example, if we cannot configure non-administered network devices), we may choose to ignore CORS.
To use the Obsidian API (also known as the Non-Native API) to bypass CORS, we can enable the toggle ``Use Request API to avoid `inevitable` CORS problem``.
<!-- Add **Long explanation of CORS** here for integrity -->
### CORS configuration with reverse proxy
- IMPORTANT: CouchDB handles CORS by itself. Do not process CORS on the reverse
@@ -235,6 +245,19 @@ To shrink the database size, `Rebuild everything` only reliably and effectively.
But do not worry, if we have synchronised well. We have the actual and real
files. Only it takes a bit of time and traffics.
### How to launch the DevTools
#### On Desktop Devices
We can launch the DevTools by pressing `ctrl`+`shift`+`i` (`Command`+`shift`+`i` on Mac).
#### On Android
Please refer to [Remote debug Android devices](https://developer.chrome.com/docs/devtools/remote-debugging/).
Once the DevTools have been launched, everything operates the same as on a PC.
#### On iOS, iPadOS devices
If we have a Mac, we can inspect from Safari on the Mac. Please refer to [Inspecting iOS and iPadOS](https://developer.apple.com/documentation/safari-developer-tools/inspecting-ios).
### How can I use the DevTools?
#### Checking the network log

View File

@@ -1,7 +1,7 @@
{
"id": "obsidian-livesync",
"name": "Self-hosted LiveSync",
"version": "0.24.26",
"version": "0.24.29",
"minAppVersion": "0.9.12",
"description": "Community implementation of self-hosted livesync. Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.",
"author": "vorotamoroz",

4056
package-lock.json generated

File diff suppressed because it is too large Load Diff

View File

@@ -1,6 +1,6 @@
{
"name": "obsidian-livesync",
"version": "0.24.26",
"version": "0.24.29",
"description": "Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.",
"main": "main.js",
"type": "module",
@@ -81,10 +81,10 @@
"fflate": "^0.8.2",
"idb": "^8.0.3",
"minimatch": "^10.0.1",
"octagonal-wheels": "^0.1.25",
"octagonal-wheels": "^0.1.31",
"qrcode-generator": "^1.4.4",
"svelte-check": "^4.1.7",
"trystero": "^0.21.3",
"trystero": "^0.21.5",
"xxhash-wasm-102": "npm:xxhash-wasm@^1.0.2"
}
}

Submodule src/lib updated: d53cad1c68...89d9e4e3e0

View File

@@ -27,20 +27,28 @@ import {
updatePreviousExecutionTime,
} from "../../common/utils";
import { isAnyNote } from "../../lib/src/common/utils";
import { EVENT_FILE_SAVED, eventHub } from "../../common/events";
import { EVENT_FILE_SAVED, EVENT_SETTING_SAVED, eventHub } from "../../common/events";
import type { LiveSyncAbstractReplicator } from "../../lib/src/replication/LiveSyncAbstractReplicator";
import { globalSlipBoard } from "../../lib/src/bureau/bureau";
import { $msg } from "../../lib/src/common/i18n";
const KEY_REPLICATION_ON_EVENT = "replicationOnEvent";
const REPLICATION_ON_EVENT_FORECASTED_TIME = 5000;
export class ModuleReplicator extends AbstractModule implements ICoreModule {
_replicatorType?: string;
$everyOnloadAfterLoadSettings(): Promise<boolean> {
eventHub.onEvent(EVENT_FILE_SAVED, () => {
if (this.settings.syncOnSave && !this.core.$$isSuspended()) {
scheduleTask("perform-replicate-after-save", 250, () => this.core.$$replicateByEvent());
}
});
eventHub.onEvent(EVENT_SETTING_SAVED, (setting) => {
if (this._replicatorType !== setting.remoteType) {
void this.setReplicator();
}
});
return Promise.resolve(true);
}
@@ -50,7 +58,12 @@ export class ModuleReplicator extends AbstractModule implements ICoreModule {
this._log($msg("Replicator.Message.InitialiseFatalError"), LOG_LEVEL_NOTICE);
return false;
}
if (this.core.replicator) {
await this.core.replicator.closeReplication();
this._log("Replicator closed for changing", LOG_LEVEL_VERBOSE);
}
this.core.replicator = replicator;
this._replicatorType = this.settings.remoteType;
await yieldMicrotask();
return true;
}
@@ -231,34 +244,49 @@ Even if you choose to clean up, you will see this option again if you exit Obsid
async loadQueuedFiles() {
if (this.settings.suspendParseReplicationResult) return;
if (!this.settings.isConfigured) return;
const kvDBKey = "queued-files";
// const ids = [...new Set(JSON.parse(localStorage.getItem(lsKey) || "[]"))] as string[];
const ids = [...new Set((await this.core.kvDB.get<string[]>(kvDBKey)) ?? [])];
const batchSize = 100;
const chunkedIds = arrayToChunkedArray(ids, batchSize);
try {
const kvDBKey = "queued-files";
// const ids = [...new Set(JSON.parse(localStorage.getItem(lsKey) || "[]"))] as string[];
const ids = [...new Set((await this.core.kvDB.get<string[]>(kvDBKey)) ?? [])];
const batchSize = 100;
const chunkedIds = arrayToChunkedArray(ids, batchSize);
// suspendParseReplicationResult is true, so we have to resume it if it is suspended.
if (this.replicationResultProcessor.isSuspended) {
this.replicationResultProcessor.resume();
}
for await (const idsBatch of chunkedIds) {
const ret = await this.localDatabase.allDocsRaw<EntryDoc>({
keys: idsBatch,
include_docs: true,
limit: 100,
});
const docs = ret.rows.filter((e) => e.doc).map((e) => e.doc) as PouchDB.Core.ExistingDocument<EntryDoc>[];
const errors = ret.rows.filter((e) => !e.doc && !e.value.deleted);
if (errors.length > 0) {
Logger("Some queued processes were not resurrected");
Logger(JSON.stringify(errors), LOG_LEVEL_VERBOSE);
// suspendParseReplicationResult is true, so we have to resume it if it is suspended.
if (this.replicationResultProcessor.isSuspended) {
this.replicationResultProcessor.resume();
}
for await (const idsBatch of chunkedIds) {
const ret = await this.localDatabase.allDocsRaw<EntryDoc>({
keys: idsBatch,
include_docs: true,
limit: 100,
});
const docs = ret.rows
.filter((e) => e.doc)
.map((e) => e.doc) as PouchDB.Core.ExistingDocument<EntryDoc>[];
const errors = ret.rows.filter((e) => !e.doc && !e.value.deleted);
if (errors.length > 0) {
Logger("Some queued processes were not resurrected");
Logger(JSON.stringify(errors), LOG_LEVEL_VERBOSE);
}
this.replicationResultProcessor.enqueueAll(docs);
}
} catch (e) {
Logger(`Failed to load queued files.`, LOG_LEVEL_NOTICE);
Logger(e, LOG_LEVEL_VERBOSE);
} finally {
// Check again before awaiting,
if (this.replicationResultProcessor.isSuspended) {
this.replicationResultProcessor.resume();
}
this.replicationResultProcessor.enqueueAll(docs);
}
if (this.replicationResultProcessor.isSuspended) {
this.replicationResultProcessor.resume();
// Wait for all queued files to be processed.
try {
await this.replicationResultProcessor.waitForAllProcessed();
} catch (e) {
Logger(`Failed to wait for all queued files to be processed.`, LOG_LEVEL_NOTICE);
Logger(e, LOG_LEVEL_VERBOSE);
}
await this.replicationResultProcessor.waitForAllProcessed();
}
replicationResultProcessor = new QueueProcessor(

View File

@@ -1,4 +1,4 @@
import { LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE } from "octagonal-wheels/common/logger.js";
import { LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE } from "octagonal-wheels/common/logger";
import { type ObsidianLiveSyncSettings } from "../../lib/src/common/types.js";
import {
EVENT_REQUEST_OPEN_P2P,

View File

@@ -138,6 +138,8 @@ export class ModuleObsidianSettings extends AbstractObsidianModule implements IO
jwtKid: settings.jwtKid,
jwtExpDuration: settings.jwtExpDuration,
jwtSub: settings.jwtSub,
useRequestAPI: settings.useRequestAPI,
bucketPrefix: settings.bucketPrefix,
};
settings.encryptedCouchDBConnection = await this.encryptConfigurationItem(
JSON.stringify(connectionSetting),

View File

@@ -1410,6 +1410,15 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
(paneEl) => {
void addPanel(paneEl, $msg("obsidianLiveSyncSettingTab.titleRemoteServer")).then((paneEl) => {
// const containerRemoteDatabaseEl = containerEl.createDiv();
this.createEl(
paneEl,
"div",
{
text: $msg("obsidianLiveSyncSettingTab.msgSettingsUnchangeableDuringSync"),
},
undefined,
visibleOnly(() => isAnySyncEnabled())
).addClass("op-warn-info");
new Setting(paneEl).autoWireDropDown("remoteType", {
holdValue: true,
options: {
@@ -1475,6 +1484,10 @@ The pane also can be launched by \`P2P Replicator\` command from the Command Pal
new Setting(paneEl).autoWireText("region", { holdValue: true });
new Setting(paneEl).autoWireText("bucket", { holdValue: true });
new Setting(paneEl).autoWireText("bucketPrefix", {
holdValue: true,
placeHolder: "vaultname/",
});
new Setting(paneEl).autoWireToggle("useCustomRequestHandler", { holdValue: true });
new Setting(paneEl).autoWireTextArea("bucketCustomHeaders", {
@@ -1503,6 +1516,7 @@ The pane also can be launched by \`P2P Replicator\` command from the Command Pal
"bucket",
"useCustomRequestHandler",
"bucketCustomHeaders",
"bucketPrefix",
])
.addOnUpdate(onlyOnMinIO);
});
@@ -1535,16 +1549,6 @@ The pane also can be launched by \`P2P Replicator\` command from the Command Pal
).addClass("op-warn-info");
}
this.createEl(
paneEl,
"div",
{
text: $msg("obsidianLiveSyncSettingTab.msgSettingsUnchangeableDuringSync"),
},
undefined,
visibleOnly(() => isAnySyncEnabled())
).addClass("sls-setting-hidden");
new Setting(paneEl).autoWireText("couchDB_URI", {
holdValue: true,
onUpdate: enableOnlySyncDisabled,
@@ -1662,6 +1666,10 @@ The pane also can be launched by \`P2P Replicator\` command from the Command Pal
onUpdate: enableOnlySyncDisabled,
});
new Setting(paneEl).autoWireTextArea("couchDB_CustomHeaders", { holdValue: true });
new Setting(paneEl).autoWireToggle("useRequestAPI", {
holdValue: true,
onUpdate: enableOnlySyncDisabled,
});
new Setting(paneEl)
.setName($msg("obsidianLiveSyncSettingTab.nameTestDatabaseConnection"))
.setClass("wizardHidden")
@@ -1706,6 +1714,7 @@ The pane also can be launched by \`P2P Replicator\` command from the Command Pal
"jwtKid",
"useJWT",
"couchDB_CustomHeaders",
"useRequestAPI",
])
.addOnUpdate(onlyOnCouchDB);
});
@@ -2080,7 +2089,12 @@ The pane also can be launched by \`P2P Replicator\` command from the Command Pal
.autoWireToggle("syncAfterMerge", { onUpdate: onlyOnNonLiveSync });
});
void addPanel(paneEl, $msg("obsidianLiveSyncSettingTab.titleUpdateThinning")).then((paneEl) => {
void addPanel(
paneEl,
$msg("obsidianLiveSyncSettingTab.titleUpdateThinning"),
undefined,
visibleOnly(() => !this.isConfiguredAs("syncMode", "LIVESYNC"))
).then((paneEl) => {
paneEl.addClass("wizardHidden");
new Setting(paneEl).setClass("wizardHidden").autoWireToggle("batchSave");
new Setting(paneEl).setClass("wizardHidden").autoWireNumeric("batchSaveMinimumDelay", {
@@ -3104,9 +3118,9 @@ ${stringifyYaml(pluginConfig)}`;
onUpdate: visibleOnly(() => this.isConfiguredAs("disableWorkerForGeneratingChunks", false)),
});
});
void addPanel(paneEl, "Edge case addressing (Networking)").then((paneEl) => {
new Setting(paneEl).autoWireToggle("useRequestAPI");
});
// void addPanel(paneEl, "Edge case addressing (Networking)").then((paneEl) => {
// new Setting(paneEl).autoWireToggle("useRequestAPI");
// });
void addPanel(paneEl, "Compatibility (Trouble addressed)").then((paneEl) => {
new Setting(paneEl).autoWireToggle("disableCheckingConfigMismatch");
});
@@ -3547,6 +3561,7 @@ ${stringifyYaml(pluginConfig)}`;
const id = this.plugin.settings.accessKey;
const key = this.plugin.settings.secretKey;
const bucket = this.plugin.settings.bucket;
const prefix = this.plugin.settings.bucketPrefix;
const region = this.plugin.settings.region;
const endpoint = this.plugin.settings.endpoint;
const useCustomRequestHandler = this.plugin.settings.useCustomRequestHandler;
@@ -3556,6 +3571,7 @@ ${stringifyYaml(pluginConfig)}`;
key,
endpoint,
bucket,
prefix,
this.plugin.simpleStore,
this.plugin,
useCustomRequestHandler,

View File

@@ -385,6 +385,10 @@ export const SettingInformation: Partial<Record<keyof AllSettings, Configuration
name: "Show status icon instead of file warnings banner",
desc: "If enabled, the ⛔ icon will be shown inside the status instead of the file warnings banner. No details will be shown.",
},
bucketPrefix: {
name: "File prefix on the bucket",
desc: "Effectively a directory. Should end with `/`. e.g., `vault-name/`.",
},
};
function translateInfo(infoSrc: ConfigurationItem | undefined | false) {
if (!infoSrc) return false;

View File

@@ -1,3 +1,39 @@
## 0.24.29
### Fixed
- Synchronisation with buckets now works correctly, regardless of whether a prefix is set or the bucket has been (re-) initialised (#664).
- An information message is now displayed again, during any automatic synchronisation is enabled (#662).
### Tidied up
- Importing paths have been tidied up.
## 0.24.28
### Fixed
- Batch Update is no longer available in LiveSync mode to avoid unexpected behaviour. (#653)
- Now compatible with Cloudflare R2 again for bucket synchronisation.
- @edo-bari-ikutsu, thank you for [your contribution](https://github.com/vrtmrz/livesync-commonlib/pull/12)!
- Prevention of broken behaviour due to database connection failures added (#649).
## 0.24.27
### Improved
- We can use prefix for path for the Bucket synchronisation.
- For example, if you set the `vaultName/` as a prefix for the bucket in the root directory, all data will be transferred to the bucket under the `vaultName/` directory.
- The "Use Request API to avoid `inevitable` CORS problem" option is now promoted to the normal setting, not a niche patch.
### Fixed
- Now switching replicators applied immediately, without the need to restart Obsidian.
### Tidied up
- Some dependencies have been updated to the latest version.
## 0.24.26
This update introduces an option to circumvent Cross-Origin Resource Sharing
@@ -30,18 +66,18 @@ However, just to whisper, this is tremendously fast.
- Automatic display-language changing according to the Obsidian language
setting.
- We will be asked on the migration or first startup.
- **Note: Please revert to the default language if you report any issues.**
- Not all messages are translated yet. We welcome your contribution!
- We will be asked on the migration or first startup.
- **Note: Please revert to the default language if you report any issues.**
- Not all messages are translated yet. We welcome your contribution!
- Now we can limit files to be synchronised even in the hidden files.
- "Use Request API to avoid `inevitable` CORS problem" has been implemented.
- Less secure, please use it only if you are sure that you are in the trusted
environment and be able to ignore the CORS. No `Web viewer` or similar tools
are recommended. (To avoid the origin forged attack). If you are able to
configure the server setting, always that is recommended.
- Less secure, please use it only if you are sure that you are in the trusted
environment and be able to ignore the CORS. No `Web viewer` or similar tools
are recommended. (To avoid the origin forged attack). If you are able to
configure the server setting, always that is recommended.
- `Show status icon instead of file warnings banner` has been implemented.
- If enabled, the ⛔ icon will be shown inside the status instead of the file
warnings banner. No details will be shown.
- If enabled, the ⛔ icon will be shown inside the status instead of the file
warnings banner. No details will be shown.
### Improved
@@ -55,7 +91,7 @@ However, just to whisper, this is tremendously fast.
- We can purge the remote bucket again if we using MinIO instead of AWS S3 or
Cloudflare R2.
- Purging the remote bucket is now more reliable.
- 100 files are purged at a time.
- 100 files are purged at a time.
- Some wrong messages have been fixed.
### Behaviour changed
@@ -63,8 +99,8 @@ However, just to whisper, this is tremendously fast.
- Entering into the deeper directories to gather the hidden files is now limited
by `/` or `\/` prefixed ignore filters. (It means that directories are scanned
deeper than before).
- However, inside the these directories, the files are still limited by the
ignore filters.
- However, inside the these directories, the files are still limited by the
ignore filters.
### Etcetera
@@ -104,25 +140,25 @@ However, just to whisper, this is tremendously fast.
- Bucket synchronisation has been enhanced for better performance and
reliability.
- Now less duplicated chunks are sent to the server. Note: If you have
encountered about too less chunks, please let me know. However, you can send
it to the server by `Overwrite remote`.
- Fetching conflicted files from the server is now more reliable.
- Dependent libraries have been updated to the latest version.
- Also, let me know if you have encountered any issues with this update.
Especially you are using a device that has been in use for a little
longer.
- Now less duplicated chunks are sent to the server. Note: If you have
encountered about too less chunks, please let me know. However, you can send
it to the server by `Overwrite remote`.
- Fetching conflicted files from the server is now more reliable.
- Dependent libraries have been updated to the latest version.
- Also, let me know if you have encountered any issues with this update.
Especially you are using a device that has been in use for a little
longer.
## 0.24.23
### New Feature
- Now, we can send custom headers to the server.
- They can be sent to either CouchDB or Object Storage.
- They can be sent to either CouchDB or Object Storage.
- Authentication with JWT in CouchDB is now supported.
- I will describe steps later, but please refer to the
[CouchDB document](https://docs.couchdb.org/en/stable/config/auth.html#authentication-configuration).
- A JWT keypair for testing can be generated in the setting dialogue.
- I will describe steps later, but please refer to the
[CouchDB document](https://docs.couchdb.org/en/stable/config/auth.html#authentication-configuration).
- A JWT keypair for testing can be generated in the setting dialogue.
### Improved

View File

@@ -1,4 +1,4 @@
import { encrypt } from "npm:octagonal-wheels@0.1.11/encryption/encryption.js";
import { encrypt } from "npm:octagonal-wheels@0.1.30/encryption/encryption";
const noun = [
"waterfall",