bump

- Internal documents are now ignored.
- Merge dialogue now respond immediately to button pressing.
- Periodic processing now works fine
- The checking interval of detecting conflicted has got shorter
- Replication is now cancelled while cleaning up
- The database locking by the cleaning up is now carefully unlocked
- Missing chunks message is correctly reported

New feature:
- Suspend database reflecting has been implemented
- Now fetch suspends the reflecting database and storage changes temporarily to improve the performance.
- We can choose the action when the remote database has been cleaned
- Merge dialogue now show `↲` before the new line.

Improved:
- Now progress is reported while the cleaning up and fetch process
- Cancelled replication is now detected

README.md

@@ -1,6 +1,6 @@
 # Self-hosted LiveSync
 
-[Japanese docs](./README_ja.md) [Chinese docs](./README_cn.md).
+[Japanese docs](./README_ja.md) - [Chinese docs](./README_cn.md).
 
 Self-hosted LiveSync is a community-implemented synchronization plugin.  
 A self-hosted or purchased CouchDB acts as the intermediate server. Available on every obsidian-compatible platform.

docker-compose.traefik.yml

@@ -0,0 +1,46 @@
+# For details and other explanations about this file refer to:
+# https://github.com/vrtmrz/obsidian-livesync/blob/main/docs/setup_own_server.md#traefik
+
+version: "2.1"
+services:
+  couchdb:
+    image: couchdb:latest
+    container_name: obsidian-livesync
+    user: 1000:1000
+    environment:
+      - COUCHDB_USER=username
+      - COUCHDB_PASSWORD=password
+    volumes:
+      - ./data:/opt/couchdb/data
+      - ./local.ini:/opt/couchdb/etc/local.ini
+    # Ports not needed when already passed to Traefik
+    #ports:
+    #  - 5984:5984
+    restart: unless-stopped
+    networks:
+      - proxy
+    labels:
+      - "traefik.enable=true"
+      # The Traefik Network
+      - "traefik.docker.network=proxy"
+      # Don't forget to replace 'obsidian-livesync.example.org' with your own domain
+      - "traefik.http.routers.obsidian-livesync.rule=Host(`obsidian-livesync.example.org`)"
+      # The 'websecure' entryPoint is basically your HTTPS entrypoint. Check the next code snippet if you are encountering problems only; you probably have a working traefik configuration if this is not your first container you are reverse proxying.
+      - "traefik.http.routers.obsidian-livesync.entrypoints=websecure"
+      - "traefik.http.routers.obsidian-livesync.service=obsidian-livesync"
+      - "traefik.http.services.obsidian-livesync.loadbalancer.server.port=5984"
+      - "traefik.http.routers.obsidian-livesync.tls=true"
+      # Replace the string 'letsencrypt' with your own certificate resolver
+      - "traefik.http.routers.obsidian-livesync.tls.certresolver=letsencrypt"
+      - "traefik.http.routers.obsidian-livesync.middlewares=obsidiancors"
+      # The part needed for CORS to work on Traefik 2.x starts here
+      - "traefik.http.middlewares.obsidiancors.headers.accesscontrolallowmethods=GET,PUT,POST,HEAD,DELETE"
+      - "traefik.http.middlewares.obsidiancors.headers.accesscontrolallowheaders=accept,authorization,content-type,origin,referer"
+      - "traefik.http.middlewares.obsidiancors.headers.accesscontrolalloworiginlist=app://obsidian.md,capacitor://localhost,http://localhost"
+      - "traefik.http.middlewares.obsidiancors.headers.accesscontrolmaxage=3600"
+      - "traefik.http.middlewares.obsidiancors.headers.addvaryheader=true"
+      - "traefik.http.middlewares.obsidiancors.headers.accessControlAllowCredentials=true"
+
+networks:
+  proxy:
+    external: true

docs/setup_own_server.md

@@ -1,10 +1,22 @@
 # Setup a CouchDB server
 
+## Table of Contents
+- [Configure](#configure)
+- [Run](#run)
+  - [Docker CLI](#docker-cli)
+  - [Docker Compose](#docker-compose)
+- [Access from a mobile device](#access-from-a-mobile-device)
+  - [Testing from a mobile](#testing-from-a-mobile)
+  - [Setting up your domain](#setting-up-your-domain)
+- [Reverse Proxies](#reverse-proxies)
+  - [Traefik](#traefik)
+---
+
 ## Configure
 
 The easiest way to set up a CouchDB instance is using the official [docker image](https://hub.docker.com/_/couchdb).
 
-Some initial configuration is required. Create a `local.ini` to use Self-hosted LiveSync as follows:
+Some initial configuration is required. Create a `local.ini` to use Self-hosted LiveSync as follows ([CouchDB has to be version 3.2 or higher](https://docs.couchdb.org/en/latest/config/http.html#chttpd/enable_cors), if lower `enable_cors = true` has to be under section `[httpd]` ):
 
 ```ini
 [couchdb]
@@ -14,6 +26,7 @@ max_document_size = 50000000
 [chttpd]
 require_valid_user = true
 max_http_request_size = 4294967296
+enable_cors = true
 
 [chttpd_auth]
 require_valid_user = true
@@ -21,13 +34,13 @@ authentication_redirect = /_utils/session.html
 
 [httpd]
 WWW-Authenticate = Basic realm="couchdb"
-enable_cors = true
+bind_address = 0.0.0.0
 
 [cors]
-origins = app://obsidian.md,capacitor://localhost,http://localhost
+origins = app://obsidian.md, capacitor://localhost, http://localhost
 credentials = true
 headers = accept, authorization, content-type, origin, referer
-methods = GET, PUT, POST, HEAD, DELETE
+methods = GET,PUT,POST,HEAD,DELETE
 max_age = 3600
 ```
 
@@ -48,7 +61,7 @@ $ docker run -d --restart always -e COUCHDB_USER=admin -e COUCHDB_PASSWORD=passw
 *Remember to replace the path with the path to your local.ini*
 
 ### Docker Compose
-Create a directory, place your `local.ini` within it, and create a `docker-compose.yml` alongside it. The directory structure should look similar to this:
+Create a directory, place your `local.ini` within it, and create a `docker-compose.yml` alongside it. Make sure to have write permissions for `local.ini` and the about to be created `data` folder after the container start. The directory structure should look similar to this:
 ```
 obsidian-livesync
 ├── docker-compose.yml
@@ -127,6 +140,77 @@ Set the A record of your domain to point to your server, and host reverse proxy
 Note: Mounting CouchDB on the top directory is not recommended.  
 Using Caddy is a handy way to serve the server with SSL automatically.
 
-I have published [docker-compose.yml and ini files](https://github.com/vrtmrz/self-hosted-livesync-server) that launch Caddy and CouchDB at once. Please try it out.
+I have published [docker-compose.yml and ini files](https://github.com/vrtmrz/self-hosted-livesync-server) that launch Caddy and CouchDB at once. If you are using Traefik you can check the [Reverse Proxies](#reverse-proxies) section below.
 
 And, be sure to check the server log and be careful of malicious access.
+
+
+## Reverse Proxies
+
+### Traefik
+
+If you are using Traefik, this [docker-compose.yml](https://github.com/vrtmrz/obsidian-livesync/blob/main/docker-compose.traefik.yml) file (also pasted below) has all the right CORS parameters set. It assumes you have an external network called `proxy`.
+
+```yaml
+version: "2.1"
+services:
+  couchdb:
+    image: couchdb:latest
+    container_name: obsidian-livesync
+    user: 1000:1000
+    environment:
+      - COUCHDB_USER=username
+      - COUCHDB_PASSWORD=password
+    volumes:
+      - ./data:/opt/couchdb/data
+      - ./local.ini:/opt/couchdb/etc/local.ini
+    # Ports not needed when already passed to Traefik
+    #ports:
+    #  - 5984:5984
+    restart: unless-stopped
+    networks:
+      - proxy
+    labels:
+      - "traefik.enable=true"
+      # The Traefik Network
+      - "traefik.docker.network=proxy"
+      # Don't forget to replace 'obsidian-livesync.example.org' with your own domain
+      - "traefik.http.routers.obsidian-livesync.rule=Host(`obsidian-livesync.example.org`)"
+      # The 'websecure' entryPoint is basically your HTTPS entrypoint. Check the next code snippet if you are encountering problems only; you probably have a working traefik configuration if this is not your first container you are reverse proxying.
+      - "traefik.http.routers.obsidian-livesync.entrypoints=websecure"
+      - "traefik.http.routers.obsidian-livesync.service=obsidian-livesync"
+      - "traefik.http.services.obsidian-livesync.loadbalancer.server.port=5984"
+      - "traefik.http.routers.obsidian-livesync.tls=true"
+      # Replace the string 'letsencrypt' with your own certificate resolver
+      - "traefik.http.routers.obsidian-livesync.tls.certresolver=letsencrypt"
+      - "traefik.http.routers.obsidian-livesync.middlewares=obsidiancors"
+      # The part needed for CORS to work on Traefik 2.x starts here
+      - "traefik.http.middlewares.obsidiancors.headers.accesscontrolallowmethods=GET,PUT,POST,HEAD,DELETE"
+      - "traefik.http.middlewares.obsidiancors.headers.accesscontrolallowheaders=accept,authorization,content-type,origin,referer"
+      - "traefik.http.middlewares.obsidiancors.headers.accesscontrolalloworiginlist=app://obsidian.md,capacitor://localhost,http://localhost"
+      - "traefik.http.middlewares.obsidiancors.headers.accesscontrolmaxage=3600"
+      - "traefik.http.middlewares.obsidiancors.headers.addvaryheader=true"
+      - "traefik.http.middlewares.obsidiancors.headers.accessControlAllowCredentials=true"
+
+networks:
+  proxy:
+    external: true
+```
+
+Partial `traefik.yml` config file mentioned in above:
+```yml
+...
+
+entryPoints:
+  web:
+    address: ":80"
+    http:
+      redirections:
+        entryPoint:
+          to: "websecure"
+          scheme: "https"
+  websecure:
+    address: ":443"
+
+...
+```

manifest.json

@@ -1,7 +1,7 @@
 {
     "id": "obsidian-livesync",
     "name": "Self-hosted LiveSync",
-    "version": "0.19.12",
+    "version": "0.19.14",
     "minAppVersion": "0.9.12",
     "description": "Community implementation of self-hosted livesync. Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.",
     "author": "vorotamoroz",

package-lock.json

@@ -1,12 +1,12 @@
 {
     "name": "obsidian-livesync",
-    "version": "0.19.12",
+    "version": "0.19.14",
     "lockfileVersion": 2,
     "requires": true,
     "packages": {
         "": {
             "name": "obsidian-livesync",
-            "version": "0.19.12",
+            "version": "0.19.14",
             "license": "MIT",
             "dependencies": {
                 "diff-match-patch": "^1.0.5",

package.json

@@ -1,6 +1,6 @@
 {
     "name": "obsidian-livesync",
-    "version": "0.19.12",
+    "version": "0.19.14",
     "description": "Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.",
     "main": "main.js",
     "type": "module",

src/CmdConfigSync.ts

@@ -4,7 +4,7 @@ import { Notice, type PluginManifest, parseYaml } from "./deps";
 import type { EntryDoc, LoadedEntry, InternalFileEntry, FilePathWithPrefix, FilePath, DocumentID, AnyEntry } from "./lib/src/types";
 import { LOG_LEVEL } from "./lib/src/types";
 import { ICXHeader, PERIODIC_PLUGIN_SWEEP, } from "./types";
-import { Parallels, delay, getDocData } from "./lib/src/utils";
+import { delay, getDocData } from "./lib/src/utils";
 import { Logger } from "./lib/src/logger";
 import { WrappedNotice } from "./lib/src/wrapper";
 import { base64ToArrayBuffer, arrayBufferToBase64, readString, crc32CKHash } from "./lib/src/strbin";

src/CmdSetupLiveSync.ts

@@ -8,6 +8,7 @@ import { LiveSyncCommands } from "./LiveSyncCommands";
 import { delay } from "./lib/src/utils";
 import { confirmWithMessage } from "./dialogs";
 import { Platform } from "./deps";
+import { fetchAllUsedChunks } from "./lib/src/utils_couchdb";
 
 export class SetupLiveSync extends LiveSyncCommands {
     onunload() { }
@@ -284,6 +285,26 @@ Of course, we are able to disable these features.`
         this.plugin.settings.syncAfterMerge = false;
         //this.suspendExtraSync();
     }
+    async suspendReflectingDatabase() {
+        if (this.plugin.settings.doNotSuspendOnFetching) return;
+        Logger(`Suspending reflection: Database and storage changes will not be reflected in each other until completely finished the fetching.`, LOG_LEVEL.NOTICE);
+        this.plugin.settings.suspendParseReplicationResult = true;
+        this.plugin.settings.suspendFileWatching = true;
+        await this.plugin.saveSettings();
+    }
+    async resumeReflectingDatabase() {
+        if (this.plugin.settings.doNotSuspendOnFetching) return;
+        Logger(`Database and storage reflection has been resumed!`, LOG_LEVEL.NOTICE);
+        this.plugin.settings.suspendParseReplicationResult = false;
+        this.plugin.settings.suspendFileWatching = false;
+        await this.plugin.saveSettings();
+        if (this.plugin.settings.readChunksOnline) {
+            await this.plugin.syncAllFiles(true);
+            await this.plugin.loadQueuedFiles();
+            // Start processing
+            this.plugin.procQueuedFiles();
+        }
+    }
     async askUseNewAdapter() {
         if (!this.plugin.settings.useIndexedDBAdapter) {
             const message = `Now this plugin has been configured to use the old database adapter for keeping compatibility. Do you want to deactivate it?`;
@@ -297,9 +318,22 @@ Of course, we are able to disable these features.`
             }
         }
     }
+    async fetchRemoteChunks() {
+        if (!this.plugin.settings.doNotSuspendOnFetching && this.plugin.settings.readChunksOnline) {
+            Logger(`Fetching chunks`, LOG_LEVEL.NOTICE);
+            const remoteDB = await this.plugin.getReplicator().connectRemoteCouchDBWithSetting(this.settings, this.plugin.getIsMobile(), true);
+            if (typeof remoteDB == "string") {
+                Logger(remoteDB, LOG_LEVEL.NOTICE);
+            } else {
+                await fetchAllUsedChunks(this.localDatabase.localDatabase, remoteDB.db);
+            }
+            Logger(`Fetching chunks done`, LOG_LEVEL.NOTICE);
+        }
+    }
     async fetchLocal() {
         this.suspendExtraSync();
         this.askUseNewAdapter();
+        await this.suspendReflectingDatabase();
         await this.plugin.realizeSettingSyncMode();
         await this.plugin.resetLocalDatabase();
         await delay(1000);
@@ -310,6 +344,8 @@ Of course, we are able to disable these features.`
         await this.plugin.replicateAllFromServer(true);
         await delay(1000);
         await this.plugin.replicateAllFromServer(true);
+        await this.fetchRemoteChunks();
+        await this.resumeReflectingDatabase();
         await this.askHiddenFileConfiguration({ enableFetch: true });
     }
     async rebuildRemote() {

src/ConflictResolveModal.ts

@@ -30,11 +30,11 @@ export class ConflictResolveModal extends Modal {
             const x1 = v[0];
             const x2 = v[1];
             if (x1 == DIFF_DELETE) {
-                diff += "<span class='deleted'>" + escapeStringToHTML(x2) + "</span>";
+                diff += "<span class='deleted'>" + escapeStringToHTML(x2).replace(/\n/g, "<span class='ls-mark-cr'></span>\n") + "</span>";
             } else if (x1 == DIFF_EQUAL) {
-                diff += "<span class='normal'>" + escapeStringToHTML(x2) + "</span>";
+                diff += "<span class='normal'>" + escapeStringToHTML(x2).replace(/\n/g, "<span class='ls-mark-cr'></span>\n") + "</span>";
             } else if (x1 == DIFF_INSERT) {
-                diff += "<span class='added'>" + escapeStringToHTML(x2) + "</span>";
+                diff += "<span class='added'>" + escapeStringToHTML(x2).replace(/\n/g, "<span class='ls-mark-cr'></span>\n") + "</span>";
             }
         }
 
@@ -48,23 +48,26 @@ export class ConflictResolveModal extends Modal {
         `;
         contentEl.createEl("button", { text: "Keep A" }, (e) => {
             e.addEventListener("click", async () => {
-                await this.callback(this.result.right.rev);
+                const callback = this.callback;
                 this.callback = null;
                 this.close();
+                await callback(this.result.right.rev);
             });
         });
         contentEl.createEl("button", { text: "Keep B" }, (e) => {
             e.addEventListener("click", async () => {
-                await this.callback(this.result.left.rev);
+                const callback = this.callback;
                 this.callback = null;
                 this.close();
+                await callback(this.result.left.rev);
             });
         });
         contentEl.createEl("button", { text: "Concat both" }, (e) => {
             e.addEventListener("click", async () => {
-                await this.callback("");
+                const callback = this.callback;
                 this.callback = null;
                 this.close();
+                await callback("");
             });
         });
         contentEl.createEl("button", { text: "Not now" }, (e) => {

src/ObsidianLiveSyncSettingTab.ts

@@ -7,7 +7,7 @@ import { Logger } from "./lib/src/logger";
 import { checkSyncInfo, isCloudantURI } from "./lib/src/utils_couchdb.js";
 import { testCrypt } from "./lib/src/e2ee_v2";
 import ObsidianLiveSyncPlugin from "./main";
-import { balanceChunks, localDatabaseCleanUp, performRebuildDB, remoteDatabaseCleanup, requestToCouchDB } from "./utils";
+import { askYesNo, performRebuildDB, requestToCouchDB, scheduleTask } from "./utils";
 
 
 export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
@@ -1610,6 +1610,27 @@ ${stringifyYaml(pluginConfig)}`;
                 toggle.setValue(this.plugin.settings.suspendFileWatching).onChange(async (value) => {
                     this.plugin.settings.suspendFileWatching = value;
                     await this.plugin.saveSettings();
+                    scheduleTask("configReload", 250, async () => {
+                        if (await askYesNo(this.app, "Do you want to restart and reload Obsidian now?") == "yes") {
+                            // @ts-ignore
+                            this.app.commands.executeCommandById("app:reload")
+                        }
+                    })
+                })
+            );
+        new Setting(containerHatchEl)
+            .setName("Suspend database reflecting")
+            .setDesc("Stop reflecting database changes to storage files.")
+            .addToggle((toggle) =>
+                toggle.setValue(this.plugin.settings.suspendParseReplicationResult).onChange(async (value) => {
+                    this.plugin.settings.suspendParseReplicationResult = value;
+                    await this.plugin.saveSettings();
+                    scheduleTask("configReload", 250, async () => {
+                        if (await askYesNo(this.app, "Do you want to restart and reload Obsidian now?") == "yes") {
+                            // @ts-ignore
+                            this.app.commands.executeCommandById("app:reload")
+                        }
+                    })
                 })
             );
         new Setting(containerHatchEl)
@@ -1731,6 +1752,16 @@ ${stringifyYaml(pluginConfig)}`;
             )
             .setClass("wizardHidden");
 
+
+        new Setting(containerHatchEl)
+            .setName("Fetch database with previous behaviour")
+            .setDesc("")
+            .addToggle((toggle) =>
+                toggle.setValue(this.plugin.settings.doNotSuspendOnFetching).onChange(async (value) => {
+                    this.plugin.settings.doNotSuspendOnFetching = value;
+                    await this.plugin.saveSettings();
+                })
+            );
         addScreenElement("50", containerHatchEl);
 
 
@@ -1851,26 +1882,6 @@ ${stringifyYaml(pluginConfig)}`;
                     })
             )
 
-        new Setting(containerMaintenanceEl)
-            .setName("(Beta) Clean the remote database")
-            .setDesc("")
-            .addButton((button) =>
-                button.setButtonText("Count")
-                    .setDisabled(false)
-                    .onClick(async () => {
-                        await remoteDatabaseCleanup(this.plugin, true);
-                    })
-            ).addButton((button) =>
-                button.setButtonText("Perform cleaning")
-                    .setDisabled(false)
-                    .setWarning()
-                    .onClick(async () => {
-                        // @ts-ignore
-                        this.plugin.app.setting.close()
-                        await remoteDatabaseCleanup(this.plugin, false);
-                        await balanceChunks(this.plugin, false);
-                    })
-            );
 
         containerMaintenanceEl.createEl("h4", { text: "The local database" });
 
@@ -1887,26 +1898,6 @@ ${stringifyYaml(pluginConfig)}`;
                     })
             )
 
-        new Setting(containerMaintenanceEl)
-            .setName("(Beta) Clean the local database")
-            .setDesc("This feature requires disabling 'Use an old adapter for compatibility'")
-            .addButton((button) =>
-                button.setButtonText("Count")
-                    .setDisabled(false)
-                    .onClick(async () => {
-                        await localDatabaseCleanUp(this.plugin, false, true);
-                    })
-            ).addButton((button) =>
-                button.setButtonText("Perform cleaning")
-                    .setDisabled(false)
-                    .setWarning()
-                    .onClick(async () => {
-                        // @ts-ignore
-                        this.plugin.app.setting.close()
-                        await localDatabaseCleanUp(this.plugin, false, false);
-                    })
-            );
-
         new Setting(containerMaintenanceEl)
             .setName("Discard local database to reset or uninstall Self-hosted LiveSync")
             .addButton((button) =>
@@ -1922,6 +1913,25 @@ ${stringifyYaml(pluginConfig)}`;
 
         containerMaintenanceEl.createEl("h4", { text: "Both databases" });
 
+        new Setting(containerMaintenanceEl)
+            .setName("(Beta2) Clean up databases")
+            .setDesc("Delete unused chunks to shrink the database. This feature requires disabling 'Use an old adapter for compatibility'")
+            .addButton((button) =>
+                button.setButtonText("DryRun")
+                    .setDisabled(false)
+                    .onClick(async () => {
+                        await this.plugin.dryRunGC();
+                    })
+            ).addButton((button) =>
+                button.setButtonText("Perform cleaning")
+                    .setDisabled(false)
+                    .setWarning()
+                    .onClick(async () => {
+                        // @ts-ignore
+                        this.plugin.app.setting.close()
+                        await this.plugin.dbGC();
+                    })
+            );
         new Setting(containerMaintenanceEl)
             .setName("Rebuild everything")
             .setDesc("Rebuild local and remote database with local files.")
@@ -1934,19 +1944,6 @@ ${stringifyYaml(pluginConfig)}`;
                         await rebuildDB("rebuildBothByThisDevice");
                     })
             )
-
-        new Setting(containerMaintenanceEl)
-            .setName("(Beta) Complement each other with possible missing chunks.")
-            .setDesc("")
-            .addButton((button) =>
-                button
-                    .setButtonText("Balance")
-                    .setWarning()
-                    .setDisabled(false)
-                    .onClick(async () => {
-                        await balanceChunks(this.plugin, false);
-                    })
-            )
         applyDisplayEnabled();
         addScreenElement("70", containerMaintenanceEl);
 

src/dialogs.ts

@@ -183,7 +183,7 @@ export class MessageBox extends Modal {
         })
         contentEl.createEl("h1", { text: this.title });
         const div = contentEl.createDiv();
-        MarkdownRenderer.renderMarkdown(this.contentMd, div, "/", null);
+        MarkdownRenderer.renderMarkdown(this.contentMd, div, "/", this.plugin);
         const buttonSetting = new Setting(contentEl);
         for (const button of this.buttons) {
             buttonSetting.addButton((btn) => {

src/main.ts

@@ -10,15 +10,15 @@ import { PouchDB } from "./lib/src/pouchdb-browser.js";
 import { ConflictResolveModal } from "./ConflictResolveModal";
 import { ObsidianLiveSyncSettingTab } from "./ObsidianLiveSyncSettingTab";
 import { DocumentHistoryModal } from "./DocumentHistoryModal";
-import { applyPatch, cancelAllPeriodicTask, cancelAllTasks, cancelTask, generatePatchObj, id2path, isObjectMargeApplicable, isSensibleMargeApplicable, flattenObject, path2id, scheduleTask, tryParseJSON, createFile, modifyFile, isValidPath, getAbstractFileByPath, touch, recentlyTouched, isInternalMetadata, isPluginMetadata, stripInternalMetadataPrefix, isChunk, askSelectString, askYesNo, askString, PeriodicProcessor, clearTouched, getPath, getPathWithoutPrefix, getPathFromTFile, localDatabaseCleanUp, balanceChunks, performRebuildDB } from "./utils";
+import { applyPatch, cancelAllPeriodicTask, cancelAllTasks, cancelTask, generatePatchObj, id2path, isObjectMargeApplicable, isSensibleMargeApplicable, flattenObject, path2id, scheduleTask, tryParseJSON, createFile, modifyFile, isValidPath, getAbstractFileByPath, touch, recentlyTouched, isInternalMetadata, isPluginMetadata, stripInternalMetadataPrefix, isChunk, askSelectString, askYesNo, askString, PeriodicProcessor, clearTouched, getPath, getPathWithoutPrefix, getPathFromTFile, performRebuildDB } from "./utils";
 import { encrypt, tryDecrypt } from "./lib/src/e2ee_v2";
-import { enableEncryption, isCloudantURI, isErrorOfMissingDoc, isValidRemoteCouchDBURI } from "./lib/src/utils_couchdb";
+import { balanceChunkPurgedDBs, enableEncryption, isCloudantURI, isErrorOfMissingDoc, isValidRemoteCouchDBURI, purgeUnreferencedChunks } from "./lib/src/utils_couchdb";
 import { getGlobalStore, ObservableStore, observeStores } from "./lib/src/store";
 import { lockStore, logMessageStore, logStore, type LogEntry } from "./lib/src/stores";
 import { setNoticeClass } from "./lib/src/wrapper";
 import { base64ToString, versionNumberString2Number, base64ToArrayBuffer, arrayBufferToBase64 } from "./lib/src/strbin";
 import { addPrefix, isPlainText, shouldBeIgnored, stripAllPrefixes } from "./lib/src/path";
-import { runWithLock } from "./lib/src/lock";
+import { isLockAcquired, runWithLock } from "./lib/src/lock";
 import { Semaphore } from "./lib/src/semaphore";
 import { StorageEventManager, StorageEventManagerObsidian } from "./StorageEventManager";
 import { LiveSyncLocalDB, type LiveSyncLocalDBEnv } from "./lib/src/LiveSyncLocalDB";
@@ -240,6 +240,8 @@ export default class ObsidianLiveSyncPlugin extends Plugin
     createPouchDBInstance<T>(name?: string, options?: PouchDB.Configuration.DatabaseConfiguration): PouchDB.Database<T> {
         if (this.settings.useIndexedDBAdapter) {
             options.adapter = "indexeddb";
+            //@ts-ignore :missing def
+            options.purged_infos_limit = 1;
             return new PouchDB(name + "-indexeddb", options);
         }
         return new PouchDB(name, options);
@@ -420,11 +422,13 @@ export default class ObsidianLiveSyncPlugin extends Plugin
                     Logger(`${FLAGMD_REDFLAG3} has been detected! Self-hosted LiveSync will discard the local database and fetch everything from the remote once again.`, LOG_LEVEL.NOTICE);
                     await this.addOnSetup.fetchLocal();
                     await this.deleteRedFlag3();
-                    if (await askYesNo(this.app, "Do you want to disable Suspend file watching and restart obsidian now?") == "yes") {
-                        this.settings.suspendFileWatching = false;
-                        await this.saveSettings();
-                        // @ts-ignore
-                        this.app.commands.executeCommandById("app:reload")
+                    if (this.settings.suspendFileWatching) {
+                        if (await askYesNo(this.app, "Do you want to disable Suspend file watching and restart obsidian now?") == "yes") {
+                            this.settings.suspendFileWatching = false;
+                            await this.saveSettings();
+                            // @ts-ignore
+                            this.app.commands.executeCommandById("app:reload")
+                        }
                     }
                 } else {
                     this.settings.writeLogToTheFile = true;
@@ -437,6 +441,9 @@ export default class ObsidianLiveSyncPlugin extends Plugin
                 if (this.settings.suspendFileWatching) {
                     Logger("'Suspend file watching' turned on. Are you sure this is what you intended? Every modification on the vault will be ignored.", LOG_LEVEL.NOTICE);
                 }
+                if (this.settings.suspendParseReplicationResult) {
+                    Logger("'Suspend database reflecting' turned on. Are you sure this is what you intended? Every replicated change will be postponed until disabling this option.", LOG_LEVEL.NOTICE);
+                }
                 const isInitialized = await this.initializeDatabase(false, false);
                 if (!isInitialized) {
                     //TODO:stop all sync.
@@ -1316,12 +1323,14 @@ export default class ObsidianLiveSyncPlugin extends Plugin
         localStorage.setItem(lsKey, saveData);
     }
     async loadQueuedFiles() {
-        const lsKey = "obsidian-livesync-queuefiles-" + this.getVaultName();
-        const ids = JSON.parse(localStorage.getItem(lsKey) || "[]") as string[];
-        const ret = await this.localDatabase.allDocsRaw<EntryDoc>({ keys: ids, include_docs: true });
-        for (const doc of ret.rows) {
-            if (doc.doc && !this.queuedFiles.some((e) => e.entry._id == doc.doc._id)) {
-                await this.parseIncomingDoc(doc.doc as PouchDB.Core.ExistingDocument<EntryBody & PouchDB.Core.AllDocsMeta>);
+        if (!this.settings.suspendParseReplicationResult) {
+            const lsKey = "obsidian-livesync-queuefiles-" + this.getVaultName();
+            const ids = [...new Set(JSON.parse(localStorage.getItem(lsKey) || "[]"))] as string[];
+            const ret = await this.localDatabase.allDocsRaw<EntryDoc>({ keys: ids, include_docs: true });
+            for (const doc of ret.rows) {
+                if (doc.doc && !this.queuedFiles.some((e) => e.entry._id == doc.doc._id)) {
+                    await this.parseIncomingDoc(doc.doc as PouchDB.Core.ExistingDocument<EntryBody & PouchDB.Core.AllDocsMeta>);
+                }
             }
         }
     }
@@ -1383,6 +1392,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin
         // It is better for your own safety, not to handle the following files
         const ignoreFiles = [
             "_design/replicate",
+            "_design/chunks",
             FLAGMD_REDFLAG,
             FLAGMD_REDFLAG2,
             FLAGMD_REDFLAG3
@@ -1431,21 +1441,41 @@ export default class ObsidianLiveSyncPlugin extends Plugin
         L1:
         for (const change of docsSorted) {
             if (isChunk(change._id)) {
-                await this.parseIncomingChunk(change);
+                if (!this.settings.suspendParseReplicationResult) {
+                    await this.parseIncomingChunk(change);
+                }
                 continue;
             }
-            for (const proc of this.addOns) {
-                if (await proc.parseReplicationResultItem(change)) {
-                    continue L1;
+            if (!this.settings.suspendParseReplicationResult) {
+                for (const proc of this.addOns) {
+                    if (await proc.parseReplicationResultItem(change)) {
+                        continue L1;
+                    }
                 }
             }
             if (change._id == SYNCINFO_ID) {
                 continue;
             }
-            if (change.type != "leaf" && change.type != "versioninfo" && change.type != "milestoneinfo" && change.type != "nodeinfo") {
-                await this.parseIncomingDoc(change);
+            if (change._id.startsWith("_design")) {
                 continue;
             }
+
+            if (change.type != "leaf" && change.type != "versioninfo" && change.type != "milestoneinfo" && change.type != "nodeinfo") {
+                if (this.settings.suspendParseReplicationResult) {
+                    const newQueue = {
+                        entry: change,
+                        missingChildren: [] as string[],
+                        timeout: 0,
+                    };
+                    Logger(`Processing scheduled: ${change.path}`, LOG_LEVEL.INFO);
+                    this.queuedFiles.push(newQueue);
+                    this.saveQueuedFiles();
+                    continue;
+                } else {
+                    await this.parseIncomingDoc(change);
+                    continue;
+                }
+            }
             if (change.type == "versioninfo") {
                 if (change.version > VER) {
                     this.replicator.closeReplication();
@@ -1566,19 +1596,19 @@ export default class ObsidianLiveSyncPlugin extends Plugin
         const newMsg = typeof message == "string" ? message : this.lastMessage;
         const newLog = typeof log == "string" ? log : this.lastLog;
         if (`${this.lastMessage}-${this.lastLog}` != `${newMsg}-${newLog}`) {
-            scheduleTask("update-display", 50, () => {
-                this.statusBar.setText(newMsg.split("\n")[0]);
+            // scheduleTask("update-display", 50, () => {
+            this.statusBar.setText(newMsg.split("\n")[0]);
 
-                if (this.settings.showStatusOnEditor) {
-                    const root = activeDocument.documentElement;
-                    const q = root.querySelectorAll(`.CodeMirror-wrap,.cm-s-obsidian>.cm-editor,.canvas-wrapper`);
-                    q.forEach(e => e.setAttr("data-log", '' + (newMsg + "\n" + newLog) + ''))
-                } else {
-                    const root = activeDocument.documentElement;
-                    const q = root.querySelectorAll(`.CodeMirror-wrap,.cm-s-obsidian>.cm-editor,.canvas-wrapper`);
-                    q.forEach(e => e.setAttr("data-log", ''))
-                }
-            }, true);
+            if (this.settings.showStatusOnEditor) {
+                const root = activeDocument.documentElement;
+                const q = root.querySelectorAll(`.CodeMirror-wrap,.cm-s-obsidian>.cm-editor,.canvas-wrapper`);
+                q.forEach(e => e.setAttr("data-log", '' + (newMsg + "\n" + newLog) + ''))
+            } else {
+                const root = activeDocument.documentElement;
+                const q = root.querySelectorAll(`.CodeMirror-wrap,.cm-s-obsidian>.cm-editor,.canvas-wrapper`);
+                q.forEach(e => e.setAttr("data-log", ''))
+            }
+            // }, true);
             scheduleTask("log-hide", 3000, () => this.setStatusBarText(null, ""));
             this.lastMessage = newMsg;
             this.lastLog = newLog;
@@ -1588,8 +1618,12 @@ export default class ObsidianLiveSyncPlugin extends Plugin
 
     async replicate(showMessage?: boolean) {
         if (!this.isReady) return;
+        if (isLockAcquired("cleanup")) {
+            Logger("Database cleaning up is in process. replication has been cancelled", LOG_LEVEL.NOTICE);
+            return;
+        }
         if (this.settings.versionUpFlash != "") {
-            Logger("Open settings and check message, please.", LOG_LEVEL.NOTICE);
+            Logger("Open settings and check message, please. replication has been cancelled.", LOG_LEVEL.NOTICE);
             return;
         }
         await this.applyBatchChange();
@@ -1599,22 +1633,41 @@ export default class ObsidianLiveSyncPlugin extends Plugin
         if (!ret) {
             if (this.replicator.remoteLockedAndDeviceNotAccepted) {
                 if (this.replicator.remoteCleaned) {
-                    const message = `
-The remote database has been cleaned up.
-To synchronize, this device must also be cleaned up or fetch everything again once.
-Fetching may takes some time. Cleaning up is not stable yet but fast.
-`
-                    const CHOICE_CLEANUP = "Clean up";
-                    const CHOICE_FETCH = "Fetch again";
-                    const CHOICE_DISMISS = "Dismiss";
-                    const ret = await confirmWithMessage(this, "Locked", message, [CHOICE_CLEANUP, CHOICE_FETCH, CHOICE_DISMISS], CHOICE_DISMISS, 10);
-                    if (ret == CHOICE_CLEANUP) {
-                        await localDatabaseCleanUp(this, true, false);
-                        await balanceChunks(this, false);
-                    }
-                    if (ret == CHOICE_FETCH) {
-                        await performRebuildDB(this, "localOnly");
-                    }
+                    Logger(`The remote database has been cleaned.`, showMessage ? LOG_LEVEL.NOTICE : LOG_LEVEL.INFO);
+                    await runWithLock("cleanup", true, async () => {
+                        const count = await purgeUnreferencedChunks(this.localDatabase.localDatabase, true);
+                        const message = `The remote database has been cleaned up.
+To synchronize, this device must be also cleaned up. ${count} chunk(s) will be erased from this device.
+However, If there are many chunks to be deleted, maybe fetching again is faster.
+We will lose the history of this device if we fetch the remote database again.
+Even if you choose to clean up, you will see this option again if you exit Obsidian and then synchronise again.`
+                        const CHOICE_FETCH = "Fetch again";
+                        const CHOICE_CLEAN = "Cleanup";
+                        const CHOICE_DISMISS = "Dismiss";
+                        const ret = await confirmWithMessage(this, "Cleaned", message, [CHOICE_FETCH, CHOICE_CLEAN, CHOICE_DISMISS], CHOICE_DISMISS, 30);
+                        if (ret == CHOICE_FETCH) {
+                            await performRebuildDB(this, "localOnly");
+                        }
+                        if (ret == CHOICE_CLEAN) {
+                            const remoteDB = await this.getReplicator().connectRemoteCouchDBWithSetting(this.settings, this.getIsMobile(), true);
+                            if (typeof remoteDB == "string") {
+                                Logger(remoteDB, LOG_LEVEL.NOTICE);
+                                return false;
+                            }
+
+                            await purgeUnreferencedChunks(this.localDatabase.localDatabase, false);
+                            // Perform the synchronisation once.
+                            if (await this.replicator.openReplication(this.settings, false, showMessage, true)) {
+                                await balanceChunkPurgedDBs(this.localDatabase.localDatabase, remoteDB.db);
+                                await purgeUnreferencedChunks(this.localDatabase.localDatabase, false);
+                                await this.getReplicator().markRemoteResolved(this.settings);
+                                Logger("The local database has been cleaned up.", showMessage ? LOG_LEVEL.NOTICE : LOG_LEVEL.INFO)
+                            } else {
+                                Logger("Replication has been cancelled. Please try it again.", showMessage ? LOG_LEVEL.NOTICE : LOG_LEVEL.INFO)
+                            }
+
+                        }
+                    });
                 } else {
                     const message = `
 The remote database has been rebuilt.
@@ -2166,7 +2219,7 @@ Or if you are sure know what had been happened, we can unlock the database from
                         setTimeout(() => {
                             //resolved, check again.
                             this.showIfConflicted(filename);
-                        }, 500);
+                        }, 50);
                     } else if (toDelete == null) {
                         Logger("Leave it still conflicted");
                     } else {
@@ -2179,7 +2232,7 @@ Or if you are sure know what had been happened, we can unlock the database from
                         setTimeout(() => {
                             //resolved, check again.
                             this.showIfConflicted(filename);
-                        }, 500);
+                        }, 50);
                     }
 
                     return res(true);
@@ -2223,7 +2276,7 @@ Or if you are sure know what had been happened, we can unlock the database from
                 Logger("conflict:Automatically merged, but we have to check it again");
                 setTimeout(() => {
                     this.showIfConflicted(filename);
-                }, 500);
+                }, 50);
                 return;
             }
             //there conflicts, and have to resolve ;
@@ -2490,6 +2543,33 @@ Or if you are sure know what had been happened, we can unlock the database from
             return this.localDatabase.isTargetFile(file);
         }
     }
+    async dryRunGC() {
+        await runWithLock("cleanup", true, async () => {
+            const remoteDBConn = await this.getReplicator().connectRemoteCouchDBWithSetting(this.settings, this.isMobile)
+            if (typeof (remoteDBConn) == "string") {
+                Logger(remoteDBConn);
+                return;
+            }
+            await purgeUnreferencedChunks(remoteDBConn.db, true, this.settings, false);
+            await purgeUnreferencedChunks(this.localDatabase.localDatabase, true);
+        });
+    }
 
+    async dbGC() {
+        // Lock the remote completely once.
+        await runWithLock("cleanup", true, async () => {
+            this.getReplicator().markRemoteLocked(this.settings, true, true);
+            const remoteDBConn = await this.getReplicator().connectRemoteCouchDBWithSetting(this.settings, this.isMobile)
+            if (typeof (remoteDBConn) == "string") {
+                Logger(remoteDBConn);
+                return;
+            }
+            await purgeUnreferencedChunks(remoteDBConn.db, false, this.settings, true);
+            await purgeUnreferencedChunks(this.localDatabase.localDatabase, false);
+            await balanceChunkPurgedDBs(this.localDatabase.localDatabase, remoteDBConn.db);
+            this.localDatabase.refreshSettings();
+            Logger("The remote database has been cleaned up! Other devices will be cleaned up on the next synchronisation.")
+        });
+    }
 }
 

src/utils.ts

@@ -2,11 +2,10 @@ import { type DataWriteOptions, normalizePath, TFile, Platform, TAbstractFile, A
 import { path2id_base, id2path_base, isValidFilenameInLinux, isValidFilenameInDarwin, isValidFilenameInWidows, isValidFilenameInAndroid, stripAllPrefixes } from "./lib/src/path";
 
 import { Logger } from "./lib/src/logger";
-import { type AnyEntry, type DocumentID, type EntryDoc, type EntryHasPath, type FilePath, type FilePathWithPrefix, LOG_LEVEL, type NewEntry } from "./lib/src/types";
+import { type AnyEntry, type DocumentID, type EntryHasPath, type FilePath, type FilePathWithPrefix, LOG_LEVEL } from "./lib/src/types";
 import { CHeader, ICHeader, ICHeaderLength, PSCHeader } from "./types";
 import { InputStringDialog, PopoverSelectString } from "./dialogs";
 import ObsidianLiveSyncPlugin from "./main";
-import { runWithLock } from "./lib/src/lock";
 
 // For backward compatibility, using the path for determining id.
 // Only CouchDB unacceptable ID (that starts with an underscore) has been prefixed with "/".
@@ -431,20 +430,15 @@ export class PeriodicProcessor {
     enable(interval: number) {
         this.disable();
         if (interval == 0) return;
-        this._timer = window.setInterval(() => this._process().then(() => { }), interval);
+        this._timer = window.setInterval(() => this.process().then(() => { }), interval);
         this._plugin.registerInterval(this._timer);
     }
     disable() {
-        if (this._timer) clearInterval(this._timer);
+        if (this._timer !== undefined) window.clearInterval(this._timer);
+        this._timer = undefined;
     }
 }
 
-function sizeToHumanReadable(size: number | undefined) {
-    if (!size) return "-";
-    const i = Math.floor(Math.log(size) / Math.log(1024));
-    return Number.parseInt((size / Math.pow(1024, i)).toFixed(2)) + ' ' + ['B', 'kB', 'MB', 'GB', 'TB'][i];
-}
-
 export const _requestToCouchDBFetch = async (baseUri: string, username: string, password: string, path?: string, body?: string | any, method?: string) => {
     const utf8str = String.fromCharCode.apply(null, new TextEncoder().encode(`${username}:${password}`));
     const encoded = window.btoa(utf8str);
@@ -492,249 +486,3 @@ export async function performRebuildDB(plugin: ObsidianLiveSyncPlugin, method: "
         await plugin.addOnSetup.rebuildEverything();
     }
 }
-
-export const gatherChunkUsage = async (db: PouchDB.Database<EntryDoc>) => {
-    const used = new Map();
-    const unreferenced = new Map();
-    const removed = new Map();
-    const missing = new Map();
-    const xx = await db.allDocs({ startkey: "h:", endkey: `h:\u{10ffff}` });
-    for (const xxd of xx.rows) {
-        const chunk = xxd.id
-        unreferenced.set(chunk, xxd.value.rev);
-    }
-
-    const x = await db.find({ limit: 999999999, selector: { children: { $exists: true, $type: "array" } }, fields: ["_id", "path", "mtime", "children"] });
-    for (const temp of x.docs) {
-        for (const chunk of (temp as NewEntry).children) {
-            used.set(chunk, (used.has(chunk) ? used.get(chunk) : 0) + 1);
-            if (unreferenced.has(chunk)) {
-                removed.set(chunk, unreferenced.get(chunk));
-                unreferenced.delete(chunk);
-            } else {
-                if (!removed.has(chunk)) {
-                    if (!missing.has(temp._id)) {
-                        missing.set(temp._id, []);
-                    }
-                    missing.get(temp._id).push(chunk);
-                }
-            }
-        }
-    }
-
-    return { used, unreferenced, missing };
-}
-
-export const localDatabaseCleanUp = async (plugin: ObsidianLiveSyncPlugin, force: boolean, dryRun: boolean) => {
-
-    await runWithLock("clean-up:local", true, async () => {
-        const db = plugin.localDatabase.localDatabase;
-        if ((db as any)?.adapter != "indexeddb") {
-            if (force && !dryRun) {
-                Logger("Fetch from the remote database", LOG_LEVEL.NOTICE, "clean-up-db");
-                await performRebuildDB(plugin, "localOnly");
-                return;
-            } else {
-                Logger("This feature requires disabling `Use an old adapter for compatibility`.", LOG_LEVEL.NOTICE, "clean-up-db");
-                return;
-            }
-        }
-        Logger(`The remote database has been locked for garbage collection`, LOG_LEVEL.NOTICE, "clean-up-db");
-        Logger(`Gathering chunk usage information`, LOG_LEVEL.NOTICE, "clean-up-db");
-
-        const { unreferenced, missing } = await gatherChunkUsage(db);
-        if (missing.size != 0) {
-            Logger(`Some chunks are not found! We have to rescue`, LOG_LEVEL.NOTICE);
-            Logger(missing, LOG_LEVEL.VERBOSE);
-        } else {
-            Logger(`All chunks are OK`, LOG_LEVEL.NOTICE);
-        }
-        const payload = {} as Record<string, string[]>;
-        for (const [id, rev] of unreferenced) {
-            payload[id] = [rev];
-        }
-        const removeItems = Object.keys(payload).length;
-        if (removeItems == 0) {
-            Logger(`No unreferenced chunks found (Local)`, LOG_LEVEL.NOTICE);
-            await plugin.markRemoteResolved();
-        }
-        if (dryRun) {
-            Logger(`There are ${removeItems} unreferenced chunks (Local)`, LOG_LEVEL.NOTICE);
-            return;
-        }
-        Logger(`Deleting unreferenced chunks: ${removeItems}`, LOG_LEVEL.NOTICE, "clean-up-db");
-        for (const [id, rev] of unreferenced) {
-            //@ts-ignore
-            const ret = await db.purge(id, rev);
-            Logger(ret, LOG_LEVEL.VERBOSE);
-        }
-        plugin.localDatabase.refreshSettings();
-        Logger(`Compacting local database...`, LOG_LEVEL.NOTICE, "clean-up-db");
-        await db.compact();
-        await plugin.markRemoteResolved();
-        Logger("Done!", LOG_LEVEL.NOTICE, "clean-up-db");
-    })
-}
-
-
-export const balanceChunks = async (plugin: ObsidianLiveSyncPlugin, dryRun: boolean) => {
-
-    await runWithLock("clean-up:balance", true, async () => {
-        const localDB = plugin.localDatabase.localDatabase;
-        Logger(`Gathering chunk usage information`, LOG_LEVEL.NOTICE, "clean-up-db");
-
-        const ret = await plugin.replicator.connectRemoteCouchDBWithSetting(plugin.settings, plugin.isMobile);
-        if (typeof ret === "string") {
-            Logger(`Connect error: ${ret}`, LOG_LEVEL.NOTICE, "clean-up-db");
-            return;
-        }
-        const localChunks = new Map<string, string>();
-        const xx = await localDB.allDocs({ startkey: "h:", endkey: `h:\u{10ffff}` });
-        for (const xxd of xx.rows) {
-            const chunk = xxd.id
-            localChunks.set(chunk, xxd.value.rev);
-        }
-        // const info = ret.info;
-        const remoteDB = ret.db;
-        const remoteChunks = new Map<string, string>();
-        const xxr = await remoteDB.allDocs({ startkey: "h:", endkey: `h:\u{10ffff}` });
-        for (const xxd of xxr.rows) {
-            const chunk = xxd.id
-            remoteChunks.set(chunk, xxd.value.rev);
-        }
-        const localToRemote = new Map<string, string>([...localChunks]);
-        const remoteToLocal = new Map<string, string>([...remoteChunks]);
-        for (const id of new Set([...localChunks.keys(), ...remoteChunks.keys()])) {
-            if (remoteChunks.has(id)) {
-                localToRemote.delete(id);
-            }
-            if (localChunks.has(id)) {
-                remoteToLocal.delete(id);
-            }
-        }
-
-        function arrayToChunkedArray<T>(src: T[], size = 25) {
-            const ret = [] as T[][];
-            let i = 0;
-            while (i < src.length) {
-                ret.push(src.slice(i, i += size));
-            }
-            return ret;
-        }
-
-        if (localToRemote.size == 0) {
-            Logger(`No chunks need to be sent`, LOG_LEVEL.NOTICE);
-        } else {
-            Logger(`${localToRemote.size} chunks need to be sent`, LOG_LEVEL.NOTICE);
-            if (!dryRun) {
-                const w = arrayToChunkedArray([...localToRemote]);
-                for (const chunk of w) {
-                    for (const [id,] of chunk) {
-                        const queryRet = await localDB.allDocs({ keys: [id], include_docs: true });
-                        const docs = queryRet.rows.filter(e => !("error" in e)).map(x => x.doc);
-
-                        const ret = await remoteDB.bulkDocs(docs, { new_edits: false });
-                        Logger(ret, LOG_LEVEL.VERBOSE);
-                    }
-                }
-                Logger(`Done! ${remoteToLocal.size} chunks have been sent`, LOG_LEVEL.NOTICE);
-            }
-        }
-        if (remoteToLocal.size == 0) {
-            Logger(`No chunks need to be retrieved`, LOG_LEVEL.NOTICE);
-        } else {
-            Logger(`${remoteToLocal.size} chunks need to be retrieved`, LOG_LEVEL.NOTICE);
-            if (!dryRun) {
-                const w = arrayToChunkedArray([...remoteToLocal]);
-                for (const chunk of w) {
-                    for (const [id,] of chunk) {
-                        const queryRet = await remoteDB.allDocs({ keys: [id], include_docs: true });
-                        const docs = queryRet.rows.filter(e => !("error" in e)).map(x => x.doc);
-
-                        const ret = await localDB.bulkDocs(docs, { new_edits: false });
-                        Logger(ret, LOG_LEVEL.VERBOSE);
-                    }
-                }
-                Logger(`Done! ${remoteToLocal.size} chunks have been retrieved`, LOG_LEVEL.NOTICE);
-            }
-        }
-    })
-}
-
-export const remoteDatabaseCleanup = async (plugin: ObsidianLiveSyncPlugin, dryRun: boolean) => {
-    const getSize = function (info: PouchDB.Core.DatabaseInfo, key: "active" | "external" | "file") {
-        return Number.parseInt((info as any)?.sizes?.[key] ?? 0);
-    }
-    await runWithLock("clean-up:remote", true, async () => {
-        const CHUNK_SIZE = 100;
-        function makeChunkedArrayFromArray<T>(items: T[]): T[][] {
-            const chunked = [];
-            for (let i = 0; i < items.length; i += CHUNK_SIZE) {
-                chunked.push(items.slice(i, i + CHUNK_SIZE));
-            }
-            return chunked;
-        }
-        try {
-            const ret = await plugin.replicator.connectRemoteCouchDBWithSetting(plugin.settings, plugin.isMobile);
-            if (typeof ret === "string") {
-                Logger(`Connect error: ${ret}`, LOG_LEVEL.NOTICE, "clean-up-db");
-                return;
-            }
-            const info = ret.info;
-            Logger(JSON.stringify(info), LOG_LEVEL.VERBOSE, "clean-up-db");
-            Logger(`Database active-size: ${sizeToHumanReadable(getSize(info, "active"))}, external-size:${sizeToHumanReadable(getSize(info, "external"))}, file-size: ${sizeToHumanReadable(getSize(info, "file"))}`, LOG_LEVEL.NOTICE);
-
-            if (!dryRun) {
-                Logger(`The remote database has been locked for garbage collection`, LOG_LEVEL.NOTICE, "clean-up-db");
-                await plugin.markRemoteLocked(true);
-            }
-            Logger(`Gathering chunk usage information`, LOG_LEVEL.NOTICE, "clean-up-db");
-            const db = ret.db;
-
-            const { unreferenced, missing } = await gatherChunkUsage(db);
-            if (missing.size != 0) {
-                Logger(`Some chunks are not found! We have to rescue`, LOG_LEVEL.NOTICE);
-                Logger(missing, LOG_LEVEL.VERBOSE);
-            } else {
-                Logger(`All chunks are OK`, LOG_LEVEL.NOTICE);
-            }
-            const payload = {} as Record<string, string[]>;
-            for (const [id, rev] of unreferenced) {
-                payload[id] = [rev];
-            }
-            const removeItems = Object.keys(payload).length;
-            if (removeItems == 0) {
-                Logger(`No unreferenced chunk found (Remote)`, LOG_LEVEL.NOTICE);
-                return;
-            }
-            if (dryRun) {
-                Logger(`There are ${removeItems} unreferenced chunks (Remote)`, LOG_LEVEL.NOTICE);
-                return;
-            }
-            Logger(`Deleting unreferenced chunks: ${removeItems}`, LOG_LEVEL.NOTICE, "clean-up-db");
-            const buffer = makeChunkedArrayFromArray(Object.entries(payload));
-            for (const chunkedPayload of buffer) {
-                const rets = await _requestToCouchDBFetch(
-                    `${plugin.settings.couchDB_URI}/${plugin.settings.couchDB_DBNAME}`,
-                    plugin.settings.couchDB_USER,
-                    plugin.settings.couchDB_PASSWORD,
-                    "_purge",
-                    chunkedPayload.reduce((p, c) => ({ ...p, [c[0]]: c[1] }), {}), "POST");
-                // const result = await rets();
-                Logger(JSON.stringify(await rets.json()), LOG_LEVEL.VERBOSE);
-            }
-            Logger(`Compacting database...`, LOG_LEVEL.NOTICE, "clean-up-db");
-            await db.compact();
-            const endInfo = await db.info();
-
-            Logger(`Processed database active-size: ${sizeToHumanReadable(getSize(endInfo, "active"))}, external-size:${sizeToHumanReadable(getSize(endInfo, "external"))}, file-size: ${sizeToHumanReadable(getSize(endInfo, "file"))}`, LOG_LEVEL.NOTICE);
-            Logger(`Reduced sizes: active-size: ${sizeToHumanReadable(getSize(info, "active") - getSize(endInfo, "active"))}, external-size:${sizeToHumanReadable(getSize(info, "external") - getSize(endInfo, "external"))}, file-size: ${sizeToHumanReadable(getSize(info, "file") - getSize(endInfo, "file"))}`, LOG_LEVEL.NOTICE);
-            Logger(JSON.stringify(endInfo), LOG_LEVEL.VERBOSE, "clean-up-db");
-            Logger(`Local database cleaning up...`);
-            await localDatabaseCleanUp(plugin, true, false);
-        } catch (ex) {
-            Logger("Failed to clean up db.")
-            Logger(ex, LOG_LEVEL.VERBOSE);
-        }
-    });
-}

styles.css

@@ -260,3 +260,14 @@ div.sls-setting-menu-btn {
 .password-input > .setting-item-control >input {
   -webkit-text-security: disc;
 }
+
+span.ls-mark-cr::after {
+    user-select: none;
+    content: "↲";
+    color: var(--text-muted);
+    font-size: 0.8em;
+}
+
+.deleted span.ls-mark-cr::after {
+    color: var(--text-on-accent);
+}

updates.md

@@ -14,32 +14,37 @@ I hope you will give it a try.
 
 #### Minors
 
-- 0.19.1 to 0.19.6 has been moved into the updates_old.md
-- 0.19.7
-  - Fixed:
-    - The initial pane of Setting dialogue is now changed to General Settings.
-    - The Setup Wizard is now able to flush existing settings and get into the mode again.
-- 0.19.8
-  - New feature:
-    - Vault history: A tab has been implemented to give a birds-eye view of the changes that have occurred in the vault.
+- 0.19.1 to 0.19.11 has been moved into the updates_old.md
+- 0.19.12
   - Improved:
-    - Now the passphrases on the dialogue masked out. Thank you @antoKeinanen!
-    - Log dialogue is now shown as one of tabs.
+    - Boot-up performance has been improved.
+    - Customisation sync performance has been improved.
+    - Synchronising performance has been improved.
+- 0.19.13
+  - Implemented:
+    - Database clean-up is now in beta 2!
+      We can shrink the remote database by deleting unused chunks, with keeping history.
+      Note: Local database is not cleaned up totally. We have to `Fetch` again to let it done.
+      **Note2**: Still in beta. Please back your vault up anything before.
   - Fixed:
-    - Some minor issues has been fixed.
-- 0.19.9
-  - New feature (For fixing a problem):
-    - We can fix the database obfuscated and plain paths that have been mixed up.
-  - Improvements
-    - Customisation Sync performance has been improved.
-- 0.19.10
-  - Fixed
-    - Fixed the issue about fixing the database.
-- 0.19.11
-  - Improvements:
-    - Hashing ChunkID has been improved.
-    - Logging keeps 400 lines now.
-  - Refactored:
-    - Import statement has been fixed about types.
+    - The log updates are not thinned out now.
+- 0.19.14
+  - Fixed:
+    - Internal documents are now ignored.
+    - Merge dialogue now respond immediately to button pressing.
+    - Periodic processing now works fine.
+    - The checking interval of detecting conflicted has got shorter.
+    - Replication is now cancelled while cleaning up.
+    - The database locking by the cleaning up is now carefully unlocked.
+    - Missing chunks message is correctly reported.
+  - New feature:
+    - Suspend database reflecting has been implemented.
+      - This can be disabled by `Fetch database with previous behaviour`.
+    - Now fetch suspends the reflecting database and storage changes temporarily to improve the performance.
+    - We can choose the action when the remote database has been cleaned
+    - Merge dialogue now show `↲` before the new line.
+  - Improved:
+    - Now progress is reported while the cleaning up and fetch process.
+    - Cancelled replication is now detected.
 
 ... To continue on to `updates_old.md`.

updates_old.md

@@ -56,7 +56,32 @@ I hope you will give it a try.
       - Items that have no meaning in the settings are no longer displayed.
       - Some items have been reordered for clarity.
       - Each configuration has been grouped.
-
+- 0.19.7
+  - Fixed:
+    - The initial pane of Setting dialogue is now changed to General Settings.
+    - The Setup Wizard is now able to flush existing settings and get into the mode again.
+- 0.19.8
+  - New feature:
+    - Vault history: A tab has been implemented to give a birds-eye view of the changes that have occurred in the vault.
+  - Improved:
+    - Now the passphrases on the dialogue masked out. Thank you @antoKeinanen!
+    - Log dialogue is now shown as one of tabs.
+  - Fixed:
+    - Some minor issues has been fixed.
+- 0.19.9
+  - New feature (For fixing a problem):
+    - We can fix the database obfuscated and plain paths that have been mixed up.
+  - Improvements
+    - Customisation Sync performance has been improved.
+- 0.19.10
+  - Fixed
+    - Fixed the issue about fixing the database.
+- 0.19.11
+  - Improvements:
+    - Hashing ChunkID has been improved.
+    - Logging keeps 400 lines now.
+  - Refactored:
+    - Import statement has been fixed about types.
 ### 0.18.0
 
 #### Now, paths of files in the database can now be obfuscated. (Experimental Feature)