Add some note.

- Fixed ignoring changes on replicating.
- Disabled `Skip old files on sync` temporary (so fragile between multiple devices)

README.md

@@ -36,23 +36,63 @@ Especially, in some companies, people have to store all data to their fully cont
 -   When the device's storage has been run out, Database corruption may happen.
 -   When editing hidden files or any other invisible files from obsidian, the file wouldn't be kept in the database. (**Or be deleted.**)
 
-## Supplements
-
--   When the file has been deleted, the deletion of the file is replicated to other devices.
--   When the folder became empty by replication, The folder will be deleted in the default setting. But you can change this behaivour. Check the [Settings](docs/settings.md).
--   LiveSync drains many batteries in mobile devices.
--   Mobile Obsidian can not connect to the non-secure(HTTP) or local CA-signed servers, even though the certificate is stored in the device store.
--   There are no 'exclude_folders' like configurations.
-
 ## How to use
 
-1. Install from Obsidian, or download from this repo's releases, copy `main.js`, `styles.css` and `manifest.json` into `[your-vault]/.obsidian/plugins/`
-2. Get your database. IBM Cloudant is preferred for testing. Or you can use your own server with CouchDB.
-   For more information, refer below:
-    1. [Setup IBM Cloudant](docs/setup_cloudant.md)
-    2. [Setup your CouchDB](docs/setup_own_server.md)
-3. Enter connection information to Plugin's setting dialog. In details, refer [Settings of Self-hosted LiveSync](docs/settings.md)
-4. Enable LiveSync or other Synchronize method as you like.
+
+### Get your database ready.
+
+First, get your database ready. IBM Cloudant is preferred for testing. Or you can use your own server with CouchDB. For more information, refer below:
+    1. [Setup IBM Cloudant](docs/setup_cloudant.md)
+    2. [Setup your CouchDB](docs/setup_own_server.md)
+
+### First device
+
+1. Install the plugin on your device.
+2. Configure with the remote database.
+	1. Fill your server's information into the `Remote Database configuration pane`.
+	2. Enabling `End to End Encryption is recommended. After inputting the passphrase, you have to press `Just apply`.
+	3. Hit `Test Database Connection` and make sure that the plugin says `Connected`.
+	4. Hit `Check database configuration` and make sure all tests have been passed.
+3. Configure how to synchronize on `Sync setting`. (You can leave these  configures later)
+	1. If you want to synchronize in real-time, enable `LiveSync`.
+	2. Or, set up the synchronization as you like.
+	3. Additional configuration is also here. I recommend enabling `Use Trash for deleted files, but you can leave all configurations disabled.
+4. Configure miscellaneous features.
+	1. Enabling `Show staus inside editor` bring you information. While edit mode, you can see the status on the top-right of the editor. (Recommended)
+	2. Enabling `Use history` let you see the diffs between your edit and synchronization. (Recommended)
+5. Back to the editor. I hope that initial scan is in the progress or done.
+6. When status became stabilized (All ⏳ and 🧩 disappeared), you are ready to synchronize with the server.
+7. Press the replicate icon on the Ribbon or run `Replicate now` from the Command pallet. You'll send all your data to the server.
+8. Open the command palette and run `Copy setup uri`. And share copied URI to your other devices.
+**IMPORTANT NOTICE: DO NOT SHARE THIS URI. THIS CONTAINS YOUR CREDENTIALS.**
+
+### Subsequent Devices
+
+Strongly recommend using the vault in which all files are completely synchronized including timestamps. Otherwise, some files will be corrupted if failed to resolve conflicts. To simplify, I recommend using a new empty vault.
+
+1. Install the plug-in.
+2. Open the link that you had been copied to the other device.
+3. The plug-in asks you that are you sure to apply the configurations. Please answer `Yes` and the following instruction below:
+	1. Answer `Yes` to `Keep local DB?`.
+		*Note: If you started with existed vault, you have to answer `No`. And `No` to `Rebuild the database?`.*
+	2. Answer `Yes` to `Keep remote DB?`.
+	3. Answer `Yes` to `Replicate once?`.
+	Yes, you have to answer `Yes` to everything.
+	Then, all your settings are copied from the first device.
+4. Your notes will arrive soon.
+
+## Something looks corrupted...
+
+Please open the link again and Answer as below:
+- If your local database looks corrupted
+(in the other words, your Obsidian getting werid even off the line)
+	- Answer `No` to `Keep local DB?`
+- If your remote database looks corrupted
+(in the other words, something happens while replicating)
+	- Answer `No` to `Keep remote DB?`
+
+If you answered `No` to both, your databases will be rebuilt by the content on your device. And the remote database will lock out other devices. You have to synchronize all your devices again. (When this time, almost all your files should be synchronized including a timestamp. So you can use the existed vault).
+
 
 ## Test Server
 
@@ -76,10 +116,17 @@ Synchronization status is shown in statusbar.
 -   ↑ Uploaded pieces
 -   ↓ Downloaded pieces
 -   ⏳ Count of the pending process  
+-   🧩 Count of the files that waiting for their chunks.
 If you have deleted or renamed files, please wait until this disappears.
 
-# More supplements
 
+## Supplements
+(This section will be rewrited soon)
+-   When the file has been deleted, the deletion of the file is replicated to other devices.
+-   When the folder became empty by replication, The folder will be deleted in the default setting. But you can change this behaivour. Check the [Settings](docs/settings.md).
+-   LiveSync drains many batteries in mobile devices.
+-   Mobile Obsidian can not connect to the non-secure(HTTP) or local CA-signed servers, even though the certificate is stored in the device store.
+-   There are no 'exclude_folders' like configurations.
 -   When synchronized, files are compared by their modified times and overwritten by the newer ones once. Then plugin checks the conflicts and if a merge is needed, the dialog will open.
 -   Rarely, the file in the database would be broken. The plugin will not write storage when it looks broken, so some old files must be on your device. If you edit the file, it will be cured. But if the file does not exist on any device, can not rescue it. So you can delete these items from the setting dialog.
 -   If your database looks corrupted, try "Drop History". Usually, It is the easiest way.

manifest.json

@@ -1,7 +1,7 @@
 {
     "id": "obsidian-livesync",
     "name": "Self-hosted LiveSync",
-    "version": "0.10.0",
+    "version": "0.11.4",
     "minAppVersion": "0.9.12",
     "description": "Community implementation of self-hosted livesync. Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.",
     "author": "vorotamoroz",

package-lock.json

@@ -1,12 +1,12 @@
 {
     "name": "obsidian-livesync",
-    "version": "0.10.0",
+    "version": "0.11.4",
     "lockfileVersion": 2,
     "requires": true,
     "packages": {
         "": {
             "name": "obsidian-livesync",
-            "version": "0.10.0",
+            "version": "0.11.4",
             "license": "MIT",
             "dependencies": {
                 "diff-match-patch": "^1.0.5",

package.json

@@ -1,6 +1,6 @@
 {
     "name": "obsidian-livesync",
-    "version": "0.10.0",
+    "version": "0.11.4",
     "description": "Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.",
     "main": "main.js",
     "type": "module",
@@ -37,7 +37,7 @@
         "diff-match-patch": "^1.0.5",
         "esbuild": "0.13.12",
         "esbuild-svelte": "^0.6.0",
-		"idb": "^7.0.1",
+        "idb": "^7.0.1",
         "svelte-preprocess": "^4.10.2",
         "xxhash-wasm": "^0.4.2"
     }

src/KeyValueDB.ts

@@ -0,0 +1,50 @@
+import { deleteDB, IDBPDatabase, openDB } from "idb";
+export interface KeyValueDatabase {
+    get<T>(key: string): Promise<T>;
+    set<T>(key: string, value: T): Promise<IDBValidKey>;
+    del(key: string): Promise<void>;
+    clear(): Promise<void>;
+    keys(query?: IDBValidKey | IDBKeyRange, count?: number): Promise<IDBValidKey[]>;
+    close(): void;
+    destroy(): void;
+}
+const databaseCache: { [key: string]: IDBPDatabase<any> } = {};
+export const OpenKeyValueDatabase = (dbKey: string): KeyValueDatabase => {
+    if (dbKey in databaseCache) {
+        databaseCache[dbKey].close();
+        delete databaseCache[dbKey];
+    }
+    const storeKey = dbKey;
+    const dbPromise = openDB(dbKey, 1, {
+        upgrade(db) {
+            db.createObjectStore(storeKey);
+        },
+    });
+    ~(async () => (databaseCache[dbKey] = await dbPromise))();
+    return {
+        async get<T>(key: string): Promise<T> {
+            return (await dbPromise).get(storeKey, key);
+        },
+        async set<T>(key: string, value: T) {
+            return (await dbPromise).put(storeKey, value, key);
+        },
+        async del(key: string) {
+            return (await dbPromise).delete(storeKey, key);
+        },
+        async clear() {
+            return (await dbPromise).clear(storeKey);
+        },
+        async keys(query?: IDBValidKey | IDBKeyRange, count?: number) {
+            return (await dbPromise).getAllKeys(storeKey, query, count);
+        },
+        async close() {
+            delete databaseCache[dbKey];
+            return (await dbPromise).close();
+        },
+        async destroy() {
+            delete databaseCache[dbKey];
+            (await dbPromise).close();
+            await deleteDB(dbKey);
+        },
+    };
+};

src/LocalPouchDB.ts

@@ -16,7 +16,6 @@ import {
     MAX_DOC_SIZE,
     MAX_DOC_SIZE_BIN,
     NODEINFO_DOCID,
-    RECENT_MOFIDIED_DOCS_QTY,
     VER,
     MILSTONE_DOCID,
     DatabaseConnectingStatus,
@@ -27,6 +26,7 @@ import { path2id } from "./utils";
 import { Logger } from "./lib/src/logger";
 import { checkRemoteVersion, connectRemoteCouchDBWithSetting, getLastPostFailedBySize } from "./utils_couchdb";
 import { openDB, deleteDB, IDBPDatabase } from "idb";
+import { KeyValueDatabase, OpenKeyValueDatabase } from "./KeyValueDB";
 
 type ReplicationCallback = (e: PouchDB.Core.ExistingDocument<EntryDoc>[]) => Promise<void>;
 class LRUCache {
@@ -76,6 +76,7 @@ export class LocalPouchDB {
     dbname: string;
     settings: RemoteDBSettings;
     localDatabase: PouchDB.Database<EntryDoc>;
+    kvDB: KeyValueDatabase;
     nodeid = "";
     isReady = false;
 
@@ -116,6 +117,7 @@ export class LocalPouchDB {
         return null;
     }
     onunload() {
+        this.kvDB.close();
         this.recentModifiedDocs = [];
         this.leafArrivedCallbacks;
         this.changeHandler = this.cancelHandler(this.changeHandler);
@@ -140,25 +142,13 @@ export class LocalPouchDB {
         if (this.localDatabase != null) {
             this.localDatabase.close();
         }
+        this.kvDB.close();
     }
 
-    updateRecentModifiedDocs(id: string, rev: string, deleted: boolean) {
-        const idrev = id + rev;
-        if (deleted) {
-            this.recentModifiedDocs = this.recentModifiedDocs.filter((e) => e != idrev);
-        } else {
-            this.recentModifiedDocs.push(idrev);
-            this.recentModifiedDocs = this.recentModifiedDocs.slice(0 - RECENT_MOFIDIED_DOCS_QTY);
-        }
-    }
-    isSelfModified(id: string, rev: string): boolean {
-        const idrev = id + rev;
-        return this.recentModifiedDocs.indexOf(idrev) !== -1;
-    }
     async isOldDatabaseExists() {
         const db = new PouchDB<EntryDoc>(this.dbname + "-livesync", {
             auto_compaction: this.settings.useHistory ? false : true,
-            revs_limit: 100,
+            revs_limit: 20,
             deterministic_revs: true,
             skip_setup: true,
         });
@@ -181,6 +171,7 @@ export class LocalPouchDB {
             revs_limit: 100,
             deterministic_revs: true,
         });
+        this.kvDB = OpenKeyValueDatabase(this.dbname + "-livesync-kv");
         Logger("Database info", LOG_LEVEL.VERBOSE);
         Logger(await this.localDatabase.info(), LOG_LEVEL.VERBOSE);
         Logger("Open Database...");
@@ -253,9 +244,9 @@ export class LocalPouchDB {
                 Logger(newDbStatus);
 
                 if (this.settings.encrypt) {
-                    enableEncryption(old, this.settings.passphrase);
+                    enableEncryption(old, this.settings.passphrase, true);
                 }
-                const rep = old.replicate.to(this.localDatabase);
+                const rep = old.replicate.to(this.localDatabase, { batch_size: 25, batches_limit: 10 });
                 rep.on("change", (e) => {
                     progress.setMessage(`Converting ${e.docs_written} docs...`);
                     Logger(`Converting ${e.docs_written} docs...`, LOG_LEVEL.VERBOSE);
@@ -267,7 +258,7 @@ export class LocalPouchDB {
                     Logger("Conversion completed!", LOG_LEVEL.NOTICE);
                     old.destroy(); // delete the old database.
                     this.isReady = true;
-                    return nextSeq();
+                    return await nextSeq();
                 } else {
                     throw new Error("Conversion failed!");
                 }
@@ -279,7 +270,7 @@ export class LocalPouchDB {
                 return false;
             }
         } else {
-            return nextSeq();
+            return await nextSeq();
         }
     }
 
@@ -305,7 +296,7 @@ export class LocalPouchDB {
     waitForLeafReady(id: string): Promise<boolean> {
         return new Promise((res, rej) => {
             // Set timeout.
-            const timer = setTimeout(() => rej(new Error(`Leaf timed out:${id}`)), LEAF_WAIT_TIMEOUT);
+            const timer = setTimeout(() => rej(new Error(`Chunk reading timed out:${id}`)), LEAF_WAIT_TIMEOUT);
             if (typeof this.leafArrivedCallbacks[id] == "undefined") {
                 this.leafArrivedCallbacks[id] = [];
             }
@@ -329,21 +320,21 @@ export class LocalPouchDB {
                 this.hashCaches.set(id, w.data);
                 return w.data;
             }
-            throw new Error(`retrive leaf, but it was not leaf.`);
+            throw new Error(`Corrupted chunk detected: ${id}`);
         } catch (ex) {
             if (ex.status && ex.status == 404) {
                 if (waitForReady) {
                     // just leaf is not ready.
                     // wait for on
                     if ((await this.waitForLeafReady(id)) === false) {
-                        throw new Error(`time out (waiting leaf)`);
+                        throw new Error(`time out (waiting chunk)`);
                     }
                     return this.getDBLeaf(id, false);
                 } else {
-                    throw new Error("Leaf was not found");
+                    throw new Error(`Chunk was not found: ${id}`);
                 }
             } else {
-                Logger(`Something went wrong on retriving leaf`);
+                Logger(`Something went wrong on retriving chunk`);
                 throw ex;
             }
         }
@@ -447,11 +438,11 @@ export class LocalPouchDB {
                     try {
                         childrens = await Promise.all(obj.children.map((e) => this.getDBLeaf(e, waitForReady)));
                         if (dump) {
-                            Logger(`childrens:`);
+                            Logger(`Chunks:`);
                             Logger(childrens);
                         }
                     } catch (ex) {
-                        Logger(`Something went wrong on reading elements of ${obj._id} from database:`, LOG_LEVEL.NOTICE);
+                        Logger(`Something went wrong on reading chunks of ${obj._id} from database, see verbose info for detail.`, LOG_LEVEL.NOTICE);
                         Logger(ex, LOG_LEVEL.VERBOSE);
                         this.corruptedEntries[obj._id] = obj;
                         return false;
@@ -515,7 +506,7 @@ export class LocalPouchDB {
                 if (!obj.type || (obj.type && obj.type == "notes")) {
                     obj._deleted = true;
                     const r = await this.localDatabase.put(obj);
-                    this.updateRecentModifiedDocs(r.id, r.rev, true);
+                    Logger(`entry removed:${obj._id}-${r.rev}`);
                     if (typeof this.corruptedEntries[obj._id] != "undefined") {
                         delete this.corruptedEntries[obj._id];
                     }
@@ -526,7 +517,6 @@ export class LocalPouchDB {
                     obj._deleted = true;
                     const r = await this.localDatabase.put(obj);
                     Logger(`entry removed:${obj._id}-${r.rev}`);
-                    this.updateRecentModifiedDocs(r.id, r.rev, true);
                     if (typeof this.corruptedEntries[obj._id] != "undefined") {
                         delete this.corruptedEntries[obj._id];
                     }
@@ -579,7 +569,6 @@ export class LocalPouchDB {
                     const item = await this.localDatabase.get(v);
                     item._deleted = true;
                     await this.localDatabase.put(item);
-                    this.updateRecentModifiedDocs(item._id, item._rev, true);
                 });
 
                 deleteCount++;
@@ -702,21 +691,21 @@ export class LocalPouchDB {
             try {
                 const result = await this.localDatabase.bulkDocs(newLeafs);
                 for (const item of result) {
-                    if ((item as any).ok) {
-                        this.updateRecentModifiedDocs(item.id, item.rev, false);
-                        Logger(`save ok:id:${item.id} rev:${item.rev}`, LOG_LEVEL.VERBOSE);
-                    } else {
+                    if (!(item as any).ok) {
                         if ((item as any).status && (item as any).status == 409) {
                             // conflicted, but it would be ok in childrens.
                         } else {
-                            Logger(`save failed:id:${item.id} rev:${item.rev}`, LOG_LEVEL.NOTICE);
+                            Logger(`Save failed:id:${item.id} rev:${item.rev}`, LOG_LEVEL.NOTICE);
                             Logger(item);
                             saved = false;
                         }
                     }
                 }
+                if (saved) {
+                    Logger(`Chunk saved:${newLeafs.length} chunks`);
+                }
             } catch (ex) {
-                Logger("ERROR ON SAVING LEAVES:", LOG_LEVEL.NOTICE);
+                Logger("Chunk save failed:", LOG_LEVEL.NOTICE);
                 Logger(ex, LOG_LEVEL.NOTICE);
                 saved = false;
             }
@@ -748,7 +737,6 @@ export class LocalPouchDB {
                     }
                 }
                 const r = await this.localDatabase.put(newDoc, { force: true });
-                this.updateRecentModifiedDocs(r.id, r.rev, newDoc._deleted);
                 if (typeof this.corruptedEntries[note._id] != "undefined") {
                     delete this.corruptedEntries[note._id];
                 }
@@ -1125,6 +1113,7 @@ export class LocalPouchDB {
         Logger("Database closed for reset Database.");
         this.isReady = false;
         await this.localDatabase.destroy();
+        await this.kvDB.destroy();
         this.localDatabase = null;
         await this.initializeDatabase();
         Logger("Local Database Reset", LOG_LEVEL.NOTICE);

src/ObsidianLiveSyncSettingTab.ts

@@ -710,6 +710,24 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
                 })
             );
 
+        // new Setting(containerSyncSettingEl)
+        //     .setName("Skip old files on sync")
+        //     .setDesc("Skip old incoming if incoming changes older than storage.")
+        //     .addToggle((toggle) =>
+        //         toggle.setValue(this.plugin.settings.skipOlderFilesOnSync).onChange(async (value) => {
+        //             this.plugin.settings.skipOlderFilesOnSync = value;
+        //             await this.plugin.saveSettings();
+        //         })
+        //     );
+        new Setting(containerSyncSettingEl)
+            .setName("Check conflict only on opening file.")
+            .setDesc("Do not check conflict while replication")
+            .addToggle((toggle) =>
+                toggle.setValue(this.plugin.settings.checkConflictOnlyOnOpen).onChange(async (value) => {
+                    this.plugin.settings.checkConflictOnlyOnOpen = value;
+                    await this.plugin.saveSettings();
+                })
+            );
         containerSyncSettingEl.createEl("div", {
             text: sanitizeHTMLToDom(`Advanced settings<br>
             If you reached the payload size limit when using IBM Cloudant, please set batch size and batch limit to a lower value.`),
@@ -1119,8 +1137,8 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
 
         const containerCorruptedDataEl = containerEl.createDiv();
 
-        containerCorruptedDataEl.createEl("h3", { text: "Corrupted data" });
-
+        containerCorruptedDataEl.createEl("h3", { text: "Corrupted or missing data" });
+        containerCorruptedDataEl.createEl("h4", { text: "Corrupted" });
         if (Object.keys(this.plugin.localDatabase.corruptedEntries).length > 0) {
             const cx = containerCorruptedDataEl.createEl("div", { text: "If you have copy of these items on any device, simply edit once or twice. Or not, delete this. sorry.." });
             for (const k in this.plugin.localDatabase.corruptedEntries) {
@@ -1149,6 +1167,38 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
         } else {
             containerCorruptedDataEl.createEl("div", { text: "There is no corrupted data." });
         }
+        containerCorruptedDataEl.createEl("h4", { text: "Missing or waiting" });
+        if (Object.keys(this.plugin.queuedFiles).length > 0) {
+            const cx = containerCorruptedDataEl.createEl("div", {
+                text: "These files have missing or waiting chunks. Perhaps almost chunks will be found in a while after replication. But if there're no chunk, you have to restore database entry from existed file by hitting the button below.",
+            });
+            const files = [...new Set([...this.plugin.queuedFiles.map((e) => e.entry._id)])];
+            for (const k of files) {
+                const xx = cx.createEl("div", { text: `${id2path(k)}` });
+
+                const ba = xx.createEl("button", { text: `Delete this` }, (e) => {
+                    e.addEventListener("click", async () => {
+                        await this.plugin.localDatabase.deleteDBEntry(k);
+                        xx.remove();
+                    });
+                });
+                ba.addClass("mod-warning");
+                xx.createEl("button", { text: `Restore from file` }, (e) => {
+                    e.addEventListener("click", async () => {
+                        const f = await this.app.vault.getFiles().filter((e) => path2id(e.path) == k);
+                        if (f.length == 0) {
+                            Logger("Not found in vault", LOG_LEVEL.NOTICE);
+                            return;
+                        }
+                        await this.plugin.updateIntoDB(f[0]);
+                        xx.remove();
+                    });
+                });
+                xx.addClass("mod-warning");
+            }
+        } else {
+            containerCorruptedDataEl.createEl("div", { text: "There is no missing or waiting chunk." });
+        }
         applyDisplayEnabled();
         addScreenElement("70", containerCorruptedDataEl);
         changeDisplay("0");

src/PluginPane.svelte

@@ -16,7 +16,7 @@
     export let plugin: ObsidianLiveSyncPlugin;
     let plugins: PluginDataEntry[] = [];
     let deviceAndPlugins: { [key: string]: PluginDataEntryDisp[] } = {};
-    let devicePluginList: [string, PluginDataEntryDisp[]][] = [];
+    let devicePluginList: [string, PluginDataEntryDisp[]][] = null;
     let ownPlugins: DevicePluginList = null;
     let showOwnPlugins = false;
     let targetList: { [key: string]: boolean } = {};
@@ -205,58 +205,63 @@
                 <th class="sls-plugins-tbl-device-head">Info</th>
                 <th class="sls-plugins-tbl-device-head">Target</th>
             </tr>
-            {#if devicePluginList.length == 0}
+            {#if !devicePluginList}
                 <tr>
                     <td colspan="3" class="sls-table-tail tcenter"> Retrieving... </td>
                 </tr>
-            {/if}
-            {#each devicePluginList as [deviceName, devicePlugins]}
+            {:else if devicePluginList.length == 0}
                 <tr>
-                    <th colspan="2" class="sls-plugins-tbl-device-head">{deviceName}</th>
-                    <th class="sls-plugins-tbl-device-head">
-                        <button class="mod-cta" on:click={() => toggleAll(deviceName)}>✔</button>
-                    </th>
+                    <td colspan="3" class="sls-table-tail tcenter"> No plugins found. </td>
                 </tr>
-                {#each devicePlugins as plugin}
+            {:else}
+                {#each devicePluginList as [deviceName, devicePlugins]}
                     <tr>
-                        <td class="sls-table-head">{plugin.manifest.name}</td>
-                        <td class="sls-table-tail tcenter">{plugin.versionInfo}{getDispString(plugin.versionFlag)}</td>
-                        <td class="sls-table-tail tcenter">
-                            {#if plugin.versionFlag === "EVEN" || plugin.versionFlag === ""}
-                                -
-                            {:else}
-                                <div class="wrapToggle">
-                                    <div
-                                        class="checkbox-container"
-                                        class:is-enabled={targetList[plugin.deviceVaultName + "---" + plugin.manifest.id + "---plugin"]}
-                                        on:click={() => toggleTarget(plugin.deviceVaultName + "---" + plugin.manifest.id + "---plugin")}
-                                    />
-                                </div>
-                            {/if}
-                        </td>
-                    </tr>
-                    <tr>
-                        <td class="sls-table-head">Settings</td>
-                        <td class="sls-table-tail tcenter">{plugin.mtimeInfo}{getDispString(plugin.mtimeFlag)}</td>
-                        <td class="sls-table-tail tcenter">
-                            {#if plugin.mtimeFlag === "EVEN" || plugin.mtimeFlag === ""}
-                                -
-                            {:else}
-                                <div class="wrapToggle">
-                                    <div
-                                        class="checkbox-container"
-                                        class:is-enabled={targetList[plugin.deviceVaultName + "---" + plugin.manifest.id + "---setting"]}
-                                        on:click={() => toggleTarget(plugin.deviceVaultName + "---" + plugin.manifest.id + "---setting")}
-                                    />
-                                </div>
-                            {/if}
-                        </td>
-                    </tr>
-                    <tr class="divider">
-                        <th colspan="3" />
+                        <th colspan="2" class="sls-plugins-tbl-device-head">{deviceName}</th>
+                        <th class="sls-plugins-tbl-device-head">
+                            <button class="mod-cta" on:click={() => toggleAll(deviceName)}>✔</button>
+                        </th>
                     </tr>
+                    {#each devicePlugins as plugin}
+                        <tr>
+                            <td class="sls-table-head">{plugin.manifest.name}</td>
+                            <td class="sls-table-tail tcenter">{plugin.versionInfo}{getDispString(plugin.versionFlag)}</td>
+                            <td class="sls-table-tail tcenter">
+                                {#if plugin.versionFlag === "EVEN" || plugin.versionFlag === ""}
+                                    -
+                                {:else}
+                                    <div class="wrapToggle">
+                                        <div
+                                            class="checkbox-container"
+                                            class:is-enabled={targetList[plugin.deviceVaultName + "---" + plugin.manifest.id + "---plugin"]}
+                                            on:click={() => toggleTarget(plugin.deviceVaultName + "---" + plugin.manifest.id + "---plugin")}
+                                        />
+                                    </div>
+                                {/if}
+                            </td>
+                        </tr>
+                        <tr>
+                            <td class="sls-table-head">Settings</td>
+                            <td class="sls-table-tail tcenter">{plugin.mtimeInfo}{getDispString(plugin.mtimeFlag)}</td>
+                            <td class="sls-table-tail tcenter">
+                                {#if plugin.mtimeFlag === "EVEN" || plugin.mtimeFlag === ""}
+                                    -
+                                {:else}
+                                    <div class="wrapToggle">
+                                        <div
+                                            class="checkbox-container"
+                                            class:is-enabled={targetList[plugin.deviceVaultName + "---" + plugin.manifest.id + "---setting"]}
+                                            on:click={() => toggleTarget(plugin.deviceVaultName + "---" + plugin.manifest.id + "---setting")}
+                                        />
+                                    </div>
+                                {/if}
+                            </td>
+                        </tr>
+                        <tr class="divider">
+                            <th colspan="3" />
+                        </tr>
+                    {/each}
                 {/each}
-            {/each}
+            {/if}
         </table>
     </div>
     <div class="ols-plugins-div-buttons">

src/main.ts

@@ -1,4 +1,4 @@
-import { debounce, Notice, Plugin, TFile, addIcon, TFolder, normalizePath, TAbstractFile, Editor, MarkdownView, PluginManifest, Modal, App } from "obsidian";
+import { debounce, Notice, Plugin, TFile, addIcon, TFolder, normalizePath, TAbstractFile, Editor, MarkdownView, PluginManifest, Modal, App, FuzzySuggestModal } from "obsidian";
 import { diff_match_patch } from "diff-match-patch";
 
 import { EntryDoc, LoadedEntry, ObsidianLiveSyncSettings, diff_check_result, diff_result_leaf, EntryBody, LOG_LEVEL, VER, DEFAULT_SETTINGS, diff_result, FLAGMD_REDFLAG, SYNCINFO_ID } from "./lib/src/types";
@@ -26,8 +26,10 @@ import { ConflictResolveModal } from "./ConflictResolveModal";
 import { ObsidianLiveSyncSettingTab } from "./ObsidianLiveSyncSettingTab";
 import { DocumentHistoryModal } from "./DocumentHistoryModal";
 
+//@ts-ignore
 import PluginPane from "./PluginPane.svelte";
 import { id2path, path2id } from "./utils";
+import { decrypt, encrypt } from "./lib/src/e2ee";
 const isDebug = false;
 setNoticeClass(Notice);
 class PluginDialogModal extends Modal {
@@ -57,6 +59,45 @@ class PluginDialogModal extends Modal {
         }
     }
 }
+class PopoverYesNo extends FuzzySuggestModal<string> {
+    app: App;
+    callback: (e: string) => void = () => {};
+
+    constructor(app: App, note: string, callback: (e: string) => void) {
+        super(app);
+        this.app = app;
+        this.setPlaceholder("y/n) " + note);
+        this.callback = callback;
+    }
+
+    getItems(): string[] {
+        return ["yes", "no"];
+    }
+
+    getItemText(item: string): string {
+        return item;
+    }
+
+    onChooseItem(item: string, evt: MouseEvent | KeyboardEvent): void {
+        // debugger;
+        this.callback(item);
+        this.callback = null;
+    }
+    onClose(): void {
+        setTimeout(() => {
+            if (this.callback != null) {
+                this.callback("");
+            }
+        }, 100);
+    }
+}
+
+const askYesNo = (app: App, message: string): Promise<"yes" | "no"> => {
+    return new Promise((res) => {
+        const popover = new PopoverYesNo(app, message, (result) => res(result as "yes" | "no"));
+        popover.open();
+    });
+};
 
 export default class ObsidianLiveSyncPlugin extends Plugin {
     settings: ObsidianLiveSyncSettings;
@@ -200,6 +241,81 @@ export default class ObsidianLiveSyncPlugin extends Plugin {
                     Logger(ex, LOG_LEVEL.VERBOSE);
                 }
         });
+        this.addCommand({
+            id: "livesync-exportconfig",
+            name: "Copy setup uri (beta)",
+            callback: async () => {
+                const encryptedSetting = encodeURIComponent(await encrypt(JSON.stringify(this.settings), "---"));
+                const uri = `obsidian://setuplivesync?settings=${encryptedSetting}`;
+                await navigator.clipboard.writeText(uri);
+                Logger("Setup uri copied to clipboard", LOG_LEVEL.NOTICE);
+            },
+        });
+        this.registerObsidianProtocolHandler("setuplivesync", async (conf: any) => {
+            try {
+                const oldConf = JSON.parse(JSON.stringify(this.settings));
+                const newconf = await JSON.parse(await decrypt(conf.settings, "---"));
+                if (newconf) {
+                    const result = await askYesNo(this.app, "Importing LiveSync's conf, OK?");
+                    if (result == "yes") {
+                        const newSettingW = Object.assign({}, this.settings, newconf);
+                        // stopping once.
+                        this.localDatabase.closeReplication();
+                        this.settings.suspendFileWatching = true;
+                        console.dir(newSettingW);
+                        const keepLocalDB = await askYesNo(this.app, "Keep local DB?");
+                        const keepRemoteDB = await askYesNo(this.app, "Keep remote DB?");
+                        if (keepLocalDB == "yes" && keepRemoteDB == "yes") {
+                            // nothing to do. so peaceful.
+                            this.settings = newSettingW;
+                            await this.saveSettings();
+                            Logger("Configuration loaded.", LOG_LEVEL.NOTICE);
+                            return;
+                        }
+                        if (keepLocalDB == "no" && keepRemoteDB == "no") {
+                            const reset = await askYesNo(this.app, "Drop everything?");
+                            if (reset != "yes") {
+                                Logger("Cancelled", LOG_LEVEL.NOTICE);
+                                this.settings = oldConf;
+                                return;
+                            }
+                        }
+                        let initDB;
+                        await this.saveSettings();
+                        if (keepLocalDB == "no") {
+                            this.resetLocalOldDatabase();
+                            this.resetLocalDatabase();
+                            this.localDatabase.initializeDatabase();
+                            const rebuild = await askYesNo(this.app, "Rebuild the database?");
+                            if (rebuild == "yes") {
+                                initDB = this.initializeDatabase(true);
+                            } else {
+                                this.markRemoteResolved();
+                            }
+                        }
+                        if (keepRemoteDB == "no") {
+                            await this.tryResetRemoteDatabase();
+                            await this.markRemoteLocked();
+                        }
+                        if (keepLocalDB == "no" || keepRemoteDB == "no") {
+                            const replicate = await askYesNo(this.app, "Replicate once?");
+                            if (replicate == "yes") {
+                                if (initDB != null) {
+                                    await initDB;
+                                }
+                                await this.replicate(true);
+                            }
+                        }
+                    }
+
+                    Logger("Configuration loaded.", LOG_LEVEL.NOTICE);
+                } else {
+                    Logger("Cancelled.", LOG_LEVEL.NOTICE);
+                }
+            } catch (ex) {
+                Logger("Couldn't parse configuration uri.", LOG_LEVEL.NOTICE);
+            }
+        });
         this.addCommand({
             id: "livesync-replicate",
             name: "Replicate now",
@@ -587,8 +703,8 @@ export default class ObsidianLiveSyncPlugin extends Plugin {
     //--> Basic document Functions
     notifies: { [key: string]: { notice: Notice; timer: NodeJS.Timeout; count: number } } = {};
 
-    // eslint-disable-next-line require-await
     lastLog = "";
+    // eslint-disable-next-line require-await
     async addLog(message: any, level: LOG_LEVEL = LOG_LEVEL.INFO) {
         if (level == LOG_LEVEL.DEBUG && !isDebug) {
             return;
@@ -607,9 +723,9 @@ export default class ObsidianLiveSyncPlugin extends Plugin {
         this.logMessage = [].concat(this.logMessage).concat([newmessage]).slice(-100);
         console.log(valutName + ":" + newmessage);
         this.setStatusBarText(null, messagecontent.substring(0, 30));
-        if (message instanceof Error) {
-            console.trace(message);
-        }
+        // if (message instanceof Error) {
+        //     console.trace(message);
+        // }
 
         if (level >= LOG_LEVEL.NOTICE) {
             if (messagecontent in this.notifies) {
@@ -808,12 +924,119 @@ export default class ObsidianLiveSyncPlugin extends Plugin {
             const doc = change;
             const file = targetFile;
             await this.doc2storage_modify(doc, file);
-            this.queueConflictedCheck(file);
+            if (!this.settings.checkConflictOnlyOnOpen) {
+                this.queueConflictedCheck(file);
+            } else {
+                const af = app.workspace.getActiveFile();
+                if (af && af.path == file.path) {
+                    this.queueConflictedCheck(file);
+                }
+            }
         } else {
             Logger(`${id2path(change._id)} is already exist as the folder`);
         }
     }
 
+    queuedFiles: {
+        entry: EntryBody;
+        missingChildren: string[];
+        timeout?: number;
+        done?: boolean;
+        warned?: boolean;
+    }[] = [];
+    chunkWaitTimeout = 60000;
+
+    async saveQueuedFiles() {
+        const saveData = JSON.stringify(this.queuedFiles.filter((e) => !e.done).map((e) => e.entry._id));
+        const lsname = "obsidian-livesync-queuefiles-" + this.app.vault.getName();
+        localStorage.setItem(lsname, saveData);
+    }
+    async loadQueuedFiles() {
+        const lsname = "obsidian-livesync-queuefiles-" + this.app.vault.getName();
+        const ids = JSON.parse(localStorage.getItem(lsname) || "[]") as string[];
+        const ret = await this.localDatabase.localDatabase.allDocs({ keys: ids, include_docs: true });
+        for (const doc of ret.rows) {
+            if (doc.doc && !this.queuedFiles.some((e) => e.entry._id == doc.doc._id)) {
+                await this.parseIncomingDoc(doc.doc as PouchDB.Core.ExistingDocument<EntryBody & PouchDB.Core.AllDocsMeta>);
+            }
+        }
+    }
+    async procQueuedFiles() {
+        await runWithLock("procQueue", false, async () => {
+            this.saveQueuedFiles();
+            for (const queue of this.queuedFiles) {
+                if (queue.done) continue;
+                const now = new Date().getTime();
+                if (queue.missingChildren.length == 0) {
+                    queue.done = true;
+                    if (isValidPath(id2path(queue.entry._id))) {
+                        Logger(`Applying ${queue.entry._id} (${queue.entry._rev}) change...`);
+                        await this.handleDBChanged(queue.entry);
+                    }
+                } else if (now > queue.timeout) {
+                    if (!queue.warned) Logger(`Timed out: ${queue.entry._id} could not collect ${queue.missingChildren.length} chunks. plugin keeps watching, but you have to check the file after the replication.`, LOG_LEVEL.NOTICE);
+                    queue.warned = true;
+                    continue;
+                }
+            }
+            this.queuedFiles = this.queuedFiles.filter((e) => !e.done);
+            this.saveQueuedFiles();
+        });
+    }
+    parseIncomingChunk(chunk: PouchDB.Core.ExistingDocument<EntryDoc>) {
+        const now = new Date().getTime();
+        let isNewFileCompleted = false;
+
+        for (const queue of this.queuedFiles) {
+            if (queue.done) continue;
+            if (queue.missingChildren.indexOf(chunk._id) !== -1) {
+                queue.missingChildren = queue.missingChildren.filter((e) => e != chunk._id);
+                queue.timeout = now + this.chunkWaitTimeout;
+            }
+            if (queue.missingChildren.length == 0) {
+                for (const e of this.queuedFiles) {
+                    if (e.entry._id == queue.entry._id && e.entry.mtime < queue.entry.mtime) {
+                        e.done = true;
+                    }
+                }
+                isNewFileCompleted = true;
+            }
+        }
+        if (isNewFileCompleted) this.procQueuedFiles();
+    }
+    async parseIncomingDoc(doc: PouchDB.Core.ExistingDocument<EntryBody>) {
+        const skipOldFile = this.settings.skipOlderFilesOnSync && false; //patched temporary.
+        if (skipOldFile) {
+            const info = this.app.vault.getAbstractFileByPath(id2path(doc._id));
+
+            if (info && info instanceof TFile) {
+                const localMtime = ~~((info as TFile).stat.mtime / 1000);
+                const docMtime = ~~(doc.mtime / 1000);
+                //TODO: some margin required.
+                if (localMtime >= docMtime) {
+                    Logger(`${doc._id} Skipped, older than storage.`, LOG_LEVEL.VERBOSE);
+                    return;
+                }
+            }
+        }
+        const now = new Date().getTime();
+        const newQueue = {
+            entry: doc,
+            missingChildren: [] as string[],
+            timeout: now + this.chunkWaitTimeout,
+        };
+        if ("children" in doc) {
+            const c = await this.localDatabase.localDatabase.allDocs({ keys: doc.children, include_docs: false });
+            const missing = c.rows.filter((e) => "error" in e).map((e) => e.key);
+            Logger(`${doc._id}(${doc._rev}) Queued (waiting ${missing.length} items)`, LOG_LEVEL.VERBOSE);
+            newQueue.missingChildren = missing;
+            this.queuedFiles.push(newQueue);
+        } else {
+            this.queuedFiles.push(newQueue);
+        }
+        this.saveQueuedFiles();
+        this.procQueuedFiles();
+    }
     periodicSyncHandler: number = null;
 
     //---> Sync
@@ -827,14 +1050,15 @@ export default class ObsidianLiveSyncPlugin extends Plugin {
                 continue;
             }
             if (change._id.startsWith("h:")) {
+                await this.parseIncomingChunk(change);
                 continue;
             }
             if (change._id == SYNCINFO_ID) {
                 continue;
             }
             if (change.type != "leaf" && change.type != "versioninfo" && change.type != "milestoneinfo" && change.type != "nodeinfo") {
-                Logger("replication change arrived", LOG_LEVEL.VERBOSE);
-                await this.handleDBChanged(change);
+                await this.parseIncomingDoc(change);
+                continue;
             }
             if (change.type == "versioninfo") {
                 if (change.version > VER) {
@@ -971,12 +1195,31 @@ export default class ObsidianLiveSyncPlugin extends Plugin {
             waiting = " " + this.batchFileChange.map((e) => "🛫").join("");
             waiting = waiting.replace(/(🛫){10}/g, "🚀");
         }
+        let queued = "";
+        const queue = Object.entries(this.queuedFiles).filter((e) => !e[1].warned);
+        const queuedCount = queue.length;
+
+        if (queuedCount) {
+            const pieces = queue.map((e) => e[1].missingChildren).reduce((prev, cur) => prev + cur.length, 0);
+            queued = ` 🧩 ${queuedCount} (${pieces})`;
+        }
         const procs = getProcessingCounts();
         const procsDisp = procs == 0 ? "" : ` ⏳${procs}`;
-        const message = `Sync:${w} ↑${sent} ↓${arrived}${waiting}${procsDisp}`;
+        const message = `Sync:${w} ↑${sent} ↓${arrived}${waiting}${procsDisp}${queued}`;
         const locks = getLocks();
-        const pendingTask = locks.pending.length ? `\nPending:${locks.pending.join(", ")}` : "";
-        const runningTask = locks.running.length ? `\nRunning:${locks.running.join(", ")}` : "";
+        const pendingTask = locks.pending.length
+            ? "\nPending: " +
+              Object.entries([...new Set([...locks.pending])].reduce((p, c) => ({ ...p, [c]: p[c] ?? 0 + 1 }), {} as { [key: string]: number }))
+                  .map((e) => `${e[0]}${e[1] == 1 ? "" : `(${e[1]})`}`)
+                  .join(", ")
+            : "";
+
+        const runningTask = locks.running.length
+            ? "\nRunning: " +
+              Object.entries([...new Set([...locks.running])].reduce((p, c) => ({ ...p, [c]: p[c] ?? 0 + 1 }), {} as { [key: string]: number }))
+                  .map((e) => `${e[0]}${e[1] == 1 ? "" : `(${e[1]})`}`)
+                  .join(", ")
+            : "";
         this.setStatusBarText(message + pendingTask + runningTask);
     }
 
@@ -1014,6 +1257,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin {
         if (this.settings.autoSweepPlugins) {
             await this.sweepPlugin(false);
         }
+        await this.loadQueuedFiles();
         this.localDatabase.openReplication(this.settings, false, showMessage, this.parseReplicationResult);
     }
 
@@ -1053,6 +1297,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin {
         if (showingNotice) {
             notice = NewNotice("Initializing", 0);
         }
+
         const filesStorage = this.app.vault.getFiles();
         const filesStorageName = filesStorage.map((e) => e.path);
         const wf = await this.localDatabase.localDatabase.allDocs();
@@ -1094,6 +1339,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin {
             });
 
             await allSettledWithConcurrencyLimit(procs, 10);
+            Logger(`${procedurename} done.`);
         };
         await runAll("UPDATE DATABASE", onlyInStorage, async (e) => {
             Logger(`Update into ${e.path}`);
@@ -1379,23 +1625,30 @@ export default class ObsidianLiveSyncPlugin extends Plugin {
 
         const storageMtime = ~~(file.stat.mtime / 1000);
         const docMtime = ~~(doc.mtime / 1000);
-        if (storageMtime > docMtime) {
-            //newer local file.
-            Logger("STORAGE -> DB :" + file.path);
-            Logger(`${storageMtime} > ${docMtime}`);
-            await this.updateIntoDB(file);
-        } else if (storageMtime < docMtime) {
-            //newer database file.
-            Logger("STORAGE <- DB :" + file.path);
-            Logger(`${storageMtime} < ${docMtime}`);
-            const docx = await this.localDatabase.getDBEntry(file.path, null, false, false);
-            if (docx != false) {
-                await this.doc2storage_modify(docx, file);
-            }
+        const dK = `${file.path}-diff`;
+        const isLastDiff = (await this.localDatabase.kvDB.get<{ storageMtime: number; docMtime: number }>(dK)) || { storageMtime: 0, docMtime: 0 };
+        if (isLastDiff.docMtime == docMtime && isLastDiff.storageMtime == storageMtime) {
+            // Logger("CHECKED       :" + file.path, LOG_LEVEL.VERBOSE);
         } else {
-            // Logger("EVEN :" + file.path, LOG_LEVEL.VERBOSE);
-            // Logger(`${storageMtime} = ${docMtime}`, LOG_LEVEL.VERBOSE);
-            //eq.case
+            if (storageMtime > docMtime) {
+                //newer local file.
+                Logger("STORAGE -> DB :" + file.path);
+                Logger(`${storageMtime} > ${docMtime}`);
+                await this.updateIntoDB(file);
+            } else if (storageMtime < docMtime) {
+                //newer database file.
+                Logger("STORAGE <- DB :" + file.path);
+                Logger(`${storageMtime} < ${docMtime}`);
+                const docx = await this.localDatabase.getDBEntry(file.path, null, false, false);
+                if (docx != false) {
+                    await this.doc2storage_modify(docx, file);
+                }
+            } else {
+                // Logger("EVEN :" + file.path, LOG_LEVEL.VERBOSE);
+                // Logger(`${storageMtime} = ${docMtime}`, LOG_LEVEL.VERBOSE);
+                //eq.case
+            }
+            await this.localDatabase.kvDB.set(dK, { storageMtime, docMtime });
         }
     }
 
@@ -1440,6 +1693,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin {
         });
         if (isNotChanged) return;
         await this.localDatabase.putDBEntry(d);
+        this.queuedFiles = this.queuedFiles.map((e) => ({ ...e, ...(e.entry._id == d._id ? { done: true } : {}) }));
 
         Logger("put database:" + fullpath + "(" + datatype + ") ");
         if (this.settings.syncOnSave && !this.suspended) {
@@ -1500,7 +1754,8 @@ export default class ObsidianLiveSyncPlugin extends Plugin {
 
     async sweepPlugin(showMessage = false) {
         if (!this.settings.usePluginSync) return;
-        await runWithLock("sweepplugin", false, async () => {
+        if (!this.localDatabase.isReady) return;
+        await runWithLock("sweepplugin", true, async () => {
             const logLevel = showMessage ? LOG_LEVEL.NOTICE : LOG_LEVEL.INFO;
             if (!this.settings.encrypt) {
                 Logger("You have to encrypt the database to use plugin setting sync.", LOG_LEVEL.NOTICE);
@@ -1517,7 +1772,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin {
                 endkey: `ps:${this.deviceAndVaultName}.`,
                 include_docs: true,
             });
-            Logger("OLD DOCS.", LOG_LEVEL.VERBOSE);
+            // Logger("OLD DOCS.", LOG_LEVEL.VERBOSE);
             // sweep current plugin.
             // @ts-ignore
             const pl = this.app.plugins;

src/utils_couchdb.ts

@@ -134,6 +134,7 @@ const connectRemoteCouchDB = async (uri: string, auth: { username: string; passw
             // return await fetch(url, opts);
         },
     };
+
     const db: PouchDB.Database<EntryDoc> = new PouchDB<EntryDoc>(uri, conf);
     if (passphrase && typeof passphrase === "string") {
         enableEncryption(db, passphrase);