bump

- Fixed a bug on `fetch chunks on demand` that could not fetch the chunks on demand.
- Improved:
  - `fetch chunks on demand` works more smoothly.
  - Initialisation `Fetch` is now more efficient.
- Tidied:
  - Removed some meaningless codes.

README.md

@@ -1,4 +1,4 @@
-<!-- For translation: 20240209r0 -->
+<!-- For translation: 20240227r0 -->
 # Self-hosted LiveSync
 [Japanese docs](./README_ja.md) - [Chinese docs](./README_cn.md).
 
@@ -12,7 +12,7 @@ Note: This plugin cannot synchronise with the official "Obsidian Sync".
 
 - Synchronize vaults very efficiently with less traffic.
 - Good at conflicted modification.
-- Automatic merging for simple conflicts.
+  - Automatic merging for simple conflicts.
 - Using OSS solution for the server.
   - Compatible solutions can be used.
 - Supporting End-to-end encryption.
@@ -50,8 +50,10 @@ This plug-in might be useful for researchers, engineers, and developers with a n
 
 ## Information in StatusBar
 
-Synchronization status is shown in statusbar.
+Synchronization status is shown in the status bar with the following icons.
 
+-   Activity Indicator
+    -   📲 Network request
 -   Status
     -   ⏹️ Stopped
     -   💤 LiveSync enabled. Waiting for changes
@@ -70,7 +72,7 @@ Synchronization status is shown in statusbar.
      -   🧩 Waiting chunks
      -   🔌 Working Customisation items (Configuration, snippets, and plug-ins)
 
-To prevent file and database corruption, please wait until all progress indicators have disappeared. Especially in case of if you have deleted or renamed files.
+To prevent file and database corruption, please wait to stop Obsidian until all progress indicators have disappeared as possible (The plugin will also try to resume, though). Especially in case of if you have deleted or renamed files.
 
 
 
@@ -79,4 +81,4 @@ If you are having problems getting the plugin working see: [Tips and Troubleshoo
 
 ## License
 
-The source code is licensed under the MIT License.
+Licensed under the MIT License.

README_ja.md

@@ -1,84 +1,85 @@
+<!-- For translation: 20240227r0 -->
 # Self-hosted LiveSync
+[英語版ドキュメント](./README.md) - [中国語版ドキュメント](./README_cn.md).
 
-**旧): obsidian-livesync**
+Obsidianで利用可能なすべてのプラットフォームで使える、CouchDBをサーバに使用する、コミュニティ版の同期プラグイン
 
-セルフホストしたデータベースを使って、双方向のライブシンクするObsidianのプラグイン。  
-**公式のSyncとは互換性はありません**  
 ![obsidian_live_sync_demo](https://user-images.githubusercontent.com/45774780/137355323-f57a8b09-abf2-4501-836c-8cb7d2ff24a3.gif)
 
-**インストールする前に、Vaultのバックアップを確実に取得してください**
-
-[英語版](./README.md)
-
-## こんなことができるプラグインです。
-- Windows, Mac, iPad, iPhone, Android, Chromebookで動く
-- セルフホストしたデータベースに同期して
-- 複数端末で同時にその変更をほぼリアルタイムで配信し
-- さらに、他の端末での変更も別の端末に配信する、双方向リアルタイムなLiveSyncを実現でき、
-- 発生した変更の衝突はその場で解決できます。
-- 同期先のホストにはCouchDBまたはその互換DBaaSのIBM Cloudantをサーバーに使用できます。あなたのデータは、あなたのものです。
-- もちろんLiveではない同期もできます。
-- 万が一のために、サーバーに送る内容を暗号化できます(betaです)。
--  [Webクリッパー](https://chrome.google.com/webstore/detail/obsidian-livesync-webclip/jfpaflmpckblieefkegjncjoceapakdf) もあります(End-to-End暗号化対象外です)
-
-NDAや類似の契約や義務、倫理を守る必要のある、研究者、設計者、開発者のような方に特にオススメです。  
-特にエンタープライズでは、たとえEnd to Endの暗号化が行われていても、管理下にあるサーバーにのみデータを格納することが求められる場合があります。  
-
-# 重要なお知らせ
-
-- ❌ファイルの重複や破損を避けるため、複数の同期手段を同時に使用しないでください。  
-これは、Vaultをクラウド管理下のフォルダに置くことも含みます。(例えば、iCloudの管理フォルダ内に入れたり)。  
-- ⚠️このプラグインは、端末間でのノートの反映を目的として作成されました。バックアップ等が目的ではありません。そのため、バックアップは必ず別のソリューションで行うようにしてください。
-- ストレージの空き容量が枯渇した場合、データベースが破損することがあります。
-
-# このプラグインの使い方
-
-1. Community Pluginsから、Self-holsted LiveSyncと検索しインストールするか、このリポジトリのReleasesから`main.js`, `manifest.json`, `style.css` をダウンロードしvaultの中の`.obsidian/plugins/obsidian-livesync`に入れて、Obsidianを再起動してください。
-2. サーバーをセットアップします。IBM Cloudantがお手軽かつ堅牢で便利です。完全にセルフホストする際にはお持ちのサーバーにCouchDBをインストールする必要があります。詳しくは下記を参照してください
-  1. [IBM Cloudantのセットアップ](docs/setup_cloudant_ja.md)
-  2. [独自のCouchDBのセットアップ](docs/setup_own_server_ja.md)
-
-備考： IBM Cloudantのアカウント登録が出来ないケースがあるようです。代替を探していて、今 [using fly.io](https://github.com/vrtmrz/obsidian-livesync/discussions/85)を検討しています。
-
-1. [Quick setup](docs/quick_setup_ja.md)から、セットアップウィザード使ってセットアップしてください。
-
-# テストサーバー
-
-もし、CouchDBをインストールしたり、Cloudantのインスタンスをセットアップしたりするのに気が引ける場合、[Self-hosted LiveSyncのテストサーバー](https://olstaste.vrtmrz.net/)を作りましたので、使ってみてください。
-
-備考: 制限事項をよく確認して使用してください。くれぐれも、本当に使用している自分のVaultを同期しないようにしてください。
-
-# WebClipperあります
-Self-hosted LiveSync用にWebClipperも作りました。Chrome Web Storeからダウンロードできます。
-
-[obsidian-livesync-webclip](https://chrome.google.com/webstore/detail/obsidian-livesync-webclip/jfpaflmpckblieefkegjncjoceapakdf)
-
-リポジトリはこちらです: [obsidian-livesync-webclip](https://github.com/vrtmrz/obsidian-livesync-webclip)。
-
-相変わらずドキュメントは間に合っていません。
-
-# ステータスバーの情報
-右下のステータスバーに、同期の状態が表示されます
-
-- 同期状態
-  - ⏹️ 同期は停止しています
-  - 💤 同期はLiveSync中で、なにか起こるのを待っています
-  - ⚡️ 同期中です
-  - ⚠ エラーが発生しています
-- ↑ 送信したデータ数
-- ↓ 受信したデータ数
--   ⏳ 保留している処理の数です  
-ファイルを削除したりリネームした場合、この表示が消えるまでお待ちください。
-
-# さらなる補足
-- ファイルは同期された後、タイムスタンプを比較して新しければいったん新しい方で上書きされます。その後、衝突が発生したかによって、マージが行われます。
-- まれにファイルが破損することがあります。破損したファイルに関してはディスクへの反映を試みないため、実際には使用しているデバイスには少し古いファイルが残っていることが多いです。そのファイルを再度更新してもらうと、データベースが更新されて問題なくなるケースがあります。ファイルがどの端末にも存在しない場合は、設定画面から、削除できます。
-- データベースの復旧中に再起動した場合など、うまくローカルデータベースを修正できない際には、Vaultのトップに`redflag.md`というファイルを置いてください。起動時のシーケンスがスキップされます。
-- データベースが大きくなってきてるんだけど、小さくできる？→各ノートは、それぞれの古い100リビジョンとともに保存されています。例えば、しばらくオフラインだったあるデバイスが、久しぶりに同期したと想定してみてください。そのとき、そのデバイスは最新とは少し異なるリビジョンを持ってるはずです。その場合でも、リモートのリビジョン履歴にリモートのものが存在した場合、安全にマージできます。もしリビジョン履歴に存在しなかった場合、確認しなければいけない差分も、対象を存在して持っている共通のリビジョン以降のみに絞れます。ちょうどGitのような方法で、衝突を解決している形になるのです。そのため、肥大化したリポジトリの解消と同様に、本質的にデータベースを小さくしたい場合は、データベースの作り直しが必要です。
-- その他の技術的なお話は、[技術的な内容](docs/tech_info_ja.md)に書いてあります。
+※公式のSyncと同期することはできません。
 
 
-# ライセンス
+## 機能
+- 高効率・低トラフィックでVault同士を同期
+- 競合解決がいい感じ
+  - 単純な競合なら自動マージします
+- OSSソリューションを同期サーバに使用
+  - 互換ソリューションも使用可能です
+- End-to-End暗号化実装済み
+- 設定・スニペット・テーマ、プラグインの同期が可能
+-  [Webクリッパー](https://chrome.google.com/webstore/detail/obsidian-livesync-webclip/jfpaflmpckblieefkegjncjoceapakdf) もあります
 
-The source code is licensed MIT.
- 
+NDAや類似の契約や義務、倫理を守る必要のある、研究者、設計者、開発者のような方に特にオススメです。
+
+
+>[!IMPORTANT]
+> - インストール・アップデート前には必ずVaultをバックアップしてください
+> - 複数の同期ソリューションを同時に有効にしないでください（これはiCloudや公式のSyncも含みます）
+> - このプラグインは同期プラグインです。バックアップとして使用しないでください
+
+
+## このプラグインの使い方
+
+### 3分セットアップ - CouchDB on fly.io
+
+**はじめての方におすすめ**
+
+[![LiveSync Setup onto Fly.io SpeedRun 2024 using Google Colab](https://img.youtube.com/vi/7sa_I1832Xc/0.jpg)](https://www.youtube.com/watch?v=7sa_I1832Xc)
+
+1. [Fly.ioにCouchDBをセットアップする](docs/setup_flyio.md)
+2. [Quick Setup](docs/quick_setup_ja.md)でプラグインを設定する
+
+
+### Manually Setup
+
+1. サーバのセットアップ
+   1. [Fly.ioにCouchDBをセットアップする](docs/setup_flyio.md)
+   2. [CouchDBをセットアップする](docs/setup_own_server_ja.md)
+2. [Quick Setup](docs/quick_setup_ja.md)でプラグインを設定する
+
+> [!TIP]
+> IBM Cloudantもまだ使用できますが、いくつかの理由で現在はおすすめしていません。[IBM Cloudantのセットアップ](docs/setup_cloudant_ja.md)はまだあります。
+
+## ステータスバーの説明
+
+同期ステータスはステータスバーに、下記のアイコンとともに表示されます
+
+-   アクティビティー
+    -   📲 ネットワーク接続中
+-   同期ステータス
+    -   ⏹️ 停止中
+    -   💤 変更待ち（LiveSync中）
+    -   ⚡️ 同期の進行中
+    -   ⚠ エラー
+-   統計情報
+     -   ↑ アップロードしたチャンクとメタデータ数
+     -   ↓ ダウンロードしたチャンクとメタデータ数
+-   進捗情報
+     -   📥 転送後、未処理の項目数
+     -   📄 稼働中データベース操作数
+     -   💾 稼働中のストレージ書き込み数操作数
+     -   ⏳ 稼働中のストレージ読み込み数操作数
+     -   🛫 待機中のストレージ読み込み数操作数
+     -   ⚙️ 隠しファイルの操作数（待機・稼働中合計）
+     -   🧩 取得待ちを行っているチャンク数
+     -   🔌 設定同期関連の操作数
+
+データベースやファイルの破損を避けるため、Obsidianの終了は進捗情報が表示されなくなるまで待ってください（プラグインも復帰を試みますが）。特にファイルを削除やリネームした場合は気をつけてください。
+
+
+## Tips and Troubleshooting
+何かこまったら、[Tips and Troubleshooting](docs/troubleshooting.md)をご参照ください。
+
+## License
+
+Licensed under the MIT License.

manifest.json

@@ -1,10 +1,10 @@
 {
     "id": "obsidian-livesync",
     "name": "Self-hosted LiveSync",
-    "version": "0.22.6",
+    "version": "0.22.9",
     "minAppVersion": "0.9.12",
     "description": "Community implementation of self-hosted livesync. Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.",
     "author": "vorotamoroz",
     "authorUrl": "https://github.com/vrtmrz",
     "isDesktopOnly": false
-}
+}

package-lock.json

@@ -1,12 +1,12 @@
 {
     "name": "obsidian-livesync",
-    "version": "0.22.6",
+    "version": "0.22.9",
     "lockfileVersion": 2,
     "requires": true,
     "packages": {
         "": {
             "name": "obsidian-livesync",
-            "version": "0.22.6",
+            "version": "0.22.9",
             "license": "MIT",
             "dependencies": {
                 "diff-match-patch": "^1.0.5",

package.json

@@ -1,6 +1,6 @@
 {
     "name": "obsidian-livesync",
-    "version": "0.22.6",
+    "version": "0.22.9",
     "description": "Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.",
     "main": "main.js",
     "type": "module",

src/CmdConfigSync.ts

@@ -1,5 +1,5 @@
 import { writable } from 'svelte/store';
-import { Notice, type PluginManifest, parseYaml, normalizePath } from "./deps";
+import { Notice, type PluginManifest, parseYaml, normalizePath, type ListedFiles } from "./deps";
 
 import type { EntryDoc, LoadedEntry, InternalFileEntry, FilePathWithPrefix, FilePath, DocumentID, AnyEntry, SavingEntry } from "./lib/src/types";
 import { LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE, MODE_SELECTIVE } from "./lib/src/types";
@@ -178,9 +178,7 @@ export class ConfigSync extends LiveSyncCommands {
     get kvDB() {
         return this.plugin.kvDB;
     }
-    ensureDirectoryEx(fullPath: string) {
-        return this.plugin.ensureDirectoryEx(fullPath);
-    }
+
     pluginDialog: PluginDialogModal = null;
     periodicPluginSweepProcessor = new PeriodicProcessor(this.plugin, async () => await this.scanAllConfigFiles(false));
 
@@ -433,7 +431,7 @@ export class ConfigSync extends LiveSyncCommands {
                 try {
                     // console.dir(f);
                     const path = `${baseDir}/${f.filename}`;
-                    await this.ensureDirectoryEx(path);
+                    await this.vaultAccess.ensureDirectory(path);
                     if (!content) {
                         const dt = decodeBinary(f.data);
                         await this.vaultAccess.adapterWrite(path, dt);
@@ -814,7 +812,14 @@ export class ConfigSync extends LiveSyncCommands {
         lastDepth: number
     ) {
         if (lastDepth == -1) return [];
-        const w = await this.app.vault.adapter.list(path);
+        let w: ListedFiles;
+        try {
+            w = await this.app.vault.adapter.list(path);
+        } catch (ex) {
+            Logger(`Could not traverse(ConfigSync):${path}`, LOG_LEVEL_INFO);
+            Logger(ex, LOG_LEVEL_VERBOSE);
+            return [];
+        }
         let files = [
             ...w.files
         ];

src/CmdHiddenFileSync.ts

@@ -1,5 +1,5 @@
-import { normalizePath, type PluginManifest } from "./deps";
-import { type EntryDoc, type LoadedEntry, type InternalFileEntry, type FilePathWithPrefix, type FilePath, LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE, MODE_SELECTIVE, MODE_PAUSED, type SavingEntry } from "./lib/src/types";
+import { normalizePath, type PluginManifest, type ListedFiles } from "./deps";
+import { type EntryDoc, type LoadedEntry, type InternalFileEntry, type FilePathWithPrefix, type FilePath, LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE, MODE_SELECTIVE, MODE_PAUSED, type SavingEntry, type DocumentID } from "./lib/src/types";
 import { type InternalFileInfo, ICHeader, ICHeaderEnd } from "./types";
 import { createBinaryBlob, isDocContentSame, sendSignal } from "./lib/src/utils";
 import { Logger } from "./lib/src/logger";
@@ -20,9 +20,6 @@ export class HiddenFileSync extends LiveSyncCommands {
     get kvDB() {
         return this.plugin.kvDB;
     }
-    ensureDirectoryEx(fullPath: string) {
-        return this.plugin.ensureDirectoryEx(fullPath);
-    }
     getConflictedDoc(path: FilePathWithPrefix, rev: string) {
         return this.plugin.getConflictedDoc(path, rev);
     }
@@ -102,11 +99,13 @@ export class HiddenFileSync extends LiveSyncCommands {
         }
         const stat = await this.vaultAccess.adapterStat(path);
         // sometimes folder is coming.
-        if (stat && stat.type != "file")
+        if (stat != null && stat.type != "file") {
             return;
-        const storageMTime = ~~((stat && stat.mtime || 0) / 1000);
+        }
+        const mtime = stat == null ? 0 : stat?.mtime ?? 0;
+        const storageMTime = ~~((mtime) / 1000);
         const key = `${path}-${storageMTime}`;
-        if (this.recentProcessedInternalFiles.contains(key)) {
+        if (mtime != 0 && this.recentProcessedInternalFiles.contains(key)) {
             //If recently processed, it may caused by self.
             return;
         }
@@ -150,6 +149,27 @@ export class HiddenFileSync extends LiveSyncCommands {
         await this.conflictResolutionProcessor.startPipeline().waitForPipeline();
     }
 
+    async resolveByNewerEntry(id: DocumentID, path: FilePathWithPrefix, currentDoc: EntryDoc, currentRev: string, conflictedRev: string) {
+        const conflictedDoc = await this.localDatabase.getRaw(id, { rev: conflictedRev });
+        // determine which revision should been deleted.
+        // simply check modified time
+        const mtimeCurrent = ("mtime" in currentDoc && currentDoc.mtime) || 0;
+        const mtimeConflicted = ("mtime" in conflictedDoc && conflictedDoc.mtime) || 0;
+        // Logger(`Revisions:${new Date(mtimeA).toLocaleString} and ${new Date(mtimeB).toLocaleString}`);
+        // console.log(`mtime:${mtimeA} - ${mtimeB}`);
+        const delRev = mtimeCurrent < mtimeConflicted ? currentRev : conflictedRev;
+        // delete older one.
+        await this.localDatabase.removeRevision(id, delRev);
+        Logger(`Older one has been deleted:${path}`);
+        const cc = await this.localDatabase.getRaw(id, { conflicts: true });
+        if (cc._conflicts.length == 0) {
+            await this.extractInternalFileFromDatabase(stripAllPrefixes(path))
+        } else {
+            this.conflictResolutionProcessor.enqueue(path);
+        }
+        // check the file again
+
+    }
     conflictResolutionProcessor = new QueueProcessor(async (paths: FilePathWithPrefix[]) => {
         const path = paths[0];
         sendSignal(`cancel-internal-conflict:${path}`);
@@ -179,33 +199,22 @@ export class HiddenFileSync extends LiveSyncCommands {
                     const filename = stripAllPrefixes(path);
                     const isExists = await this.plugin.vaultAccess.adapterExists(filename);
                     if (!isExists) {
-                        await this.ensureDirectoryEx(filename);
+                        await this.vaultAccess.ensureDirectory(filename);
                     }
                     await this.plugin.vaultAccess.adapterWrite(filename, result);
                     const stat = await this.vaultAccess.adapterStat(filename);
                     await this.storeInternalFileToDatabase({ path: filename, ...stat });
                     await this.extractInternalFileFromDatabase(filename);
-                    await this.localDatabase.removeRaw(id, revB);
+                    await this.localDatabase.removeRevision(id, revB);
                     this.conflictResolutionProcessor.enqueue(path);
                     return;
                 } else {
                     Logger(`Object merge is not applicable.`, LOG_LEVEL_VERBOSE);
                 }
-                return [{ path, revA, revB }];
+                return [{ path, revA, revB, id, doc }];
             }
-            const revBDoc = await this.localDatabase.getRaw(id, { rev: revB });
-            // determine which revision should been deleted.
-            // simply check modified time
-            const mtimeA = ("mtime" in doc && doc.mtime) || 0;
-            const mtimeB = ("mtime" in revBDoc && revBDoc.mtime) || 0;
-            // Logger(`Revisions:${new Date(mtimeA).toLocaleString} and ${new Date(mtimeB).toLocaleString}`);
-            // console.log(`mtime:${mtimeA} - ${mtimeB}`);
-            const delRev = mtimeA < mtimeB ? revA : revB;
-            // delete older one.
-            await this.localDatabase.removeRaw(id, delRev);
-            Logger(`Older one has been deleted:${path}`);
-            // check the file again 
-            this.conflictResolutionProcessor.enqueue(path);
+            // When not JSON file, resolve conflicts by choosing a newer one.
+            await this.resolveByNewerEntry(id, path, doc, revA, revB);
             return;
         } catch (ex) {
             Logger(`Failed to resolve conflict (Hidden): ${path}`);
@@ -215,7 +224,7 @@ export class HiddenFileSync extends LiveSyncCommands {
     }, {
         suspended: false, batchSize: 1, concurrentLimit: 5, delay: 10, keepResultUntilDownstreamConnected: true, yieldThreshold: 10,
         pipeTo: new QueueProcessor(async (results) => {
-            const { path, revA, revB } = results[0]
+            const { id, doc, path, revA, revB } = results[0];
             const docAMerge = await this.localDatabase.getDBEntry(path, { rev: revA });
             const docBMerge = await this.localDatabase.getDBEntry(path, { rev: revB });
             if (docAMerge != false && docBMerge != false) {
@@ -224,6 +233,9 @@ export class HiddenFileSync extends LiveSyncCommands {
                     this.conflictResolutionProcessor.enqueue(path);
                 }
                 return;
+            } else {
+                // If either revision could not read, force resolving by the newer one.
+                await this.resolveByNewerEntry(id, path, doc, revA, revB);
             }
         }, { suspended: false, batchSize: 1, concurrentLimit: 1, delay: 10, keepResultUntilDownstreamConnected: false, yieldThreshold: 10 })
     })
@@ -361,7 +373,11 @@ export class HiddenFileSync extends LiveSyncCommands {
                         }
                     }
                 } else if (xFileOnStorage && !xFileOnDatabase) {
-                    await this.storeInternalFileToDatabase(xFileOnStorage);
+                    if (direction == "push" || direction == "pushForce" || direction == "safe") {
+                        await this.storeInternalFileToDatabase(xFileOnStorage);
+                    } else {
+                        await this.extractInternalFileFromDatabase(xFileOnStorage.path);
+                    }
                 } else {
                     throw new Error("Invalid state on hidden file sync");
                     // Something corrupted?
@@ -479,7 +495,7 @@ export class HiddenFileSync extends LiveSyncCommands {
                         type: "newnote",
                     };
                 }
-                const ret = await this.localDatabase.putDBEntry(saveData, true);
+                const ret = await this.localDatabase.putDBEntry(saveData);
                 Logger(`STORAGE --> DB:${file.path}: (hidden) Done`);
                 return ret;
             } catch (ex) {
@@ -513,6 +529,14 @@ export class HiddenFileSync extends LiveSyncCommands {
                         type: "newnote",
                     };
                 } else {
+                    // Remove all conflicted before deleting.
+                    const conflicts = await this.localDatabase.getRaw(old._id, { conflicts: true });
+                    if ("_conflicts" in conflicts) {
+                        for (const conflictRev of conflicts._conflicts) {
+                            await this.localDatabase.removeRevision(old._id, conflictRev);
+                            Logger(`STORAGE -x> DB:${filename}: (hidden) conflict removed ${old._rev} =>  ${conflictRev}`, LOG_LEVEL_VERBOSE);
+                        }
+                    }
                     if (old.deleted) {
                         Logger(`STORAGE -x> DB:${filename}: (hidden) already deleted`);
                         return;
@@ -546,8 +570,7 @@ export class HiddenFileSync extends LiveSyncCommands {
         return await serialized("file-" + prefixedFileName, async () => {
             try {
                 // Check conflicted status 
-                //TODO option
-                const fileOnDB = await this.localDatabase.getDBEntry(prefixedFileName, { conflicts: true }, false, true);
+                const fileOnDB = await this.localDatabase.getDBEntry(prefixedFileName, { conflicts: true }, false, true, true);
                 if (fileOnDB === false)
                     throw new Error(`File not found on database.:${filename}`);
                 // Prevent overwrite for Prevent overwriting while some conflicted revision exists.
@@ -555,7 +578,7 @@ export class HiddenFileSync extends LiveSyncCommands {
                     Logger(`Hidden file ${filename} has conflicted revisions, to keep in safe, writing to storage has been prevented`, LOG_LEVEL_INFO);
                     return;
                 }
-                const deleted = "deleted" in fileOnDB ? fileOnDB.deleted : false;
+                const deleted = fileOnDB.deleted || fileOnDB._deleted || false;
                 if (deleted) {
                     if (!isExists) {
                         Logger(`STORAGE <x- DB:${filename}: deleted (hidden) Deleted on DB, but the file is  already not found on storage.`);
@@ -573,7 +596,7 @@ export class HiddenFileSync extends LiveSyncCommands {
                     return true;
                 }
                 if (!isExists) {
-                    await this.ensureDirectoryEx(filename);
+                    await this.vaultAccess.ensureDirectory(filename);
                     await this.plugin.vaultAccess.adapterWrite(filename, decodeBinary(fileOnDB.data), { mtime: fileOnDB.mtime, ctime: fileOnDB.ctime });
                     try {
                         //@ts-ignore internalAPI
@@ -642,7 +665,7 @@ export class HiddenFileSync extends LiveSyncCommands {
                     if (!keep && result) {
                         const isExists = await this.plugin.vaultAccess.adapterExists(filename);
                         if (!isExists) {
-                            await this.ensureDirectoryEx(filename);
+                            await this.vaultAccess.ensureDirectory(filename);
                         }
                         await this.plugin.vaultAccess.adapterWrite(filename, result);
                         const stat = await this.plugin.vaultAccess.adapterStat(filename);
@@ -709,8 +732,14 @@ export class HiddenFileSync extends LiveSyncCommands {
         filter: RegExp[],
         ignoreFilter: RegExp[]
     ) {
-
-        const w = await this.app.vault.adapter.list(path);
+        let w: ListedFiles;
+        try {
+            w = await this.app.vault.adapter.list(path);
+        } catch (ex) {
+            Logger(`Could not traverse(HiddenSync):${path}`, LOG_LEVEL_INFO);
+            Logger(ex, LOG_LEVEL_VERBOSE);
+            return [];
+        }
         const filesSrc = [
             ...w.files
                 .filter((e) => !ignoreList.some((ee) => e.endsWith(ee)))

src/CmdSetupLiveSync.ts

@@ -359,7 +359,7 @@ Of course, we are able to disable these features.`
             Logger(`Fetching chunks done`, LOG_LEVEL_NOTICE);
         }
     }
-    async fetchLocal() {
+    async fetchLocal(makeLocalChunkBeforeSync?: boolean) {
         this.suspendExtraSync();
         await this.askUseNewAdapter();
         this.plugin.settings.isConfigured = true;
@@ -367,9 +367,12 @@ Of course, we are able to disable these features.`
         await this.plugin.realizeSettingSyncMode();
         await this.resetLocalDatabase();
         await delay(1000);
-        await this.plugin.markRemoteResolved();
         await this.plugin.openDatabase();
         this.plugin.isReady = true;
+        if (makeLocalChunkBeforeSync) {
+            await this.plugin.initializeDatabase(true);
+        }
+        await this.plugin.markRemoteResolved();
         await delay(500);
         await this.plugin.replicateAllFromServer(true);
         await delay(1000);
@@ -379,24 +382,7 @@ Of course, we are able to disable these features.`
         await this.askHiddenFileConfiguration({ enableFetch: true });
     }
     async fetchLocalWithKeepLocal() {
-        this.suspendExtraSync();
-        await this.askUseNewAdapter();
-        this.plugin.settings.isConfigured = true;
-        await this.suspendReflectingDatabase();
-        await this.plugin.realizeSettingSyncMode();
-        await this.resetLocalDatabase();
-        await delay(1000);
-        await this.plugin.initializeDatabase(true);
-        await this.plugin.markRemoteResolved();
-        await this.plugin.openDatabase();
-        this.plugin.isReady = true;
-        await delay(500);
-        await this.plugin.replicateAllFromServer(true);
-        await delay(1000);
-        await this.plugin.replicateAllFromServer(true);
-        await this.fetchRemoteChunks();
-        await this.resumeReflectingDatabase();
-        await this.askHiddenFileConfiguration({ enableFetch: true });
+        return await this.fetchLocal(true);
     }
     async rebuildRemote() {
         this.suspendExtraSync();

src/DocumentHistoryModal.ts

@@ -21,6 +21,7 @@ function isComparableTextDecode(path: string) {
     return ["json"].includes(ext)
 }
 function readDocument(w: LoadedEntry) {
+    if (w.data.length == 0) return "";
     if (isImage(w.path)) {
         return new Uint8Array(decodeBinary(w.data));
     }
@@ -71,7 +72,7 @@ export class DocumentHistoryModal extends Modal {
         }
         const db = this.plugin.localDatabase;
         try {
-            const w = await db.localDatabase.get(this.id, { revs_info: true });
+            const w = await db.getRaw(this.id, { revs_info: true });
             this.revs_info = w._revs_info?.filter((e) => e?.status == "available") ?? [];
             this.range.max = `${Math.max(this.revs_info.length - 1, 0)}`;
             this.range.value = this.range.max;

src/ObsidianLiveSyncSettingTab.ts

@@ -663,7 +663,7 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
                     .setWarning()
                     .setDisabled(false)
                     .onClick(async () => {
-                        await rebuildDB("localOnly");
+                        await rebuildDB("localOnlyWithChunks");
                     })
             )
             .addButton((button) =>
@@ -732,7 +732,7 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
             }
         };
 
-        const rebuildDB = async (method: "localOnly" | "remoteOnly" | "rebuildBothByThisDevice") => {
+        const rebuildDB = async (method: "localOnly" | "remoteOnly" | "rebuildBothByThisDevice" | "localOnlyWithChunks") => {
             if (encrypt && passphrase == "") {
                 Logger("If you enable encryption, you have to set the passphrase", LOG_LEVEL_NOTICE);
                 return;
@@ -1053,9 +1053,9 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
                             if (!this.plugin.settings.isConfigured) {
                                 this.plugin.settings.isConfigured = true;
                                 await this.plugin.saveSettings();
-                                await rebuildDB("localOnly");
+                                await rebuildDB("localOnlyWithChunks");
                                 Logger("All done! Please set up subsequent devices with 'Copy current settings as a new setup URI' and 'Use the copied setup URI'.", LOG_LEVEL_NOTICE);
-                                await this.plugin.addOnSetup.command_openSetupURI();
+                                await this.plugin.addOnSetup.command_copySetupURI();
                             } else {
                                 this.askReload();
                             }
@@ -1736,7 +1736,7 @@ ${stringifyYaml(pluginConfig)}`;
 
                                     el.appendChild(el.createEl("button", { text: "Storage -> Database" }, buttonEl => {
                                         buttonEl.onClickEvent(() => {
-                                            this.plugin.updateIntoDB(file, false, undefined, true);
+                                            this.plugin.updateIntoDB(file, undefined, true);
                                             el.remove();
                                         })
                                     }))
@@ -1867,15 +1867,15 @@ ${stringifyYaml(pluginConfig)}`;
                 })
             );
 
-        new Setting(containerHatchEl)
-            .setName("Do not pace synchronization")
-            .setDesc("If this toggle enabled, synchronisation will not be paced by queued entries. If synchronisation has been deadlocked, please make this enabled once.")
-            .addToggle((toggle) =>
-                toggle.setValue(this.plugin.settings.doNotPaceReplication).onChange(async (value) => {
-                    this.plugin.settings.doNotPaceReplication = value;
-                    await this.plugin.saveSettings();
-                })
-            );
+        // new Setting(containerHatchEl)
+        //     .setName("Do not pace synchronization")
+        //     .setDesc("If this toggle enabled, synchronisation will not be paced by queued entries. If synchronisation has been deadlocked, please make this enabled once.")
+        //     .addToggle((toggle) =>
+        //         toggle.setValue(this.plugin.settings.doNotPaceReplication).onChange(async (value) => {
+        //             this.plugin.settings.doNotPaceReplication = value;
+        //             await this.plugin.saveSettings();
+        //         })
+        //     );
         containerHatchEl.createEl("h4", {
             text: sanitizeHTMLToDom(`Compatibility`),
             cls: "wizardHidden"
@@ -1922,7 +1922,7 @@ ${stringifyYaml(pluginConfig)}`;
                 toggle.setValue(!this.plugin.settings.useIndexedDBAdapter).onChange(async (value) => {
                     this.plugin.settings.useIndexedDBAdapter = !value;
                     await this.plugin.saveSettings();
-                    await rebuildDB("localOnly");
+                    await rebuildDB("localOnlyWithChunks");
                 })
             );
 
@@ -2119,6 +2119,19 @@ ${stringifyYaml(pluginConfig)}`;
                     .setButtonText("Fetch")
                     .setWarning()
                     .setDisabled(false)
+                    .onClick(async () => {
+                        await rebuildDB("localOnlyWithChunks");
+                    })
+            )
+
+        new Setting(containerMaintenanceEl)
+            .setName("Fetch rebuilt DB with all remote chunks")
+            .setDesc("Restore or reconstruct local database from remote database but use remote chunk .")
+            .addButton((button) =>
+                button
+                    .setButtonText("Fetch all")
+                    .setWarning()
+                    .setDisabled(false)
                     .onClick(async () => {
                         await rebuildDB("localOnly");
                     })

src/SerializedFileAccess.ts

@@ -1,5 +1,6 @@
 import { type App, TFile, type DataWriteOptions, TFolder, TAbstractFile } from "./deps";
 import { serialized } from "./lib/src/lock";
+import { Logger } from "./lib/src/logger";
 import type { FilePath } from "./lib/src/types";
 import { createBinaryBlob, isDocContentSame } from "./lib/src/utils";
 import type { InternalFileInfo } from "./types";
@@ -107,6 +108,15 @@ export class SerializedFileAccess {
             return await processWriteFile(path, () => this.app.vault.createBinary(path, toArrayBuffer(data), options));
         }
     }
+
+    trigger(name: string, ...data: any[]) {
+        return this.app.vault.trigger(name, ...data);
+    }
+
+    async adapterAppend(normalizedPath: string, data: string, options?: DataWriteOptions) {
+        return await this.app.vault.adapter.append(normalizedPath, data, options)
+    }
+
     async delete(file: TFile | TFolder, force = false) {
         return await processWriteFile(file, () => this.app.vault.delete(file, force));
     }
@@ -127,6 +137,31 @@ export class SerializedFileAccess {
         // }
     }
 
+    getFiles() {
+        return this.app.vault.getFiles();
+    }
+
+    async ensureDirectory(fullPath: string) {
+        const pathElements = fullPath.split("/");
+        pathElements.pop();
+        let c = "";
+        for (const v of pathElements) {
+            c += v;
+            try {
+                await this.app.vault.adapter.mkdir(c);
+            } catch (ex) {
+                // basically skip exceptions.
+                if (ex.message && ex.message == "Folder already exists.") {
+                    // especially this message is.
+                } else {
+                    Logger("Folder Create Error");
+                    Logger(ex);
+                }
+            }
+            c += "/";
+        }
+    }
+
 
     touchedFiles: string[] = [];
 

src/deps.ts

@@ -4,7 +4,7 @@ export {
     addIcon, App, debounce, Editor, FuzzySuggestModal, MarkdownRenderer, MarkdownView, Modal, Notice, Platform, Plugin, PluginSettingTab, requestUrl, sanitizeHTMLToDom, Setting, stringifyYaml, TAbstractFile, TextAreaComponent, TFile, TFolder,
     parseYaml, ItemView, WorkspaceLeaf
 } from "obsidian";
-export type { DataWriteOptions, PluginManifest, RequestUrlParam, RequestUrlResponse, MarkdownFileInfo } from "obsidian";
+export type { DataWriteOptions, PluginManifest, RequestUrlParam, RequestUrlResponse, MarkdownFileInfo, ListedFiles } from "obsidian";
 import {
     normalizePath as normalizePath_
 } from "obsidian";

src/main.ts

@@ -84,7 +84,6 @@ export default class ObsidianLiveSyncPlugin extends Plugin
         this._suspended = value;
     }
     deviceAndVaultName = "";
-    isMobile = false;
     isReady = false;
     packageVersion = "";
     manifestVersion = "";
@@ -117,6 +116,8 @@ export default class ObsidianLiveSyncPlugin extends Plugin
         return this.isMobile;
     }
 
+    requestCount = reactiveSource(0);
+    responseCount = reactiveSource(0);
     processReplication = (e: PouchDB.Core.ExistingDocument<EntryDoc>[]) => this.parseReplicationResult(e);
     async connectRemoteCouchDB(uri: string, auth: { username: string; password: string }, disableRequestURI: boolean, passphrase: string | false, useDynamicIterationCount: boolean, performSetup: boolean, skipInfo: boolean): Promise<string | { db: PouchDB.Database<EntryDoc>; info: PouchDB.Core.DatabaseInfo }> {
         if (!isValidRemoteCouchDBURI(uri)) return "Remote URI is not valid";
@@ -172,6 +173,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin
                     };
 
                     try {
+                        this.requestCount.value = this.requestCount.value + 1;
                         const r = await fetchByAPI(requestParam);
                         if (method == "POST" || method == "PUT") {
                             this.last_successful_post = r.status - (r.status % 100) == 200;
@@ -193,12 +195,15 @@ export default class ObsidianLiveSyncPlugin extends Plugin
                         }
                         Logger(ex);
                         throw ex;
+                    } finally {
+                        this.responseCount.value = this.responseCount.value + 1;
                     }
                 }
 
                 // -old implementation
 
                 try {
+                    this.requestCount.value = this.requestCount.value + 1;
                     const response: Response = await fetch(url, opts);
                     if (method == "POST" || method == "PUT") {
                         this.last_successful_post = response.ok;
@@ -215,6 +220,8 @@ export default class ObsidianLiveSyncPlugin extends Plugin
                     }
                     Logger(ex);
                     throw ex;
+                } finally {
+                    this.responseCount.value = this.responseCount.value + 1;
                 }
                 // return await fetch(url, opts);
             },
@@ -240,6 +247,22 @@ export default class ObsidianLiveSyncPlugin extends Plugin
         }
     }
 
+    get isMobile() {
+        // @ts-ignore: internal API
+        return this.app.isMobile
+    }
+
+    get vaultName() {
+        return this.app.vault.getName()
+    }
+    getActiveFile() {
+        return this.app.workspace.getActiveFile();
+    }
+
+    get appId() {
+        return `${("appId" in this.app ? this.app.appId : "")}`;
+    }
+
     id2path(id: DocumentID, entry?: EntryHasPath, stripPrefix?: boolean): FilePathWithPrefix {
         const tempId = id2path(id, entry);
         if (stripPrefix && isInternalMetadata(tempId)) {
@@ -301,13 +324,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin
     // end interfaces
 
     getVaultName(): string {
-        return this.app.vault.getName() + (this.settings?.additionalSuffixOfDatabaseName ? ("-" + this.settings.additionalSuffixOfDatabaseName) : "");
-    }
-
-    setInterval(handler: () => any, timeout?: number): number {
-        const timer = window.setInterval(handler, timeout);
-        this.registerInterval(timer);
-        return timer;
+        return this.vaultName + (this.settings?.additionalSuffixOfDatabaseName ? ("-" + this.settings.additionalSuffixOfDatabaseName) : "");
     }
 
     isFlagFileExist(path: string) {
@@ -353,7 +370,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin
         }
         notes.sort((a, b) => b.mtime - a.mtime);
         const notesList = notes.map(e => e.dispPath);
-        const target = await askSelectString(this.app, "File to view History", notesList);
+        const target = await this.askSelectString("File to view History", notesList);
         if (target) {
             const targetId = notes.find(e => e.dispPath == target);
             this.showHistory(targetId.path, targetId.id);
@@ -371,7 +388,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin
             Logger("There are no conflicted documents", LOG_LEVEL_NOTICE);
             return false;
         }
-        const target = await askSelectString(this.app, "File to resolve conflict", notesList);
+        const target = await this.askSelectString("File to resolve conflict", notesList);
         if (target) {
             const targetItem = notes.find(e => e.dispPath == target);
             this.resolveConflicted(targetItem.path);
@@ -450,10 +467,7 @@ Click anywhere to stop counting down.
             const ret = await confirmWithMessage(this, "Welcome to Self-hosted LiveSync", message, [USE_SETUP, OPEN_SETUP, DISMISS], DISMISS, 40);
             if (ret === OPEN_SETUP) {
                 try {
-                    //@ts-ignore: undocumented api
-                    this.app.setting.open();
-                    //@ts-ignore: undocumented api
-                    this.app.setting.openTabById("obsidian-livesync");
+                    this.openSetting();
                 } catch (ex) {
                     Logger("Something went wrong on opening setting dialog, please open it manually", LOG_LEVEL_NOTICE);
                 }
@@ -474,22 +488,20 @@ Click anywhere to stop counting down.
                     Logger(`${FLAGMD_REDFLAG2} or ${FLAGMD_REDFLAG2_HR} has been detected! Self-hosted LiveSync suspends all sync and rebuild everything.`, LOG_LEVEL_NOTICE);
                     await this.addOnSetup.rebuildEverything();
                     await this.deleteRedFlag2();
-                    if (await askYesNo(this.app, "Do you want to disable Suspend file watching and restart obsidian now?") == "yes") {
+                    if (await this.askYesNo("Do you want to disable Suspend file watching and restart obsidian now?") == "yes") {
                         this.settings.suspendFileWatching = false;
                         await this.saveSettings();
-                        // @ts-ignore
-                        this.app.commands.executeCommandById("app:reload")
+                        this.performAppReload();
                     }
                 } else if (this.isRedFlag3Raised()) {
                     Logger(`${FLAGMD_REDFLAG3} or ${FLAGMD_REDFLAG3_HR} has been detected! Self-hosted LiveSync will discard the local database and fetch everything from the remote once again.`, LOG_LEVEL_NOTICE);
                     await this.addOnSetup.fetchLocal();
                     await this.deleteRedFlag3();
                     if (this.settings.suspendFileWatching) {
-                        if (await askYesNo(this.app, "Do you want to disable Suspend file watching and restart obsidian now?") == "yes") {
+                        if (await this.askYesNo("Do you want to disable Suspend file watching and restart obsidian now?") == "yes") {
                             this.settings.suspendFileWatching = false;
                             await this.saveSettings();
-                            // @ts-ignore
-                            this.app.commands.executeCommandById("app:reload")
+                            this.performAppReload();
                         }
                     }
                 } else {
@@ -538,8 +550,7 @@ Click anywhere to stop counting down.
             this.askInPopup(`conflicting-detected-on-safety`, `Some files have been left conflicted! Press {HERE} to resolve them, or you can do it later by "Pick a file to resolve conflict`, (anchor) => {
                 anchor.text = "HERE";
                 anchor.addEventListener("click", () => {
-                    // @ts-ignore
-                    this.app.commands.executeCommandById("obsidian-livesync:livesync-all-conflictcheck");
+                    this.performCommand("obsidian-livesync:livesync-all-conflictcheck");
                 });
             }
             );
@@ -621,7 +632,7 @@ Note: We can always able to read V1 format. It will be progressively converted.
             //     this.localDatabase.getDBEntry(getPathFromTFile(file), {}, true, false);
             // },
             callback: () => {
-                const file = this.app.workspace.getActiveFile();
+                const file = this.getActiveFile();
                 if (!file) return;
                 this.localDatabase.getDBEntry(getPathFromTFile(file), {}, true, false);
             },
@@ -670,7 +681,7 @@ Note: We can always able to read V1 format. It will be progressively converted.
             id: "livesync-history",
             name: "Show history",
             callback: () => {
-                const file = this.app.workspace.getActiveFile();
+                const file = this.getActiveFile();
                 if (file) this.showHistory(file, null);
             }
         });
@@ -783,8 +794,7 @@ Note: We can always able to read V1 format. It will be progressively converted.
         }
 
         //@ts-ignore
-        if (this.app.isMobile) {
-            this.isMobile = true;
+        if (this.isMobile) {
             this.settings.disableRequestURI = true;
         }
         if (last_version && Number(last_version) < VER) {
@@ -861,8 +871,6 @@ Note: We can always able to read V1 format. It will be progressively converted.
         }
         const vaultName = this.getVaultName();
         Logger("Waiting for ready...");
-        //@ts-ignore
-        this.isMobile = this.app.isMobile;
         this.localDatabase = new LiveSyncLocalDB(vaultName, this);
         initializeStores(vaultName);
         return await this.localDatabase.initializeDatabase();
@@ -925,7 +933,7 @@ Note: We can always able to read V1 format. It will be progressively converted.
             if (JSON.stringify(settings) !== JSON.stringify(DEFAULT_SETTINGS)) {
                 settings.isConfigured = true;
             } else {
-                settings.additionalSuffixOfDatabaseName = `${("appId" in this.app ? this.app.appId : "")}`
+                settings.additionalSuffixOfDatabaseName = this.appId;
                 settings.isConfigured = false;
             }
         }
@@ -1049,7 +1057,7 @@ Note: We can always able to read V1 format. It will be progressively converted.
     }
 
     async parseSettingFromMarkdown(filename: string, data?: string) {
-        const file = this.app.vault.getAbstractFileByPath(filename);
+        const file = this.vaultAccess.getAbstractFileByPath(filename);
         if (!(file instanceof TFile)) return {
             preamble: "",
             body: "",
@@ -1058,7 +1066,7 @@ Note: We can always able to read V1 format. It will be progressively converted.
         if (data) {
             return this.extractSettingFromWholeText(data);
         }
-        const parseData = data ?? await this.app.vault.read(file);
+        const parseData = data ?? await this.vaultAccess.vaultRead(file);
         return this.extractSettingFromWholeText(parseData);
     }
 
@@ -1107,7 +1115,7 @@ Note: We can always able to read V1 format. It will be progressively converted.
                 const APPLY_AND_REBUILD = "Apply settings and restart obsidian with red_flag_rebuild.md";
                 const APPLY_AND_FETCH = "Apply settings and restart obsidian with red_flag_fetch.md";
                 const CANCEL = "Cancel";
-                const result = await askSelectString(this.app, "Ready for apply the setting.", [APPLY_AND_RESTART, APPLY_ONLY, APPLY_AND_FETCH, APPLY_AND_REBUILD, CANCEL]);
+                const result = await this.askSelectString("Ready for apply the setting.", [APPLY_AND_RESTART, APPLY_ONLY, APPLY_AND_FETCH, APPLY_AND_REBUILD, CANCEL]);
                 if (result == APPLY_ONLY || result == APPLY_AND_RESTART || result == APPLY_AND_REBUILD || result == APPLY_AND_FETCH) {
                     this.settings = settingToApply;
                     await this.saveSettingData();
@@ -1116,13 +1124,12 @@ Note: We can always able to read V1 format. It will be progressively converted.
                         return;
                     }
                     if (result == APPLY_AND_REBUILD) {
-                        await this.app.vault.create(FLAGMD_REDFLAG2_HR, "");
+                        await this.vaultAccess.vaultCreate(FLAGMD_REDFLAG2_HR, "");
                     }
                     if (result == APPLY_AND_FETCH) {
-                        await this.app.vault.create(FLAGMD_REDFLAG3_HR, "");
+                        await this.vaultAccess.vaultCreate(FLAGMD_REDFLAG3_HR, "");
                     }
-                    // @ts-ignore
-                    this.app.commands.executeCommandById("app:reload");
+                    this.performAppReload();
                 }
             }
             )
@@ -1142,11 +1149,11 @@ Note: We can always able to read V1 format. It will be progressively converted.
 
     async saveSettingToMarkdown(filename: string) {
         const saveData = this.generateSettingForMarkdown();
-        let file = this.app.vault.getAbstractFileByPath(filename);
+        let file = this.vaultAccess.getAbstractFileByPath(filename);
 
 
         if (!file) {
-            await this.ensureDirectoryEx(filename);
+            await this.vaultAccess.ensureDirectory(filename);
             const initialContent = `This file contains Self-hosted LiveSync settings as YAML.
 Except for the \`livesync-setting\` code block, we can add a note for free.
 
@@ -1159,21 +1166,21 @@ We can perform a command in this file.
 
 
 `
-            file = await this.app.vault.create(filename, initialContent + SETTING_HEADER + "\n" + SETTING_FOOTER);
+            file = await this.vaultAccess.vaultCreate(filename, initialContent + SETTING_HEADER + "\n" + SETTING_FOOTER);
         }
         if (!(file instanceof TFile)) {
             Logger(`Markdown Setting: ${filename} already exists as a folder`, LOG_LEVEL_NOTICE);
             return;
         }
 
-        const data = await this.app.vault.read(file);
+        const data = await this.vaultAccess.vaultRead(file);
         const { preamble, body, postscript } = this.extractSettingFromWholeText(data);
         const newBody = stringifyYaml(saveData);
 
         if (newBody == body) {
             Logger("Markdown setting: Nothing had been changed", LOG_LEVEL_VERBOSE);
         } else {
-            await this.app.vault.modify(file, preamble + SETTING_HEADER + newBody + SETTING_FOOTER + postscript);
+            await this.vaultAccess.vaultModify(file, preamble + SETTING_HEADER + newBody + SETTING_FOOTER + postscript);
             Logger(`Markdown setting: ${filename} has been updated!`, LOG_LEVEL_VERBOSE);
         }
     }
@@ -1216,8 +1223,7 @@ We can perform a command in this file.
         const _this = this;
         //@ts-ignore
         window.CodeMirrorAdapter.commands.save = () => {
-            //@ts-ignore
-            _this.app.commands.executeCommandById('editor:save-file');
+            _this.performCommand('editor:save-file');
         };
     }
     registerWatchEvents() {
@@ -1227,7 +1233,6 @@ We can perform a command in this file.
         this.registerDomEvent(window, "offline", this.watchOnline);
     }
 
-
     watchOnline() {
         scheduleTask("watch-online", 500, () => fireAndForget(() => this.watchOnlineAsync()));
     }
@@ -1329,7 +1334,7 @@ We can perform a command in this file.
                 fireAndForget(() => this.checkAndApplySettingFromMarkdown(queue.args.file.path, true));
                 const keyD1 = `file-last-proc-DELETED-${file.path}`;
                 await this.kvDB.set(keyD1, mtime);
-                if (!await this.updateIntoDB(targetFile, false, cache)) {
+                if (!await this.updateIntoDB(targetFile, cache)) {
                     Logger(`STORAGE -> DB: failed, cancel the relative operations: ${targetFile.path}`, LOG_LEVEL_INFO);
                     // cancel running queues and remove one of atomic operation
                     this.cancelRelativeEvent(queue);
@@ -1402,7 +1407,7 @@ We can perform a command in this file.
         if (file instanceof TFile) {
             try {
                 // Logger(`RENAMING.. ${file.path} into db`);
-                if (await this.updateIntoDB(file, false, cache)) {
+                if (await this.updateIntoDB(file, cache)) {
                     // Logger(`deleted ${oldFile} from db`);
                     await this.deleteFromDBbyPath(oldFile);
                 } else {
@@ -1448,9 +1453,9 @@ We can perform a command in this file.
             const logDate = `${PREFIXMD_LOGFILE}${time}.md`;
             const file = this.vaultAccess.getAbstractFileByPath(normalizePath(logDate));
             if (!file) {
-                this.app.vault.adapter.append(normalizePath(logDate), "```\n");
+                this.vaultAccess.adapterAppend(normalizePath(logDate), "```\n");
             }
-            this.app.vault.adapter.append(normalizePath(logDate), vaultName + ":" + newMessage + "\n");
+            this.vaultAccess.adapterAppend(normalizePath(logDate), vaultName + ":" + newMessage + "\n");
         }
         recentLogProcessor.enqueue(newMessage);
 
@@ -1488,28 +1493,6 @@ We can perform a command in this file.
             })
         }
     }
-
-    async ensureDirectory(fullPath: string) {
-        const pathElements = fullPath.split("/");
-        pathElements.pop();
-        let c = "";
-        for (const v of pathElements) {
-            c += v;
-            try {
-                await this.app.vault.createFolder(c);
-            } catch (ex) {
-                // basically skip exceptions.
-                if (ex.message && ex.message == "Folder already exists.") {
-                    // especially this message is.
-                } else {
-                    Logger("Folder Create Error");
-                    Logger(ex);
-                }
-            }
-            c += "/";
-        }
-    }
-
     async processEntryDoc(docEntry: EntryBody, file: TFile | undefined, force?: boolean) {
         const mode = file == undefined ? "create" : "modify";
 
@@ -1569,7 +1552,7 @@ We can perform a command in this file.
             return;
         }
         const writeData = doc.datatype == "newnote" ? decodeBinary(doc.data) : getDocData(doc.data);
-        await this.ensureDirectoryEx(path);
+        await this.vaultAccess.ensureDirectory(path);
         try {
             let outFile;
             let isChanged = true;
@@ -1584,7 +1567,7 @@ We can perform a command in this file.
             if (isChanged) {
                 Logger(msg + path);
                 this.vaultAccess.touch(outFile);
-                this.app.vault.trigger(mode, outFile);
+                this.vaultAccess.trigger(mode, outFile);
             } else {
                 Logger(msg + "Skipped, the file is the same: " + path, LOG_LEVEL_VERBOSE);
             }
@@ -1618,7 +1601,7 @@ We can perform a command in this file.
     queueConflictCheck(file: FilePathWithPrefix | TFile) {
         const path = file instanceof TFile ? getPathFromTFile(file) : file;
         if (this.settings.checkConflictOnlyOnOpen) {
-            const af = this.app.workspace.getActiveFile();
+            const af = this.getActiveFile();
             if (af && af.path != path) {
                 Logger(`${file} is conflicted, merging process has been postponed.`, LOG_LEVEL_NOTICE);
                 return;
@@ -1785,6 +1768,10 @@ We can perform a command in this file.
             const labelConflictProcessCount = conflictProcessCount ? `🔩${conflictProcessCount} ` : "";
             return `${labelReplication}${labelDBCount}${labelStorageCount}${labelChunkCount}${labelPluginScanCount}${labelHiddenFilesCount}${labelConflictProcessCount}`;
         })
+        const requestingStatLabel = reactive(() => {
+            const diff = this.requestCount.value - this.responseCount.value;
+            return diff != 0 ? "📲 " : "";
+        })
 
         const replicationStatLabel = reactive(() => {
             const e = this.replicationStat.value;
@@ -1834,8 +1821,9 @@ We can perform a command in this file.
             const { w, sent, pushLast, arrived, pullLast } = replicationStatLabel.value;
             const queued = queueCountLabel.value;
             const waiting = waitingLabel.value;
+            const networkActivity = requestingStatLabel.value;
             return {
-                message: `Sync: ${w} ↑${sent}${pushLast} ↓${arrived}${pullLast}${waiting} ${queued}`,
+                message: `${networkActivity}Sync: ${w} ↑${sent}${pushLast} ↓${arrived}${pullLast}${waiting} ${queued}`,
             };
         })
         const statusBarLabels = reactive(() => {
@@ -1914,7 +1902,7 @@ Even if you choose to clean up, you will see this option again if you exit Obsid
                         const CHOICE_DISMISS = "Dismiss";
                         const ret = await confirmWithMessage(this, "Cleaned", message, [CHOICE_FETCH, CHOICE_CLEAN, CHOICE_DISMISS], CHOICE_DISMISS, 30);
                         if (ret == CHOICE_FETCH) {
-                            await performRebuildDB(this, "localOnly");
+                            await performRebuildDB(this, "localOnlyWithChunks");
                         }
                         if (ret == CHOICE_CLEAN) {
                             const remoteDB = await this.getReplicator().connectRemoteCouchDBWithSetting(this.settings, this.getIsMobile(), true);
@@ -1948,7 +1936,7 @@ Or if you are sure know what had been happened, we can unlock the database from
                     const CHOICE_DISMISS = "Dismiss";
                     const ret = await confirmWithMessage(this, "Locked", message, [CHOICE_FETCH, CHOICE_DISMISS], CHOICE_DISMISS, 10);
                     if (ret == CHOICE_FETCH) {
-                        await performRebuildDB(this, "localOnly");
+                        await performRebuildDB(this, "localOnlyWithChunks");
                     }
                 }
             }
@@ -2014,7 +2002,6 @@ Or if you are sure know what had been happened, we can unlock the database from
             }
             return;
         }
-        let initialScan = false;
         if (showingNotice) {
             Logger("Initializing", LOG_LEVEL_NOTICE, "syncAll");
         }
@@ -2024,7 +2011,7 @@ Or if you are sure know what had been happened, we can unlock the database from
         await this.collectDeletedFiles();
 
         Logger("Collecting local files on the storage", LOG_LEVEL_VERBOSE);
-        const filesStorageSrc = this.app.vault.getFiles();
+        const filesStorageSrc = this.vaultAccess.getFiles();
 
         const filesStorage = [] as typeof filesStorageSrc;
         for (const f of filesStorageSrc) {
@@ -2047,11 +2034,7 @@ Or if you are sure know what had been happened, we can unlock the database from
         }
         Logger("Opening the key-value database", LOG_LEVEL_VERBOSE);
         const isInitialized = await (this.kvDB.get<boolean>("initialized")) || false;
-        // Make chunk bigger if it is the initial scan. There must be non-active docs.
-        if (filesDatabase.length == 0 && !isInitialized) {
-            initialScan = true;
-            Logger("Database looks empty, save files as initial sync data");
-        }
+
         const onlyInStorage = filesStorage.filter((e) => filesDatabase.indexOf(getPathFromTFile(e)) == -1);
         const onlyInDatabase = filesDatabase.filter((e) => filesStorageName.indexOf(e) == -1);
 
@@ -2092,69 +2075,62 @@ Or if you are sure know what had been happened, we can unlock the database from
         }
         initProcess.push(runAll("UPDATE DATABASE", onlyInStorage, async (e) => {
             if (!this.isFileSizeExceeded(e.stat.size)) {
-                await this.updateIntoDB(e, initialScan);
+                await this.updateIntoDB(e);
                 fireAndForget(() => this.checkAndApplySettingFromMarkdown(e.path, true));
             } else {
                 Logger(`UPDATE DATABASE: ${e.path} has been skipped due to file size exceeding the limit`, logLevel);
             }
         }));
-        if (!initialScan) {
-            initProcess.push(runAll("UPDATE STORAGE", onlyInDatabase, async (e) => {
-                const w = await this.localDatabase.getDBEntryMeta(e, {}, true);
-                if (w && !(w.deleted || w._deleted)) {
-                    if (!this.isFileSizeExceeded(w.size)) {
-                        await this.pullFile(e, filesStorage, false, null, false);
-                        fireAndForget(() => this.checkAndApplySettingFromMarkdown(e, true));
-                        Logger(`Check or pull from db:${e} OK`);
-                    } else {
-                        Logger(`UPDATE STORAGE: ${e} has been skipped due to file size exceeding the limit`, logLevel);
-                    }
-                } else if (w) {
-                    Logger(`Deletion history skipped: ${e}`, LOG_LEVEL_VERBOSE);
+        initProcess.push(runAll("UPDATE STORAGE", onlyInDatabase, async (e) => {
+            const w = await this.localDatabase.getDBEntryMeta(e, {}, true);
+            if (w && !(w.deleted || w._deleted)) {
+                if (!this.isFileSizeExceeded(w.size)) {
+                    await this.pullFile(e, filesStorage, false, null, false);
+                    fireAndForget(() => this.checkAndApplySettingFromMarkdown(e, true));
+                    Logger(`Check or pull from db:${e} OK`);
                 } else {
-                    Logger(`entry not found: ${e}`);
+                    Logger(`UPDATE STORAGE: ${e} has been skipped due to file size exceeding the limit`, logLevel);
                 }
-            }));
-        }
-        if (!initialScan) {
-            // let caches: { [key: string]: { storageMtime: number; docMtime: number } } = {};
-            // caches = await this.kvDB.get<{ [key: string]: { storageMtime: number; docMtime: number } }>("diff-caches") || {};
-            type FileDocPair = { file: TFile, id: DocumentID };
+            } else if (w) {
+                Logger(`Deletion history skipped: ${e}`, LOG_LEVEL_VERBOSE);
+            } else {
+                Logger(`entry not found: ${e}`);
+            }
+        }));
+        type FileDocPair = { file: TFile, id: DocumentID };
 
-            const processPrepareSyncFile = new QueueProcessor(
-                async (files) => {
-                    const file = files[0];
-                    const id = await this.path2id(getPathFromTFile(file));
-                    const pair: FileDocPair = { file, id };
-                    return [pair];
-                    // processSyncFile.enqueue(pair);
-                }
-                , { batchSize: 1, concurrentLimit: 10, delay: 0, suspended: true }, syncFiles);
-            processPrepareSyncFile
-                .pipeTo(
-                    new QueueProcessor(
-                        async (pairs) => {
-                            const docs = await this.localDatabase.allDocsRaw<EntryDoc>({ keys: pairs.map(e => e.id), include_docs: true });
-                            const docsMap = docs.rows.reduce((p, c) => ({ ...p, [c.id]: c.doc }), {} as Record<DocumentID, EntryDoc>);
-                            const syncFilesToSync = pairs.map((e) => ({ file: e.file, doc: docsMap[e.id] as LoadedEntry }));
-                            return syncFilesToSync;
-                        }
-                        , { batchSize: 10, concurrentLimit: 5, delay: 10, suspended: false }))
-                .pipeTo(
-                    new QueueProcessor(
-                        async (loadedPairs) => {
-                            const e = loadedPairs[0];
-                            await this.syncFileBetweenDBandStorage(e.file, e.doc, initialScan);
-                            return;
-                        }, { batchSize: 1, concurrentLimit: 5, delay: 10, suspended: false }
-                    ))
+        const processPrepareSyncFile = new QueueProcessor(
+            async (files) => {
+                const file = files[0];
+                const id = await this.path2id(getPathFromTFile(file));
+                const pair: FileDocPair = { file, id };
+                return [pair];
+                // processSyncFile.enqueue(pair);
+            }
+            , { batchSize: 1, concurrentLimit: 10, delay: 0, suspended: true }, syncFiles);
+        processPrepareSyncFile
+            .pipeTo(
+                new QueueProcessor(
+                    async (pairs) => {
+                        const docs = await this.localDatabase.allDocsRaw<EntryDoc>({ keys: pairs.map(e => e.id), include_docs: true });
+                        const docsMap = docs.rows.reduce((p, c) => ({ ...p, [c.id]: c.doc }), {} as Record<DocumentID, EntryDoc>);
+                        const syncFilesToSync = pairs.map((e) => ({ file: e.file, doc: docsMap[e.id] as LoadedEntry }));
+                        return syncFilesToSync;
+                    }
+                    , { batchSize: 10, concurrentLimit: 5, delay: 10, suspended: false }))
+            .pipeTo(
+                new QueueProcessor(
+                    async (loadedPairs) => {
+                        const e = loadedPairs[0];
+                        await this.syncFileBetweenDBandStorage(e.file, e.doc);
+                        return;
+                    }, { batchSize: 1, concurrentLimit: 5, delay: 10, suspended: false }
+                ))
 
-            processPrepareSyncFile.startPipeline();
-            initProcess.push(async () => {
-                await processPrepareSyncFile.waitForPipeline();
-                // await this.kvDB.set("diff-caches", caches);
-            })
-        }
+        processPrepareSyncFile.startPipeline();
+        initProcess.push(async () => {
+            await processPrepareSyncFile.waitForPipeline();
+        })
         await Promise.all(initProcess);
 
         // this.setStatusBarText(`NOW TRACKING!`);
@@ -2533,7 +2509,7 @@ Or if you are sure know what had been happened, we can unlock the database from
                 return;
             }
             if (this.settings.showMergeDialogOnlyOnActive) {
-                const af = this.app.workspace.getActiveFile();
+                const af = this.getActiveFile();
                 if (af && af.path != filename) {
                     Logger(`${filename} is conflicted. Merging process has been postponed to the file have got opened.`, LOG_LEVEL_NOTICE);
                     return;
@@ -2647,7 +2623,7 @@ Or if you are sure know what had been happened, we can unlock the database from
         //when to opened file;
     }
 
-    async syncFileBetweenDBandStorage(file: TFile, doc: LoadedEntry, initialScan: boolean) {
+    async syncFileBetweenDBandStorage(file: TFile, doc: LoadedEntry) {
         if (!doc) {
             throw new Error(`Missing doc:${(file as any).path}`)
         }
@@ -2665,7 +2641,7 @@ Or if you are sure know what had been happened, we can unlock the database from
             case BASE_IS_NEW:
                 if (!this.isFileSizeExceeded(file.stat.size)) {
                     Logger("STORAGE -> DB :" + file.path);
-                    await this.updateIntoDB(file, initialScan);
+                    await this.updateIntoDB(file);
                     fireAndForget(() => this.checkAndApplySettingFromMarkdown(file.path, true));
                 } else {
                     Logger(`STORAGE -> DB : ${file.path} has been skipped due to file size exceeding the limit`, LOG_LEVEL_NOTICE);
@@ -2694,7 +2670,7 @@ Or if you are sure know what had been happened, we can unlock the database from
 
     }
 
-    async updateIntoDB(file: TFile, initialScan?: boolean, cache?: CacheData, force?: boolean) {
+    async updateIntoDB(file: TFile, cache?: CacheData, force?: boolean) {
         if (!await this.isTargetFile(file)) return true;
         if (shouldBeIgnored(file.path)) {
             return true;
@@ -2779,7 +2755,7 @@ Or if you are sure know what had been happened, we can unlock the database from
             Logger(msg + " Skip " + fullPath, LOG_LEVEL_VERBOSE);
             return true;
         }
-        const ret = await this.localDatabase.putDBEntry(d, initialScan);
+        const ret = await this.localDatabase.putDBEntry(d);
         if (ret !== false) {
             Logger(msg + fullPath);
             if (this.settings.syncOnSave && !this.suspended) {
@@ -2819,27 +2795,6 @@ Or if you are sure know what had been happened, we can unlock the database from
         await this.replicator.tryCreateRemoteDatabase(this.settings);
     }
 
-    async ensureDirectoryEx(fullPath: string) {
-        const pathElements = fullPath.split("/");
-        pathElements.pop();
-        let c = "";
-        for (const v of pathElements) {
-            c += v;
-            try {
-                await this.app.vault.adapter.mkdir(c);
-            } catch (ex) {
-                // basically skip exceptions.
-                if (ex.message && ex.message == "Folder already exists.") {
-                    // especially this message is.
-                } else {
-                    Logger("Folder Create Error");
-                    Logger(ex);
-                }
-            }
-            c += "/";
-        }
-    }
-
     filterTargetFiles(files: InternalFileInfo[], targetFiles: string[] | false = false) {
         const ignorePatterns = this.settings.syncInternalFilesIgnorePatterns
             .replace(/\n| /g, "")
@@ -2864,7 +2819,7 @@ Or if you are sure know what had been happened, we can unlock the database from
         const mtimeB = ("mtime" in revBDoc && revBDoc.mtime) || 0;
         const delRev = mtimeA < mtimeB ? revA : revB;
         // delete older one.
-        await this.localDatabase.removeRaw(id, delRev);
+        await this.localDatabase.removeRevision(id, delRev);
         Logger(`Older one has been deleted:${this.getPath(doc)}`);
         return true;
     }
@@ -2948,6 +2903,12 @@ Or if you are sure know what had been happened, we can unlock the database from
         });
     }
 
+    askYesNo(message: string): Promise<"yes" | "no"> {
+        return askYesNo(this.app, message);
+    }
+    askSelectString(message: string, items: string[]): Promise<string> {
+        return askSelectString(this.app, message, items);
+    }
 
     askInPopup(key: string, dialogText: string, anchorCallback: (anchor: HTMLAnchorElement) => void) {
 
@@ -2966,7 +2927,6 @@ Or if you are sure know what had been happened, we can unlock the database from
         const popupKey = "popup-" + key;
         scheduleTask(popupKey, 1000, async () => {
             const popup = await memoIfNotExist(popupKey, () => new Notice(fragment, 0));
-            //@ts-ignore
             const isShown = popup?.noticeEl?.isShown();
             if (!isShown) {
                 memoObject(popupKey, new Notice(fragment, 0));
@@ -2975,7 +2935,6 @@ Or if you are sure know what had been happened, we can unlock the database from
                 const popup = retrieveMemoObject<Notice>(popupKey);
                 if (!popup)
                     return;
-                //@ts-ignore
                 if (popup?.noticeEl?.isShown()) {
                     popup.hide();
                 }
@@ -2983,5 +2942,19 @@ Or if you are sure know what had been happened, we can unlock the database from
             });
         });
     }
+    openSetting() {
+        //@ts-ignore: undocumented api
+        this.app.setting.open();
+        //@ts-ignore: undocumented api
+        this.app.setting.openTabById("obsidian-livesync");
+    }
+
+    performAppReload() {
+        this.performCommand("app:reload");
+    }
+    performCommand(id: string) {
+        // @ts-ignore
+        this.app.commands.executeCommandById(id)
+    }
 }
 

src/utils.ts

@@ -405,10 +405,13 @@ export const requestToCouchDB = async (baseUri: string, username: string, passwo
     return await _requestToCouchDB(baseUri, username, password, origin, uri, body, method);
 };
 
-export async function performRebuildDB(plugin: ObsidianLiveSyncPlugin, method: "localOnly" | "remoteOnly" | "rebuildBothByThisDevice") {
+export async function performRebuildDB(plugin: ObsidianLiveSyncPlugin, method: "localOnly" | "remoteOnly" | "rebuildBothByThisDevice" | "localOnlyWithChunks") {
     if (method == "localOnly") {
         await plugin.addOnSetup.fetchLocal();
     }
+    if (method == "localOnlyWithChunks") {
+        await plugin.addOnSetup.fetchLocal(true);
+    }
     if (method == "remoteOnly") {
         await plugin.addOnSetup.rebuildRemote();
     }

updates.md

@@ -10,6 +10,35 @@ Note: we got a very performance improvement.
 Note at 0.22.2: **Now, to rescue mobile devices, Maximum file size is set to 50 by default**. Please configure the limit as you need. If you do not want to limit the sizes, set zero manually, please.
 
 #### Version history
+- 0.22.9
+  - Fixed:
+    - Fixed a bug on `fetch chunks on demand` that could not fetch the chunks on demand.
+  - Improved:
+    - `fetch chunks on demand` works more smoothly.
+    - Initialisation `Fetch` is now more efficient.
+  - Tidied:
+    - Removed some meaningless codes.
+- 0.22.8
+  - Fixed:
+    - Now fetch and unlock the locked remote database works well again.
+    - No longer crash on symbolic links inside hidden folders.
+  - Improved:
+    - Chunks are now created more efficiently.
+      - Splitting old notes into a larger chunk.
+    - Better performance in saving notes.
+    - Network activities are indicated as an icon.
+    - Less memory used for binary processing.
+  - Tidied:
+    - Cleaned unused functions up.
+    - Sorting out the codes that have become nonsense.
+  - Changed:
+    - Now no longer `fetch chunks on demand` needs `Pacing replication`
+      - The setting `Do not pace synchronization` has been deleted.
+- 0.22.7
+  - Fixed:
+    - No longer deleted hidden files were ignored.
+    - The document history dialogue is now able to process the deleted revisions.
+    - Deletion of a hidden file is now surely performed even if the file is already conflicted.
 - 0.22.6
   - Fixed:
     - Fixed a problem with synchronisation taking a long time to start in some cases.