An minor note added

### Fixed

- Off-loaded chunking have been fixed to ensure proper functionality (#693).
- Chunk document ID assignment has been fixed.
- Replication prevention message during version up detection has been improved (#686).
- `Keep A` and `Keep B` on Conflict resolving dialogue has been renamed to `Use Base` and `Use Conflicted` (#691).

### Improved

- Metadata and content-size unmatched documents are now detected and reported, prevented to be applied to the storage.

### New Features

- `Scan for Broken files` has been implemented on `Hatch` -> `TroubleShooting`.

### Refactored

- Off-loaded processes have been refactored for the better maintainability.
- Removed unused code.

docs/design_docs/chunk_aggregation_by_prefix.md

@@ -0,0 +1,122 @@
+# [WITHDRAWN] Chunk Aggregation by Prefix 
+
+## Goal
+
+To address the "document explosion" and storage bloat issues caused by the current chunking mechanism, while preserving the benefits of content-addressable storage and efficient delta synchronisation. This design aims to significantly reduce the number of documents in the database and simplify Garbage Collection (GC).
+
+## Motivation
+
+Our current synchronisation solution splits files into content-defined chunks, with each chunk stored as a separate document in CouchDB, identified by its hash. This architecture effectively leverages CouchDB's replication for automatic deduplication and efficient transfer.
+
+However, this approach faces significant challenges as the number of files and edits increases:
+1.  **Document Explosion:** A large vault can generate millions of chunk documents, severely degrading CouchDB's performance, particularly during view building and replication.
+2.  **Storage Bloat & GC Difficulty:** Obsolete chunks generated during edits are difficult to identify and remove. Since CouchDB's deletion (`_deleted: true`) is a soft delete, and compaction is a heavy, space-intensive operation, unused chunks perpetually consume storage, making GC impractical for many users.
+3.  **The "Eden" Problem:** A previous attempt, "Keep newborn chunks in Eden", aimed to mitigate this by embedding volatile chunks within the parent document. While it reduced the number of standalone chunks, it introduced a new issue: the parent document's history (`_revs_info`) became excessively large, causing its own form of database bloat and making compaction equally necessary but difficult to manage.
+
+This new design addresses the root cause—the sheer number of documents—by aggregating chunks into sets.
+
+## Prerequisites
+
+- The new implementation must maintain the core benefit of deduplication to ensure efficient synchronisation.
+- The solution must not introduce a single point of bottleneck and should handle concurrent writes from multiple clients gracefully.
+- The system must provide a clear and feasible strategy for Garbage Collection.
+- The design should be forward-compatible, allowing for a smooth migration path for existing users.
+
+## Outlined Methods and Implementation Plans
+
+### Abstract
+
+This design introduces a two-tiered document structure to manage chunks: **Index Documents** and **Data Documents**. Chunks are no longer stored as individual documents. Instead, they are grouped into `Data Documents` based on a common hash prefix. The existence and location of each chunk are tracked by `Index Documents`, which are also grouped by the same prefix. This approach dramatically reduces the total document count.
+
+### Detailed Implementation
+
+**1. Document Structure:**
+
+-   **Index Document:** Maps chunk hashes to their corresponding Data Document ID. Identified by a prefix of the chunk hash.
+    -   `_id`: `idx:{prefix}` (e.g., `idx:a9f1b`)
+    -   Content:
+        ```json
+        {
+            "_id": "idx:a9f1b",
+            "_rev": "...",
+            "chunks": {
+                "a9f1b12...": "dat:a9f1b-001",
+                "a9f1b34...": "dat:a9f1b-001",
+                "a9f1b56...": "dat:a9f1b-002"
+            }
+        }
+        ```
+-   **Data Document:** Contains the actual chunk data as base64-encoded strings. Identified by a prefix and a sequential number.
+    -   `_id`: `dat:{prefix}-{sequence}` (e.g., `dat:a9f1b-001`)
+    -   Content:
+        ```json
+        {
+            "_id": "dat:a9f1b-001",
+            "_rev": "...",
+            "chunks": {
+                "a9f1b12...": "...", // base64 data
+                "a9f1b34...": "..."  // base64 data
+            }
+        }
+        ```
+
+**2. Configuration:**
+
+-   `chunk_prefix_length`: The number of characters from the start of a chunk hash to use as a prefix (e.g., `5`). This determines the granularity of aggregation.
+-   `data_doc_size_limit`: The maximum size for a single Data Document to prevent it from becoming too large (e.g., 1MB). When this limit is reached, a new Data Document with an incremented sequence number is created.
+
+**3. Write/Save Operation Flow:**
+
+When a client creates new chunks:
+1.  For each new chunk, determine its hash prefix.
+2.  Read the corresponding `Index Document` (e.g., `idx:a9f1b`).
+3.  From the index, determine which of the new chunks already exist in the database.
+4.  For the **truly new chunks only**:
+    a. Read the last `Data Document` for that prefix (e.g., `dat:a9f1b-005`).
+    b. If it is nearing its size limit, create a new one (`dat:a9f1b-006`).
+    c. Add the new chunk data to the Data Document and save it.
+5.  Update the `Index Document` with the locations of the newly added chunks.
+
+**4. Handling Write Conflicts:**
+
+Concurrent writes to the same `Index Document` or `Data Document` from multiple clients will cause conflicts (409 Conflict). This is expected and must be handled gracefully. Since additions are incremental, the client application must implement a **retry-and-merge loop**:
+1.  Attempt to save the document.
+2.  On a conflict, re-fetch the latest version of the document from the server.
+3.  Merge its own changes into the latest version.
+4.  Attempt to save again.
+5.  Repeat until successful or a retry limit is reached.
+
+**5. Garbage Collection (GC):**
+
+GC becomes a manageable, periodic batch process:
+1.  Scan all file metadata documents to build a master set of all *currently referenced* chunk hashes.
+2.  Iterate through all `Index Documents`. For each chunk listed:
+    a. If the chunk hash is not in the master reference set, it is garbage.
+    b. Remove the garbage entry from the `Index Document`.
+    c. Remove the corresponding data from its `Data Document`.
+3.  If a `Data Document` becomes empty after this process, it can be deleted.
+
+## Test Strategy
+
+1.  **Unit Tests:** Implement tests for the conflict resolution logic (retry-and-merge loop) to ensure robustness.
+2.  **Integration Tests:**
+    -   Verify that concurrent writes from multiple simulated clients result in a consistent, merged state without data loss.
+    -   Run a full synchronisation scenario and confirm the resulting database has a significantly lower document count compared to the previous implementation.
+3.  **GC Test:** Simulate a scenario where files are deleted, run the GC process, and verify that orphaned chunks are correctly removed from both Index and Data documents, and that storage is reclaimed after compaction.
+4.  **Migration Test:** Develop and test a "rebuild" process for existing users, which migrates their chunk data into the new aggregated structure.
+
+## Documentation Strategy
+
+-   This design document will be published to explain the new architecture.
+-   The configuration options (`chunk_prefix_length`, etc.) will be documented for advanced users.
+-   A guide for the migration/rebuild process will be provided.
+
+## Future Work
+
+The separation of index and data opens up a powerful possibility. While this design initially implements both within CouchDB, the `Data Documents` could be offloaded to a dedicated object storage service such as **S3, MinIO, or Cloudflare R2**.
+
+In such a hybrid model, CouchDB would handle only the lightweight `Index Documents` and file metadata, serving as a high-speed synchronisation and coordination layer. The bulky chunk data would reside in a more cost-effective and scalable blob store. This would represent the ultimate evolution of this architecture, combining the best of both worlds.
+
+## Consideration and Conclusion
+
+This design directly addresses the scalability limitations of the original chunk-per-document model. By aggregating chunks into sets, it significantly reduces the document count, which in turn improves database performance and makes maintenance feasible. The explicit handling of write conflicts and a clear strategy for garbage collection make this a robust and sustainable long-term solution. It effectively resolves the problems identified in previous approaches, including the "Eden" experiment, by tackling the root cause of database bloat. This architecture provides a solid foundation for future growth and scalability.

docs/design_docs/intention_of_chunks.md

@@ -0,0 +1,127 @@
+# [WIP] The design intent explanation for using metadata and chunks
+
+## Abstract
+
+## Goal
+
+- To explain the following:
+  - What metadata and chunks are
+  - The design intent of using metadata and chunks
+
+## Background and Motivation
+
+We are using PouchDB and CouchDB for storing files and synchronising them. PouchDB is a JavaScript database that stores data on the device (browser, and of course, Obsidian), while CouchDB is a NoSQL database that stores data on the server. The two databases can be synchronised to keep data consistent across devices via the CouchDB replication protocol. This is a powerful and flexible way to store and synchronise data, including conflict management, but it is not well suited for files. Therefore, we needed to manage how to store files and synchronise them.
+
+## Terminology
+
+- Password:
+  - A string used to authenticate the user.
+
+- Passphrase:
+  - A string used to encrypt and decrypt data.
+  - This is not a password.
+
+- Encrypt:
+  - To convert data into a format that is unreadable to anyone.
+  - Can be decrypted by the user who has the passphrase.
+  - Should be 1:n, containing random data to ensure that even the same data, when encrypted, results in different outputs.
+
+- Obfuscate:
+  - To convert data into a format that is not easily readable.
+  - Can be decrypted by the user who has the passphrase.
+  - Should be 1:1, containing no random data, and the same data is always obfuscated to the same result. It is necessarily unreadable.
+
+- Hash:
+  - To convert data into a fixed-length string that is not easily readable.
+  - Cannot be decrypted.
+  - Should be 1:1, containing no random data, and the same data is always hashed to the same result.
+
+## Designs
+
+### Principles
+
+- To synchronise and handle conflicts, we should keep the history of modifications.
+- No data should be lost. Even though some extra data may be stored, it should be removed later, safely.
+- Each stored data item should be as small as possible to transfer efficiently, but not so small as to be inefficient.
+- Any type of file should be supported, including binary files.
+- Encryption should be supported efficiently.
+- This method should not depart too far from the PouchDB/CouchDB philosophy. It needs to leave room for other `remote`s, to benefit from custom replicators.
+
+As a result, we have adopted the following design.
+
+- Files are stored as one metadata entry and multiple chunks.
+- Chunks are content-addressable, and the metadata contains the ids of the chunks.
+- Chunks may be referenced from multiple metadata entries. They should be efficiently managed to avoid redundancy.
+
+### Metadata Design
+
+The metadata contains the following information:
+
+| Field    | Type                 | Description                  | Note                                                                                                  |
+| -------- | -------------------- | ---------------------------- | ----------------------------------------------------------------------------------------------------- |
+| _id      | string               | The id of the metadata       | It is created from the file path                                                                      |
+| _rev     | string               | The revision of the metadata | It is created by PouchDB                                                                              |
+| children | [string]             | The ids of the chunks        |                                                                                                       |
+| path     | string               | The path of the file         | If Obfuscate path has been enabled, it has been encrypted                                             |
+| size     | number               | The size of the metadata     | Not respected; for troubleshooting                                                                    |
+| ctime    | string               | The creation timestamp       | This is not used to compare files, but when writing to storage, it will be used                       |
+| mtime    | string               | The modification timestamp   | This will be used to compare files, and will be written to storage                                    |
+| type     | `plain` \| `newnote` | The type of the file         | Children of type `plain` will not be base64 encoded, while `newnote` will be                          |
+| e_       | boolean              | The file is encrypted        | Encryption is processed during transfer to the remote. In local storage, this property does not exist |
+
+#### Decision Rule for `_id` of Metadata
+
+```ts
+// Note: This is pseudo code.
+let _id = PATH;
+if (!HANDLE_FILES_AS_CASE_SENSITIVE) {
+  _id = _id.toLowerCase();
+}
+if (_id.startsWith("_")) {
+  _id = "/" + _id;
+}
+if (OBFUSCATE_PATH) {
+  _id = `f:${OBFUSCATE_PATH(_id, E2EE_PASSPHRASE)}`;
+}
+return _id;
+```
+
+#### Expected Questions
+
+- Why do we need to handle files as case-sensitive?
+  - Some filesystems are case-sensitive, while others are not. For example, Windows is not case-sensitive, while Linux is. Therefore, we need to handle files as case-sensitive to manage conflicts.
+  - The trade-off is that you will not be able to manage files with different cases, so this can be disabled if you only have case-sensitive terminals.
+- Why obfuscate the path?
+  - E2EE only encrypts the content of the file, not metadata. Hence, E2EE alone is not enough to protect the vault completely. The path is also part of the metadata, so it should be obfuscated. This is a trade-off between security and performance. However, if you title a note with sensitive information, you should obfuscate the path.
+- What is `f:`?
+  - It is a prefix to indicate that the path is obfuscated. It is used to distinguish between normal paths and obfuscated paths. Due to file enumeration, Self-hosted LiveSync should scan the files to find the metadata, excluding chunks and other information.
+  - Why does an unobfuscated path not start with `f:`?
+  - For compatibility. Self-hosted LiveSync, by its nature, must also be able to handle files created with newer versions as far as possible.
+
+### Chunk Design
+
+#### Chunk Structure
+
+The chunk contains the following information:
+
+| Field | Type         | Description               | Note                                                                                                  |
+| ----- | ------------ | ------------------------- | ----------------------------------------------------------------------------------------------------- |
+| _id   | `h:{string}` | The id of the chunk       | It is created from the hash of the chunk content                                                      |
+| _rev  | string       | The revision of the chunk | It is created by PouchDB                                                                              |
+| data  | string       | The content of the chunk  |                                                                                                       |
+| type  | `leaf`       | Fixed                     |                                                                                                       |
+| e_    | boolean      | The chunk is encrypted    | Encryption is processed during transfer to the remote. In local storage, this property does not exist |
+
+**SORRY, TO BE WRITTEN, BUT WE HAVE IMPLEMENTED `v2`, WHICH REQUIRES MORE INFORMATION.**
+
+### How they are unified
+
+## Deduplication and Optimisation
+
+## Synchronisation Strategy
+
+## Performance Considerations
+
+## Security and Privacy
+
+## Edge Cases

docs/design_docs/tired_chunk_pack.md

@@ -0,0 +1,117 @@
+# [IN DESIGN] Tiered Chunk Storage with Live Compaction
+
+** VERY IMPORTANT NOTE: This design must be used with the new journal synchronisation method. Otherwise, we risk introducing the bloat of changes from hot-pack into the Bucket. (CouchDB/PouchDB can synchronise only the most recent changes, or resolve conflicts.) Previous Journal Sync **IS NOT**. Please proceed with caution. **
+
+## Goal
+
+To establish a highly efficient, robust, and scalable synchronisation architecture by introducing a tiered storage system inspired by Log-Structured Merge-Trees (LSM-Trees). This design aims to address the challenges of real-time synchronisation, specifically the massive generation of transient data, while minimising storage bloat and ensuring high performance.
+
+## Motivation
+
+Our previous designs, including "Chunk Aggregation by Prefix", successfully addressed the "document explosion" problem. However, the introduction of real-time editor synchronisation exposed a new, critical challenge: the constant generation of short-lived "garbage" chunks during user input. This "garbage storm" places immense pressure on storage, I/O, and the Garbage Collection (GC) process.
+
+A simple aggregation strategy is insufficient because it treats all data equally, mixing valuable, stable chunks with transient, garbage chunks in permanent storage. This leads to storage bloat and inefficient compaction. We require a system that can intelligently distinguish between "hot" (volatile) and "cold" (stable) data, processing them in the most efficient manner possible.
+
+## Outlined Methods and Implementation Plans
+
+### Abstract
+
+This design implements a two-tiered storage system within CouchDB.
+1.  **Level 0 – Hot Storage:** A set of "Hot-Packs", one for each active client. These act as fast, append-only logs for all newly created chunks. They serve as a temporary staging area, absorbing the "garbage storm" of real-time editing.
+2.  **Level 1 – Cold Storage:** The permanent, immutable storage for stable chunks, consisting of **Index Documents** for fast lookups and **Data Documents (Cold-Packs)** for storing chunk data.
+
+A background "Compaction" process continuously promotes stable chunks from Hot Storage to Cold Storage, while automatically discarding garbage. This keeps the permanent storage clean and highly optimised.
+
+### Detailed Implementation
+
+**1. Document Structure:**
+
+-   **Hot-Pack Document (Level 0):** A per-client, append-only log.
+  -   `_id`: `hotpack:{client_id}` (`client_id` could be the same as the `deviceNodeID` used in the `accepted_nodes` in MILESTONE_DOC; enables database 'lockout' for safe synchronisation)
+  -   Content: A log of chunk creation events.
+    ```json
+    {
+      "_id": "hotpack:a9f1b12...",
+      "_rev": "...",
+      "log": [
+      { "hash": "abc...", "data": "...", "ts": ..., "file_id": "file1" },
+      { "hash": "def...", "data": "...", "ts": ..., "file_id": "file2" }
+      ]
+    }
+    ```
+
+-   **Index Document (Level 1):** A fast, prefix-based lookup table for stable chunks.
+  -   `_id`: `idx:{prefix}` (e.g., `idx:a9f1b`)
+  -   Content: Maps a chunk hash to the ID of the Cold-Pack it resides in.
+    ```json
+    {
+      "_id": "idx:a9f1b",
+      "chunks": { "a9f1b12...": "dat:1678886400" }
+    }
+    ```
+
+-   **Cold-Pack Document (Level 1):** An immutable data block created by the compaction process.
+  -   `_id`: `dat:{timestamp_or_uuid}` (e.g., `dat:1678886400123`)
+  -   Content: A collection of stable chunks.
+    ```json
+    {
+      "_id": "dat:1678886400123",
+      "chunks": { "a9f1b12...": "...", "c3d4e5f...": "..." }
+    }
+    ```
+
+-   **Hot-Pack List Document:** A central registry of all active Hot-Packs. This might be a computed document that clients maintain in memory on startup.
+  -   `_id`: `hotpack_list`
+  -   Content: `{"active_clients": ["hotpack:a9f1b12...", "hotpack:c3d4e5f..."]}`
+
+**2. Write/Save Operation Flow (Real-time Editing):**
+
+1.  A client generates a new chunk.
+2.  It **immediately appends** the chunk object (`{hash, data, ts, file_id}`) to its **own** Hot-Pack document's `log` array within its local PouchDB. This operation is extremely fast.
+3.  The PouchDB synchronisation process replicates this change to the remote CouchDB and other clients in the background. No other Hot-Packs are consulted during this write operation.
+
+**3. Read/Load Operation Flow:**
+
+To find a chunk's data:
+1.  The client first consults its in-memory list of active Hot-Pack IDs (see section 5).
+2.  It searches for the chunk hash in all **Hot-Pack documents**, starting from its own, then others. It reads them in reverse log order (newest first).
+3.  If not found, it consults the appropriate **Index Document (`idx:...`)** to get the ID of the Cold-Pack.
+4.  It then reads the chunk data from the corresponding **Cold-Pack document (`dat:...`)**.
+
+**4. Compaction & Promotion Process (The "GC"):**
+
+This is a background task run periodically by clients, or triggered when the number of unprocessed log entries exceeds a threshold (to maintain the ability to synchronise with the remote database, which has a limited document size).
+1.  The client takes its own Hot-Pack (`hotpack:{client_id}`) and scans its `log` array from the beginning (oldest first).
+2.  For each chunk in the log, it checks if the chunk is still referenced in the latest revision of any file.
+  -   **If not referenced (Garbage):** The log entry is simply discarded.
+  -   **If referenced (Stable):** The chunk is added to a "promotion batch".
+3.  After scanning a certain number of log entries, the client takes the "promotion batch".
+4.  It creates one or more new, immutable **Cold-Pack (`dat:...`)** documents to store the chunk data from the batch.
+5.  It updates the corresponding **Index (`idx:...`)** documents to point to the new Cold-Pack(s).
+6.  Once the promotion is successfully saved to the database, it **removes the processed entries from its Hot-Pack's `log` array**. This is a critical step to prevent reprocessing and keep the Hot-Pack small.
+
+**5. Hot-Pack List Management:**
+
+To know which Hot-Packs to read, clients will:
+1.  On startup, load the `hotpack_list` document into memory.
+2.  Use PouchDB's live `changes` feed to monitor the creation of new `hotpack:*` documents.
+3.  Upon detecting an unknown Hot-Pack, the client updates its in-memory list and attempts to update the central `hotpack_list` document (on a best-effort basis, with conflict resolution).
+
+## Planned Test Strategy
+
+1.  **Unit Tests:** Test the Compaction/Promotion logic extensively. Ensure garbage is correctly identified and stable chunks are promoted correctly.
+2.  **Integration Tests:** Simulate a multi-client real-time editing session.
+  -   Verify that writes are fast and responsive.
+  -   Confirm that transient garbage chunks do not pollute the Cold Storage.
+  -   Confirm that after a period of inactivity, compaction runs and the Hot-Packs shrink.
+3.  **Stress Tests:** Simulate many clients joining and leaving to test the robustness of the `hotpack_list` management.
+
+## Documentation Strategy
+
+-   This design document will serve as the core architectural reference.
+-   The roles of each document type (Hot-Pack, Index, Cold-Pack, List) will be clearly explained for future developers.
+-   The logic of the Compaction/Promotion process will be detailed.
+
+## Consideration and Conclusion
+
+This tiered storage design is a direct evolution, born from the lessons of previous architectures. It embraces the ephemeral nature of data in real-time applications. By creating a "staging area" (Hot-Packs) for volatile data, it protects the integrity and performance of the permanent "cold" storage. The Compaction process acts as a self-cleaning mechanism, ensuring that only valuable, stable data is retained long-term. This is not just an optimisation; it is a fundamental shift that enables robust, high-performance, and scalable real-time synchronisation on top of CouchDB.

docs/design_docs/tired_chunk_pack_bucket.md

@@ -0,0 +1,97 @@
+# [IN DESIGN] Tiered Chunk Storage for Bucket Sync
+
+## Goal
+
+To evolve the "Journal Sync" mechanism by integrating the Tiered Storage architecture. This design aims to drastically reduce the size and number of sync packs, minimise storage consumption on the backend bucket, and establish a clear, efficient process for Garbage Collection, all while remaining protocol-agnostic.
+
+## Motivation
+
+The original "Journal Sync" liberates us from CouchDB's protocol, but it still packages and transfers entire document changes, including bulky and often transient chunk data. In a real-time or frequent-editing scenario, this results in:
+1.  **Bloated Sync Packs:** Packs become large with redundant or short-lived chunk data, increasing upload and download times.
+2.  **Inefficient Storage:** The backend bucket stores numerous packs containing overlapping and obsolete chunk data, wasting space.
+3.  **Impractical Garbage Collection:** Identifying and purging obsolete *chunk data* from within the pack-based journal history is extremely difficult.
+
+This new design addresses these problems by fundamentally changing *what* is synchronised in the journal packs. We will synchronise lightweight metadata and logs, while handling bulk data separately.
+
+## Outlined methods and implementation plans
+
+### Abstract
+
+This design adapts the Tiered Storage model for a bucket-based backend. The backend bucket is partitioned into distinct areas for different data types. The "Journal Sync" process is now responsible for synchronising only the "hot" volatile data and lightweight metadata. A separate, asynchronous "Compaction" process, which can be run by any client, is responsible for migrating stable data into permanent, deduplicated "cold" storage.
+
+### Detailed Implementation
+
+**1. Bucket Structure:**
+
+The backend bucket will have four distinct logical areas (prefixes):
+-   `packs/`: For "Journal Sync" packs, containing the journal of metadata and Hot-Log changes.
+-   `hot_logs/`: A dedicated area for each client's "Hot-Log," containing newly created, volatile chunks.
+-   `indices/`: For prefix-based Index files, mapping chunk hashes to their permanent location in Cold Storage.
+-   `cold_chunks/`: For deduplicated, stable chunk data, stored by content hash.
+
+**2. Data Structures (Client-side PouchDB & Backend Bucket):**
+
+-   **Client Metadata:** Standard file metadata documents, kept in the client's PouchDB.
+-   **Hot-Log (in `hot_logs/`):** A per-client, append-only log file on the bucket.
+    -   Path: `hot_logs/{client_id}.jsonlog`
+    -   Content: A sequence of JSON objects, one per line, representing chunk creation events. `{"hash": "...", "data": "...", "ts": ..., "file_id": "..."}`
+
+-   **Index File (in `indices/`):** A JSON file for a given hash prefix.
+    -   Path: `indices/{prefix}.json`
+    -   Content: Maps a chunk hash to its content hash (which is its key in `cold_chunks/`). `{"hash_abc...": true, "hash_def...": true}`
+
+-   **Cold Chunk (in `cold_chunks/`):** The raw, immutable, deduplicated chunk data.
+    -   Path: `cold_chunks/{chunk_hash}`
+
+**3. "Journal Sync" - Send/Receive Operation (Not Live):**
+
+This process is now extremely lightweight.
+1.  **Send:**
+    a. The client takes all newly generated chunks and **appends them to its own Hot-Log file (`hot_logs/{client_id}.jsonlog`)** on the bucket.
+    b. The client updates its local file metadata in PouchDB.
+    c. It then creates a "Journal Sync" pack containing **only the PouchDB journal of the file metadata changes.** This pack is very small as it contains no chunk data.
+    d. The pack is uploaded to `packs/`.
+
+2.  **Receive:**
+    a. The client downloads new packs from `packs/` and applies the metadata journal to its local PouchDB.
+    b. It downloads the latest versions of all **other clients' Hot-Log files** from `hot_logs/`.
+    c. Now the client has a complete, up-to-date view of all metadata and all "hot" chunks.
+
+**4. Read/Load Operation Flow:**
+
+To find a chunk's data:
+1.  The client searches for the chunk hash in its local copy of all **Hot-Logs**.
+2.  If not found, it downloads and consults the appropriate **Index file (`indices/{prefix}.json`)**.
+3.  If the index confirms existence, it downloads the data from **`cold_chunks/{chunk_hash}`**.
+
+**5. Compaction & Promotion Process (Asynchronous "GC"):**
+
+This is a deliberate, offline-capable process that any client can choose to run.
+1.  The client "leases" its own Hot-Log for compaction.
+2.  It reads its entire `hot_logs/{client_id}.jsonlog`.
+3.  For each chunk in the log, it checks if the chunk is referenced in the *current, latest state* of the file metadata.
+    -   **If not referenced (Garbage):** The log entry is discarded.
+    -   **If referenced (Stable):** The chunk is added to a "promotion batch."
+4.  For each chunk in the promotion batch:
+    a. It checks the corresponding `indices/{prefix}.json` to see if the chunk already exists in Cold Storage.
+    b. If it does not exist, it **uploads the chunk data to `cold_chunks/{chunk_hash}`** and updates the `indices/{prefix}.json` file.
+5.  Once the entire Hot-Log has been processed, the client **deletes its `hot_logs/{client_id}.jsonlog` file** (or truncates it to empty), effectively completing the cycle.
+
+## Test strategy
+
+1.  **Component Tests:** Test the Compaction process independently. Ensure it correctly identifies stable versus garbage chunks and populates the `cold_chunks/` and `indices/` areas correctly.
+2.  **Integration Tests:**
+    -   Simulate a multi-client sync cycle. Verify that sync packs in `packs/` are small.
+    -   Confirm that `hot_logs/` are correctly created and updated.
+    -   Run the Compaction process and verify that data migrates correctly to cold storage and the hot log is cleared.
+3.  **Conflict Tests:** Simulate two clients trying to compact the same index file simultaneously and ensure the outcome is consistent (for example, via a locking mechanism or last-write-wins).
+
+## Documentation strategy
+
+-   This design document will be the primary reference for the bucket-based architecture.
+-   The structure of the backend bucket (`packs/`, `hot_logs/`, etc.) will be clearly defined.
+-   A detailed description of how to run the Compaction process will be provided to users.
+
+## Consideration and Conclusion
+
+By applying the Tiered Storage model to "Journal Sync", we transform it into a remarkably efficient system. The synchronisation of everyday changes becomes extremely fast and lightweight, as only metadata journals are exchanged. The heavy lifting of data deduplication and permanent storage is offloaded to a separate, asynchronous Compaction process. This clear separation of concerns makes the system highly scalable, minimises storage costs, and finally provides a practical, robust solution for Garbage Collection in a protocol-agnostic, bucket-based environment.

docs/setup_own_server.md

@@ -5,10 +5,15 @@
 - [Setup a CouchDB server](#setup-a-couchdb-server)
   - [Table of Contents](#table-of-contents)
   - [1. Prepare CouchDB](#1-prepare-couchdb)
-    - [A. Using Docker container](#a-using-docker-container)
+    - [A. Using Docker](#a-using-docker)
       - [1. Prepare](#1-prepare)
       - [2. Run docker container](#2-run-docker-container)
-    - [B. Install CouchDB directly](#b-install-couchdb-directly)
+    - [B. Using Docker Compose](#b-using-docker-compose)
+      - [1. Prepare](#1-prepare-1)
+      - [2. Creating Compose file](#2-create-a-docker-composeyml-file-with-the-following-added-to-it)
+      - [3. Boot check](#3-run-the-docker-compose-file-to-boot-check)
+      - [4. Starting Docker Compose in background](#4-run-the-docker-compose-file-in-the-background)
+    - [C. Install CouchDB directly](#c-install-couchdb-directly)
   - [2. Run couchdb-init.sh for initialise](#2-run-couchdb-initsh-for-initialise)
   - [3. Expose CouchDB to the Internet](#3-expose-couchdb-to-the-internet)
   - [4. Client Setup](#4-client-setup)
@@ -21,44 +26,56 @@
 ---
 
 ## 1. Prepare CouchDB
-### A. Using Docker container
+### A. Using Docker
 
 #### 1. Prepare
 ```bash
 
-# Prepare environment variables.
+# Adding environment variables.
 export hostname=localhost:5984
 export username=goojdasjdas     #Please change as you like.
 export password=kpkdasdosakpdsa #Please change as you like
 
-# Prepare directories which save data and configurations.
+# Creating the save data & configuration directories.
 mkdir couchdb-data
 mkdir couchdb-etc
 ```
 
 #### 2. Run docker container
-
 1. Boot Check.
 ```
 $ docker run --name couchdb-for-ols --rm -it -e COUCHDB_USER=${username} -e COUCHDB_PASSWORD=${password} -v ${PWD}/couchdb-data:/opt/couchdb/data -v ${PWD}/couchdb-etc:/opt/couchdb/etc/local.d -p 5984:5984 couchdb
 ```
-If your container has been exited, please check the permission of couchdb-data, and couchdb-etc.  
-Once CouchDB run, these directories will be owned by uid:`5984`. Please chown it for you again.
+> [!WARNING]
+> If your container threw an error or exited unexpectedly, please check the permission of couchdb-data, and couchdb-etc.  
+> Once CouchDB starts, these directories will be owned by uid:`5984`. Please chown it for that uid again.
 
 2. Enable it in the background
 ```
 $ docker run --name couchdb-for-ols -d --restart always -e COUCHDB_USER=${username} -e COUCHDB_PASSWORD=${password} -v ${PWD}/couchdb-data:/opt/couchdb/data -v ${PWD}/couchdb-etc:/opt/couchdb/etc/local.d -p 5984:5984 couchdb
 ```
-If you prefer a compose file instead of docker run, here is the equivalent below:
+
+Congrats, move on to [step 2](#2-run-couchdb-initsh-for-initialise)
+### B. Using Docker Compose
+
+#### 1. Prepare
+
+```
+# Creating the save data & configuration directories.
+mkdir couchdb-data
+mkdir couchdb-etc
+```
+
+#### 2. Create a `docker-compose.yml` file with the following added to it
 ```
 services:
   couchdb:
     image: couchdb:latest
     container_name: couchdb-for-ols
-    user: 1000:1000
+    user: 5984:5984
     environment:
-      - COUCHDB_USER=${username}
-      - COUCHDB_PASSWORD=${password}
+      - COUCHDB_USER=<INSERT USERNAME HERE>  #Please change as you like.
+      - COUCHDB_PASSWORD=<INSERT PASSWORD HERE> #Please change as you like.
     volumes:
       - ./couchdb-data:/opt/couchdb/data
       - ./couchdb-etc:/opt/couchdb/etc/local.d
@@ -66,7 +83,30 @@ services:
       - 5984:5984
     restart: unless-stopped
 ```
-### B. Install CouchDB directly
+
+#### 3. Run the Docker Compose file to boot check
+
+```
+docker compose up
+# Or if using the old version
+docker-compose up
+```
+> [!WARNING]
+> If your container threw an error or exited unexpectedly, please check the permission of couchdb-data, and couchdb-etc.  
+> Once CouchDB starts, these directories will be owned by uid:`5984`. Please chown it for that uid again.
+
+#### 4. Run the Docker Compose file in the background
+If all went well and didn't throw any errors, `CTRL+C` out of it, and then run this command
+```
+docker compose up -d
+# Or if using the old version
+docker-compose up -d
+```
+
+Congrats, move on to [step 2](#2-run-couchdb-initsh-for-initialise)
+
+
+### C. Install CouchDB directly
 Please refer to the [official document](https://docs.couchdb.org/en/stable/install/index.html). However, we do not have to configure it fully. Just the administrator needs to be configured.
 
 ## 2. Run couchdb-init.sh for initialise

manifest-beta.json

@@ -1,7 +1,7 @@
 {
     "id": "obsidian-livesync",
     "name": "Self-hosted LiveSync",
-    "version": "0.25.0",
+    "version": "0.25.2",
     "minAppVersion": "0.9.12",
     "description": "Community implementation of self-hosted livesync. Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.",
     "author": "vorotamoroz",

manifest.json

@@ -1,7 +1,7 @@
 {
     "id": "obsidian-livesync",
     "name": "Self-hosted LiveSync",
-    "version": "0.25.0",
+    "version": "0.25.7",
     "minAppVersion": "0.9.12",
     "description": "Community implementation of self-hosted livesync. Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.",
     "author": "vorotamoroz",

package.json

@@ -1,6 +1,6 @@
 {
     "name": "obsidian-livesync",
-    "version": "0.25.0",
+    "version": "0.25.7",
     "description": "Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.",
     "main": "main.js",
     "type": "module",
@@ -23,7 +23,8 @@
         "pretty": "npm run prettyNoWrite -- --write --log-level error",
         "prettyCheck": "npm run prettyNoWrite -- --check",
         "prettyNoWrite": "prettier --config ./.prettierrc \"**/*.js\" \"**/*.ts\" \"**/*.json\" ",
-        "check": "npm run lint && npm run svelte-check && npm run tsc-check"
+        "check": "npm run lint && npm run svelte-check && npm run tsc-check",
+        "unittest": "deno test -A --no-check --coverage=cov_profile  --v8-flags=--expose-gc  --trace-leaks  ./src/"
     },
     "keywords": [],
     "author": "vorotamoroz",
@@ -60,6 +61,7 @@
         "pouchdb-adapter-http": "^9.0.0",
         "pouchdb-adapter-idb": "^9.0.0",
         "pouchdb-adapter-indexeddb": "^9.0.0",
+        "pouchdb-adapter-memory": "^9.0.0",
         "pouchdb-core": "^9.0.0",
         "pouchdb-errors": "^9.0.0",
         "pouchdb-find": "^9.0.0",
@@ -75,7 +77,8 @@
         "tslib": "^2.8.1",
         "tsx": "^4.19.4",
         "typescript": "5.7.3",
-        "yaml": "^2.8.0"
+        "yaml": "^2.8.0",
+        "@types/deno": "^2.3.0"
     },
     "dependencies": {
         "@aws-sdk/client-s3": "^3.808.0",
@@ -89,7 +92,7 @@
         "fflate": "^0.8.2",
         "idb": "^8.0.3",
         "minimatch": "^10.0.1",
-        "octagonal-wheels": "^0.1.35",
+        "octagonal-wheels": "^0.1.37",
         "qrcode-generator": "^1.4.4",
         "svelte-check": "^4.1.7",
         "trystero": "^0.21.5",

src/common/KeyValueDB.ts

@@ -1,13 +1,5 @@
 import { deleteDB, type IDBPDatabase, openDB } from "idb";
-export interface KeyValueDatabase {
-    get<T>(key: IDBValidKey): Promise<T>;
-    set<T>(key: IDBValidKey, value: T): Promise<IDBValidKey>;
-    del(key: IDBValidKey): Promise<void>;
-    clear(): Promise<void>;
-    keys(query?: IDBValidKey | IDBKeyRange, count?: number): Promise<IDBValidKey[]>;
-    close(): void;
-    destroy(): Promise<void>;
-}
+import type { KeyValueDatabase } from "../lib/src/interfaces/KeyValueDatabase.ts";
 const databaseCache: { [key: string]: IDBPDatabase<any> } = {};
 export const OpenKeyValueDatabase = async (dbKey: string): Promise<KeyValueDatabase> => {
     if (dbKey in databaseCache) {

src/common/events.ts

@@ -20,6 +20,7 @@ export const EVENT_REQUEST_OPEN_P2P = "request-open-p2p";
 export const EVENT_REQUEST_CLOSE_P2P = "request-close-p2p";
 
 export const EVENT_REQUEST_RUN_DOCTOR = "request-run-doctor";
+export const EVENT_REQUEST_RUN_FIX_INCOMPLETE = "request-run-fix-incomplete";
 
 // export const EVENT_FILE_CHANGED = "file-changed";
 
@@ -38,6 +39,7 @@ declare global {
         [EVENT_REQUEST_COPY_SETUP_URI]: undefined;
         [EVENT_REQUEST_SHOW_SETUP_QR]: undefined;
         [EVENT_REQUEST_RUN_DOCTOR]: string;
+        [EVENT_REQUEST_RUN_FIX_INCOMPLETE]: undefined;
     }
 }
 

src/common/utils.ts

@@ -28,11 +28,12 @@ import type ObsidianLiveSyncPlugin from "../main.ts";
 import { writeString } from "../lib/src/string_and_binary/convert.ts";
 import { fireAndForget } from "../lib/src/common/utils.ts";
 import { sameChangePairs } from "./stores.ts";
-import type { KeyValueDatabase } from "./KeyValueDB.ts";
+
 import { scheduleTask } from "octagonal-wheels/concurrency/task";
 import { EVENT_PLUGIN_UNLOADED, eventHub } from "./events.ts";
 import { promiseWithResolver, type PromiseWithResolvers } from "octagonal-wheels/promises";
 import { AuthorizationHeaderGenerator } from "../lib/src/replication/httplib.ts";
+import type { KeyValueDatabase } from "../lib/src/interfaces/KeyValueDatabase.ts";
 
 export { scheduleTask, cancelTask, cancelAllTasks } from "../lib/src/concurrency/task.ts";
 

src/features/LocalDatabaseMainte/CmdLocalDatabaseMainte.ts

@@ -28,7 +28,7 @@ export class LocalDatabaseMaintenance extends LiveSyncCommands implements IObsid
         return this.localDatabase.localDatabase;
     }
     clearHash() {
-        this.localDatabase.hashCaches.clear();
+        this.localDatabase.clearCaches();
     }
 
     async confirm(title: string, message: string, affirmative = "Yes", negative = "No") {

src/main.ts

@@ -27,7 +27,7 @@ import {
     LiveSyncAbstractReplicator,
     type LiveSyncReplicatorEnv,
 } from "./lib/src/replication/LiveSyncAbstractReplicator.js";
-import { type KeyValueDatabase } from "./common/KeyValueDB.ts";
+import { type KeyValueDatabase } from "./lib/src/interfaces/KeyValueDatabase.ts";
 import { LiveSyncCommands } from "./features/LiveSyncCommands.ts";
 import { HiddenFileSync } from "./features/HiddenFileSync/CmdHiddenFileSync.ts";
 import { ConfigSync } from "./features/ConfigSync/CmdConfigSync.ts";
@@ -472,6 +472,14 @@ export default class ObsidianLiveSyncPlugin
     $$clearUsedPassphrase(): void {
         throwShouldBeOverridden();
     }
+
+    $$decryptSettings(settings: ObsidianLiveSyncSettings): Promise<ObsidianLiveSyncSettings> {
+        throwShouldBeOverridden();
+    }
+    $$adjustSettings(settings: ObsidianLiveSyncSettings): Promise<ObsidianLiveSyncSettings> {
+        throwShouldBeOverridden();
+    }
+
     $$loadSettings(): Promise<void> {
         throwShouldBeOverridden();
     }
@@ -546,6 +554,10 @@ export default class ObsidianLiveSyncPlugin
     $everyAfterResumeProcess(): Promise<boolean> {
         return InterceptiveEvery;
     }
+
+    $$fetchRemotePreferredTweakValues(trialSetting: RemoteDBSettings): Promise<TweakValues | false> {
+        throwShouldBeOverridden();
+    }
     $$checkAndAskResolvingMismatchedTweaks(preferred: Partial<TweakValues>): Promise<[TweakValues | boolean, boolean]> {
         throwShouldBeOverridden();
     }
@@ -579,7 +591,11 @@ export default class ObsidianLiveSyncPlugin
         throwShouldBeOverridden();
     }
 
-    $$initializeDatabase(showingNotice: boolean = false, reopenDatabase = true): Promise<boolean> {
+    $$initializeDatabase(
+        showingNotice: boolean = false,
+        reopenDatabase = true,
+        ignoreSuspending: boolean = false
+    ): Promise<boolean> {
         throwShouldBeOverridden();
     }
 
@@ -616,7 +632,7 @@ export default class ObsidianLiveSyncPlugin
         throwShouldBeOverridden();
     }
 
-    $$performFullScan(showingNotice?: boolean): Promise<void> {
+    $$performFullScan(showingNotice?: boolean, ignoreSuspending?: boolean): Promise<void> {
         throwShouldBeOverridden();
     }
 

src/modules/core/ModuleDatabaseFileAccess.ts

@@ -53,7 +53,7 @@ export class ModuleDatabaseFileAccess extends AbstractModule implements IObsidia
             async () => await this.storeContent("autoTest.md" as FilePathWithPrefix, testString)
         );
         // For test, we need to clear the caches.
-        await this.localDatabase.hashCaches.clear();
+        this.localDatabase.clearCaches();
         await this._test("readContent", async () => {
             const content = await this.fetch("autoTest.md" as FilePathWithPrefix);
             if (!content) return "File not found";

src/modules/core/ModuleFileHandler.ts

@@ -18,7 +18,7 @@ import {
     getStoragePathFromUXFileInfo,
     markChangesAreSame,
 } from "../../common/utils";
-import { getDocDataAsArray, isDocContentSame, readContent } from "../../lib/src/common/utils";
+import { getDocDataAsArray, isDocContentSame, readAsBlob, readContent } from "../../lib/src/common/utils";
 import { shouldBeIgnored } from "../../lib/src/string_and_binary/path";
 import type { ICoreModule } from "../ModuleTypes";
 import { Semaphore } from "octagonal-wheels/concurrency/semaphore";
@@ -259,6 +259,16 @@ export class ModuleFileHandler extends AbstractModule implements ICoreModule {
         const docData = readContent(docRead);
 
         if (existOnStorage && !force) {
+            // If we want to process size mismatched files -- in case of having files created by some integrations, enable the toggle.
+            if (!this.settings.processSizeMismatchedFiles) {
+                // Check the file is not corrupted
+                // (Zero is a special case, may be created by some APIs and it might be acceptable).
+                if (docRead.size != 0 && docRead.size !== readAsBlob(docRead).size) {
+                    this._log(`File ${path} seems to be corrupted! Writing prevented.`, LOG_LEVEL_NOTICE);
+                    return false;
+                }
+            }
+
             // The file is exist on the storage. Let's check the difference between the file and the entry.
             // But, if force is true, then it should be updated.
             // Ok, we have to compare.

src/modules/core/ModuleRebuilder.ts

@@ -73,7 +73,7 @@ export class ModuleRebuilder extends AbstractModule implements ICoreModule, Rebu
         await this.core.$$realizeSettingSyncMode();
         await this.resetLocalDatabase();
         await delay(1000);
-        await this.core.$$initializeDatabase(true);
+        await this.core.$$initializeDatabase(true, true, true);
         await this.core.$$markRemoteLocked();
         await this.core.$$tryResetRemoteDatabase();
         await this.core.$$markRemoteLocked();
@@ -190,7 +190,7 @@ export class ModuleRebuilder extends AbstractModule implements ICoreModule, Rebu
         if (makeLocalChunkBeforeSync) {
             await this.core.fileHandler.createAllChunks(true);
         } else if (!preventMakeLocalFilesBeforeSync) {
-            await this.core.$$initializeDatabase(true);
+            await this.core.$$initializeDatabase(true, true, true);
         } else {
             // Do not create local file entries before sync (Means use remote information)
         }

src/modules/core/ModuleReplicator.ts

@@ -30,8 +30,9 @@ import {
 import { isAnyNote } from "../../lib/src/common/utils";
 import { EVENT_FILE_SAVED, EVENT_SETTING_SAVED, eventHub } from "../../common/events";
 import type { LiveSyncAbstractReplicator } from "../../lib/src/replication/LiveSyncAbstractReplicator";
-import { globalSlipBoard } from "../../lib/src/bureau/bureau";
+
 import { $msg } from "../../lib/src/common/i18n";
+import { clearHandlers } from "../../lib/src/replication/SyncParamsHandler";
 
 const KEY_REPLICATION_ON_EVENT = "replicationOnEvent";
 const REPLICATION_ON_EVENT_FORECASTED_TIME = 5000;
@@ -66,6 +67,8 @@ export class ModuleReplicator extends AbstractModule implements ICoreModule {
         this.core.replicator = replicator;
         this._replicatorType = this.settings.remoteType;
         await yieldMicrotask();
+        // Clear any existing sync parameter handlers (means clearing key-deriving salt).
+        clearHandlers();
         return true;
     }
 
@@ -88,8 +91,9 @@ export class ModuleReplicator extends AbstractModule implements ICoreModule {
 
     async $everyBeforeReplicate(showMessage: boolean): Promise<boolean> {
         // Checking salt
-        if (!(await this.ensureReplicatorPBKDF2Salt(showMessage))) {
-            Logger("Failed to ensure PBKDF2 salt for replication.", LOG_LEVEL_NOTICE);
+        // Showing message is false: that because be shown here. (And it is a fatal error, no way to hide it).
+        if (!(await this.ensureReplicatorPBKDF2Salt(false))) {
+            Logger("Failed to initialise the encryption key, preventing replication.", LOG_LEVEL_NOTICE);
             return false;
         }
         await this.loadQueuedFiles();
@@ -146,12 +150,12 @@ Even if you choose to clean up, you will see this option again if you exit Obsid
                 }
 
                 await purgeUnreferencedChunks(this.localDatabase.localDatabase, false);
-                this.localDatabase.hashCaches.clear();
+                this.localDatabase.clearCaches();
                 // Perform the synchronisation once.
                 if (await this.core.replicator.openReplication(this.settings, false, showMessage, true)) {
                     await balanceChunkPurgedDBs(this.localDatabase.localDatabase, remoteDB.db);
                     await purgeUnreferencedChunks(this.localDatabase.localDatabase, false);
-                    this.localDatabase.hashCaches.clear();
+                    this.localDatabase.clearCaches();
                     await this.core.$$getReplicator().markRemoteResolved(this.settings);
                     Logger("The local database has been cleaned up.", showMessage ? LOG_LEVEL_NOTICE : LOG_LEVEL_INFO);
                 } else {
@@ -306,7 +310,7 @@ Even if you choose to clean up, you will see this option again if you exit Obsid
             const change = docs[0];
             if (!change) return;
             if (isChunk(change._id)) {
-                globalSlipBoard.submit("read-chunk", change._id, change as EntryLeaf);
+                this.localDatabase.onNewLeaf(change as EntryLeaf);
                 return;
             }
             if (await this.core.$anyModuleParsedReplicationResultItem(change)) return;

src/modules/coreFeatures/ModuleRedFlag.ts

@@ -6,6 +6,7 @@ import {
     FLAGMD_REDFLAG2_HR,
     FLAGMD_REDFLAG3,
     FLAGMD_REDFLAG3_HR,
+    type ObsidianLiveSyncSettings,
 } from "../../lib/src/common/types.ts";
 import { AbstractModule } from "../AbstractModule.ts";
 import type { ICoreModule } from "../ModuleTypes.ts";
@@ -133,6 +134,34 @@ export class ModuleRedFlag extends AbstractModule implements ICoreModule {
                         return false;
                     }
 
+                    const optionFetchRemoteConf = $msg("RedFlag.FetchRemoteConfig.Buttons.Fetch");
+                    const optionCancel = $msg("RedFlag.FetchRemoteConfig.Buttons.Cancel");
+                    const fetchRemote = await this.core.confirm.askSelectStringDialogue(
+                        $msg("RedFlag.FetchRemoteConfig.Message"),
+                        [optionFetchRemoteConf, optionCancel],
+                        {
+                            defaultAction: optionFetchRemoteConf,
+                            timeout: 0,
+                            title: $msg("RedFlag.FetchRemoteConfig.Title"),
+                        }
+                    );
+                    if (fetchRemote === optionFetchRemoteConf) {
+                        this._log("Fetching remote configuration", LOG_LEVEL_NOTICE);
+                        const newSettings = JSON.parse(JSON.stringify(this.core.settings)) as ObsidianLiveSyncSettings;
+                        const remoteConfig = await this.core.$$fetchRemotePreferredTweakValues(newSettings);
+                        if (remoteConfig) {
+                            this._log("Remote configuration found.", LOG_LEVEL_NOTICE);
+                            const mergedSettings = {
+                                ...this.core.settings,
+                                ...remoteConfig,
+                            } satisfies ObsidianLiveSyncSettings;
+                            this._log("Remote configuration applied.", LOG_LEVEL_NOTICE);
+                            this.core.settings = mergedSettings;
+                        } else {
+                            this._log("Remote configuration not applied.", LOG_LEVEL_NOTICE);
+                        }
+                    }
+
                     await this.core.rebuilder.$fetchLocal(makeLocalChunkBeforeSync, !makeLocalFilesBeforeSync);
 
                     await this.deleteRedFlag3();

src/modules/coreFeatures/ModuleResolveMismatchedTweaks.ts

@@ -164,22 +164,28 @@ export class ModuleResolvingMismatchedTweaks extends AbstractModule implements I
         return "IGNORE";
     }
 
-    async $$checkAndAskUseRemoteConfiguration(
-        trialSetting: RemoteDBSettings
-    ): Promise<{ result: false | TweakValues; requireFetch: boolean }> {
-        const replicator = await this.core.$anyNewReplicator(trialSetting);
+    async $$fetchRemotePreferredTweakValues(trialSetting: RemoteDBSettings): Promise<TweakValues | false> {
+        const replicator = await this.core.$anyNewReplicator();
         if (await replicator.tryConnectRemote(trialSetting)) {
             const preferred = await replicator.getRemotePreferredTweakValues(trialSetting);
             if (preferred) {
-                return await this.$$askUseRemoteConfiguration(trialSetting, preferred);
-            } else {
-                this._log("Failed to get the preferred tweak values from the remote server.", LOG_LEVEL_NOTICE);
+                return preferred;
             }
-            return { result: false, requireFetch: false };
-        } else {
-            this._log("Failed to connect to the remote server.", LOG_LEVEL_NOTICE);
-            return { result: false, requireFetch: false };
+            this._log("Failed to get the preferred tweak values from the remote server.", LOG_LEVEL_NOTICE);
+            return false;
         }
+        this._log("Failed to connect to the remote server.", LOG_LEVEL_NOTICE);
+        return false;
+    }
+
+    async $$checkAndAskUseRemoteConfiguration(
+        trialSetting: RemoteDBSettings
+    ): Promise<{ result: false | TweakValues; requireFetch: boolean }> {
+        const preferred = await this.core.$$fetchRemotePreferredTweakValues(trialSetting);
+        if (preferred) {
+            return await this.$$askUseRemoteConfiguration(trialSetting, preferred);
+        }
+        return { result: false, requireFetch: false };
     }
 
     async $$askUseRemoteConfiguration(

src/modules/essential/ModuleInitializerFile.ts

@@ -20,7 +20,7 @@ import { AbstractModule } from "../AbstractModule.ts";
 import type { ICoreModule } from "../ModuleTypes.ts";
 import { withConcurrency } from "octagonal-wheels/iterable/map";
 export class ModuleInitializerFile extends AbstractModule implements ICoreModule {
-    async $$performFullScan(showingNotice?: boolean): Promise<void> {
+    async $$performFullScan(showingNotice?: boolean, ignoreSuspending: boolean = false): Promise<void> {
         this._log("Opening the key-value database", LOG_LEVEL_VERBOSE);
         const isInitialized = (await this.core.kvDB.get<boolean>("initialized")) || false;
         // synchronize all files between database and storage.
@@ -34,6 +34,16 @@ export class ModuleInitializerFile extends AbstractModule implements ICoreModule
             }
             return;
         }
+        if (!ignoreSuspending && this.settings.suspendFileWatching) {
+            if (showingNotice) {
+                this._log(
+                    "Now suspending file watching. Synchronising between the storage and the local database is now prevented.",
+                    LOG_LEVEL_NOTICE,
+                    "syncAll"
+                );
+            }
+            return;
+        }
 
         if (showingNotice) {
             this._log("Initializing", LOG_LEVEL_NOTICE, "syncAll");
@@ -355,11 +365,15 @@ export class ModuleInitializerFile extends AbstractModule implements ICoreModule
         this._log(`Checking expired file history done`);
     }
 
-    async $$initializeDatabase(showingNotice: boolean = false, reopenDatabase = true): Promise<boolean> {
+    async $$initializeDatabase(
+        showingNotice: boolean = false,
+        reopenDatabase = true,
+        ignoreSuspending: boolean = false
+    ): Promise<boolean> {
         this.core.$$resetIsReady();
         if (!reopenDatabase || (await this.core.$$openDatabase())) {
             if (this.localDatabase.isReady) {
-                await this.core.$$performFullScan(showingNotice);
+                await this.core.$$performFullScan(showingNotice, ignoreSuspending);
             }
             if (!(await this.core.$everyOnDatabaseInitialized(showingNotice))) {
                 this._log(`Initializing database has been failed on some module`, LOG_LEVEL_NOTICE);

src/modules/essential/ModuleMigration.ts

@@ -1,141 +1,44 @@
-import { LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE } from "octagonal-wheels/common/logger";
-import { type ObsidianLiveSyncSettings } from "../../lib/src/common/types.js";
+import { LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE, Logger } from "../../lib/src/common/logger.ts";
 import {
     EVENT_REQUEST_OPEN_P2P,
     EVENT_REQUEST_OPEN_SETTING_WIZARD,
     EVENT_REQUEST_OPEN_SETTINGS,
     EVENT_REQUEST_OPEN_SETUP_URI,
     EVENT_REQUEST_RUN_DOCTOR,
+    EVENT_REQUEST_RUN_FIX_INCOMPLETE,
     eventHub,
 } from "../../common/events.ts";
 import { AbstractModule } from "../AbstractModule.ts";
 import type { ICoreModule } from "../ModuleTypes.ts";
 import { $msg } from "src/lib/src/common/i18n.ts";
-import { checkUnsuitableValues, RuleLevel, type RuleForType } from "../../lib/src/common/configForDoc.ts";
-import { getConfName, type AllSettingItemKey } from "../features/SettingDialogue/settingConstants.ts";
+import { performDoctorConsultation, RebuildOptions } from "../../lib/src/common/configForDoc.ts";
+import { getPath, isValidPath } from "../../common/utils.ts";
+import { isMetaEntry } from "../../lib/src/common/types.ts";
+import { isDeletedEntry, isDocContentSame, isLoadedEntry, readAsBlob } from "../../lib/src/common/utils.ts";
 
 export class ModuleMigration extends AbstractModule implements ICoreModule {
     async migrateUsingDoctor(skipRebuild: boolean = false, activateReason = "updated", forceRescan = false) {
-        const r = checkUnsuitableValues(this.core.settings);
-        if (!forceRescan && r.version == this.settings.doctorProcessedVersion) {
-            const isIssueFound = Object.keys(r.rules).length > 0;
-            const msg = isIssueFound ? "Issues found" : "No issues found";
-            this._log(`${msg} but marked as to be silent`, LOG_LEVEL_VERBOSE);
-            return;
-        }
-        const issues = Object.entries(r.rules);
-        if (issues.length == 0) {
-            this._log(
-                $msg("Doctor.Message.NoIssues"),
-                activateReason !== "updated" ? LOG_LEVEL_NOTICE : LOG_LEVEL_INFO
-            );
-            return;
-        } else {
-            const OPT_YES = `${$msg("Doctor.Button.Yes")}` as const;
-            const OPT_NO = `${$msg("Doctor.Button.No")}` as const;
-            const OPT_DISMISS = `${$msg("Doctor.Button.DismissThisVersion")}` as const;
-            // this._log(`Issues found in ${key}`, LOG_LEVEL_VERBOSE);
-            const issues = Object.keys(r.rules)
-                .map((key) => `- ${getConfName(key as AllSettingItemKey)}`)
-                .join("\n");
-            const msg = await this.core.confirm.askSelectStringDialogue(
-                $msg("Doctor.Dialogue.Main", { activateReason, issues }),
-                [OPT_YES, OPT_NO, OPT_DISMISS],
-                {
-                    title: $msg("Doctor.Dialogue.Title"),
-                    defaultAction: OPT_YES,
-                }
-            );
-            if (msg == OPT_DISMISS) {
-                this.settings.doctorProcessedVersion = r.version;
-                await this.core.saveSettings();
-                this._log("Marked as to be silent", LOG_LEVEL_VERBOSE);
-                return;
-            }
-            if (msg != OPT_YES) return;
-            let shouldRebuild = false;
-            let shouldRebuildLocal = false;
-            const issueItems = Object.entries(r.rules) as [keyof ObsidianLiveSyncSettings, RuleForType<any>][];
-            this._log(`${issueItems.length} Issue(s) found `, LOG_LEVEL_VERBOSE);
-            let idx = 0;
-            const applySettings = {} as Partial<ObsidianLiveSyncSettings>;
-            const OPT_FIX = `${$msg("Doctor.Button.Fix")}` as const;
-            const OPT_SKIP = `${$msg("Doctor.Button.Skip")}` as const;
-            const OPT_FIXBUTNOREBUILD = `${$msg("Doctor.Button.FixButNoRebuild")}` as const;
-            let skipped = 0;
-            for (const [key, value] of issueItems) {
-                const levelMap = {
-                    [RuleLevel.Necessary]: $msg("Doctor.Level.Necessary"),
-                    [RuleLevel.Recommended]: $msg("Doctor.Level.Recommended"),
-                    [RuleLevel.Optional]: $msg("Doctor.Level.Optional"),
-                    [RuleLevel.Must]: $msg("Doctor.Level.Must"),
-                };
-                const level = value.level ? levelMap[value.level] : "Unknown";
-                const options = [OPT_FIX] as [typeof OPT_FIX | typeof OPT_SKIP | typeof OPT_FIXBUTNOREBUILD];
-                if ((!skipRebuild && value.requireRebuild) || value.requireRebuildLocal) {
-                    options.push(OPT_FIXBUTNOREBUILD);
-                }
-                options.push(OPT_SKIP);
-                const note = skipRebuild
-                    ? ""
-                    : `${value.requireRebuild ? $msg("Doctor.Message.RebuildRequired") : ""}${value.requireRebuildLocal ? $msg("Doctor.Message.RebuildLocalRequired") : ""}`;
-
-                const ret = await this.core.confirm.askSelectStringDialogue(
-                    $msg("Doctor.Dialogue.MainFix", {
-                        name: getConfName(key as AllSettingItemKey),
-                        current: `${this.settings[key]}`,
-                        reason: value.reasonFunc?.(this.settings) ?? value.reason ?? " N/A ",
-                        ideal: `${value.valueDisplayFunc ? value.valueDisplayFunc(this.settings) : value.value}`,
-                        //@ts-ignore
-                        level: `${level}`,
-                        note: note,
-                    }),
-                    options,
-                    {
-                        title: $msg("Doctor.Dialogue.TitleFix", { current: `${++idx}`, total: `${issueItems.length}` }),
-                        defaultAction: OPT_FIX,
-                    }
-                );
-
-                if (ret == OPT_FIX || ret == OPT_FIXBUTNOREBUILD) {
-                    //@ts-ignore
-                    applySettings[key] = value.value;
-                    if (ret == OPT_FIX) {
-                        shouldRebuild = shouldRebuild || value.requireRebuild || false;
-                        shouldRebuildLocal = shouldRebuildLocal || value.requireRebuildLocal || false;
-                    }
-                } else {
-                    skipped++;
-                }
-            }
-            if (Object.keys(applySettings).length > 0) {
-                this.settings = {
-                    ...this.settings,
-                    ...applySettings,
-                };
-            }
-            if (skipped == 0) {
-                this.settings.doctorProcessedVersion = r.version;
-            } else {
-                if (
-                    (await this.core.confirm.askYesNoDialog($msg("Doctor.Message.SomeSkipped"), {
-                        title: $msg("Doctor.Dialogue.TitleAlmostDone"),
-                        defaultOption: "No",
-                    })) == "no"
-                ) {
-                    // Some skipped, and user wants
-                    this.settings.doctorProcessedVersion = r.version;
-                }
+        const { shouldRebuild, shouldRebuildLocal, isModified, settings } = await performDoctorConsultation(
+            this.core,
+            this.settings,
+            {
+                localRebuild: skipRebuild ? RebuildOptions.SkipEvenIfRequired : RebuildOptions.AutomaticAcceptable,
+                remoteRebuild: skipRebuild ? RebuildOptions.SkipEvenIfRequired : RebuildOptions.AutomaticAcceptable,
+                activateReason,
+                forceRescan,
             }
+        );
+        if (isModified) {
+            this.settings = settings;
             await this.core.saveSettings();
-            if (!skipRebuild) {
-                if (shouldRebuild) {
-                    await this.core.rebuilder.scheduleRebuild();
-                    await this.core.$$performRestart();
-                } else if (shouldRebuildLocal) {
-                    await this.core.rebuilder.scheduleFetch();
-                    await this.core.$$performRestart();
-                }
+        }
+        if (!skipRebuild) {
+            if (shouldRebuild) {
+                await this.core.rebuilder.scheduleRebuild();
+                await this.core.$$performRestart();
+            } else if (shouldRebuildLocal) {
+                await this.core.rebuilder.scheduleFetch();
+                await this.core.$$performRestart();
             }
         }
     }
@@ -197,12 +100,129 @@ export class ModuleMigration extends AbstractModule implements ICoreModule {
         return false;
     }
 
+    async checkIncompleteDocs(force: boolean = false): Promise<boolean> {
+        const incompleteDocsChecked = (await this.core.kvDB.get<boolean>("checkIncompleteDocs")) || false;
+        if (incompleteDocsChecked && !force) {
+            this._log("Incomplete docs check already done, skipping.", LOG_LEVEL_VERBOSE);
+            return Promise.resolve(true);
+        }
+
+        this._log("Checking for incomplete documents...", LOG_LEVEL_NOTICE, "check-incomplete");
+        const errorFiles = [];
+        for await (const metaDoc of this.localDatabase.findAllNormalDocs({ conflicts: true })) {
+            const path = getPath(metaDoc);
+
+            if (!isValidPath(path)) {
+                continue;
+            }
+            if (!(await this.core.$$isTargetFile(path, true))) {
+                continue;
+            }
+            if (!isMetaEntry(metaDoc)) {
+                continue;
+            }
+
+            const doc = await this.localDatabase.getDBEntryFromMeta(metaDoc);
+            if (!doc || !isLoadedEntry(doc)) {
+                continue;
+            }
+            if (isDeletedEntry(doc)) {
+                continue;
+            }
+            const storageFileContent = await this.core.storageAccess.readHiddenFileBinary(path);
+            // const storageFileBlob = createBlob(storageFileContent);
+            const sizeOnStorage = storageFileContent.byteLength;
+            const recordedSize = doc.size;
+            const docBlob = readAsBlob(doc);
+            const actualSize = docBlob.size;
+            if (recordedSize !== actualSize || sizeOnStorage !== actualSize || sizeOnStorage !== recordedSize) {
+                const contentMatched = await isDocContentSame(doc.data, storageFileContent);
+                errorFiles.push({ path, recordedSize, actualSize, storageSize: sizeOnStorage, contentMatched });
+                Logger(
+                    `Size mismatch for ${path}: ${recordedSize} (DB Recorded) , ${actualSize} (DB Stored) , ${sizeOnStorage} (Storage Stored), ${contentMatched ? "Content Matched" : "Content Mismatched"}`
+                );
+            }
+        }
+        if (errorFiles.length == 0) {
+            Logger("No size mismatches found", LOG_LEVEL_NOTICE);
+            await this.core.kvDB.set("checkIncompleteDocs", true);
+            return Promise.resolve(true);
+        }
+        Logger(`Found ${errorFiles.length} size mismatches`, LOG_LEVEL_NOTICE);
+        // We have to repair them following rules and situations:
+        // A. DB Recorded != DB Stored
+        //   A.1. DB Recorded == Storage Stored
+        //        Possibly recoverable from storage. Just overwrite the DB content with storage content.
+        //   A.2. Neither
+        //        Probably it cannot be resolved on this device. Even if the storage content is larger than DB Recorded, it possibly corrupted.
+        //        We do not fix it automatically. Leave it as is. Possibly other device can do this.
+        // B. DB Recorded == DB Stored ,  < Storage Stored
+        //   Very fragile, if DB Recorded size is less than Storage Stored size, we possibly repair the content (The issue was `unexpectedly shortened file`).
+        //   We do not fix it automatically, but it will be automatically overwritten in other process.
+        // C. DB Recorded == DB Stored ,  > Storage Stored
+        //   Probably restored by the user by resolving A or B on other device, We should overwrite the storage
+        //   Also do not fix it automatically. It should be overwritten by replication.
+        const recoverable = errorFiles.filter((e) => {
+            return e.recordedSize === e.storageSize;
+        });
+        const unrecoverable = errorFiles.filter((e) => {
+            return e.recordedSize !== e.storageSize;
+        });
+        const messageUnrecoverable =
+            unrecoverable.length > 0
+                ? $msg("moduleMigration.fix0256.messageUnrecoverable", {
+                      filesNotRecoverable: unrecoverable
+                          .map((e) => `- ${e.path} (M: ${e.recordedSize}, A: ${e.actualSize}, S: ${e.storageSize})`)
+                          .join("\n"),
+                  })
+                : "";
+
+        const message = $msg("moduleMigration.fix0256.message", {
+            files: recoverable
+                .map((e) => `- ${e.path} (M: ${e.recordedSize}, A: ${e.actualSize}, S: ${e.storageSize})`)
+                .join("\n"),
+            messageUnrecoverable,
+        });
+        const CHECK_IT_LATER = $msg("moduleMigration.fix0256.buttons.checkItLater");
+        const FIX = $msg("moduleMigration.fix0256.buttons.fix");
+        const DISMISS = $msg("moduleMigration.fix0256.buttons.DismissForever");
+        const ret = await this.core.confirm.askSelectStringDialogue(message, [CHECK_IT_LATER, FIX, DISMISS], {
+            title: $msg("moduleMigration.fix0256.title"),
+            defaultAction: CHECK_IT_LATER,
+        });
+        if (ret == FIX) {
+            for (const file of recoverable) {
+                // Overwrite the database with the files on the storage
+                const stubFile = this.core.storageAccess.getFileStub(file.path);
+                if (stubFile == null) {
+                    Logger(`Could not find stub file for ${file.path}`, LOG_LEVEL_NOTICE);
+                    continue;
+                }
+
+                stubFile.stat.mtime = Date.now();
+                const result = await this.core.fileHandler.storeFileToDB(stubFile, true, false);
+                if (result) {
+                    Logger(`Successfully restored ${file.path} from storage`);
+                } else {
+                    Logger(`Failed to restore ${file.path} from storage`, LOG_LEVEL_NOTICE);
+                }
+            }
+        } else if (ret === DISMISS) {
+            // User chose to dismiss the issue
+            await this.core.kvDB.set("checkIncompleteDocs", true);
+        }
+
+        return Promise.resolve(true);
+    }
+
     async $everyOnFirstInitialize(): Promise<boolean> {
         if (!this.localDatabase.isReady) {
             this._log($msg("moduleMigration.logLocalDatabaseNotReady"), LOG_LEVEL_NOTICE);
             return false;
         }
         if (this.settings.isConfigured) {
+            // TODO: Probably we have to check for insecure chunks
+            await this.checkIncompleteDocs();
             await this.migrateUsingDoctor(false);
             // await this.migrationCheck();
             await this.migrateDisableBulkSend();
@@ -221,6 +241,9 @@ export class ModuleMigration extends AbstractModule implements ICoreModule {
         eventHub.onEvent(EVENT_REQUEST_RUN_DOCTOR, async (reason) => {
             await this.migrateUsingDoctor(false, reason, true);
         });
+        eventHub.onEvent(EVENT_REQUEST_RUN_FIX_INCOMPLETE, async () => {
+            await this.checkIncompleteDocs(true);
+        });
         return Promise.resolve(true);
     }
 }

src/modules/features/InteractiveConflictResolving/ConflictResolveModal.ts

@@ -25,8 +25,8 @@ export class ConflictResolveModal extends Modal {
     title: string = "Conflicting changes";
 
     pluginPickMode: boolean = false;
-    localName: string = "Keep A";
-    remoteName: string = "Keep B";
+    localName: string = "Use Base";
+    remoteName: string = "Use Conflicted";
     offEvent?: ReturnType<typeof eventHub.onEvent>;
 
     constructor(app: App, filename: string, diff: diff_result, pluginPickMode?: boolean, remoteName?: string) {

src/modules/features/ModuleObsidianSetting.ts

@@ -11,11 +11,11 @@ import {
     SALT_OF_PASSPHRASE,
 } from "../../lib/src/common/types";
 import { LOG_LEVEL_NOTICE, LOG_LEVEL_URGENT } from "octagonal-wheels/common/logger";
-import { encrypt, tryDecrypt } from "octagonal-wheels/encryption";
 import { $msg, setLang } from "../../lib/src/common/i18n";
 import { isCloudantURI } from "../../lib/src/pouchdb/utils_couchdb";
 import { getLanguage } from "obsidian";
 import { SUPPORTED_I18N_LANGS, type I18N_LANGS } from "../../lib/src/common/rosetta.ts";
+import { decryptString, encryptString } from "@/lib/src/encryption/stringEncryption.ts";
 export class ModuleObsidianSettings extends AbstractObsidianModule implements IObsidianModule {
     async $everyOnLayoutReady(): Promise<boolean> {
         let isChanged = false;
@@ -73,7 +73,7 @@ export class ModuleObsidianSettings extends AbstractObsidianModule implements IO
     }
 
     async decryptConfigurationItem(encrypted: string, passphrase: string) {
-        const dec = await tryDecrypt(encrypted, passphrase + SALT_OF_PASSPHRASE, false);
+        const dec = await decryptString(encrypted, passphrase + SALT_OF_PASSPHRASE);
         if (dec) {
             this.usedPassphrase = passphrase;
             return dec;
@@ -83,7 +83,7 @@ export class ModuleObsidianSettings extends AbstractObsidianModule implements IO
 
     async encryptConfigurationItem(src: string, settings: ObsidianLiveSyncSettings) {
         if (this.usedPassphrase != "") {
-            return await encrypt(src, this.usedPassphrase + SALT_OF_PASSPHRASE, false);
+            return await encryptString(src, this.usedPassphrase + SALT_OF_PASSPHRASE);
         }
 
         const passphrase = await this.getPassphrase(settings);
@@ -94,7 +94,7 @@ export class ModuleObsidianSettings extends AbstractObsidianModule implements IO
             );
             return "";
         }
-        const dec = await encrypt(src, passphrase + SALT_OF_PASSPHRASE, false);
+        const dec = await encryptString(src, passphrase + SALT_OF_PASSPHRASE);
         if (dec) {
             this.usedPassphrase = passphrase;
             return dec;
@@ -174,18 +174,7 @@ export class ModuleObsidianSettings extends AbstractObsidianModule implements IO
         }
     }
 
-    async $$loadSettings(): Promise<void> {
-        const settings = Object.assign({}, DEFAULT_SETTINGS, await this.core.loadData()) as ObsidianLiveSyncSettings;
-
-        if (typeof settings.isConfigured == "undefined") {
-            // If migrated, mark true
-            if (JSON.stringify(settings) !== JSON.stringify(DEFAULT_SETTINGS)) {
-                settings.isConfigured = true;
-            } else {
-                settings.additionalSuffixOfDatabaseName = this.appId;
-                settings.isConfigured = false;
-            }
-        }
+    async $$decryptSettings(settings: ObsidianLiveSyncSettings): Promise<ObsidianLiveSyncSettings> {
         const passphrase = await this.getPassphrase(settings);
         if (passphrase === false) {
             this._log("No passphrase found for data.json! Verify configuration before syncing.", LOG_LEVEL_URGENT);
@@ -237,20 +226,62 @@ export class ModuleObsidianSettings extends AbstractObsidianModule implements IO
                 }
             }
         }
-        this.settings = settings;
+        return settings;
+    }
+
+    /**
+     * This method mutates the settings object.
+     * @param settings
+     * @returns
+     */
+    $$adjustSettings(settings: ObsidianLiveSyncSettings): Promise<ObsidianLiveSyncSettings> {
+        // Adjust settings as needed
+
+        // Delete this feature to avoid problems on mobile.
+        settings.disableRequestURI = true;
+
+        // GC is disabled.
+        settings.gcDelay = 0;
+        // So, use history is always enabled.
+        settings.useHistory = true;
+
+        if ("workingEncrypt" in settings) delete settings.workingEncrypt;
+        if ("workingPassphrase" in settings) delete settings.workingPassphrase;
+        // Splitter configurations have been replaced with chunkSplitterVersion.
+        if (settings.chunkSplitterVersion == "") {
+            if (settings.enableChunkSplitterV2) {
+                if (settings.useSegmenter) {
+                    settings.chunkSplitterVersion = "v2-segmenter";
+                } else {
+                    settings.chunkSplitterVersion = "v2";
+                }
+            } else {
+                settings.chunkSplitterVersion = "";
+            }
+        } else if (!(settings.chunkSplitterVersion in ChunkAlgorithmNames)) {
+            settings.chunkSplitterVersion = "";
+        }
+        return Promise.resolve(settings);
+    }
+
+    async $$loadSettings(): Promise<void> {
+        const settings = Object.assign({}, DEFAULT_SETTINGS, await this.core.loadData()) as ObsidianLiveSyncSettings;
+
+        if (typeof settings.isConfigured == "undefined") {
+            // If migrated, mark true
+            if (JSON.stringify(settings) !== JSON.stringify(DEFAULT_SETTINGS)) {
+                settings.isConfigured = true;
+            } else {
+                settings.additionalSuffixOfDatabaseName = this.appId;
+                settings.isConfigured = false;
+            }
+        }
+
+        this.settings = await this.core.$$decryptSettings(settings);
 
         setLang(this.settings.displayLanguage);
 
-        if ("workingEncrypt" in this.settings) delete this.settings.workingEncrypt;
-        if ("workingPassphrase" in this.settings) delete this.settings.workingPassphrase;
-
-        // Delete this feature to avoid problems on mobile.
-        this.settings.disableRequestURI = true;
-
-        // GC is disabled.
-        this.settings.gcDelay = 0;
-        // So, use history is always enabled.
-        this.settings.useHistory = true;
+        await this.core.$$adjustSettings(this.settings);
 
         const lsKey = "obsidian-live-sync-vaultanddevicename-" + this.core.$$getVaultName();
         if (this.settings.deviceAndVaultName != "") {
@@ -275,21 +306,6 @@ export class ModuleObsidianSettings extends AbstractObsidianModule implements IO
             }
         }
 
-        // Splitter configurations have been replaced with chunkSplitterVersion.
-        if (this.settings.chunkSplitterVersion == "") {
-            if (this.settings.enableChunkSplitterV2) {
-                if (this.settings.useSegmenter) {
-                    this.settings.chunkSplitterVersion = "v2-segmenter";
-                } else {
-                    this.settings.chunkSplitterVersion = "v2";
-                }
-            } else {
-                this.settings.chunkSplitterVersion = "";
-            }
-        } else if (!(this.settings.chunkSplitterVersion in ChunkAlgorithmNames)) {
-            this.settings.chunkSplitterVersion = "";
-        }
-
         // this.core.ignoreFiles = this.settings.ignoreFiles.split(",").map(e => e.trim());
         eventHub.emitEvent(EVENT_REQUEST_RELOAD_SETTING_TAB);
     }

src/modules/features/ModuleSetupObsidian.ts

@@ -7,7 +7,6 @@ import {
 } from "../../lib/src/common/types.ts";
 import { configURIBase, configURIBaseQR } from "../../common/types.ts";
 // import { PouchDB } from "../../lib/src/pouchdb/pouchdb-browser.js";
-import { decrypt, encrypt } from "../../lib/src/encryption/e2ee_v2.ts";
 import { fireAndForget } from "../../lib/src/common/utils.ts";
 import {
     EVENT_REQUEST_COPY_SETUP_URI,
@@ -19,6 +18,8 @@ import { AbstractObsidianModule, type IObsidianModule } from "../AbstractObsidia
 import { decodeAnyArray, encodeAnyArray } from "../../common/utils.ts";
 import qrcode from "qrcode-generator";
 import { $msg } from "../../lib/src/common/i18n.ts";
+import { performDoctorConsultation, RebuildOptions } from "@/lib/src/common/configForDoc.ts";
+import { encryptString, decryptString } from "@/lib/src/encryption/stringEncryption.ts";
 
 export class ModuleSetupObsidian extends AbstractObsidianModule implements IObsidianModule {
     $everyOnload(): Promise<boolean> {
@@ -129,9 +130,7 @@ export class ModuleSetupObsidian extends AbstractObsidianModule implements IObsi
                 delete setting[k];
             }
         }
-        const encryptedSetting = encodeURIComponent(
-            await encrypt(JSON.stringify(setting), encryptingPassphrase, false)
-        );
+        const encryptedSetting = encodeURIComponent(await encryptString(JSON.stringify(setting), encryptingPassphrase));
         const uri = `${configURIBase}${encryptedSetting} `;
         await navigator.clipboard.writeText(uri);
         this._log("Setup URI copied to clipboard", LOG_LEVEL_NOTICE);
@@ -150,9 +149,7 @@ export class ModuleSetupObsidian extends AbstractObsidianModule implements IObsi
             encryptedCouchDBConnection: "",
             encryptedPassphrase: "",
         };
-        const encryptedSetting = encodeURIComponent(
-            await encrypt(JSON.stringify(setting), encryptingPassphrase, false)
-        );
+        const encryptedSetting = encodeURIComponent(await encryptString(JSON.stringify(setting), encryptingPassphrase));
         const uri = `${configURIBase}${encryptedSetting} `;
         await navigator.clipboard.writeText(uri);
         this._log("Setup URI copied to clipboard", LOG_LEVEL_NOTICE);
@@ -170,6 +167,73 @@ export class ModuleSetupObsidian extends AbstractObsidianModule implements IObsi
         const config = decodeURIComponent(setupURI.substring(configURIBase.length));
         await this.setupWizard(config);
     }
+    async askSyncWithRemoteConfig(tryingSettings: ObsidianLiveSyncSettings): Promise<ObsidianLiveSyncSettings> {
+        const buttons = {
+            fetch: $msg("Setup.FetchRemoteConf.Buttons.Fetch"),
+            no: $msg("Setup.FetchRemoteConf.Buttons.Skip"),
+        } as const;
+        const fetchRemoteConf = await this.core.confirm.askSelectStringDialogue(
+            $msg("Setup.FetchRemoteConf.Message"),
+            Object.values(buttons),
+            { defaultAction: buttons.fetch, timeout: 0, title: $msg("Setup.FetchRemoteConf.Title") }
+        );
+        if (fetchRemoteConf == buttons.no) {
+            return tryingSettings;
+        }
+
+        const newSettings = JSON.parse(JSON.stringify(tryingSettings)) as ObsidianLiveSyncSettings;
+        const remoteConfig = await this.core.$$fetchRemotePreferredTweakValues(newSettings);
+        if (remoteConfig) {
+            this._log("Remote configuration found.", LOG_LEVEL_NOTICE);
+            const resultSettings = {
+                ...DEFAULT_SETTINGS,
+                ...tryingSettings,
+                ...remoteConfig,
+            } satisfies ObsidianLiveSyncSettings;
+            return resultSettings;
+        } else {
+            this._log("Remote configuration not applied.", LOG_LEVEL_NOTICE);
+            return {
+                ...DEFAULT_SETTINGS,
+                ...tryingSettings,
+            } satisfies ObsidianLiveSyncSettings;
+        }
+    }
+    async askPerformDoctor(
+        tryingSettings: ObsidianLiveSyncSettings
+    ): Promise<{ settings: ObsidianLiveSyncSettings; shouldRebuild: boolean; isModified: boolean }> {
+        const buttons = {
+            yes: $msg("Setup.Doctor.Buttons.Yes"),
+            no: $msg("Setup.Doctor.Buttons.No"),
+        } as const;
+        const performDoctor = await this.core.confirm.askSelectStringDialogue(
+            $msg("Setup.Doctor.Message"),
+            Object.values(buttons),
+            { defaultAction: buttons.yes, timeout: 0, title: $msg("Setup.Doctor.Title") }
+        );
+        if (performDoctor == buttons.no) {
+            return { settings: tryingSettings, shouldRebuild: false, isModified: false };
+        }
+
+        const newSettings = JSON.parse(JSON.stringify(tryingSettings)) as ObsidianLiveSyncSettings;
+        const { settings, shouldRebuild, isModified } = await performDoctorConsultation(this.core, newSettings, {
+            localRebuild: RebuildOptions.AutomaticAcceptable, // Because we are in the setup wizard, we can skip the confirmation.
+            remoteRebuild: RebuildOptions.SkipEvenIfRequired,
+            activateReason: "New settings from URI",
+        });
+        if (isModified) {
+            this._log("Doctor has fixed some issues!", LOG_LEVEL_NOTICE);
+            return {
+                settings,
+                shouldRebuild,
+                isModified,
+            };
+        } else {
+            this._log("Doctor detected no issues!", LOG_LEVEL_NOTICE);
+            return { settings: tryingSettings, shouldRebuild: false, isModified: false };
+        }
+    }
+
     async applySettingWizard(
         oldConf: ObsidianLiveSyncSettings,
         newConf: ObsidianLiveSyncSettings,
@@ -180,20 +244,24 @@ export class ModuleSetupObsidian extends AbstractObsidianModule implements IObsi
             {}
         );
         if (result == "yes") {
-            const newSettingW = Object.assign({}, DEFAULT_SETTINGS, newConf) as ObsidianLiveSyncSettings;
+            let newSettingW = Object.assign({}, DEFAULT_SETTINGS, newConf) as ObsidianLiveSyncSettings;
             this.core.replicator.closeReplication();
             this.settings.suspendFileWatching = true;
-            console.dir(newSettingW);
+            newSettingW = await this.askSyncWithRemoteConfig(newSettingW);
+            const { settings, shouldRebuild, isModified } = await this.askPerformDoctor(newSettingW);
+            if (isModified) {
+                newSettingW = settings;
+            }
             // Back into the default method once.
             newSettingW.configPassphraseStore = "";
             newSettingW.encryptedPassphrase = "";
             newSettingW.encryptedCouchDBConnection = "";
             newSettingW.additionalSuffixOfDatabaseName = `${"appId" in this.app ? this.app.appId : ""} `;
-            const setupJustImport = "Don't sync anything, just apply the settings.";
-            const setupAsNew = "This is a new client - sync everything from the remote server.";
-            const setupAsMerge = "This is an existing client - merge existing files with the server.";
-            const setupAgain = "Initialise new server data - ideal for new or broken servers.";
-            const setupManually = "Continue and configure manually.";
+            const setupJustImport = $msg("Setup.Apply.Buttons.OnlyApply");
+            const setupAsNew = $msg("Setup.Apply.Buttons.ApplyAndFetch");
+            const setupAsMerge = $msg("Setup.Apply.Buttons.ApplyAndMerge");
+            const setupAgain = $msg("Setup.Apply.Buttons.ApplyAndRebuild");
+            const setupCancel = $msg("Setup.Apply.Buttons.Cancel");
             newSettingW.syncInternalFiles = false;
             newSettingW.usePluginSync = false;
             newSettingW.isConfigured = true;
@@ -201,11 +269,16 @@ export class ModuleSetupObsidian extends AbstractObsidianModule implements IObsi
             if (!newSettingW.useIndexedDBAdapter) {
                 newSettingW.useIndexedDBAdapter = true;
             }
+            const warn = shouldRebuild ? $msg("Setup.Apply.WarningRebuildRecommended") : "";
+            const message = $msg("Setup.Apply.Message", {
+                method,
+                warn,
+            });
 
             const setupType = await this.core.confirm.askSelectStringDialogue(
-                "How would you like to set it up?",
-                [setupAsNew, setupAgain, setupAsMerge, setupJustImport, setupManually],
-                { defaultAction: setupAsNew }
+                message,
+                [setupAsNew, setupAsMerge, setupAgain, setupJustImport, setupCancel],
+                { defaultAction: setupAsNew, title: $msg("Setup.Apply.Title", { method }), timeout: 0 }
             );
             if (setupType == setupJustImport) {
                 this.core.settings = newSettingW;
@@ -237,71 +310,11 @@ export class ModuleSetupObsidian extends AbstractObsidianModule implements IObsi
                 await this.core.saveSettings();
                 this.core.$$clearUsedPassphrase();
                 await this.core.rebuilder.$rebuildEverything();
-            } else if (setupType == setupManually) {
-                const keepLocalDB = await this.core.confirm.askYesNoDialog("Keep local DB?", {
-                    defaultOption: "No",
-                });
-                const keepRemoteDB = await this.core.confirm.askYesNoDialog("Keep remote DB?", {
-                    defaultOption: "No",
-                });
-                if (keepLocalDB == "yes" && keepRemoteDB == "yes") {
-                    // nothing to do. so peaceful.
-                    this.core.settings = newSettingW;
-                    this.core.$$clearUsedPassphrase();
-                    await this.core.$allSuspendAllSync();
-                    await this.core.$allSuspendExtraSync();
-                    await this.core.saveSettings();
-                    const replicate = await this.core.confirm.askYesNoDialog("Unlock and replicate?", {
-                        defaultOption: "Yes",
-                    });
-                    if (replicate == "yes") {
-                        await this.core.$$replicate(true);
-                        await this.core.$$markRemoteUnlocked();
-                    }
-                    this._log("Configuration loaded.", LOG_LEVEL_NOTICE);
-                    return;
-                }
-                if (keepLocalDB == "no" && keepRemoteDB == "no") {
-                    const reset = await this.core.confirm.askYesNoDialog("Drop everything?", {
-                        defaultOption: "No",
-                    });
-                    if (reset != "yes") {
-                        this._log("Cancelled", LOG_LEVEL_NOTICE);
-                        this.core.settings = oldConf;
-                        return;
-                    }
-                }
-                let initDB;
-                this.core.settings = newSettingW;
-                this.core.$$clearUsedPassphrase();
-                await this.core.saveSettings();
-                if (keepLocalDB == "no") {
-                    await this.core.$$resetLocalDatabase();
-                    await this.core.localDatabase.initializeDatabase();
-                    const rebuild = await this.core.confirm.askYesNoDialog("Rebuild the database?", {
-                        defaultOption: "Yes",
-                    });
-                    if (rebuild == "yes") {
-                        initDB = this.core.$$initializeDatabase(true);
-                    } else {
-                        await this.core.$$markRemoteResolved();
-                    }
-                }
-                if (keepRemoteDB == "no") {
-                    await this.core.$$tryResetRemoteDatabase();
-                    await this.core.$$markRemoteLocked();
-                }
-                if (keepLocalDB == "no" || keepRemoteDB == "no") {
-                    const replicate = await this.core.confirm.askYesNoDialog("Replicate once?", {
-                        defaultOption: "Yes",
-                    });
-                    if (replicate == "yes") {
-                        if (initDB != null) {
-                            await initDB;
-                        }
-                        await this.core.$$replicate(true);
-                    }
-                }
+            } else {
+                // Explicitly cancel the operation or the dialog was closed.
+                this._log("Cancelled", LOG_LEVEL_NOTICE);
+                this.core.settings = oldConf;
+                return;
             }
             this._log("Configuration loaded.", LOG_LEVEL_NOTICE);
         } else {
@@ -320,7 +333,7 @@ export class ModuleSetupObsidian extends AbstractObsidianModule implements IObsi
                 true
             );
             if (encryptingPassphrase === false) return;
-            const newConf = await JSON.parse(await decrypt(confString, encryptingPassphrase, false));
+            const newConf = await JSON.parse(await decryptString(confString, encryptingPassphrase));
             if (newConf) {
                 await this.applySettingWizard(oldConf, newConf);
                 this._log("Configuration loaded.", LOG_LEVEL_NOTICE);

src/modules/features/SettingDialogue/ObsidianLiveSyncSettingTab.ts

@@ -17,8 +17,6 @@ import { delay, isObjectDifferent, sizeToHumanReadable } from "../../../lib/src/
 import { versionNumberString2Number } from "../../../lib/src/string_and_binary/convert.ts";
 import { Logger } from "../../../lib/src/common/logger.ts";
 import { checkSyncInfo } from "@/lib/src/pouchdb/negotiation.ts";
-import { balanceChunkPurgedDBs } from "@/lib/src/pouchdb/chunks.ts";
-import { purgeUnreferencedChunks } from "@/lib/src/pouchdb/chunks.ts";
 import { testCrypt } from "../../../lib/src/encryption/e2ee_v2.ts";
 import ObsidianLiveSyncPlugin from "../../../main.ts";
 import { scheduleTask } from "../../../common/utils.ts";
@@ -38,7 +36,6 @@ import { LiveSyncSetting as Setting } from "./LiveSyncSetting.ts";
 import { fireAndForget, yieldNextAnimationFrame } from "octagonal-wheels/promises";
 import { confirmWithMessage } from "../../coreObsidian/UILib/dialogs.ts";
 import { EVENT_REQUEST_RELOAD_SETTING_TAB, eventHub } from "../../../common/events.ts";
-import { skipIfDuplicated } from "octagonal-wheels/concurrency/lock";
 import { JournalSyncMinio } from "../../../lib/src/replication/journal/objectstore/JournalSyncMinio.ts";
 import { paneChangeLog } from "./PaneChangeLog.ts";
 import {
@@ -861,49 +858,6 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
         });
     }
 
-    async dryRunGC() {
-        await skipIfDuplicated("cleanup", async () => {
-            const replicator = this.plugin.$$getReplicator();
-            if (!(replicator instanceof LiveSyncCouchDBReplicator)) return;
-            const remoteDBConn = await replicator.connectRemoteCouchDBWithSetting(
-                this.plugin.settings,
-                this.plugin.$$isMobile()
-            );
-            if (typeof remoteDBConn == "string") {
-                Logger(remoteDBConn);
-                return;
-            }
-            await purgeUnreferencedChunks(remoteDBConn.db, true, this.plugin.settings, false);
-            await purgeUnreferencedChunks(this.plugin.localDatabase.localDatabase, true);
-            this.plugin.localDatabase.hashCaches.clear();
-        });
-    }
-
-    async dbGC() {
-        // Lock the remote completely once.
-        await skipIfDuplicated("cleanup", async () => {
-            const replicator = this.plugin.$$getReplicator();
-            if (!(replicator instanceof LiveSyncCouchDBReplicator)) return;
-            await this.plugin.$$getReplicator().markRemoteLocked(this.plugin.settings, true, true);
-            const remoteDBConnection = await replicator.connectRemoteCouchDBWithSetting(
-                this.plugin.settings,
-                this.plugin.$$isMobile()
-            );
-            if (typeof remoteDBConnection == "string") {
-                Logger(remoteDBConnection);
-                return;
-            }
-            await purgeUnreferencedChunks(remoteDBConnection.db, false, this.plugin.settings, true);
-            await purgeUnreferencedChunks(this.plugin.localDatabase.localDatabase, false);
-            this.plugin.localDatabase.hashCaches.clear();
-            await balanceChunkPurgedDBs(this.plugin.localDatabase.localDatabase, remoteDBConnection.db);
-            this.plugin.localDatabase.refreshSettings();
-            Logger(
-                "The remote database has been cleaned up! Other devices will be cleaned up on the next synchronisation."
-            );
-        });
-    }
-
     getMinioJournalSyncClient() {
         const id = this.plugin.settings.accessKey;
         const key = this.plugin.settings.secretKey;

src/modules/features/SettingDialogue/PaneAdvanced.ts

@@ -6,7 +6,7 @@ import type { PageFunctions } from "./SettingPane.ts";
 export function paneAdvanced(this: ObsidianLiveSyncSettingTab, paneEl: HTMLElement, { addPanel }: PageFunctions): void {
     void addPanel(paneEl, "Memory cache").then((paneEl) => {
         new Setting(paneEl).autoWireNumeric("hashCacheMaxCount", { clampMin: 10 });
-        new Setting(paneEl).autoWireNumeric("hashCacheMaxAmount", { clampMin: 1 });
+        // new Setting(paneEl).autoWireNumeric("hashCacheMaxAmount", { clampMin: 1 });
     });
     void addPanel(paneEl, "Local Database Tweak").then((paneEl) => {
         paneEl.addClass("wizardHidden");

src/modules/features/SettingDialogue/PaneChangeLog.ts

@@ -3,6 +3,7 @@ import { versionNumberString2Number } from "../../../lib/src/string_and_binary/c
 import { $msg } from "../../../lib/src/common/i18n.ts";
 import { fireAndForget } from "octagonal-wheels/promises";
 import type { ObsidianLiveSyncSettingTab } from "./ObsidianLiveSyncSettingTab.ts";
+import { visibleOnly } from "./SettingPane.ts";
 //@ts-ignore
 const manifestVersion: string = MANIFEST_VERSION || "-";
 //@ts-ignore
@@ -10,8 +11,34 @@ const updateInformation: string = UPDATE_INFO || "";
 
 const lastVersion = ~~(versionNumberString2Number(manifestVersion) / 1000);
 export function paneChangeLog(this: ObsidianLiveSyncSettingTab, paneEl: HTMLElement): void {
-    const informationDivEl = this.createEl(paneEl, "div", { text: "" });
+    const cx = this.createEl(
+        paneEl,
+        "div",
+        {
+            cls: "op-warn-info",
+        },
+        undefined,
+        visibleOnly(() => !this.isConfiguredAs("versionUpFlash", ""))
+    );
 
+    this.createEl(
+        cx,
+        "div",
+        {
+            text: this.editingSettings.versionUpFlash,
+        },
+        undefined
+    );
+    this.createEl(cx, "button", { text: $msg("obsidianLiveSyncSettingTab.btnGotItAndUpdated") }, (e) => {
+        e.addClass("mod-cta");
+        e.addEventListener("click", () => {
+            fireAndForget(async () => {
+                this.editingSettings.versionUpFlash = "";
+                await this.saveAllDirtySettings();
+            });
+        });
+    });
+    const informationDivEl = this.createEl(paneEl, "div", { text: "" });
     const tmpDiv = createDiv();
     // tmpDiv.addClass("sls-header-button");
     tmpDiv.addClass("op-warn-info");

src/modules/features/SettingDialogue/PaneHatch.ts

@@ -26,7 +26,7 @@ import { addPrefix, shouldBeIgnored, stripAllPrefixes } from "../../../lib/src/s
 import { $msg } from "../../../lib/src/common/i18n.ts";
 import { Semaphore } from "octagonal-wheels/concurrency/semaphore";
 import { LiveSyncSetting as Setting } from "./LiveSyncSetting.ts";
-import { EVENT_REQUEST_RUN_DOCTOR, eventHub } from "../../../common/events.ts";
+import { EVENT_REQUEST_RUN_DOCTOR, EVENT_REQUEST_RUN_FIX_INCOMPLETE, eventHub } from "../../../common/events.ts";
 import { ICHeader, ICXHeader, PSCHeader } from "../../../common/types.ts";
 import { HiddenFileSync } from "../../../features/HiddenFileSync/CmdHiddenFileSync.ts";
 import { EVENT_REQUEST_SHOW_HISTORY } from "../../../common/obsidianEvents.ts";
@@ -50,6 +50,19 @@ export function paneHatch(this: ObsidianLiveSyncSettingTab, paneEl: HTMLElement,
                         eventHub.emitEvent(EVENT_REQUEST_RUN_DOCTOR, "you wanted(Thank you)!");
                     })
             );
+        new Setting(paneEl)
+            .setName($msg("Setting.TroubleShooting.ScanBrokenFiles"))
+            .setDesc($msg("Setting.TroubleShooting.ScanBrokenFiles.Desc"))
+            .addButton((button) =>
+                button
+                    .setButtonText("Scan for Broken files")
+                    .setCta()
+                    .setDisabled(false)
+                    .onClick(() => {
+                        this.closeSetting();
+                        eventHub.emitEvent(EVENT_REQUEST_RUN_FIX_INCOMPLETE);
+                    })
+            );
         new Setting(paneEl).setName("Prepare the 'report' to create an issue").addButton((button) =>
             button
                 .setButtonText("Copy Report to clipboard")
@@ -190,7 +203,7 @@ ${stringifyYaml({
                             );
                             infoGroupEl.appendChild(
                                 this.createEl(infoGroupEl, "div", {
-                                    text: `Database: Modified: ${!fileOnDB ? `Missing:` : `${new Date(fileOnDB.mtime).toLocaleString()}, Size:${fileOnDB.size}`}`,
+                                    text: `Database: Modified: ${!fileOnDB ? `Missing:` : `${new Date(fileOnDB.mtime).toLocaleString()}, Size:${fileOnDB.size} (actual size:${readAsBlob(fileOnDB).size})`}`,
                                 })
                             );
                         })
@@ -335,7 +348,7 @@ ${stringifyYaml({
                         Logger("Start verifying all files", LOG_LEVEL_NOTICE, "verify");
                         const ignorePatterns = getFileRegExp(this.plugin.settings, "syncInternalFilesIgnorePatterns");
                         const targetPatterns = getFileRegExp(this.plugin.settings, "syncInternalFilesTargetPatterns");
-                        this.plugin.localDatabase.hashCaches.clear();
+                        this.plugin.localDatabase.clearCaches();
                         Logger("Start verifying all files", LOG_LEVEL_NOTICE, "verify");
                         const files = this.plugin.settings.syncInternalFiles
                             ? await this.plugin.storageAccess.getFilesIncludeHidden("/", targetPatterns, ignorePatterns)

src/modules/features/SettingDialogue/PanePatches.ts

@@ -28,9 +28,9 @@ export function panePatches(this: ObsidianLiveSyncSettingTab, paneEl: HTMLElemen
     void addPanel(paneEl, "Compatibility (Database structure)").then((paneEl) => {
         new Setting(paneEl).autoWireToggle("useIndexedDBAdapter", { invert: true, holdValue: true });
 
-        new Setting(paneEl)
-            .autoWireToggle("doNotUseFixedRevisionForChunks", { holdValue: true })
-            .setClass("wizardHidden");
+        // new Setting(paneEl)
+        //     .autoWireToggle("doNotUseFixedRevisionForChunks", { holdValue: true })
+        //     .setClass("wizardHidden");
         new Setting(paneEl).autoWireToggle("handleFilenameCaseSensitive", { holdValue: true }).setClass("wizardHidden");
 
         this.addOnSaved("useIndexedDBAdapter", async () => {
@@ -82,6 +82,7 @@ export function panePatches(this: ObsidianLiveSyncSettingTab, paneEl: HTMLElemen
     void addPanel(paneEl, "Edge case addressing (Behaviour)").then((paneEl) => {
         new Setting(paneEl).autoWireToggle("doNotSuspendOnFetching");
         new Setting(paneEl).setClass("wizardHidden").autoWireToggle("doNotDeleteFolder");
+        new Setting(paneEl).autoWireToggle("processSizeMismatchedFiles");
     });
 
     void addPanel(paneEl, "Edge case addressing (Processing)").then((paneEl) => {
@@ -99,13 +100,13 @@ export function panePatches(this: ObsidianLiveSyncSettingTab, paneEl: HTMLElemen
     });
 
     void addPanel(paneEl, "Remote Database Tweak (In sunset)").then((paneEl) => {
-        new Setting(paneEl).autoWireToggle("useEden").setClass("wizardHidden");
-        const onlyUsingEden = visibleOnly(() => this.isConfiguredAs("useEden", true));
-        new Setting(paneEl).autoWireNumeric("maxChunksInEden", { onUpdate: onlyUsingEden }).setClass("wizardHidden");
-        new Setting(paneEl)
-            .autoWireNumeric("maxTotalLengthInEden", { onUpdate: onlyUsingEden })
-            .setClass("wizardHidden");
-        new Setting(paneEl).autoWireNumeric("maxAgeInEden", { onUpdate: onlyUsingEden }).setClass("wizardHidden");
+        // new Setting(paneEl).autoWireToggle("useEden").setClass("wizardHidden");
+        // const onlyUsingEden = visibleOnly(() => this.isConfiguredAs("useEden", true));
+        // new Setting(paneEl).autoWireNumeric("maxChunksInEden", { onUpdate: onlyUsingEden }).setClass("wizardHidden");
+        // new Setting(paneEl)
+        //     .autoWireNumeric("maxTotalLengthInEden", { onUpdate: onlyUsingEden })
+        //     .setClass("wizardHidden");
+        // new Setting(paneEl).autoWireNumeric("maxAgeInEden", { onUpdate: onlyUsingEden }).setClass("wizardHidden");
 
         new Setting(paneEl).autoWireToggle("enableCompression").setClass("wizardHidden");
     });

src/modules/features/SettingDialogue/PaneSyncSettings.ts

@@ -7,7 +7,6 @@ import {
 import { Logger } from "../../../lib/src/common/logger.ts";
 import { $msg } from "../../../lib/src/common/i18n.ts";
 import { LiveSyncSetting as Setting } from "./LiveSyncSetting.ts";
-import { fireAndForget } from "octagonal-wheels/promises";
 import { EVENT_REQUEST_COPY_SETUP_URI, eventHub } from "../../../common/events.ts";
 import type { ObsidianLiveSyncSettingTab } from "./ObsidianLiveSyncSettingTab.ts";
 import type { PageFunctions } from "./SettingPane.ts";
@@ -17,30 +16,6 @@ export function paneSyncSettings(
     paneEl: HTMLElement,
     { addPanel, addPane }: PageFunctions
 ): void {
-    if (this.editingSettings.versionUpFlash != "") {
-        const c = this.createEl(
-            paneEl,
-            "div",
-            {
-                text: this.editingSettings.versionUpFlash,
-                cls: "op-warn sls-setting-hidden",
-            },
-            (el) => {
-                this.createEl(el, "button", { text: $msg("obsidianLiveSyncSettingTab.btnGotItAndUpdated") }, (e) => {
-                    e.addClass("mod-cta");
-                    e.addEventListener("click", () => {
-                        fireAndForget(async () => {
-                            this.editingSettings.versionUpFlash = "";
-                            await this.saveAllDirtySettings();
-                            c.remove();
-                        });
-                    });
-                });
-            },
-            visibleOnly(() => !this.isConfiguredAs("versionUpFlash", ""))
-        );
-    }
-
     this.createEl(paneEl, "div", {
         text: $msg("obsidianLiveSyncSettingTab.msgSelectAndApplyPreset"),
         cls: "wizardOnly",

src/modules/features/SettingDialogue/settingConstants.ts

@@ -1,422 +1 @@
-import { $t } from "../../../lib/src/common/i18n.ts";
-import {
-    DEFAULT_SETTINGS,
-    configurationNames,
-    type ConfigurationItem,
-    type FilterBooleanKeys,
-    type FilterNumberKeys,
-    type FilterStringKeys,
-    type ObsidianLiveSyncSettings,
-} from "../../../lib/src/common/types.ts";
-
-export type OnDialogSettings = {
-    configPassphrase: string;
-    preset: "" | "PERIODIC" | "LIVESYNC" | "DISABLE";
-    syncMode: "ONEVENTS" | "PERIODIC" | "LIVESYNC";
-    dummy: number;
-    deviceAndVaultName: string;
-};
-
-export const OnDialogSettingsDefault: OnDialogSettings = {
-    configPassphrase: "",
-    preset: "",
-    syncMode: "ONEVENTS",
-    dummy: 0,
-    deviceAndVaultName: "",
-};
-export const AllSettingDefault = { ...DEFAULT_SETTINGS, ...OnDialogSettingsDefault };
-
-export type AllSettings = ObsidianLiveSyncSettings & OnDialogSettings;
-export type AllStringItemKey = FilterStringKeys<AllSettings>;
-export type AllNumericItemKey = FilterNumberKeys<AllSettings>;
-export type AllBooleanItemKey = FilterBooleanKeys<AllSettings>;
-export type AllSettingItemKey = AllStringItemKey | AllNumericItemKey | AllBooleanItemKey;
-
-export type ValueOf<T extends AllSettingItemKey> = T extends AllStringItemKey
-    ? string
-    : T extends AllNumericItemKey
-      ? number
-      : T extends AllBooleanItemKey
-        ? boolean
-        : AllSettings[T];
-
-export const SettingInformation: Partial<Record<keyof AllSettings, ConfigurationItem>> = {
-    liveSync: {
-        name: "Sync Mode",
-    },
-    couchDB_URI: {
-        name: "Server URI",
-        placeHolder: "https://........",
-    },
-    couchDB_USER: {
-        name: "Username",
-        desc: "username",
-    },
-    couchDB_PASSWORD: {
-        name: "Password",
-        desc: "password",
-    },
-    couchDB_DBNAME: {
-        name: "Database Name",
-    },
-    passphrase: {
-        name: "Passphrase",
-        desc: "Encryption phassphrase. If changed, you should overwrite the server's database with the new (encrypted) files.",
-    },
-    showStatusOnEditor: {
-        name: "Show status inside the editor",
-        desc: "Requires restart of Obsidian.",
-    },
-    showOnlyIconsOnEditor: {
-        name: "Show status as icons only",
-    },
-    showStatusOnStatusbar: {
-        name: "Show status on the status bar",
-        desc: "Requires restart of Obsidian.",
-    },
-    lessInformationInLog: {
-        name: "Show only notifications",
-        desc: "Disables logging, only shows notifications. Please disable if you report an issue.",
-    },
-    showVerboseLog: {
-        name: "Verbose Log",
-        desc: "Show verbose log. Please enable if you report an issue.",
-    },
-    hashCacheMaxCount: {
-        name: "Memory cache size (by total items)",
-    },
-    hashCacheMaxAmount: {
-        name: "Memory cache size (by total characters)",
-        desc: "(Mega chars)",
-    },
-    writeCredentialsForSettingSync: {
-        name: "Write credentials in the file",
-        desc: "(Not recommended) If set, credentials will be stored in the file.",
-    },
-    notifyAllSettingSyncFile: {
-        name: "Notify all setting files",
-    },
-    configPassphrase: {
-        name: "Passphrase of sensitive configuration items",
-        desc: "This passphrase will not be copied to another device. It will be set to `Default` until you configure it again.",
-    },
-    configPassphraseStore: {
-        name: "Encrypting sensitive configuration items",
-    },
-    syncOnSave: {
-        name: "Sync on Save",
-        desc: "Starts synchronisation when a file is saved.",
-    },
-    syncOnEditorSave: {
-        name: "Sync on Editor Save",
-        desc: "When you save a file in the editor, start a sync automatically",
-    },
-    syncOnFileOpen: {
-        name: "Sync on File Open",
-        desc: "Forces the file to be synced when opened.",
-    },
-    syncOnStart: {
-        name: "Sync on Startup",
-        desc: "Automatically Sync all files when opening Obsidian.",
-    },
-    syncAfterMerge: {
-        name: "Sync after merging file",
-        desc: "Sync automatically after merging files",
-    },
-    trashInsteadDelete: {
-        name: "Use the trash bin",
-        desc: "Move remotely deleted files to the trash, instead of deleting.",
-    },
-    doNotDeleteFolder: {
-        name: "Keep empty folder",
-        desc: "Should we keep folders that don't have any files inside?",
-    },
-    resolveConflictsByNewerFile: {
-        name: "(BETA) Always overwrite with a newer file",
-        desc: "Testing only - Resolve file conflicts by syncing newer copies of the file, this can overwrite modified files. Be Warned.",
-    },
-    checkConflictOnlyOnOpen: {
-        name: "Delay conflict resolution of inactive files",
-        desc: "Should we only check for conflicts when a file is opened?",
-    },
-    showMergeDialogOnlyOnActive: {
-        name: "Delay merge conflict prompt for inactive files.",
-        desc: "Should we prompt you about conflicting files when a file is opened?",
-    },
-    disableMarkdownAutoMerge: {
-        name: "Always prompt merge conflicts",
-        desc: "Should we prompt you for every single merge, even if we can safely merge automatcially?",
-    },
-    writeDocumentsIfConflicted: {
-        name: "Apply Latest Change if Conflicting",
-        desc: "Enable this option to automatically apply the most recent change to documents even when it conflicts",
-    },
-    syncInternalFilesInterval: {
-        name: "Scan hidden files periodically",
-        desc: "Seconds, 0 to disable",
-    },
-    batchSave: {
-        name: "Batch database update",
-        desc: "Reducing the frequency with which on-disk changes are reflected into the DB",
-    },
-    readChunksOnline: {
-        name: "Fetch chunks on demand",
-        desc: "(ex. Read chunks online) If this option is enabled, LiveSync reads chunks online directly instead of replicating them locally. Increasing Custom chunk size is recommended.",
-    },
-    syncMaxSizeInMB: {
-        name: "Maximum file size",
-        desc: "(MB) If this is set, changes to local and remote files that are larger than this will be skipped. If the file becomes smaller again, a newer one will be used.",
-    },
-    useIgnoreFiles: {
-        name: "(Beta) Use ignore files",
-        desc: "If this is set, changes to local files which are matched by the ignore files will be skipped. Remote changes are determined using local ignore files.",
-    },
-    ignoreFiles: {
-        name: "Ignore files",
-        desc: "Comma separated `.gitignore, .dockerignore`",
-    },
-    batch_size: {
-        name: "Batch size",
-        desc: "Number of changes to sync at a time. Defaults to 50. Minimum is 2.",
-    },
-    batches_limit: {
-        name: "Batch limit",
-        desc: "Number of batches to process at a time. Defaults to 40. Minimum is 2. This along with batch size controls how many docs are kept in memory at a time.",
-    },
-    useTimeouts: {
-        name: "Use timeouts instead of heartbeats",
-        desc: "If this option is enabled, PouchDB will hold the connection open for 60 seconds, and if no change arrives in that time, close and reopen the socket, instead of holding it open indefinitely. Useful when a proxy limits request duration but can increase resource usage.",
-    },
-    concurrencyOfReadChunksOnline: {
-        name: "Batch size of on-demand fetching",
-    },
-    minimumIntervalOfReadChunksOnline: {
-        name: "The delay for consecutive on-demand fetches",
-    },
-    suspendFileWatching: {
-        name: "Suspend file watching",
-        desc: "Stop watching for file changes.",
-    },
-    suspendParseReplicationResult: {
-        name: "Suspend database reflecting",
-        desc: "Stop reflecting database changes to storage files.",
-    },
-    writeLogToTheFile: {
-        name: "Write logs into the file",
-        desc: "Warning! This will have a serious impact on performance. And the logs will not be synchronised under the default name. Please be careful with logs; they often contain your confidential information.",
-    },
-    deleteMetadataOfDeletedFiles: {
-        name: "Do not keep metadata of deleted files.",
-    },
-    useIndexedDBAdapter: {
-        name: "(Obsolete) Use an old adapter for compatibility",
-        desc: "Before v0.17.16, we used an old adapter for the local database. Now the new adapter is preferred. However, it needs local database rebuilding. Please disable this toggle when you have enough time. If leave it enabled, also while fetching from the remote database, you will be asked to disable this.",
-        obsolete: true,
-    },
-    watchInternalFileChanges: {
-        name: "Scan changes on customization sync",
-        desc: "Do not use internal API",
-    },
-    doNotSuspendOnFetching: {
-        name: "Fetch database with previous behaviour",
-    },
-    disableCheckingConfigMismatch: {
-        name: "Do not check configuration mismatch before replication",
-    },
-    usePluginSync: {
-        name: "Enable customization sync",
-    },
-    autoSweepPlugins: {
-        name: "Scan customization automatically",
-        desc: "Scan customization before replicating.",
-    },
-    autoSweepPluginsPeriodic: {
-        name: "Scan customization periodically",
-        desc: "Scan customization every 1 minute.",
-    },
-    notifyPluginOrSettingUpdated: {
-        name: "Notify customized",
-        desc: "Notify when other device has newly customized.",
-    },
-    remoteType: {
-        name: "Remote Type",
-        desc: "Remote server type",
-    },
-    endpoint: {
-        name: "Endpoint URL",
-        placeHolder: "https://........",
-    },
-    accessKey: {
-        name: "Access Key",
-    },
-    secretKey: {
-        name: "Secret Key",
-    },
-    region: {
-        name: "Region",
-        placeHolder: "auto",
-    },
-    bucket: {
-        name: "Bucket Name",
-    },
-    useCustomRequestHandler: {
-        name: "Use Custom HTTP Handler",
-        desc: "Enable this if your Object Storage doesn't support CORS",
-    },
-    maxChunksInEden: {
-        name: "Maximum Incubating Chunks",
-        desc: "The maximum number of chunks that can be incubated within the document. Chunks exceeding this number will immediately graduate to independent chunks.",
-    },
-    maxTotalLengthInEden: {
-        name: "Maximum Incubating Chunk Size",
-        desc: "The maximum total size of chunks that can be incubated within the document. Chunks exceeding this size will immediately graduate to independent chunks.",
-    },
-    maxAgeInEden: {
-        name: "Maximum Incubation Period",
-        desc: "The maximum duration for which chunks can be incubated within the document. Chunks exceeding this period will graduate to independent chunks.",
-    },
-    settingSyncFile: {
-        name: "Filename",
-        desc: "Save settings to a markdown file. You will be notified when new settings arrive. You can set different files by the platform.",
-    },
-    preset: {
-        name: "Presets",
-        desc: "Apply preset configuration",
-    },
-    syncMode: {
-        name: "Sync Mode",
-    },
-    periodicReplicationInterval: {
-        name: "Periodic Sync interval",
-        desc: "Interval (sec)",
-    },
-    syncInternalFilesBeforeReplication: {
-        name: "Scan for hidden files before replication",
-    },
-    automaticallyDeleteMetadataOfDeletedFiles: {
-        name: "Delete old metadata of deleted files on start-up",
-        desc: "(Days passed, 0 to disable automatic-deletion)",
-    },
-    additionalSuffixOfDatabaseName: {
-        name: "Database suffix",
-        desc: "LiveSync could not handle multiple vaults which have same name without different prefix, This should be automatically configured.",
-    },
-    hashAlg: {
-        name: configurationNames["hashAlg"]?.name || "",
-        desc: "xxhash64 is the current default.",
-    },
-    deviceAndVaultName: {
-        name: "Device name",
-        desc: "Unique name between all synchronized devices. To edit this setting, please disable customization sync once.",
-    },
-    displayLanguage: {
-        name: "Display Language",
-        desc: 'Not all messages have been translated. And, please revert to "Default" when reporting errors.',
-    },
-    enableChunkSplitterV2: {
-        name: "Use splitting-limit-capped chunk splitter",
-        desc: "If enabled, chunks will be split into no more than 100 items. However, dedupe is slightly weaker.",
-    },
-    disableWorkerForGeneratingChunks: {
-        name: "Do not split chunks in the background",
-        desc: "If disabled(toggled), chunks will be split on the UI thread (Previous behaviour).",
-    },
-    processSmallFilesInUIThread: {
-        name: "Process small files in the foreground",
-        desc: "If enabled, the file under 1kb will be processed in the UI thread.",
-    },
-    batchSaveMinimumDelay: {
-        name: "Minimum delay for batch database updating",
-        desc: "Seconds. Saving to the local database will be delayed until this value after we stop typing or saving.",
-    },
-    batchSaveMaximumDelay: {
-        name: "Maximum delay for batch database updating",
-        desc: "Saving will be performed forcefully after this number of seconds.",
-    },
-    notifyThresholdOfRemoteStorageSize: {
-        name: "Notify when the estimated remote storage size exceeds on start up",
-        desc: "MB (0 to disable).",
-    },
-    usePluginSyncV2: {
-        name: "Enable per-file customization sync",
-        desc: "If enabled, efficient per-file customization sync will be used. A minor migration is required when enabling this feature, and all devices must be updated to v0.23.18. Enabling this feature will result in losing compatibility with older versions.",
-    },
-    handleFilenameCaseSensitive: {
-        name: "Handle files as Case-Sensitive",
-        desc: "If this enabled, All files are handled as case-Sensitive (Previous behaviour).",
-    },
-    doNotUseFixedRevisionForChunks: {
-        name: "Compute revisions for chunks",
-        desc: "If this enabled, all chunks will be stored with the revision made from its content.",
-    },
-    sendChunksBulkMaxSize: {
-        name: "Maximum size of chunks to send in one request",
-        desc: "MB",
-    },
-    useAdvancedMode: {
-        name: "Enable advanced features",
-        // desc: "Enable advanced mode"
-    },
-    usePowerUserMode: {
-        name: "Enable poweruser features",
-        // desc: "Enable power user mode",
-        // level: LEVEL_ADVANCED
-    },
-    useEdgeCaseMode: {
-        name: "Enable edge case treatment features",
-    },
-    enableDebugTools: {
-        name: "Enable Developers' Debug Tools.",
-        desc: "While enabled, it causes very performance impact but debugging replication testing and other features will be enabled. Please disable this if you have not read the source code. Requires restart of Obsidian.",
-    },
-    suppressNotifyHiddenFilesChange: {
-        name: "Suppress notification of hidden files change",
-        desc: "If enabled, the notification of hidden files change will be suppressed.",
-    },
-    syncMinimumInterval: {
-        name: "Minimum interval for syncing",
-        desc: "The minimum interval for automatic synchronisation on event.",
-    },
-    useRequestAPI: {
-        name: "Use Request API to avoid `inevitable` CORS problem",
-        desc: "If enabled, the request API will be used to avoid `inevitable` CORS problems. This is a workaround and may not work in all cases. PLEASE READ THE DOCUMENTATION BEFORE USING THIS OPTION. This is a less-secure option.",
-    },
-    hideFileWarningNotice: {
-        name: "Show status icon instead of file warnings banner",
-        desc: "If enabled, the ⛔ icon will be shown inside the status instead of the file warnings banner. No details will be shown.",
-    },
-    bucketPrefix: {
-        name: "File prefix on the bucket",
-        desc: "Effectively a directory. Should end with `/`. e.g., `vault-name/`.",
-    },
-    chunkSplitterVersion: {
-        name: "Chunk Splitter",
-        desc: "Now we can choose how to split the chunks; V3 is the most efficient. If you have troubled, please make this Default or Legacy.",
-    },
-};
-function translateInfo(infoSrc: ConfigurationItem | undefined | false) {
-    if (!infoSrc) return false;
-    const info = { ...infoSrc };
-    info.name = $t(info.name);
-    if (info.desc) {
-        info.desc = $t(info.desc);
-    }
-    return info;
-}
-function _getConfig(key: AllSettingItemKey) {
-    if (key in configurationNames) {
-        return configurationNames[key as keyof ObsidianLiveSyncSettings];
-    }
-    if (key in SettingInformation) {
-        return SettingInformation[key as keyof ObsidianLiveSyncSettings];
-    }
-    return false;
-}
-export function getConfig(key: AllSettingItemKey) {
-    return translateInfo(_getConfig(key));
-}
-export function getConfName(key: AllSettingItemKey) {
-    const conf = getConfig(key);
-    if (!conf) return `${key} (No info)`;
-    return conf.name;
-}
+export * from "@lib/common/settingConstants.ts";

tsconfig.json

@@ -23,5 +23,5 @@
         }
     },
     "include": ["**/*.ts"],
-    "exclude": ["pouchdb-browser-webpack", "utils", "src/lib/apps"]
+    "exclude": ["pouchdb-browser-webpack", "utils", "src/lib/apps", "**/*.test.ts"]
 }

updates.md

@@ -1,4 +1,142 @@
+## 0.25.7
+
+15th August, 2025
+
+**Since the release of 0.25.6, there are two large problem. Please update immediately.**
+
+- We may have corrupted some documents during the migration process. **Please check your documents on the wizard.**
+- Due to a chunk ID assignment issue, some data has not been encrypted. **Please rebuild the database using Rebuild Everything** if you have enabled E2EE.
+
+**_So, If you have enabled E2EE, please perform `Rebuild everything`. If not, please check your documents on the wizard._**
+
+In next version, insecure chunk detection will be implemented.
+
+### Fixed
+
+- Off-loaded chunking have been fixed to ensure proper functionality (#693).
+- Chunk document ID assignment has been fixed.
+- Replication prevention message during version up detection has been improved (#686).
+- `Keep A` and `Keep B` on Conflict resolving dialogue has been renamed to `Use Base` and `Use Conflicted` (#691).
+
+### Improved
+
+- Metadata and content-size unmatched documents are now detected and reported, prevented to be applied to the storage.
+    - This behaviour can be configured in `Patch` -> `Edge case addressing (Behaviour)` -> `Process files even if seems to be corrupted`
+    - Note: this toggle is for the direct-database-manipulation users.
+
+### New Features
+
+- `Scan for Broken files` has been implemented on `Hatch` -> `TroubleShooting`.
+
+### Refactored
+
+- Off-loaded processes have been refactored for the better maintainability.
+    - Files prefixed `bg.worker` are now work on the worker threads.
+    - Files prefixed `bgWorker.` are now also controls these worker threads. (I know what you want to say... I will rename them).
+- Removed unused code.
+
+## ~~0.25.5~~ 0.25.6
+
+(0.25.5 has been withdrawn due to a bug in the `Fetch chunks on demand` feature).
+
+9th August, 2025
+
+### Fixed
+
+- Storage scanning no longer occurs when `Suspend file watching` is enabled (including boot-sequence).
+    - This change improves safety when troubleshooting or fetching the remote database.
+- `Fetch chunks on demand` is now working again (if you installed 0.25.5, other versions are not affected).
+
+### Improved
+
+- Saving notes and files now consumes less memory.
+    - Data is no longer fully buffered in memory and written at once; instead, it is now written in each over-2MB increments.
+- Chunk caching is now more efficient.
+    - Chunks are now managed solely by their count (still maintained as LRU). If memory usage becomes excessive, they will be automatically released by the system-runtime.
+    - Reverse-indexing is also no longer used. It is performed as scanning caches and act also as a WeakRef thinning.
+- Both of them (may) are effective for #692, #680, and some more.
+
+### Changed
+
+- `Incubate Chunks in Document` (also known as `Eden`) is now fully sunset.
+    - Existing chunks can still be read, but new ones will no longer be created.
+- The `Compute revisions for chunks` setting has also been removed.
+    - This feature is now always enabled and is no longer configurable (restoring the original behaviour).
+- As mentioned, `Memory cache size (by total characters)` has been removed.
+    - The `Memory cache size (by total items)` setting is now the only option available (but it has 10x ratio compared to the previous version).
+
+### Refactored
+
+- A significant refactoring of the core codebase is underway.
+    - This is part of our ongoing efforts to improve code maintainability, readability, and to unify interfaces.
+        - Previously, complex files posed a risk due to a low bus factor. Fortunately, as our devices have become faster and more capable, we can now write code that is clearer and more maintainable (And not so much costs on performance).
+    - Hashing functions have been refactored into the `HashManager` class and its derived classes.
+    - Chunk splitting functions have been refactored into the `ContentSplitterCore` class and its derived classes.
+    - Change tracking functions have been refactored into the `ChangeManager` class.
+    - Chunk read/write functions have been refactored into the `ChunkManager` class.
+    - Fetching chunks on demand is now handled separately from the `ChunkManager` and chunk reading functions. Chunks are queued by the `ChunkManager` and then processed by the `ChunkFetcher`, simplifying the process and reducing unnecessary complexity.
+    - Then, local database access via `LiveSyncLocalDB` has been refactored to use the new classes.
+- References to external sources from `commonlib` have been corrected.
+- Type definitions in `types.ts` have been refined.
+- Unit tests are being added incrementally.
+    - I am using `Deno` for testing, to simplify testing and coverage reporting.
+    - While this is not identical to the Obsidian environment, `jest` may also have limitations. It is certainly better than having no tests.
+        - In other words, recent manual scenario testing has highlighted some shortcomings.
+    - `pouchdb-test`, used for testing PouchDB with Deno, has been added, utilising the `memory` adapter.
+
+Side note: Although class-oriented programming is sometimes considered an outdated style, However, I have come to re-evaluate it as valuable from the perspectives of maintainability and readability.
+
+## 0.25.4
+
+29th July, 2025
+
+### Fixed
+
+- The PBKDF2Salt is no longer corrupted when attempting replication while the device is offline. (#686)
+    - If this issue has already occurred, please use `Maintenance` -> `Rebuilding Operations (Remote Only)` -> `Overwrite Remote` and `Send` to resolve it.
+    - Please perform this operation on the device that is most reliable.
+    - I am so sorry for the inconvenience; there are no patching workarounds. The rebuilding operation is the only solution.
+        - This issue only affects the encryption of the remote database and does not impact the local databases on any devices.
+        - (Preventing synchronisation is by design and expected behaviour, even if it is sometimes inconvenient. This is also why we should avoid using workarounds; it is, admittedly, an excuse).
+        - In any case, we can unlock the remote from the warning dialogue on receiving devices. We are performing replication, instead of simple synchronisation at the expense of a little complexity (I would love to express thank you again for your every effort to manage and maintain the settings! Your all understanding saves our notes).
+    - This process may require considerable time and bandwidth (as usual), so please wait patiently and ensure a stable network connection.
+
+### Side note
+
+The PBKDF2Salt will be referred to as the `Security Seed`, and it is used to derive the encryption key for replication. Therefore, it should be stored on the server prior to synchronisation. We apologise for the lack of explanation in previous updates!
+
+## 0.25.3
+
+22nd July, 2025
+
+### Fixed
+
+- Now the `Doctor` at migration will save the configuration.
+
+## 0.25.2 ~~0.25.1~~
+
+(0.25.1 was missed due to a mistake in the versioning process).
+19th July, 2025
+
+### Refined and New Features
+
+- Fetching the remote database on `RedFlag` now also retrieves remote configurations optionally.
+    - This is beneficial if we have already set up another device and wish to use the same configuration. We will see a much less frequent `Unmatched` dialogue.
+- The setup wizard using Set-up URI and QR code has been improved.
+    - The message is now more user-friendly.
+    - The obsolete method (manual setting application) has been removed.
+    - The `Cancel` button has been added to the setup wizard.
+    - We can now fetch the remote configuration from the server if it exists, which is useful for adding new devices.
+        - Mostly same as a `RedFlag` fetching remote configuration.
+    - We can also use the `Doctor` to check and fix the imported (and fetched) configuration before applying it.
+
+### Changes
+
+- The Set-up URI is now encrypted with a new encryption algorithm (mostly the same as `V2`).
+    - The new Set-up URI is not compatible with version 0.24.x or earlier.
+
 ## 0.25.0
+
 19th July, 2025 (beta1 in 0.25.0-beta1, 13th July, 2025)
 
 After reading Issue #668, I conducted another self-review of the E2EE-related code. In retrospect, it was clearly written by someone inexperienced, which is understandable, but it is still rather embarrassing. Three years is certainly enough time for growth.
@@ -7,93 +145,5 @@ I have now rewritten the E2EE code to be more robust and easier to understand. I
 
 As a result, this is the first time in a while that forward compatibility has been broken. We have also taken the opportunity to change all metadata to use encryption rather than obfuscation. Furthermore, the `Dynamic Iteration Count` setting is now redundant and has been moved to the `Patches` pane in the settings. Thanks to Rabin-Karp, the eden setting is also no longer necessary and has been relocated accordingly. Therefore, v0.25.0 represents a legitimate and correct evolution.
 
-### Fixed
-- The encryption algorithm now uses HKDF with a master key.
-    - This is more robust and faster than the previous implementation.
-    - It is now more secure against rainbow table attacks.
-    - The previous implementation can still be used via `Patches` -> `End-to-end encryption algorithm` -> `Force V1`.
-      - Note that `V1: Legacy` can decrypt V2, but produces V1 output.
-- `Fetch everything from the remote` now works correctly.
-    - It no longer creates local database entries before synchronisation.
-- Extra log messages during QR code decoding have been removed.
-
-### Changed
-- The following settings have been moved to the `Patches` pane:
-    - `Remote Database Tweak`
-      - `Incubate Chunks in Document`
-      - `Data Compression`
-
-### Behavioural and API Changes
-- `DirectFileManipulatorV2` now requires new settings (as you may already know, E2EEAlgorithm).
-- The database version has been increased to `12` from `10`.
-    - If an older version is detected, we will be notified and synchronisation will be paused until the update is acknowledged. (It has been a long time since this behaviour was last encountered; we always err on the side of caution, even if it is less convenient.)
-
-### Refactored
-- `couchdb_utils.ts` has been separated into several explicitly named files.
-- Some missing functions in `bgWorker.mock.ts` have been added.
-
-## 0.24.31
-10th July, 2025
-
-### Fixed
-
-- The description of `Enable Developers' Debug Tools.` has been refined.
-    - Now performance impact is more clearly stated.
-- Automatic conflict checking and resolution has been improved.
-    - It now works parallelly for each other file, instead of sequentially. It makes significantly faster on first synchronisation when with local files information.
-- Resolving conflicts dialogue will not be shown for the multiple files at once.
-    - It will be shown for each file, one by one.
-
-## 0.24.30
-
-9th July, 2025
-
-### New Feature
-
-- New chunking algorithm `V3: Fine deduplication` has been added, and will be recommended after updates.
-    - The Rabin-Karp algorithm is used for efficient chunking.
-    - This will be the default in the new installations.
-    - It is more robust and faster than the previous one.
-    - We can change it in the `Advanced` pane of the settings.
-- New language `ko` (Korean) has been added.
-    - Thank you for your contribution, [@ellixspace](https://x.com/ellixspace)!
-        - Any contributions are welcome, from any route. Please let me know if I seem to be unaware of this. It is often the case that I am not really aware of it.
-- Chinese (Simplified) translation has been updated.
-    - Thank you for your contribution, [@52sanmao](https://github.com/52sanmao)!
-
-### Fixed
-
-- Numeric settings are now never lost the focus during value changing.
-- Doctor now redacts more sensitive information on error reports.
-
-### Improved
-
-- All translations have been rewritten into YAML format, to easier to manage and contribute.
-    - We can write them with comments, newlines, and other YAML features.
-- Doctor recommendations are now shown in a user-friendly notation.
-    - We can now see the recommended as `V3: Fine deduplication` instead of `v3-rabin-karp`.
-
-### Refactored
-
-- Never-ending `ObsidianLiveSyncSettingTab.ts` has finally been separated into each pane's file.
-- Some commented-out code has been removed.
-
-### Acknowledgement
-
-- Jun Murakami, Shun Ishiguro, and Yoshihiro Oyama. 2012. Implementation and Evaluation of a Cache Deduplication Mechanism with Content-Defined Chunking. In _IPSJ SIG Technical Report_, Vol.2012-ARC-202, No.4. Information Processing Society of Japan, 1-7.
-
-## 0.24.29
-
-20th June, 2025
-
-### Fixed
-
-- Synchronisation with buckets now works correctly, regardless of whether a prefix is set or the bucket has been (re-) initialised (#664).
-- An information message is now displayed again, during any automatic synchronisation is enabled (#662).
-
-### Tidied up
-
-- Importing paths have been tidied up.
-
 Older notes are in
 [updates_old.md](https://github.com/vrtmrz/obsidian-livesync/blob/main/updates_old.md).

updates_old.md

@@ -1,3 +1,45 @@
+
+## 0.25.0
+
+19th July, 2025 (beta1 in 0.25.0-beta1, 13th July, 2025)
+
+After reading Issue #668, I conducted another self-review of the E2EE-related code. In retrospect, it was clearly written by someone inexperienced, which is understandable, but it is still rather embarrassing. Three years is certainly enough time for growth.
+
+I have now rewritten the E2EE code to be more robust and easier to understand. It is significantly more readable and should be easier to maintain in the future. The performance issue, previously considered a concern, has been addressed by introducing a master key and deriving keys using HKDF. This approach is both fast and robust, and it provides protection against rainbow table attacks. (In addition, this implementation has been [a dedicated package on the npm registry](https://github.com/vrtmrz/octagonal-wheels), and tested in 100% branch-coverage).
+
+As a result, this is the first time in a while that forward compatibility has been broken. We have also taken the opportunity to change all metadata to use encryption rather than obfuscation. Furthermore, the `Dynamic Iteration Count` setting is now redundant and has been moved to the `Patches` pane in the settings. Thanks to Rabin-Karp, the eden setting is also no longer necessary and has been relocated accordingly. Therefore, v0.25.0 represents a legitimate and correct evolution.
+
+### Fixed
+
+- The encryption algorithm now uses HKDF with a master key.
+    - This is more robust and faster than the previous implementation.
+    - It is now more secure against rainbow table attacks.
+    - The previous implementation can still be used via `Patches` -> `End-to-end encryption algorithm` -> `Force V1`.
+        - Note that `V1: Legacy` can decrypt V2, but produces V1 output.
+- `Fetch everything from the remote` now works correctly.
+    - It no longer creates local database entries before synchronisation.
+- Extra log messages during QR code decoding have been removed.
+
+### Changed
+
+- The following settings have been moved to the `Patches` pane:
+    - `Remote Database Tweak`
+        - `Incubate Chunks in Document`
+        - `Data Compression`
+
+### Behavioural and API Changes
+
+- `DirectFileManipulatorV2` now requires new settings (as you may already know, E2EEAlgorithm).
+- The database version has been increased to `12` from `10`.
+    - If an older version is detected, we will be notified and synchronisation will be paused until the update is acknowledged. (It has been a long time since this behaviour was last encountered; we always err on the side of caution, even if it is less convenient.)
+
+### Refactored
+
+- `couchdb_utils.ts` has been separated into several explicitly named files.
+- Some missing functions in `bgWorker.mock.ts` have been added.
+
+
+
 ## 0.24.0
 
 I know that we have been waiting for a long time. It is finally released!
@@ -14,6 +56,70 @@ Thank you, and I hope your troubles will be resolved!
 
 ---
 
+## 0.24.31
+
+10th July, 2025
+
+### Fixed
+
+- The description of `Enable Developers' Debug Tools.` has been refined.
+    - Now performance impact is more clearly stated.
+- Automatic conflict checking and resolution has been improved.
+    - It now works parallelly for each other file, instead of sequentially. It makes significantly faster on first synchronisation when with local files information.
+- Resolving conflicts dialogue will not be shown for the multiple files at once.
+    - It will be shown for each file, one by one.
+
+## 0.24.30
+
+9th July, 2025
+
+### New Feature
+
+- New chunking algorithm `V3: Fine deduplication` has been added, and will be recommended after updates.
+    - The Rabin-Karp algorithm is used for efficient chunking.
+    - This will be the default in the new installations.
+    - It is more robust and faster than the previous one.
+    - We can change it in the `Advanced` pane of the settings.
+- New language `ko` (Korean) has been added.
+    - Thank you for your contribution, [@ellixspace](https://x.com/ellixspace)!
+        - Any contributions are welcome, from any route. Please let me know if I seem to be unaware of this. It is often the case that I am not really aware of it.
+- Chinese (Simplified) translation has been updated.
+    - Thank you for your contribution, [@52sanmao](https://github.com/52sanmao)!
+
+### Fixed
+
+- Numeric settings are now never lost the focus during value changing.
+- Doctor now redacts more sensitive information on error reports.
+
+### Improved
+
+- All translations have been rewritten into YAML format, to easier to manage and contribute.
+    - We can write them with comments, newlines, and other YAML features.
+- Doctor recommendations are now shown in a user-friendly notation.
+    - We can now see the recommended as `V3: Fine deduplication` instead of `v3-rabin-karp`.
+
+### Refactored
+
+- Never-ending `ObsidianLiveSyncSettingTab.ts` has finally been separated into each pane's file.
+- Some commented-out code has been removed.
+
+### Acknowledgement
+
+- Jun Murakami, Shun Ishiguro, and Yoshihiro Oyama. 2012. Implementation and Evaluation of a Cache Deduplication Mechanism with Content-Defined Chunking. In _IPSJ SIG Technical Report_, Vol.2012-ARC-202, No.4. Information Processing Society of Japan, 1-7.
+
+## 0.24.29
+
+20th June, 2025
+
+### Fixed
+
+- Synchronisation with buckets now works correctly, regardless of whether a prefix is set or the bucket has been (re-) initialised (#664).
+- An information message is now displayed again, during any automatic synchronisation is enabled (#662).
+
+### Tidied up
+
+- Importing paths have been tidied up.
+
 ## 0.24.28
 
 15th June, 2025