public/obsidian-livesync
1
0
Fork 0
mirror of https://github.com/vrtmrz/obsidian-livesync.git synced 2026-05-15 12:01:16 +00:00
Code Issues Packages Projects Releases Wiki Activity

Compare commits

base: public:0.22.1
Branches Tags
public:main public:enhance_fix_p2p public:test_real_obsidian public:v0_25_62 public:update_lib public:feat-userhashsalt public:auto_conf_2 public:auto_conf public:beta public:alpha public:refactor_services_2 public:refactor_services public:dev public:svelteui public:forprivacy
public:0.25.62 public:0.25.61 public:0.25.60 public:0.25.59 public:0.25.58 public:0.25.57 public:0.25.56+patched5 public:0.25.56+patched4 public:0.25.56+patched3 public:0.25.56+patched2 public:0.25.56+patched1 public:0.25.56 public:0.25.55 public:0.25.54 public:0.25.53 public:0.25.52-patched-3 public:0.25.52-patched-2 public:0.25.52-patched-1 public:0.25.52 public:0.25.51 public:0.25.50 public:0.25.49 public:0.25.48 public:0.25.47 public:0.25.46 public:0.25.45 public:0.25.44 public:0.25.43-patched-9 public:0.25.43-patched-8 public:0.25.43-patched-7 public:0.25.43-patched-6 public:0.25.43-patched-5 public:0.25.43-patched-4 public:0.25.43-patched-3 public:0.25.43-patched-2 public:0.25.43-patched-1 public:0.25.43 public:0.25.42 public:0.25.41-patched-3 public:0.25.41-patched-2 public:0.25.41-patched-1 public:0.25.41 public:0.25.40 public:0.25.39 public:0.25.38 public:0.25.37 public:0.25.36 public:0.25.35 public:0.25.34 public:0.25.33 public:0.25.32 public:0.25.31 public:0.25.30 public:0.25.30-dev public:0.25.29 public:0.25.28 public:0.25.27 public:0.25.26 public:0.25.25 public:0.25.24 public:0.25.24.beta3 public:0.25.24.beta2 public:0.25.24.beta1 public:0.25.23 public:0.25.23.beta1 public:0.25.22 public:0.25.21 public:0.25.21.beta2 public:0.25.21.beta1 public:0.25.20 public:0.25.19 public:0.25.18 public:0.25.17 public:0.25.16 public:0.25.15 public:0.25.14 public:0.25.13 public:0.25.12 public:0.25.11 public:0.25.10 public:0.25.9 public:0.25.8 public:0.25.7 public:0.25.6 public:0.25.5 public:0.25.4 public:0.25.3 public:0.25.2 public:0.25.1 public:0.25.0 public:0.25.0-beta1 public:0.24.31 public:0.24.30 public:0.24.29 public:0.24.28 public:0.24.27 public:0.24.26 public:0.24.25 public:0.24.24 public:0.24.23 public:0.24.22 public:0.24.21 public:0.24.20 public:0.24.19 public:0.24.18 public:0.24.17 public:0.24.16 public:0.24.15 public:0.24.14 public:0.24.13 public:0.24.12 public:0.24.11 public:0.24.10 public:0.24.9 public:0.24.8 public:0.24.7 public:0.24.6 public:0.24.5 public:0.24.4 public:0.24.3 public:0.24.2 public:0.24.1 public:0.24.0 public:0.24.0.dev-rc8 public:0.24.0.dev-rc7 public:0.24.0.dev-rc6 public:0.24.0.dev-rc5 public:0.24.0.dev-rc4 public:0.24.0.dev-rc3 public:0.24.0.dev-rc2 public:0.23.23 public:0.23.22 public:0.23.21 public:0.23.20 public:0.23.19 public:0.23.18 public:0.23.17 public:0.23.16 public:0.23.15 public:0.23.14 public:0.23.13 public:0.23.12 public:0.23.11 public:0.23.10 public:0.23.9 public:0.23.8 public:0.23.7 public:0.23.6 public:0.23.5 public:0.23.4 public:0.23.3 public:0.23.2 public:0.23.1 public:0.23.0 public:0.22.19 public:0.22.18 public:0.22.17 public:0.22.16 public:0.22.15 public:0.22.14 public:0.22.13 public:0.22.12 public:0.22.11 public:0.22.10 public:0.22.9 public:0.22.8 public:0.22.7 public:0.22.6 public:0.22.5 public:0.22.4 public:0.22.3 public:0.22.2 public:0.22.1 public:0.22.0 public:0.21.5 public:0.21.4 public:0.21.3 public:0.21.2 public:0.21.1 public:0.21.0 public:0.20.7 public:0.20.6 public:0.20.5 public:0.20.4 public:0.20.3 public:0.20.2 public:0.20.1 public:0.20.0 public:0.19.23 public:0.19.22 public:0.19.21 public:0.19.20 public:0.19.19 public:0.19.18 public:0.19.17 public:0.19.16 public:0.19.15 public:0.19.14 public:0.19.13 public:0.19.12 public:0.19.11 public:0.19.10 public:0.19.9 public:0.19.8 public:0.19.7 public:0.19.6 public:0.19.5 public:0.19.4 public:0.19.3 public:0.19.2 public:0.19.1 public:0.19.0 public:0.18.6 public:0.18.5 public:0.18.4 public:0.18.3 public:0.18.2 public:0.18.1 public:0.18.0 public:0.17.34 public:0.17.33 public:0.17.32 public:0.17.31 public:0.17.30 public:0.17.29 public:0.17.28 public:0.17.27 public:0.17.26 public:0.17.25 public:0.17.24 public:0.17.23 public:0.17.22 public:0.17.21 public:0.17.20 public:0.17.19 public:0.17.18 public:0.17.17 public:0.17.16 public:0.17.15 public:0.17.14 public:0.17.13 public:0.17.12 public:0.17.11 public:0.17.10 public:0.17.9 public:0.17.8 public:0.17.7 public:0.17.6 public:0.17.5 public:0.17.4 public:0.17.3 public:0.17.2 public:0.17.1 public:0.17.0 public:0.16.8 public:0.16.7 public:0.16.6 public:0.16.5 public:0.16.4 public:0.16.3 public:0.16.2 public:0.16.1 public:0.16.0 public:0.15.10 public:0.15.9 public:0.15.8 public:0.15.7 public:0.15.6 public:0.15.5 public:0.15.4 public:0.15.3 public:0.15.2 public:0.15.1 public:0.15.0 public:0.14.8 public:0.14.7 public:0.14.6 public:0.14.5 public:0.14.4 public:0.14.3 public:0.14.2 public:0.14.1 public:0.14.0 public:0.13.4 public:0.13.3 public:0.13.2 public:0.13.1 public:0.13.0 public:0.12.5 public:0.12.4 public:0.12.3 public:0.12.2 public:0.12.1 public:0.12.0 public:0.11.10 public:0.11.9 public:0.11.8 public:0.11.7 public:0.11.6 public:0.11.5 public:0.11.4 public:0.11.3 public:0.11.2 public:0.11.1 public:0.11.0 public:0.10.1 public:0.10.0 public:0.9.0 public:0.8.8 public:0.8.7 public:0.8.6 public:0.8.5 public:0.8.3 public:0.8.2 public:0.8.1 public:0.8.0 public:0.7.2 public:0.7.1 public:0.7.0 public:0.6.1 public:0.6.0 public:0.5.0 public:0.4.1 public:0.4.0 public:0.3.1 public:0.3.0 public:0.2.1 public:0.2.0 public:0.1.26 public:0.1.25 public:0.1.24 public:0.1.23 public:0.1.22 public:0.1.21 public:0.1.20 public:0.1.19 public:0.1.18 public:0.1.17 public:0.1.16 public:0.1.15 public:0.1.14 public:0.1.13 public:0.1.12 public:0.1.11 public:0.1.10 public:0.1.9 public:0.1.8 public:0.1.7 public:0.1.6 public:0.1.5 public:0.1.4 public:0.1.3 public:0.1.2 public:0.1.1 public:0.1.0 public:0.0.9 public:0.0.8 public:0.0.7 public:0.0.6
...
compare: public:0.22.4
Branches Tags
public:enhance_fix_p2p public:test_real_obsidian public:main public:v0_25_62 public:update_lib public:feat-userhashsalt public:auto_conf_2 public:auto_conf public:beta public:alpha public:refactor_services_2 public:refactor_services public:dev public:svelteui public:forprivacy
public:0.25.62 public:0.25.61 public:0.25.60 public:0.25.59 public:0.25.58 public:0.25.57 public:0.25.56+patched5 public:0.25.56+patched4 public:0.25.56+patched3 public:0.25.56+patched2 public:0.25.56+patched1 public:0.25.56 public:0.25.55 public:0.25.54 public:0.25.53 public:0.25.52-patched-3 public:0.25.52-patched-2 public:0.25.52-patched-1 public:0.25.52 public:0.25.51 public:0.25.50 public:0.25.49 public:0.25.48 public:0.25.47 public:0.25.46 public:0.25.45 public:0.25.44 public:0.25.43-patched-9 public:0.25.43-patched-8 public:0.25.43-patched-7 public:0.25.43-patched-6 public:0.25.43-patched-5 public:0.25.43-patched-4 public:0.25.43-patched-3 public:0.25.43-patched-2 public:0.25.43-patched-1 public:0.25.43 public:0.25.42 public:0.25.41-patched-3 public:0.25.41-patched-2 public:0.25.41-patched-1 public:0.25.41 public:0.25.40 public:0.25.39 public:0.25.38 public:0.25.37 public:0.25.36 public:0.25.35 public:0.25.34 public:0.25.33 public:0.25.32 public:0.25.31 public:0.25.30 public:0.25.30-dev public:0.25.29 public:0.25.28 public:0.25.27 public:0.25.26 public:0.25.25 public:0.25.24 public:0.25.24.beta3 public:0.25.24.beta2 public:0.25.24.beta1 public:0.25.23 public:0.25.23.beta1 public:0.25.22 public:0.25.21 public:0.25.21.beta2 public:0.25.21.beta1 public:0.25.20 public:0.25.19 public:0.25.18 public:0.25.17 public:0.25.16 public:0.25.15 public:0.25.14 public:0.25.13 public:0.25.12 public:0.25.11 public:0.25.10 public:0.25.9 public:0.25.8 public:0.25.7 public:0.25.6 public:0.25.5 public:0.25.4 public:0.25.3 public:0.25.2 public:0.25.1 public:0.25.0 public:0.25.0-beta1 public:0.24.31 public:0.24.30 public:0.24.29 public:0.24.28 public:0.24.27 public:0.24.26 public:0.24.25 public:0.24.24 public:0.24.23 public:0.24.22 public:0.24.21 public:0.24.20 public:0.24.19 public:0.24.18 public:0.24.17 public:0.24.16 public:0.24.15 public:0.24.14 public:0.24.13 public:0.24.12 public:0.24.11 public:0.24.10 public:0.24.9 public:0.24.8 public:0.24.7 public:0.24.6 public:0.24.5 public:0.24.4 public:0.24.3 public:0.24.2 public:0.24.1 public:0.24.0 public:0.24.0.dev-rc8 public:0.24.0.dev-rc7 public:0.24.0.dev-rc6 public:0.24.0.dev-rc5 public:0.24.0.dev-rc4 public:0.24.0.dev-rc3 public:0.24.0.dev-rc2 public:0.23.23 public:0.23.22 public:0.23.21 public:0.23.20 public:0.23.19 public:0.23.18 public:0.23.17 public:0.23.16 public:0.23.15 public:0.23.14 public:0.23.13 public:0.23.12 public:0.23.11 public:0.23.10 public:0.23.9 public:0.23.8 public:0.23.7 public:0.23.6 public:0.23.5 public:0.23.4 public:0.23.3 public:0.23.2 public:0.23.1 public:0.23.0 public:0.22.19 public:0.22.18 public:0.22.17 public:0.22.16 public:0.22.15 public:0.22.14 public:0.22.13 public:0.22.12 public:0.22.11 public:0.22.10 public:0.22.9 public:0.22.8 public:0.22.7 public:0.22.6 public:0.22.5 public:0.22.4 public:0.22.3 public:0.22.2 public:0.22.1 public:0.22.0 public:0.21.5 public:0.21.4 public:0.21.3 public:0.21.2 public:0.21.1 public:0.21.0 public:0.20.7 public:0.20.6 public:0.20.5 public:0.20.4 public:0.20.3 public:0.20.2 public:0.20.1 public:0.20.0 public:0.19.23 public:0.19.22 public:0.19.21 public:0.19.20 public:0.19.19 public:0.19.18 public:0.19.17 public:0.19.16 public:0.19.15 public:0.19.14 public:0.19.13 public:0.19.12 public:0.19.11 public:0.19.10 public:0.19.9 public:0.19.8 public:0.19.7 public:0.19.6 public:0.19.5 public:0.19.4 public:0.19.3 public:0.19.2 public:0.19.1 public:0.19.0 public:0.18.6 public:0.18.5 public:0.18.4 public:0.18.3 public:0.18.2 public:0.18.1 public:0.18.0 public:0.17.34 public:0.17.33 public:0.17.32 public:0.17.31 public:0.17.30 public:0.17.29 public:0.17.28 public:0.17.27 public:0.17.26 public:0.17.25 public:0.17.24 public:0.17.23 public:0.17.22 public:0.17.21 public:0.17.20 public:0.17.19 public:0.17.18 public:0.17.17 public:0.17.16 public:0.17.15 public:0.17.14 public:0.17.13 public:0.17.12 public:0.17.11 public:0.17.10 public:0.17.9 public:0.17.8 public:0.17.7 public:0.17.6 public:0.17.5 public:0.17.4 public:0.17.3 public:0.17.2 public:0.17.1 public:0.17.0 public:0.16.8 public:0.16.7 public:0.16.6 public:0.16.5 public:0.16.4 public:0.16.3 public:0.16.2 public:0.16.1 public:0.16.0 public:0.15.10 public:0.15.9 public:0.15.8 public:0.15.7 public:0.15.6 public:0.15.5 public:0.15.4 public:0.15.3 public:0.15.2 public:0.15.1 public:0.15.0 public:0.14.8 public:0.14.7 public:0.14.6 public:0.14.5 public:0.14.4 public:0.14.3 public:0.14.2 public:0.14.1 public:0.14.0 public:0.13.4 public:0.13.3 public:0.13.2 public:0.13.1 public:0.13.0 public:0.12.5 public:0.12.4 public:0.12.3 public:0.12.2 public:0.12.1 public:0.12.0 public:0.11.10 public:0.11.9 public:0.11.8 public:0.11.7 public:0.11.6 public:0.11.5 public:0.11.4 public:0.11.3 public:0.11.2 public:0.11.1 public:0.11.0 public:0.10.1 public:0.10.0 public:0.9.0 public:0.8.8 public:0.8.7 public:0.8.6 public:0.8.5 public:0.8.3 public:0.8.2 public:0.8.1 public:0.8.0 public:0.7.2 public:0.7.1 public:0.7.0 public:0.6.1 public:0.6.0 public:0.5.0 public:0.4.1 public:0.4.0 public:0.3.1 public:0.3.0 public:0.2.1 public:0.2.0 public:0.1.26 public:0.1.25 public:0.1.24 public:0.1.23 public:0.1.22 public:0.1.21 public:0.1.20 public:0.1.19 public:0.1.18 public:0.1.17 public:0.1.16 public:0.1.15 public:0.1.14 public:0.1.13 public:0.1.12 public:0.1.11 public:0.1.10 public:0.1.9 public:0.1.8 public:0.1.7 public:0.1.6 public:0.1.5 public:0.1.4 public:0.1.3 public:0.1.2 public:0.1.1 public:0.1.0 public:0.0.9 public:0.0.8 public:0.0.7 public:0.0.6

12 Commits
0.22.1 ... 0.22.4

Author SHA1 Message Date
vorotamoroz
b9527ccab0 bump 2024-01-30 17:31:52 +00:00
vorotamoroz
fa3aa2702c Fixed: 
- Now the result of conflict resolution could be surely written into the storage.
- Deleted files can be handled correctly again in the history dialogue and conflict dialogue.
- Some wrong log messages were fixed.
- Change handling now has become more stable.
- Some event handling became to be safer.
Improved:
- Dumping document information shows conflicts and revisions.
- The timestamp-only differences can be surely cached.
- Timestamp difference detection can be rounded by two seconds.
Refactored:
- A bit of organisation to write the test.
 2024-01-30 17:31:02 +00:00
vorotamoroz
93e7cbb133 bump. 2024-01-29 08:41:03 +00:00
vorotamoroz
716ae32e02 Fixed: 
- Deletion of files is now reliably synchronised.
 2024-01-29 08:40:41 +00:00
vorotamoroz
d6d8cbcf5a bump 2024-01-29 07:57:02 +00:00
vorotamoroz
efd348b266 Fixed: 
- No longer detects storage changes which have been caused by Self-hosted LiveSync itself.
- Setting sync file will be detected only if it has been configured now.
  - And its log will be shown only while the verbose log is enabled.
- Customisation file enumeration has got less blingy.
Fixed and improved:
- In-editor-status is now shown in the following areas:
  - Note editing pane (Source mode and live-preview mode).
  - New tab pane.
  - Canvas pane.
 2024-01-29 07:56:02 +00:00
vorotamoroz
8969b1800a bump 2024-01-24 08:53:00 +00:00
vorotamoroz
2c8e026e29 Fixed: 
- Now the results of resolving conflicts are surely synchronised.
Modified:
- Some setting items got new clear names.
New feature:
- We can limit the synchronising files by their size.
- Now the settings could be stored in a specific markdown file to synchronise or switch it
- Customisation of the obsoleted device is now able to be deleted at once.
 2024-01-24 08:52:47 +00:00
vorotamoroz
a6c27eab3d Merge pull request #367 from calvinbui/patch-1 
Skip workspace-mobile.json for cross-platform sync
 2024-01-24 15:55:50 +09:00
vorotamoroz
9b5c57d540 Merge pull request #336 from toon159/patch-1 
Lower payload size limit and batch limit from 10 to 2
 2024-01-22 13:03:29 +09:00
Calvin Bui
c251c596e8 Skip workspace-mobile.json for cross-platform sync 2024-01-18 13:23:48 +11:00
Vichaya Raksakunpanich
5802ed31be Update ObsidianLiveSyncSettingTab.ts 
Lower payload size limit and batch limit to 2 due to IBM Cloudant read/write limitation. Hope it will fix the "Replication error".
 2023-11-24 16:25:16 +07:00
14 changed files with 688 additions and 219 deletions
Expand all files Collapse all files

2
manifest.json
View File

@@ -1,7 +1,7 @@
{ {
"id": "obsidian-livesync", "id": "obsidian-livesync",
"name": "Self-hosted LiveSync", "name": "Self-hosted LiveSync",
"version": "0.22.1", "version": "0.22.4",
"minAppVersion": "0.9.12", "minAppVersion": "0.9.12",
"description": "Community implementation of self-hosted livesync. Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.", "description": "Community implementation of self-hosted livesync. Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.",
"author": "vorotamoroz", "author": "vorotamoroz",

4
package-lock.json generated
View File

@@ -1,12 +1,12 @@
{ {
"name": "obsidian-livesync", "name": "obsidian-livesync",
"version": "0.22.1", "version": "0.22.4",
"lockfileVersion": 2, "lockfileVersion": 2,
"requires": true, "requires": true,
"packages": { "packages": {
"": { "": {
"name": "obsidian-livesync", "name": "obsidian-livesync",
"version": "0.22.1", "version": "0.22.4",
"license": "MIT", "license": "MIT",
"dependencies": { "dependencies": {
"diff-match-patch": "^1.0.5", "diff-match-patch": "^1.0.5",

2
package.json
View File

@@ -1,6 +1,6 @@
{ {
"name": "obsidian-livesync", "name": "obsidian-livesync",
"version": "0.22.1", "version": "0.22.4",
"description": "Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.", "description": "Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.",
"main": "main.js", "main": "main.js",
"type": "module", "type": "module",

12
src/CmdConfigSync.ts
View File

@@ -4,7 +4,7 @@ import { Notice, type PluginManifest, parseYaml, normalizePath } from "./deps";
import type { EntryDoc, LoadedEntry, InternalFileEntry, FilePathWithPrefix, FilePath, DocumentID, AnyEntry, SavingEntry } from "./lib/src/types"; import type { EntryDoc, LoadedEntry, InternalFileEntry, FilePathWithPrefix, FilePath, DocumentID, AnyEntry, SavingEntry } from "./lib/src/types";
import { LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE, MODE_SELECTIVE } from "./lib/src/types"; import { LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE, MODE_SELECTIVE } from "./lib/src/types";
import { ICXHeader, PERIODIC_PLUGIN_SWEEP, } from "./types"; import { ICXHeader, PERIODIC_PLUGIN_SWEEP, } from "./types";
import { createTextBlob, delay, getDocData } from "./lib/src/utils"; import { createTextBlob, delay, getDocData, sendSignal, waitForSignal } from "./lib/src/utils";
import { Logger } from "./lib/src/logger"; import { Logger } from "./lib/src/logger";
import { WrappedNotice } from "./lib/src/wrapper"; import { WrappedNotice } from "./lib/src/wrapper";
import { readString, decodeBinary, arrayBufferToBase64, sha1 } from "./lib/src/strbin"; import { readString, decodeBinary, arrayBufferToBase64, sha1 } from "./lib/src/strbin";
@@ -335,7 +335,9 @@ export class ConfigSync extends LiveSyncCommands {
return; return;
}, { suspended: true, batchSize: 1, concurrentLimit: 5, delay: 300, yieldThreshold: 10 }).pipeTo( }, { suspended: true, batchSize: 1, concurrentLimit: 5, delay: 300, yieldThreshold: 10 }).pipeTo(
new QueueProcessor( new QueueProcessor(
(pluginDataList) => { async (pluginDataList) => {
// Concurrency is two, therefore, we can unlock the previous awaiting.
sendSignal("plugin-next-load");
let newList = [...this.pluginList]; let newList = [...this.pluginList];
for (const item of pluginDataList) { for (const item of pluginDataList) {
newList = newList.filter(x => x.documentPath != item.documentPath); newList = newList.filter(x => x.documentPath != item.documentPath);
@@ -343,9 +345,13 @@ export class ConfigSync extends LiveSyncCommands {
} }
this.pluginList = newList; this.pluginList = newList;
pluginList.set(newList); pluginList.set(newList);
if (pluginDataList.length != 10) {
// If the queue is going to be empty, await subsequent for a while.
await waitForSignal("plugin-next-load", 1000);
}
return; return;
} }
, { suspended: true, batchSize: 1000, concurrentLimit: 10, delay: 200, yieldThreshold: 25, totalRemainingReactiveSource: pluginScanningCount })).startPipeline().root.onIdle(() => { , { suspended: true, batchSize: 10, concurrentLimit: 2, delay: 250, yieldThreshold: 25, totalRemainingReactiveSource: pluginScanningCount })).startPipeline().root.onIdle(() => {
Logger(`All files enumerated`, LOG_LEVEL_INFO, "get-plugins"); Logger(`All files enumerated`, LOG_LEVEL_INFO, "get-plugins");
this.createMissingConfigurationEntry(); this.createMissingConfigurationEntry();
}); });

135
src/ObsidianLiveSyncSettingTab.ts
View File

@@ -8,6 +8,7 @@ import { checkSyncInfo, isCloudantURI } from "./lib/src/utils_couchdb";
import { testCrypt } from "./lib/src/e2ee_v2"; import { testCrypt } from "./lib/src/e2ee_v2";
import ObsidianLiveSyncPlugin from "./main"; import ObsidianLiveSyncPlugin from "./main";
import { askYesNo, performRebuildDB, requestToCouchDB, scheduleTask } from "./utils"; import { askYesNo, performRebuildDB, requestToCouchDB, scheduleTask } from "./utils";
import type { ButtonComponent } from "obsidian";
export class ObsidianLiveSyncSettingTab extends PluginSettingTab { export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
@@ -34,6 +35,10 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
containerEl.empty(); containerEl.empty();
// const preferred_setting = isCloudantURI(this.plugin.settings.couchDB_URI) ? PREFERRED_SETTING_CLOUDANT : PREFERRED_SETTING_SELF_HOSTED;
// const default_setting = { ...DEFAULT_SETTINGS };
containerEl.createEl("h2", { text: "Settings for Self-hosted LiveSync." }); containerEl.createEl("h2", { text: "Settings for Self-hosted LiveSync." });
containerEl.addClass("sls-setting"); containerEl.addClass("sls-setting");
containerEl.removeClass("isWizard"); containerEl.removeClass("isWizard");
@@ -819,6 +824,53 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
text.inputEl.setAttribute("type", "number"); text.inputEl.setAttribute("type", "number");
}); });
containerGeneralSettingsEl.createEl("h4", { text: "Share settings via markdown" });
let settingSyncFile = this.plugin.settings.settingSyncFile;
let buttonApplyFilename: ButtonComponent;
new Setting(containerGeneralSettingsEl)
.setName("Filename")
.setDesc("If you set this, all settings are saved in a markdown file. You will also be notified when new settings were arrived. You can set different files by the platform.")
.addText((text) => {
text.setPlaceholder("livesync/setting.md")
.setValue(settingSyncFile)
.onChange((value) => {
settingSyncFile = value;
if (settingSyncFile == this.plugin.settings.settingSyncFile) {
buttonApplyFilename.removeCta()
buttonApplyFilename.setDisabled(true);
} else {
buttonApplyFilename.setCta()
buttonApplyFilename.setDisabled(false);
}
})
}).addButton(button => {
button.setButtonText("Apply")
.onClick(async () => {
this.plugin.settings.settingSyncFile = settingSyncFile;
await this.plugin.saveSettings();
this.display();
})
buttonApplyFilename = button;
})
new Setting(containerGeneralSettingsEl)
.setName("Write credentials in the file")
.setDesc("(Not recommended) If set, credentials will be stored in the file.")
.addToggle(toggle => {
toggle.setValue(this.plugin.settings.writeCredentialsForSettingSync)
.onChange(async (value) => {
this.plugin.settings.writeCredentialsForSettingSync = value;
await this.plugin.saveSettings();
})
});
new Setting(containerGeneralSettingsEl)
.setName("Notify all setting files")
.addToggle(toggle => {
toggle.setValue(this.plugin.settings.notifyAllSettingSyncFile)
.onChange(async (value) => {
this.plugin.settings.notifyAllSettingSyncFile = value;
await this.plugin.saveSettings();
})
});
containerGeneralSettingsEl.createEl("h4", { text: "Advanced Confidentiality" }); containerGeneralSettingsEl.createEl("h4", { text: "Advanced Confidentiality" });
@@ -1238,7 +1290,7 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
}); });
let skipPatternTextArea: TextAreaComponent = null; let skipPatternTextArea: TextAreaComponent = null;
const defaultSkipPattern = "\\/node_modules\\/, \\/\\.git\\/, \\/obsidian-livesync\\/"; const defaultSkipPattern = "\\/node_modules\\/, \\/\\.git\\/, \\/obsidian-livesync\\/";
const defaultSkipPatternXPlat = defaultSkipPattern + ",\\/workspace$ ,\\/workspace.json$"; const defaultSkipPatternXPlat = defaultSkipPattern + ",\\/workspace$ ,\\/workspace.json$,\\/workspace-mobile.json$";
new Setting(containerSyncSettingEl) new Setting(containerSyncSettingEl)
.setName("Folders and files to ignore") .setName("Folders and files to ignore")
.setDesc( .setDesc(
@@ -1322,11 +1374,35 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
}); });
containerSyncSettingEl.createEl("h4", { containerSyncSettingEl.createEl("h4", {
text: sanitizeHTMLToDom(`Synchronization target filters`), text: sanitizeHTMLToDom(`Targets`),
}).addClass("wizardHidden"); }).addClass("wizardHidden");
new Setting(containerSyncSettingEl) new Setting(containerSyncSettingEl)
.setName("Regular expression to ignore files") .setName("Synchronising files")
.setDesc("If this is set, any changes to local and remote files that match this will be skipped.") .setDesc("(RegExp) Empty to sync all files. set filter as a regular expression to limit synchronising files.")
.setClass("wizardHidden")
.addTextArea((text) => {
text
.setValue(this.plugin.settings.syncOnlyRegEx)
.setPlaceholder("\\.md$|\\.txt")
.onChange(async (value) => {
let isValidRegExp = false;
try {
new RegExp(value);
isValidRegExp = true;
} catch (_) {
// NO OP.
}
if (isValidRegExp || value.trim() == "") {
this.plugin.settings.syncOnlyRegEx = value;
await this.plugin.saveSettings();
}
})
return text;
}
);
new Setting(containerSyncSettingEl)
.setName("Non-Synchronising files")
.setDesc("(RegExp) If this is set, any changes to local and remote files that match this will be skipped.")
.setClass("wizardHidden") .setClass("wizardHidden")
.addTextArea((text) => { .addTextArea((text) => {
text text
@@ -1349,29 +1425,22 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
} }
); );
new Setting(containerSyncSettingEl) new Setting(containerSyncSettingEl)
.setName("Regular expression for restricting synchronization targets") .setName("Maximum file size")
.setDesc("If this is set, changes to local and remote files that only match this will be processed.") .setDesc("(MB) If this is set, changes to local and remote files that are larger than this will be skipped. If the file becomes smaller again, a newer one will be used.")
.setClass("wizardHidden") .setClass("wizardHidden")
.addTextArea((text) => { .addText((text) => {
text text.setPlaceholder("")
.setValue(this.plugin.settings.syncOnlyRegEx) .setValue(this.plugin.settings.syncMaxSizeInMB + "")
.setPlaceholder("\\.md$|\\.txt")
.onChange(async (value) => { .onChange(async (value) => {
let isValidRegExp = false; let v = Number(value);
try { if (isNaN(v) || v < 1) {
new RegExp(value); v = 0;
isValidRegExp = true;
} catch (_) {
// NO OP.
} }
if (isValidRegExp || value.trim() == "") { this.plugin.settings.syncMaxSizeInMB = v;
this.plugin.settings.syncOnlyRegEx = value; await this.plugin.saveSettings();
await this.plugin.saveSettings(); });
} text.inputEl.setAttribute("type", "number");
}) });
return text;
}
);
new Setting(containerSyncSettingEl) new Setting(containerSyncSettingEl)
.setName("(Beta) Use ignore files") .setName("(Beta) Use ignore files")
.setDesc("If this is set, changes to local files which are matched by the ignore files will be skipped. Remote changes are determined using local ignore files.") .setDesc("If this is set, changes to local files which are matched by the ignore files will be skipped. Remote changes are determined using local ignore files.")
@@ -1412,15 +1481,15 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
}).addClass("wizardHidden"); }).addClass("wizardHidden");
new Setting(containerSyncSettingEl) new Setting(containerSyncSettingEl)
.setName("Batch size") .setName("Batch size")
.setDesc("Number of change feed items to process at a time. Defaults to 50.") .setDesc("Number of change feed items to process at a time. Defaults to 50. Minimum is 2.")
.setClass("wizardHidden") .setClass("wizardHidden")
.addText((text) => { .addText((text) => {
text.setPlaceholder("") text.setPlaceholder("")
.setValue(this.plugin.settings.batch_size + "") .setValue(this.plugin.settings.batch_size + "")
.onChange(async (value) => { .onChange(async (value) => {
let v = Number(value); let v = Number(value);
if (isNaN(v) || v < 10) { if (isNaN(v) || v < 2) {
v = 10; v = 2;
} }
this.plugin.settings.batch_size = v; this.plugin.settings.batch_size = v;
await this.plugin.saveSettings(); await this.plugin.saveSettings();
@@ -1430,15 +1499,15 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
new Setting(containerSyncSettingEl) new Setting(containerSyncSettingEl)
.setName("Batch limit") .setName("Batch limit")
.setDesc("Number of batches to process at a time. Defaults to 40. This along with batch size controls how many docs are kept in memory at a time.") .setDesc("Number of batches to process at a time. Defaults to 40. Minimum is 2. This along with batch size controls how many docs are kept in memory at a time.")
.setClass("wizardHidden") .setClass("wizardHidden")
.addText((text) => { .addText((text) => {
text.setPlaceholder("") text.setPlaceholder("")
.setValue(this.plugin.settings.batches_limit + "") .setValue(this.plugin.settings.batches_limit + "")
.onChange(async (value) => { .onChange(async (value) => {
let v = Number(value); let v = Number(value);
if (isNaN(v) || v < 10) { if (isNaN(v) || v < 2) {
v = 10; v = 2;
} }
this.plugin.settings.batches_limit = v; this.plugin.settings.batches_limit = v;
await this.plugin.saveSettings(); await this.plugin.saveSettings();
@@ -1535,8 +1604,10 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
pluginConfig.encryptedPassphrase = REDACTED; pluginConfig.encryptedPassphrase = REDACTED;
pluginConfig.encryptedCouchDBConnection = REDACTED; pluginConfig.encryptedCouchDBConnection = REDACTED;
pluginConfig.pluginSyncExtendedSetting = {}; pluginConfig.pluginSyncExtendedSetting = {};
const obsidianInfo = navigator.userAgent;
const msgConfig = `----remote config---- const msgConfig = `---- Obsidian info ----
${obsidianInfo}
---- remote config ----
${stringifyYaml(responseConfig)} ${stringifyYaml(responseConfig)}
---- Plug-in config --- ---- Plug-in config ---
version:${manifestVersion} version:${manifestVersion}

46
src/PluginPane.svelte
View File

@@ -27,8 +27,10 @@
async function requestReload() { async function requestReload() {
await addOn.reloadPluginList(true); await addOn.reloadPluginList(true);
} }
let allTerms = [] as string[];
pluginList.subscribe((e) => { pluginList.subscribe((e) => {
list = e; list = e;
allTerms = unique(list.map((e) => e.term));
}); });
pluginIsEnumerating.subscribe((e) => { pluginIsEnumerating.subscribe((e) => {
loading = e; loading = e;
@@ -172,7 +174,7 @@
.filter((e) => `${e.category}/${e.name}` == key) .filter((e) => `${e.category}/${e.name}` == key)
.map((e) => e.files) .map((e) => e.files)
.flat() .flat()
.map((e) => e.filename) .map((e) => e.filename),
); );
automaticList.set(key, mode); automaticList.set(key, mode);
automaticListDisp = automaticList; automaticListDisp = automaticList;
@@ -218,6 +220,16 @@
.sort((a, b) => (a.displayName ?? a.name).localeCompare(b.displayName ?? b.name)) .sort((a, b) => (a.displayName ?? a.name).localeCompare(b.displayName ?? b.name))
.reduce((p, c) => ({ ...p, [c.category]: unique(c.category in p ? [...p[c.category], c.displayName ?? c.name] : [c.displayName ?? c.name]) }), {} as Record<string, string[]>); .reduce((p, c) => ({ ...p, [c.category]: unique(c.category in p ? [...p[c.category], c.displayName ?? c.name] : [c.displayName ?? c.name]) }), {} as Record<string, string[]>);
} }
let deleteTerm = "";
async function deleteAllItems(term: string) {
const deleteItems = list.filter((e) => e.term == term);
for (const item of deleteItems) {
await deleteData(item);
}
addOn.reloadPluginList(true);
}
</script> </script>
<div> <div>
@@ -322,6 +334,29 @@
</div> </div>
{/if} {/if}
</div> </div>
{#if isMaintenanceMode}
<div class="list">
<div>
<h3>Maintenance Commands</h3>
<div class="maintenancerow">
<label for="">Delete All of </label>
<select bind:value={deleteTerm}>
{#each allTerms as term}
<option value={term}>{term}</option>
{/each}
</select>
<button
class="status"
on:click={(evt) => {
deleteAllItems(deleteTerm);
}}
>
🗑️
</button>
</div>
</div>
</div>
{/if}
<div class="buttons"> <div class="buttons">
<label><span>Hide not applicable items</span><input type="checkbox" bind:checked={hideEven} /></label> <label><span>Hide not applicable items</span><input type="checkbox" bind:checked={hideEven} /></label>
</div> </div>
@@ -423,4 +458,13 @@
align-items: center; align-items: center;
min-height: 3em; min-height: 3em;
} }
.maintenancerow {
display: flex;
justify-content: flex-end;
align-items: center;
}
.maintenancerow label {
margin-right: 0.5em;
margin-left: 0.5em;
}
</style> </style>

54
src/SerializedFileAccess.ts
View File

@@ -2,6 +2,9 @@ import { type App, TFile, type DataWriteOptions, TFolder, TAbstractFile } from "
import { serialized } from "./lib/src/lock"; import { serialized } from "./lib/src/lock";
import type { FilePath } from "./lib/src/types"; import type { FilePath } from "./lib/src/types";
import { createBinaryBlob, isDocContentSame } from "./lib/src/utils"; import { createBinaryBlob, isDocContentSame } from "./lib/src/utils";
import type { InternalFileInfo } from "./types";
import { markChangesAreSame } from "./utils";
function getFileLockKey(file: TFile | TFolder | string) { function getFileLockKey(file: TFile | TFolder | string) {
return `fl:${typeof (file) == "string" ? file : file.path}`; return `fl:${typeof (file) == "string" ? file : file.path}`;
} }
@@ -15,6 +18,15 @@ function toArrayBuffer(arr: Uint8Array | ArrayBuffer | DataView): ArrayBufferLik
return arr; return arr;
} }
async function processReadFile<T>(file: TFile | TFolder | string, proc: () => Promise<T>) {
const ret = await serialized(getFileLockKey(file), () => proc());
return ret;
}
async function processWriteFile<T>(file: TFile | TFolder | string, proc: () => Promise<T>) {
const ret = await serialized(getFileLockKey(file), () => proc());
return ret;
}
export class SerializedFileAccess { export class SerializedFileAccess {
app: App app: App
constructor(app: App) { constructor(app: App) {
@@ -23,60 +35,64 @@ export class SerializedFileAccess {
async adapterStat(file: TFile | string) { async adapterStat(file: TFile | string) {
const path = file instanceof TFile ? file.path : file; const path = file instanceof TFile ? file.path : file;
return await serialized(getFileLockKey(path), () => this.app.vault.adapter.stat(path)); return await processReadFile(file, () => this.app.vault.adapter.stat(path));
} }
async adapterExists(file: TFile | string) { async adapterExists(file: TFile | string) {
const path = file instanceof TFile ? file.path : file; const path = file instanceof TFile ? file.path : file;
return await serialized(getFileLockKey(path), () => this.app.vault.adapter.exists(path)); return await processReadFile(file, () => this.app.vault.adapter.exists(path));
} }
async adapterRemove(file: TFile | string) { async adapterRemove(file: TFile | string) {
const path = file instanceof TFile ? file.path : file; const path = file instanceof TFile ? file.path : file;
return await serialized(getFileLockKey(path), () => this.app.vault.adapter.remove(path)); return await processReadFile(file, () => this.app.vault.adapter.remove(path));
} }
async adapterRead(file: TFile | string) { async adapterRead(file: TFile | string) {
const path = file instanceof TFile ? file.path : file; const path = file instanceof TFile ? file.path : file;
return await serialized(getFileLockKey(path), () => this.app.vault.adapter.read(path)); return await processReadFile(file, () => this.app.vault.adapter.read(path));
} }
async adapterReadBinary(file: TFile | string) { async adapterReadBinary(file: TFile | string) {
const path = file instanceof TFile ? file.path : file; const path = file instanceof TFile ? file.path : file;
return await serialized(getFileLockKey(path), () => this.app.vault.adapter.readBinary(path)); return await processReadFile(file, () => this.app.vault.adapter.readBinary(path));
} }
async adapterWrite(file: TFile | string, data: string | ArrayBuffer | Uint8Array, options?: DataWriteOptions) { async adapterWrite(file: TFile | string, data: string | ArrayBuffer | Uint8Array, options?: DataWriteOptions) {
const path = file instanceof TFile ? file.path : file; const path = file instanceof TFile ? file.path : file;
if (typeof (data) === "string") { if (typeof (data) === "string") {
return await serialized(getFileLockKey(path), () => this.app.vault.adapter.write(path, data, options)); return await processWriteFile(file, () => this.app.vault.adapter.write(path, data, options));
} else { } else {
return await serialized(getFileLockKey(path), () => this.app.vault.adapter.writeBinary(path, toArrayBuffer(data), options)); return await processWriteFile(file, () => this.app.vault.adapter.writeBinary(path, toArrayBuffer(data), options));
} }
} }
async vaultCacheRead(file: TFile) { async vaultCacheRead(file: TFile) {
return await serialized(getFileLockKey(file), () => this.app.vault.cachedRead(file)); return await processReadFile(file, () => this.app.vault.cachedRead(file));
} }
async vaultRead(file: TFile) { async vaultRead(file: TFile) {
return await serialized(getFileLockKey(file), () => this.app.vault.read(file)); return await processReadFile(file, () => this.app.vault.read(file));
} }
async vaultReadBinary(file: TFile) { async vaultReadBinary(file: TFile) {
return await serialized(getFileLockKey(file), () => this.app.vault.readBinary(file)); return await processReadFile(file, () => this.app.vault.readBinary(file));
} }
async vaultModify(file: TFile, data: string | ArrayBuffer | Uint8Array, options?: DataWriteOptions) { async vaultModify(file: TFile, data: string | ArrayBuffer | Uint8Array, options?: DataWriteOptions) {
if (typeof (data) === "string") { if (typeof (data) === "string") {
return await serialized(getFileLockKey(file), async () => { return await processWriteFile(file, async () => {
const oldData = await this.app.vault.read(file); const oldData = await this.app.vault.read(file);
if (data === oldData) return false if (data === oldData) {
markChangesAreSame(file, file.stat.mtime, options.mtime);
return false
}
await this.app.vault.modify(file, data, options) await this.app.vault.modify(file, data, options)
return true; return true;
} }
); );
} else { } else {
return await serialized(getFileLockKey(file), async () => { return await processWriteFile(file, async () => {
const oldData = await this.app.vault.readBinary(file); const oldData = await this.app.vault.readBinary(file);
if (await isDocContentSame(createBinaryBlob(oldData), createBinaryBlob(data))) { if (await isDocContentSame(createBinaryBlob(oldData), createBinaryBlob(data))) {
markChangesAreSame(file, file.stat.mtime, options.mtime);
return false; return false;
} }
await this.app.vault.modifyBinary(file, toArrayBuffer(data), options) await this.app.vault.modifyBinary(file, toArrayBuffer(data), options)
@@ -86,16 +102,16 @@ export class SerializedFileAccess {
} }
async vaultCreate(path: string, data: string | ArrayBuffer | Uint8Array, options?: DataWriteOptions): Promise<TFile> { async vaultCreate(path: string, data: string | ArrayBuffer | Uint8Array, options?: DataWriteOptions): Promise<TFile> {
if (typeof (data) === "string") { if (typeof (data) === "string") {
return await serialized(getFileLockKey(path), () => this.app.vault.create(path, data, options)); return await processWriteFile(path, () => this.app.vault.create(path, data, options));
} else { } else {
return await serialized(getFileLockKey(path), () => this.app.vault.createBinary(path, toArrayBuffer(data), options)); return await processWriteFile(path, () => this.app.vault.createBinary(path, toArrayBuffer(data), options));
} }
} }
async delete(file: TFile | TFolder, force = false) { async delete(file: TFile | TFolder, force = false) {
return await serialized(getFileLockKey(file), () => this.app.vault.delete(file, force)); return await processWriteFile(file, () => this.app.vault.delete(file, force));
} }
async trash(file: TFile | TFolder, force = false) { async trash(file: TFile | TFolder, force = false) {
return await serialized(getFileLockKey(file), () => this.app.vault.trash(file, force)); return await processWriteFile(file, () => this.app.vault.trash(file, force));
} }
getAbstractFileByPath(path: FilePath | string): TAbstractFile | null { getAbstractFileByPath(path: FilePath | string): TAbstractFile | null {
@@ -121,8 +137,8 @@ export class SerializedFileAccess {
this.touchedFiles.unshift(key); this.touchedFiles.unshift(key);
this.touchedFiles = this.touchedFiles.slice(0, 100); this.touchedFiles = this.touchedFiles.slice(0, 100);
} }
recentlyTouched(file: TFile) { recentlyTouched(file: TFile | InternalFileInfo) {
const key = `${file.path}-${file.stat.mtime}-${file.stat.size}`; const key = file instanceof TFile ? `${file.path}-${file.stat.mtime}-${file.stat.size}` : `${file.path}-${file.mtime}-${file.size}`;
if (this.touchedFiles.indexOf(key) == -1) return false; if (this.touchedFiles.indexOf(key) == -1) return false;
return true; return true;
} }

14
src/StorageEventManager.ts
View File

@@ -1,8 +1,10 @@
import type { SerializedFileAccess } from "./SerializedFileAccess"; import type { SerializedFileAccess } from "./SerializedFileAccess";
import { Plugin, TAbstractFile, TFile, TFolder } from "./deps"; import { Plugin, TAbstractFile, TFile, TFolder } from "./deps";
import { Logger } from "./lib/src/logger";
import { isPlainText, shouldBeIgnored } from "./lib/src/path"; import { isPlainText, shouldBeIgnored } from "./lib/src/path";
import type { KeyedQueueProcessor } from "./lib/src/processor"; import type { KeyedQueueProcessor } from "./lib/src/processor";
import { type FilePath, type ObsidianLiveSyncSettings } from "./lib/src/types"; import { LOG_LEVEL_NOTICE, type FilePath, type ObsidianLiveSyncSettings } from "./lib/src/types";
import { delay } from "./lib/src/utils";
import { type FileEventItem, type FileEventType, type FileInfo, type InternalFileInfo } from "./types"; import { type FileEventItem, type FileEventType, type FileInfo, type InternalFileInfo } from "./types";
@@ -17,7 +19,8 @@ type LiveSyncForStorageEventManager = Plugin &
vaultAccess: SerializedFileAccess vaultAccess: SerializedFileAccess
} & { } & {
isTargetFile: (file: string | TAbstractFile) => Promise<boolean>, isTargetFile: (file: string | TAbstractFile) => Promise<boolean>,
fileEventQueue: KeyedQueueProcessor<FileEventItem, any> fileEventQueue: KeyedQueueProcessor<FileEventItem, any>,
isFileSizeExceeded: (size: number) => boolean;
}; };
@@ -96,6 +99,11 @@ export class StorageEventManagerObsidian extends StorageEventManager {
const type = param.type; const type = param.type;
const file = param.file; const file = param.file;
const oldPath = param.oldPath; const oldPath = param.oldPath;
const size = file instanceof TFile ? file.stat.size : (file as InternalFileInfo)?.size ?? 0;
if (this.plugin.isFileSizeExceeded(size) && (type == "CREATE" || type == "CHANGED")) {
Logger(`The storage file has been changed but exceeds the maximum size. Skipping: ${param.file.path}`, LOG_LEVEL_NOTICE);
continue;
}
if (file instanceof TFolder) continue; if (file instanceof TFolder) continue;
if (!await this.plugin.isTargetFile(file.path)) continue; if (!await this.plugin.isTargetFile(file.path)) continue;
if (this.plugin.settings.suspendFileWatching) continue; if (this.plugin.settings.suspendFileWatching) continue;
@@ -103,6 +111,8 @@ export class StorageEventManagerObsidian extends StorageEventManager {
let cache: null | string | ArrayBuffer; let cache: null | string | ArrayBuffer;
// new file or something changed, cache the changes. // new file or something changed, cache the changes.
if (file instanceof TFile && (type == "CREATE" || type == "CHANGED")) { if (file instanceof TFile && (type == "CREATE" || type == "CHANGED")) {
// Wait for a bit while to let the writer has marked `touched` at the file.
await delay(10);
if (this.plugin.vaultAccess.recentlyTouched(file)) { if (this.plugin.vaultAccess.recentlyTouched(file)) {
continue; continue;
} }

2
src/lib

Submodule src/lib updated: 57e663b6dc...1d6c0cc6aa

521
src/main.ts
View File

@@ -1,16 +1,16 @@
const isDebug = false; const isDebug = false;
import { type Diff, DIFF_DELETE, DIFF_EQUAL, DIFF_INSERT, diff_match_patch } from "./deps"; import { type Diff, DIFF_DELETE, DIFF_EQUAL, DIFF_INSERT, diff_match_patch, stringifyYaml, parseYaml } from "./deps";
import { debounce, Notice, Plugin, TFile, addIcon, TFolder, normalizePath, TAbstractFile, Editor, MarkdownView, type RequestUrlParam, type RequestUrlResponse, requestUrl, type MarkdownFileInfo } from "./deps"; import { Notice, Plugin, TFile, addIcon, TFolder, normalizePath, TAbstractFile, Editor, MarkdownView, type RequestUrlParam, type RequestUrlResponse, requestUrl, type MarkdownFileInfo } from "./deps";
import { type EntryDoc, type LoadedEntry, type ObsidianLiveSyncSettings, type diff_check_result, type diff_result_leaf, type EntryBody, LOG_LEVEL, VER, DEFAULT_SETTINGS, type diff_result, FLAGMD_REDFLAG, SYNCINFO_ID, SALT_OF_PASSPHRASE, type ConfigPassphraseStore, type CouchDBConnection, FLAGMD_REDFLAG2, FLAGMD_REDFLAG3, PREFIXMD_LOGFILE, type DatabaseConnectingStatus, type EntryHasPath, type DocumentID, type FilePathWithPrefix, type FilePath, type AnyEntry, LOG_LEVEL_DEBUG, LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_URGENT, LOG_LEVEL_VERBOSE, type SavingEntry, MISSING_OR_ERROR, NOT_CONFLICTED, AUTO_MERGED, CANCELLED, LEAVE_TO_SUBSEQUENT, } from "./lib/src/types"; import { type EntryDoc, type LoadedEntry, type ObsidianLiveSyncSettings, type diff_check_result, type diff_result_leaf, type EntryBody, LOG_LEVEL, VER, DEFAULT_SETTINGS, type diff_result, FLAGMD_REDFLAG, SYNCINFO_ID, SALT_OF_PASSPHRASE, type ConfigPassphraseStore, type CouchDBConnection, FLAGMD_REDFLAG2, FLAGMD_REDFLAG3, PREFIXMD_LOGFILE, type DatabaseConnectingStatus, type EntryHasPath, type DocumentID, type FilePathWithPrefix, type FilePath, type AnyEntry, LOG_LEVEL_DEBUG, LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_URGENT, LOG_LEVEL_VERBOSE, type SavingEntry, MISSING_OR_ERROR, NOT_CONFLICTED, AUTO_MERGED, CANCELLED, LEAVE_TO_SUBSEQUENT, FLAGMD_REDFLAG2_HR, FLAGMD_REDFLAG3_HR, } from "./lib/src/types";
import { type InternalFileInfo, type CacheData, type FileEventItem, FileWatchEventQueueMax } from "./types"; import { type InternalFileInfo, type CacheData, type FileEventItem, FileWatchEventQueueMax } from "./types";
import { createBinaryBlob, createTextBlob, fireAndForget, getDocData, isDocContentSame, sendValue } from "./lib/src/utils"; import { arrayToChunkedArray, createBinaryBlob, createTextBlob, fireAndForget, getDocData, isDocContentSame, isObjectDifferent, sendValue } from "./lib/src/utils";
import { Logger, setGlobalLogFunction } from "./lib/src/logger"; import { Logger, setGlobalLogFunction } from "./lib/src/logger";
import { PouchDB } from "./lib/src/pouchdb-browser.js"; import { PouchDB } from "./lib/src/pouchdb-browser.js";
import { ConflictResolveModal } from "./ConflictResolveModal"; import { ConflictResolveModal } from "./ConflictResolveModal";
import { ObsidianLiveSyncSettingTab } from "./ObsidianLiveSyncSettingTab"; import { ObsidianLiveSyncSettingTab } from "./ObsidianLiveSyncSettingTab";
import { DocumentHistoryModal } from "./DocumentHistoryModal"; import { DocumentHistoryModal } from "./DocumentHistoryModal";
import { applyPatch, cancelAllPeriodicTask, cancelAllTasks, cancelTask, generatePatchObj, id2path, isObjectMargeApplicable, isSensibleMargeApplicable, flattenObject, path2id, scheduleTask, tryParseJSON, isValidPath, isInternalMetadata, isPluginMetadata, stripInternalMetadataPrefix, isChunk, askSelectString, askYesNo, askString, PeriodicProcessor, getPath, getPathWithoutPrefix, getPathFromTFile, performRebuildDB, memoIfNotExist, memoObject, retrieveMemoObject, disposeMemoObject, isCustomisationSyncMetadata } from "./utils"; import { applyPatch, cancelAllPeriodicTask, cancelAllTasks, cancelTask, generatePatchObj, id2path, isObjectMargeApplicable, isSensibleMargeApplicable, flattenObject, path2id, scheduleTask, tryParseJSON, isValidPath, isInternalMetadata, isPluginMetadata, stripInternalMetadataPrefix, isChunk, askSelectString, askYesNo, askString, PeriodicProcessor, getPath, getPathWithoutPrefix, getPathFromTFile, performRebuildDB, memoIfNotExist, memoObject, retrieveMemoObject, disposeMemoObject, isCustomisationSyncMetadata, compareFileFreshness, BASE_IS_NEW, TARGET_IS_NEW, EVEN, compareMTime, markChangesAreSame } from "./utils";
import { encrypt, tryDecrypt } from "./lib/src/e2ee_v2"; import { encrypt, tryDecrypt } from "./lib/src/e2ee_v2";
import { balanceChunkPurgedDBs, enableEncryption, isCloudantURI, isErrorOfMissingDoc, isValidRemoteCouchDBURI, purgeUnreferencedChunks } from "./lib/src/utils_couchdb"; import { balanceChunkPurgedDBs, enableEncryption, isCloudantURI, isErrorOfMissingDoc, isValidRemoteCouchDBURI, purgeUnreferencedChunks } from "./lib/src/utils_couchdb";
import { logStore, type LogEntry, collectingChunks, pluginScanningCount, hiddenFilesProcessingCount, hiddenFilesEventCount, logMessages } from "./lib/src/stores"; import { logStore, type LogEntry, collectingChunks, pluginScanningCount, hiddenFilesProcessingCount, hiddenFilesEventCount, logMessages } from "./lib/src/stores";
@@ -33,6 +33,7 @@ import { LRUCache } from "./lib/src/LRUCache";
import { SerializedFileAccess } from "./SerializedFileAccess.js"; import { SerializedFileAccess } from "./SerializedFileAccess.js";
import { KeyedQueueProcessor, QueueProcessor, type QueueItemWithKey } from "./lib/src/processor.js"; import { KeyedQueueProcessor, QueueProcessor, type QueueItemWithKey } from "./lib/src/processor.js";
import { reactive, reactiveSource } from "./lib/src/reactive.js"; import { reactive, reactiveSource } from "./lib/src/reactive.js";
import { initializeStores } from "./stores.js";
setNoticeClass(Notice); setNoticeClass(Notice);
@@ -63,6 +64,10 @@ async function fetchByAPI(request: RequestUrlParam): Promise<RequestUrlResponse>
} }
return ret; return ret;
} }
const SETTING_HEADER = "````yaml:livesync-setting\n";
const SETTING_FOOTER = "\n````";
export default class ObsidianLiveSyncPlugin extends Plugin export default class ObsidianLiveSyncPlugin extends Plugin
implements LiveSyncLocalDBEnv, LiveSyncReplicatorEnv { implements LiveSyncLocalDBEnv, LiveSyncReplicatorEnv {
@@ -299,38 +304,36 @@ export default class ObsidianLiveSyncPlugin extends Plugin
return timer; return timer;
} }
isRedFlagRaised(): boolean { isFlagFileExist(path: string) {
const redflag = this.vaultAccess.getAbstractFileByPath(normalizePath(FLAGMD_REDFLAG)); const redflag = this.vaultAccess.getAbstractFileByPath(normalizePath(path));
if (redflag != null) { if (redflag != null && redflag instanceof TFile) {
return true; return true;
} }
return false; return false;
} }
isRedFlag2Raised(): boolean { async deleteFlagFile(path: string) {
const redflag = this.vaultAccess.getAbstractFileByPath(normalizePath(FLAGMD_REDFLAG2)); try {
if (redflag != null) { const redflag = this.vaultAccess.getAbstractFileByPath(normalizePath(path));
return true; if (redflag != null && redflag instanceof TFile) {
await this.vaultAccess.delete(redflag, true);
}
} catch (ex) {
Logger(`Could not delete ${path}`);
Logger(ex, LOG_LEVEL_VERBOSE);
} }
return false;
} }
isRedFlagRaised = () => this.isFlagFileExist(FLAGMD_REDFLAG)
isRedFlag2Raised = () => this.isFlagFileExist(FLAGMD_REDFLAG2) || this.isFlagFileExist(FLAGMD_REDFLAG2_HR)
isRedFlag3Raised = () => this.isFlagFileExist(FLAGMD_REDFLAG3) || this.isFlagFileExist(FLAGMD_REDFLAG3_HR)
async deleteRedFlag2() { async deleteRedFlag2() {
const redflag = this.vaultAccess.getAbstractFileByPath(normalizePath(FLAGMD_REDFLAG2)); await this.deleteFlagFile(FLAGMD_REDFLAG2);
if (redflag != null && redflag instanceof TFile) { await this.deleteFlagFile(FLAGMD_REDFLAG2_HR);
await this.vaultAccess.delete(redflag, true);
}
}
isRedFlag3Raised(): boolean {
const redflag = this.vaultAccess.getAbstractFileByPath(normalizePath(FLAGMD_REDFLAG3));
if (redflag != null) {
return true;
}
return false;
} }
async deleteRedFlag3() { async deleteRedFlag3() {
const redflag = this.vaultAccess.getAbstractFileByPath(normalizePath(FLAGMD_REDFLAG3)); await this.deleteFlagFile(FLAGMD_REDFLAG3);
if (redflag != null && redflag instanceof TFile) { await this.deleteFlagFile(FLAGMD_REDFLAG3_HR);
await this.vaultAccess.delete(redflag, true);
}
} }
showHistory(file: TFile | FilePathWithPrefix, id?: DocumentID) { showHistory(file: TFile | FilePathWithPrefix, id?: DocumentID) {
@@ -426,7 +429,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin
this.settings.suspendFileWatching = true; this.settings.suspendFileWatching = true;
await this.saveSettings(); await this.saveSettings();
if (this.isRedFlag2Raised()) { if (this.isRedFlag2Raised()) {
Logger(`${FLAGMD_REDFLAG2} has been detected! Self-hosted LiveSync suspends all sync and rebuild everything.`, LOG_LEVEL_NOTICE); Logger(`${FLAGMD_REDFLAG2} or ${FLAGMD_REDFLAG2_HR} has been detected! Self-hosted LiveSync suspends all sync and rebuild everything.`, LOG_LEVEL_NOTICE);
await this.addOnSetup.rebuildEverything(); await this.addOnSetup.rebuildEverything();
await this.deleteRedFlag2(); await this.deleteRedFlag2();
if (await askYesNo(this.app, "Do you want to disable Suspend file watching and restart obsidian now?") == "yes") { if (await askYesNo(this.app, "Do you want to disable Suspend file watching and restart obsidian now?") == "yes") {
@@ -436,7 +439,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin
this.app.commands.executeCommandById("app:reload") this.app.commands.executeCommandById("app:reload")
} }
} else if (this.isRedFlag3Raised()) { } else if (this.isRedFlag3Raised()) {
Logger(`${FLAGMD_REDFLAG3} has been detected! Self-hosted LiveSync will discard the local database and fetch everything from the remote once again.`, LOG_LEVEL_NOTICE); Logger(`${FLAGMD_REDFLAG3} or ${FLAGMD_REDFLAG3_HR} has been detected! Self-hosted LiveSync will discard the local database and fetch everything from the remote once again.`, LOG_LEVEL_NOTICE);
await this.addOnSetup.fetchLocal(); await this.addOnSetup.fetchLocal();
await this.deleteRedFlag3(); await this.deleteRedFlag3();
if (this.settings.suspendFileWatching) { if (this.settings.suspendFileWatching) {
@@ -679,6 +682,28 @@ Note: We can always able to read V1 format. It will be progressively converted.
this.showGlobalHistory() this.showGlobalHistory()
} }
}) })
this.addCommand({
id: "livesync-export-config",
name: "Write setting markdown manually",
checkCallback: (checking) => {
if (checking) {
return this.settings.settingSyncFile != "";
}
this.saveSettingData();
}
})
this.addCommand({
id: "livesync-import-config",
name: "Parse setting file",
editorCheckCallback: (checking, editor, ctx) => {
if (checking) {
const doc = editor.getValue();
const ret = this.extractSettingFromWholeText(doc);
return ret.body != "";
}
this.checkAndApplySettingFromMarkdown(ctx.file.path, false);
},
})
this.registerView( this.registerView(
VIEW_TYPE_GLOBAL_HISTORY, VIEW_TYPE_GLOBAL_HISTORY,
@@ -704,9 +729,9 @@ Note: We can always able to read V1 format. It will be progressively converted.
this.packageVersion = packageVersion; this.packageVersion = packageVersion;
Logger(`Self-hosted LiveSync v${manifestVersion} ${packageVersion} `); Logger(`Self-hosted LiveSync v${manifestVersion} ${packageVersion} `);
await this.loadSettings();
const lsKey = "obsidian-live-sync-ver" + this.getVaultName(); const lsKey = "obsidian-live-sync-ver" + this.getVaultName();
const last_version = localStorage.getItem(lsKey); const last_version = localStorage.getItem(lsKey);
await this.loadSettings();
this.observeForLogs(); this.observeForLogs();
this.statusBar = this.addStatusBarItem(); this.statusBar = this.addStatusBarItem();
this.statusBar.addClass("syncstatusbar"); this.statusBar.addClass("syncstatusbar");
@@ -733,9 +758,9 @@ Note: We can always able to read V1 format. It will be progressively converted.
} }
localStorage.setItem(lsKey, `${VER}`); localStorage.setItem(lsKey, `${VER}`);
await this.openDatabase(); await this.openDatabase();
this.watchWorkspaceOpen = debounce(this.watchWorkspaceOpen.bind(this), 1000, false); this.watchWorkspaceOpen = this.watchWorkspaceOpen.bind(this);
this.watchWindowVisibility = debounce(this.watchWindowVisibility.bind(this), 1000, false); this.watchWindowVisibility = this.watchWindowVisibility.bind(this)
this.watchOnline = debounce(this.watchOnline.bind(this), 500, false); this.watchOnline = this.watchOnline.bind(this);
this.realizeSettingSyncMode = this.realizeSettingSyncMode.bind(this); this.realizeSettingSyncMode = this.realizeSettingSyncMode.bind(this);
this.parseReplicationResult = this.parseReplicationResult.bind(this); this.parseReplicationResult = this.parseReplicationResult.bind(this);
@@ -797,6 +822,7 @@ Note: We can always able to read V1 format. It will be progressively converted.
//@ts-ignore //@ts-ignore
this.isMobile = this.app.isMobile; this.isMobile = this.app.isMobile;
this.localDatabase = new LiveSyncLocalDB(vaultName, this); this.localDatabase = new LiveSyncLocalDB(vaultName, this);
initializeStores(vaultName);
return await this.localDatabase.initializeDatabase(); return await this.localDatabase.initializeDatabase();
} }
@@ -944,9 +970,163 @@ Note: We can always able to read V1 format. It will be progressively converted.
this.localDatabase.settings = this.settings; this.localDatabase.settings = this.settings;
this.fileEventQueue.delay = this.settings.batchSave ? 5000 : 100; this.fileEventQueue.delay = this.settings.batchSave ? 5000 : 100;
this.ignoreFiles = this.settings.ignoreFiles.split(",").map(e => e.trim()); this.ignoreFiles = this.settings.ignoreFiles.split(",").map(e => e.trim());
if (this.settings.settingSyncFile != "") {
fireAndForget(() => this.saveSettingToMarkdown(this.settings.settingSyncFile));
}
} }
extractSettingFromWholeText(data: string): { preamble: string, body: string, postscript: string } {
if (data.indexOf(SETTING_HEADER) === -1) {
return {
preamble: data,
body: "",
postscript: ""
}
}
const startMarkerPos = data.indexOf(SETTING_HEADER);
const dataStartPos = startMarkerPos == -1 ? data.length : startMarkerPos;
const endMarkerPos = startMarkerPos == -1 ? data.length : data.indexOf(SETTING_FOOTER, dataStartPos);
const dataEndPos = endMarkerPos == -1 ? data.length : endMarkerPos;
const body = data.substring(dataStartPos + SETTING_HEADER.length, dataEndPos);
const ret = {
preamble: data.substring(0, dataStartPos),
body,
postscript: data.substring(dataEndPos + SETTING_FOOTER.length + 1)
}
return ret;
}
async parseSettingFromMarkdown(filename: string, data?: string) {
const file = this.app.vault.getAbstractFileByPath(filename);
if (!(file instanceof TFile)) return {
preamble: "",
body: "",
postscript: "",
};
if (data) {
return this.extractSettingFromWholeText(data);
}
const parseData = data ?? await this.app.vault.read(file);
return this.extractSettingFromWholeText(parseData);
}
async checkAndApplySettingFromMarkdown(filename: string, automated?: boolean) {
if (automated && !this.settings.notifyAllSettingSyncFile) {
if (!this.settings.settingSyncFile || this.settings.settingSyncFile != filename) {
Logger(`Setting file (${filename}) is not matched to the current configuration. skipped.`, LOG_LEVEL_VERBOSE);
return;
}
}
const { body } = await this.parseSettingFromMarkdown(filename);
let newSetting = {} as Partial<ObsidianLiveSyncSettings>;
try {
newSetting = parseYaml(body);
} catch (ex) {
Logger("Could not parse YAML", LOG_LEVEL_NOTICE);
Logger(ex, LOG_LEVEL_VERBOSE);
return;
}
if ("settingSyncFile" in newSetting && newSetting.settingSyncFile != filename) {
Logger("This setting file seems to backed up one. Please fix the filename or settingSyncFile value.", automated ? LOG_LEVEL_INFO : LOG_LEVEL_NOTICE);
return;
}
let settingToApply = { ...DEFAULT_SETTINGS } as ObsidianLiveSyncSettings;
settingToApply = { ...settingToApply, ...newSetting }
if (!(settingToApply?.writeCredentialsForSettingSync)) {
//New setting does not contains credentials.
settingToApply.couchDB_USER = this.settings.couchDB_USER;
settingToApply.couchDB_PASSWORD = this.settings.couchDB_PASSWORD;
settingToApply.passphrase = this.settings.passphrase;
}
const oldSetting = this.generateSettingForMarkdown(this.settings, settingToApply.writeCredentialsForSettingSync);
if (!isObjectDifferent(oldSetting, this.generateSettingForMarkdown(settingToApply))) {
Logger("Setting markdown has been detected, but not changed.", automated ? LOG_LEVEL_INFO : LOG_LEVEL_NOTICE);
return
}
const addMsg = this.settings.settingSyncFile != filename ? " (This is not-active file)" : "";
this.askInPopup("apply-setting-from-md", `Setting markdown ${filename}${addMsg} has been detected. Apply this from {HERE}.`, (anchor) => {
anchor.text = "HERE";
anchor.addEventListener("click", async () => {
const APPLY_ONLY = "Apply settings";
const APPLY_AND_RESTART = "Apply settings and restart obsidian";
const APPLY_AND_REBUILD = "Apply settings and restart obsidian with red_flag_rebuild.md";
const APPLY_AND_FETCH = "Apply settings and restart obsidian with red_flag_fetch.md";
const CANCEL = "Cancel";
const result = await askSelectString(this.app, "Ready for apply the setting.", [APPLY_AND_RESTART, APPLY_ONLY, APPLY_AND_FETCH, APPLY_AND_REBUILD, CANCEL]);
if (result == APPLY_ONLY || result == APPLY_AND_RESTART || result == APPLY_AND_REBUILD || result == APPLY_AND_FETCH) {
this.settings = settingToApply;
await this.saveSettingData();
if (result == APPLY_ONLY) {
Logger("Loaded settings have been applied!", LOG_LEVEL_NOTICE);
return;
}
if (result == APPLY_AND_REBUILD) {
await this.app.vault.create(FLAGMD_REDFLAG2_HR, "");
}
if (result == APPLY_AND_FETCH) {
await this.app.vault.create(FLAGMD_REDFLAG3_HR, "");
}
// @ts-ignore
this.app.commands.executeCommandById("app:reload");
}
}
)
})
}
generateSettingForMarkdown(settings?: ObsidianLiveSyncSettings, keepCredential?: boolean): Partial<ObsidianLiveSyncSettings> {
const saveData = { ...(settings ? settings : this.settings) };
delete saveData.encryptedCouchDBConnection;
delete saveData.encryptedPassphrase;
if (!saveData.writeCredentialsForSettingSync && !keepCredential) {
delete saveData.couchDB_USER;
delete saveData.couchDB_PASSWORD;
delete saveData.passphrase;
}
return saveData;
}
async saveSettingToMarkdown(filename: string) {
const saveData = this.generateSettingForMarkdown();
let file = this.app.vault.getAbstractFileByPath(filename);
if (!file) {
await this.ensureDirectoryEx(filename);
const initialContent = `This file contains Self-hosted LiveSync settings as YAML.
Except for the \`livesync-setting\` code block, we can add a note for free.
If the name of this file matches the value of the "settingSyncFile" setting inside the \`livesync-setting\` block, LiveSync will tell us whenever the settings change. We can decide to accept or decline the remote setting. (In other words, we can back up this file by renaming it to another name).
We can perform a command in this file.
- \`Parse setting file\` : load the setting from the file.
**Note** Please handle it with all of your care if you have configured to write credentials in.
`
file = await this.app.vault.create(filename, initialContent + SETTING_HEADER + "\n" + SETTING_FOOTER);
}
if (!(file instanceof TFile)) {
Logger(`Markdown Setting: ${filename} already exists as a folder`, LOG_LEVEL_NOTICE);
return;
}
const data = await this.app.vault.read(file);
const { preamble, body, postscript } = this.extractSettingFromWholeText(data);
const newBody = stringifyYaml(saveData);
if (newBody == body) {
Logger("Markdown setting: Nothing had been changed", LOG_LEVEL_VERBOSE);
} else {
await this.app.vault.modify(file, preamble + SETTING_HEADER + newBody + SETTING_FOOTER + postscript);
Logger(`Markdown setting: ${filename} has been updated!`, LOG_LEVEL_VERBOSE);
}
}
async saveSettings() { async saveSettings() {
await this.saveSettingData(); await this.saveSettingData();
fireAndForget(() => this.realizeSettingSyncMode()); fireAndForget(() => this.realizeSettingSyncMode());
@@ -997,7 +1177,7 @@ Note: We can always able to read V1 format. It will be progressively converted.
watchOnline() { watchOnline() {
this.watchOnlineAsync(); scheduleTask("watch-online", 500, () => fireAndForget(() => this.watchOnlineAsync()));
} }
async watchOnlineAsync() { async watchOnlineAsync() {
// If some files were failed to retrieve, scan files again. // If some files were failed to retrieve, scan files again.
@@ -1008,7 +1188,7 @@ Note: We can always able to read V1 format. It will be progressively converted.
} }
} }
watchWindowVisibility() { watchWindowVisibility() {
this.watchWindowVisibilityAsync(); scheduleTask("watch-window-visibility", 500, () => fireAndForget(() => this.watchWindowVisibilityAsync()));
} }
async watchWindowVisibilityAsync() { async watchWindowVisibilityAsync() {
@@ -1094,6 +1274,7 @@ Note: We can always able to read V1 format. It will be progressively converted.
const cache = queue.args.cache; const cache = queue.args.cache;
if (queue.type == "CREATE" || queue.type == "CHANGED") { if (queue.type == "CREATE" || queue.type == "CHANGED") {
fireAndForget(() => this.checkAndApplySettingFromMarkdown(queue.args.file.path, true));
const keyD1 = `file-last-proc-DELETED-${file.path}`; const keyD1 = `file-last-proc-DELETED-${file.path}`;
await this.kvDB.set(keyD1, mtime); await this.kvDB.set(keyD1, mtime);
if (!await this.updateIntoDB(targetFile, false, cache)) { if (!await this.updateIntoDB(targetFile, false, cache)) {
@@ -1112,10 +1293,11 @@ Note: We can always able to read V1 format. It will be progressively converted.
} }
pendingFileEventCount = reactiveSource(0); pendingFileEventCount = reactiveSource(0);
processingFileEventCount = reactiveSource(0);
fileEventQueue = fileEventQueue =
new KeyedQueueProcessor( new KeyedQueueProcessor(
(items: FileEventItem[]) => this.handleFileEvent(items[0]), (items: FileEventItem[]) => this.handleFileEvent(items[0]),
{ suspended: true, batchSize: 1, concurrentLimit: 5, delay: 100, yieldThreshold: FileWatchEventQueueMax, totalRemainingReactiveSource: this.pendingFileEventCount } { suspended: true, batchSize: 1, concurrentLimit: 5, delay: 100, yieldThreshold: FileWatchEventQueueMax, totalRemainingReactiveSource: this.pendingFileEventCount, processingEntitiesReactiveSource: this.processingFileEventCount }
).replaceEnqueueProcessor((items, newItem) => this.queueNextFileEvent(items, newItem)); ).replaceEnqueueProcessor((items, newItem) => this.queueNextFileEvent(items, newItem));
@@ -1127,7 +1309,7 @@ Note: We can always able to read V1 format. It will be progressively converted.
if (this.settings.suspendFileWatching) return; if (this.settings.suspendFileWatching) return;
if (!this.isReady) return; if (!this.isReady) return;
if (!file) return; if (!file) return;
this.watchWorkspaceOpenAsync(file); scheduleTask("watch-workspace-open", 500, () => fireAndForget(() => this.watchWorkspaceOpenAsync(file)));
} }
async watchWorkspaceOpenAsync(file: TFile) { async watchWorkspaceOpenAsync(file: TFile) {
@@ -1274,7 +1456,7 @@ Note: We can always able to read V1 format. It will be progressively converted.
} }
} }
async doc2storage(docEntry: EntryBody, file?: TFile, force?: boolean) { async processEntryDoc(docEntry: EntryBody, file: TFile | undefined, force?: boolean) {
const mode = file == undefined ? "create" : "modify"; const mode = file == undefined ? "create" : "modify";
const path = this.getPath(docEntry); const path = this.getPath(docEntry);
@@ -1282,42 +1464,52 @@ Note: We can always able to read V1 format. It will be progressively converted.
return; return;
} }
if (!await this.isTargetFile(path)) return; if (!await this.isTargetFile(path)) return;
if (docEntry._deleted || docEntry.deleted) {
// This occurs not only when files are deleted, but also when conflicts are resolved. // Conflict resolution check
// We have to check no other revisions are left. const existDoc = await this.localDatabase.getDBEntry(path, { conflicts: true });
const existDoc = await this.localDatabase.getDBEntry(path, { conflicts: true }); const msg = `STORAGE <- DB (${mode}${force ? ",force" : ""},${existDoc ? existDoc?.datatype : "--"}) `;
if (path != file.path) { // let performPullFileAgain = false;
Logger(`delete skipped: ${file.path} :Not exactly matched`, LOG_LEVEL_VERBOSE); if (existDoc && existDoc._conflicts) {
if (this.settings.writeDocumentsIfConflicted) {
Logger(`Processing: ${path}: Conflicted revision has been deleted, but there were more conflicts. `, LOG_LEVEL_INFO);
await this.processEntryDoc(docEntry, file, true);
return;
} else if (force != true) {
Logger(`Processing: ${path}: Conflicted revision has been deleted, but there were more conflicts...`);
this.queueConflictCheck(path);
return;
}
}
// If there are no conflicts, or forced to overwrite.
if (docEntry._deleted || docEntry.deleted || existDoc === false) {
if (!file) {
Logger(`delete skipped: ${path} :Already not exist on storage`, LOG_LEVEL_VERBOSE);
return;
}
if (file.path != path) {
Logger(`delete skipped: ${path} :Not exactly matched`, LOG_LEVEL_VERBOSE);
return;
} }
if (existDoc === false) { if (existDoc === false) {
await this.deleteVaultItem(file); await this.deleteVaultItem(file);
} else { } else {
if (existDoc._conflicts) { // Conflict has been resolved at this time,
if (this.settings.writeDocumentsIfConflicted) { await this.pullFile(path, null, true);
Logger(`Delete: ${file.path}: Conflicted revision has been deleted, but there were more conflicts. `, LOG_LEVEL_INFO);
await this.pullFile(path, null, true);
} else {
Logger(`Delete: ${file.path}: Conflicted revision has been deleted, but there were more conflicts...`);
this.queueConflictCheck(file);
}
} else {
Logger(`Delete: ${file.path}: Conflict revision has been deleted and resolved`);
await this.pullFile(path, null, true);
}
} }
return; return;
} }
const localMtime = ~~((file?.stat?.mtime || 0) / 1000);
const docMtime = ~~(docEntry.mtime / 1000);
const doc = await this.localDatabase.getDBEntry(path, { rev: docEntry._rev }); const compareResult = compareFileFreshness(file, docEntry);
if (doc === false) return;
const msg = `STORAGE <- DB (${mode}${force ? ",force" : ""},${doc.datatype}) `; const doc = existDoc;
if (doc.datatype != "newnote" && doc.datatype != "plain") { if (doc.datatype != "newnote" && doc.datatype != "plain") {
Logger(msg + "ERROR, Invalid datatype: " + path + "(" + doc.datatype + ")", LOG_LEVEL_NOTICE); Logger(msg + "ERROR, Invalid datatype: " + path + "(" + doc.datatype + ")", LOG_LEVEL_NOTICE);
return; return;
} }
if (!force && localMtime >= docMtime) return; // if (!force && localMtime >= docMtime) return;
if (!force && (compareResult == BASE_IS_NEW || compareResult == EVEN)) return;
if (!isValidPath(path)) { if (!isValidPath(path)) {
Logger(msg + "ERROR, invalid path: " + path, LOG_LEVEL_NOTICE); Logger(msg + "ERROR, invalid path: " + path, LOG_LEVEL_NOTICE);
return; return;
@@ -1327,7 +1519,7 @@ Note: We can always able to read V1 format. It will be progressively converted.
try { try {
let outFile; let outFile;
let isChanged = true; let isChanged = true;
if (mode == "create") { if (!file) {
const normalizedPath = normalizePath(path); const normalizedPath = normalizePath(path);
await this.vaultAccess.vaultCreate(normalizedPath, writeData, { ctime: doc.ctime, mtime: doc.mtime, }); await this.vaultAccess.vaultCreate(normalizedPath, writeData, { ctime: doc.ctime, mtime: doc.mtime, });
outFile = this.vaultAccess.getAbstractFileByPath(normalizedPath) as TFile; outFile = this.vaultAccess.getAbstractFileByPath(normalizedPath) as TFile;
@@ -1390,9 +1582,12 @@ Note: We can always able to read V1 format. It will be progressively converted.
if (!this.settings.suspendParseReplicationResult) { if (!this.settings.suspendParseReplicationResult) {
const lsKey = "obsidian-livesync-queuefiles-" + this.getVaultName(); const lsKey = "obsidian-livesync-queuefiles-" + this.getVaultName();
const ids = [...new Set(JSON.parse(localStorage.getItem(lsKey) || "[]"))] as string[]; const ids = [...new Set(JSON.parse(localStorage.getItem(lsKey) || "[]"))] as string[];
const ret = await this.localDatabase.allDocsRaw<EntryDoc>({ keys: ids, include_docs: true }); const batchSize = 100;
for (const doc of ret.rows) { const chunkedIds = arrayToChunkedArray(ids, batchSize);
this.replicationResultProcessor.enqueue(doc.doc); for await (const idsBatch of chunkedIds) {
const ret = await this.localDatabase.allDocsRaw<EntryDoc>({ keys: idsBatch, include_docs: true, limit: 100 });
this.replicationResultProcessor.enqueueAll(ret.rows.map(doc => doc.doc));
await this.replicationResultProcessor.waitForPipeline();
} }
} }
} }
@@ -1425,32 +1620,15 @@ Note: We can always able to read V1 format. It will be progressively converted.
storageApplyingProcessor = new KeyedQueueProcessor(async (docs: LoadedEntry[]) => { storageApplyingProcessor = new KeyedQueueProcessor(async (docs: LoadedEntry[]) => {
const entry = docs[0]; const entry = docs[0];
const path = this.getPath(entry); const path = this.getPath(entry);
Logger(`Applying ${path} (${entry._id.substring(0, 8)}: ${entry._rev?.substring(0, 5)}) change...`, LOG_LEVEL_VERBOSE); Logger(`Processing ${path} (${entry._id.substring(0, 8)}: ${entry._rev?.substring(0, 5)}) change...`, LOG_LEVEL_VERBOSE);
const targetFile = this.vaultAccess.getAbstractFileByPath(this.getPathWithoutPrefix(entry)); const targetFile = this.vaultAccess.getAbstractFileByPath(this.getPathWithoutPrefix(entry));
if (targetFile == null) { if (targetFile instanceof TFolder) {
if (entry._deleted || entry.deleted) {
return;
}
const doc = entry;
await this.doc2storage(doc);
} else if (targetFile instanceof TFile) {
const doc = entry;
const file = targetFile;
if (this.settings.writeDocumentsIfConflicted) {
await this.doc2storage(doc, file);
this.queueConflictCheck(file);
} else {
const d = await this.localDatabase.getDBEntryMeta(this.getPath(entry), { conflicts: true }, true);
if (d && !d._conflicts) {
await this.doc2storage(doc, file);
} else {
this.queueConflictCheck(file);
}
}
} else {
Logger(`${this.getPath(entry)} is already exist as the folder`); Logger(`${this.getPath(entry)} is already exist as the folder`);
} else {
await this.processEntryDoc(entry, targetFile instanceof TFile ? targetFile : undefined);
Logger(`Processing ${path} (${entry._id.substring(0, 8)}:${entry._rev?.substring(0, 5)}) `);
} }
Logger(`Applied ${path} (${entry._id.substring(0, 8)}:${entry._rev?.substring(0, 5)}) change...`);
return; return;
}, { suspended: true, batchSize: 1, concurrentLimit: 2, yieldThreshold: 1, delay: 0, totalRemainingReactiveSource: this.storageApplyingCount }).startPipeline() }, { suspended: true, batchSize: 1, concurrentLimit: 2, yieldThreshold: 1, delay: 0, totalRemainingReactiveSource: this.storageApplyingCount }).startPipeline()
@@ -1487,20 +1665,26 @@ Note: We can always able to read V1 format. It will be progressively converted.
if (this.databaseQueuedProcessor._isSuspended) { if (this.databaseQueuedProcessor._isSuspended) {
Logger(`Processing scheduled: ${change.path}`, LOG_LEVEL_INFO); Logger(`Processing scheduled: ${change.path}`, LOG_LEVEL_INFO);
} }
const size = change.size;
if (this.isFileSizeExceeded(size)) {
Logger(`Processing ${change.path} has been skipped due to file size exceeding the limit`, LOG_LEVEL_NOTICE);
return;
}
this.databaseQueuedProcessor.enqueueWithKey(change.path, change); this.databaseQueuedProcessor.enqueueWithKey(change.path, change);
} }
return; return;
}, { batchSize: 1, suspended: true, concurrentLimit: 1, delay: 0, totalRemainingReactiveSource: this.replicationResultCount }).startPipeline().onUpdateProgress(() => { }, { batchSize: 1, suspended: true, concurrentLimit: 100, delay: 0, totalRemainingReactiveSource: this.replicationResultCount }).startPipeline().onUpdateProgress(() => {
this.saveQueuedFiles(); this.saveQueuedFiles();
}); });
//---> Sync //---> Sync
parseReplicationResult(docs: Array<PouchDB.Core.ExistingDocument<EntryDoc>>) { parseReplicationResult(docs: Array<PouchDB.Core.ExistingDocument<EntryDoc>>) {
if (this.settings.suspendParseReplicationResult) { if (this.settings.suspendParseReplicationResult) {
this.replicationResultProcessor.suspend() this.replicationResultProcessor.suspend()
} else {
this.replicationResultProcessor.resume()
} }
this.replicationResultProcessor.enqueueAll(docs); this.replicationResultProcessor.enqueueAll(docs);
if (!this.settings.suspendParseReplicationResult) {
this.replicationResultProcessor.resume()
}
} }
@@ -1585,7 +1769,7 @@ Note: We can always able to read V1 format. It will be progressively converted.
}) })
const waitingLabel = reactive(() => { const waitingLabel = reactive(() => {
const e = this.pendingFileEventCount.value; const e = this.pendingFileEventCount.value;
const proc = this.fileEventQueue.processingEntities; const proc = this.processingFileEventCount.value;
const pend = e - proc; const pend = e - proc;
const labelProc = proc != 0 ? `⏳${proc} ` : ""; const labelProc = proc != 0 ? `⏳${proc} ` : "";
const labelPend = pend != 0 ? ` 🛫${pend}` : ""; const labelPend = pend != 0 ? ` 🛫${pend}` : "";
@@ -1626,13 +1810,19 @@ Note: We can always able to read V1 format. It will be progressively converted.
const newLog = log; const newLog = log;
// scheduleTask("update-display", 50, () => { // scheduleTask("update-display", 50, () => {
this.statusBar?.setText(newMsg.split("\n")[0]); this.statusBar?.setText(newMsg.split("\n")[0]);
const selector = `.CodeMirror-wrap,` +
`.markdown-preview-view.cm-s-obsidian,` +
`.markdown-source-view.cm-s-obsidian,` +
`.canvas-wrapper,` +
`.empty-state`
;
if (this.settings.showStatusOnEditor) { if (this.settings.showStatusOnEditor) {
const root = activeDocument.documentElement; const root = activeDocument.documentElement;
const q = root.querySelectorAll(`.CodeMirror-wrap,.cm-s-obsidian>.cm-editor,.canvas-wrapper`); const q = root.querySelectorAll(selector);
q.forEach(e => e.setAttr("data-log", '' + (newMsg + "\n" + newLog) + '')) q.forEach(e => e.setAttr("data-log", '' + (newMsg + "\n" + newLog) + ''))
} else { } else {
const root = activeDocument.documentElement; const root = activeDocument.documentElement;
const q = root.querySelectorAll(`.CodeMirror-wrap,.cm-s-obsidian>.cm-editor,.canvas-wrapper`); const q = root.querySelectorAll(selector);
q.forEach(e => e.setAttr("data-log", '')) q.forEach(e => e.setAttr("data-log", ''))
} }
// }, true); // }, true);
@@ -1752,6 +1942,15 @@ Or if you are sure know what had been happened, we can unlock the database from
return await this.replicator.markRemoteResolved(this.settings); return await this.replicator.markRemoteResolved(this.settings);
} }
isFileSizeExceeded(size: number) {
if (this.settings.syncMaxSizeInMB > 0 && size > 0) {
if (this.settings.syncMaxSizeInMB * 1024 * 1024 < size) {
return true;
}
}
return false;
}
async syncAllFiles(showingNotice?: boolean) { async syncAllFiles(showingNotice?: boolean) {
// synchronize all files between database and storage. // synchronize all files between database and storage.
let initialScan = false; let initialScan = false;
@@ -1799,9 +1998,9 @@ Or if you are sure know what had been happened, we can unlock the database from
const syncFiles = filesStorage.filter((e) => onlyInStorageNames.indexOf(e.path) == -1); const syncFiles = filesStorage.filter((e) => onlyInStorageNames.indexOf(e.path) == -1);
Logger("Updating database by new files"); Logger("Updating database by new files");
// this.setStatusBarText(`UPDATE DATABASE`);
const initProcess = []; const initProcess = [];
const logLevel = showingNotice ? LOG_LEVEL_NOTICE : LOG_LEVEL_INFO;
const runAll = async<T>(procedureName: string, objects: T[], callback: (arg: T) => Promise<void>) => { const runAll = async<T>(procedureName: string, objects: T[], callback: (arg: T) => Promise<void>) => {
if (objects.length == 0) { if (objects.length == 0) {
Logger(`${procedureName}: Nothing to do`); Logger(`${procedureName}: Nothing to do`);
@@ -1812,7 +2011,6 @@ Or if you are sure know what had been happened, we can unlock the database from
let success = 0; let success = 0;
let failed = 0; let failed = 0;
const step = 10; const step = 10;
const logLevel = showingNotice ? LOG_LEVEL_NOTICE : LOG_LEVEL_INFO;
const processor = new QueueProcessor(async (e) => { const processor = new QueueProcessor(async (e) => {
try { try {
await callback(e[0]); await callback(e[0]);
@@ -1832,15 +2030,24 @@ Or if you are sure know what had been happened, we can unlock the database from
Logger(`${procedureName} All done: DONE:${success}, FAILED:${failed}`, logLevel, `log-${procedureName}`); Logger(`${procedureName} All done: DONE:${success}, FAILED:${failed}`, logLevel, `log-${procedureName}`);
} }
initProcess.push(runAll("UPDATE DATABASE", onlyInStorage, async (e) => { initProcess.push(runAll("UPDATE DATABASE", onlyInStorage, async (e) => {
await this.updateIntoDB(e, initialScan); if (!this.isFileSizeExceeded(e.stat.size)) {
await this.updateIntoDB(e, initialScan);
fireAndForget(() => this.checkAndApplySettingFromMarkdown(e.path, true));
} else {
Logger(`UPDATE DATABASE: ${e.path} has been skipped due to file size exceeding the limit`, logLevel);
}
})); }));
if (!initialScan) { if (!initialScan) {
initProcess.push(runAll("UPDATE STORAGE", onlyInDatabase, async (e) => { initProcess.push(runAll("UPDATE STORAGE", onlyInDatabase, async (e) => {
const w = await this.localDatabase.getDBEntryMeta(e, {}, true); const w = await this.localDatabase.getDBEntryMeta(e, {}, true);
if (w && !(w.deleted || w._deleted)) { if (w && !(w.deleted || w._deleted)) {
Logger(`Check or pull from db:${e}`); if (!this.isFileSizeExceeded(w.size)) {
await this.pullFile(e, filesStorage, false, null, false); await this.pullFile(e, filesStorage, false, null, false);
Logger(`Check or pull from db:${e} OK`); fireAndForget(() => this.checkAndApplySettingFromMarkdown(e, true));
Logger(`Check or pull from db:${e} OK`);
} else {
Logger(`UPDATE STORAGE: ${e} has been skipped due to file size exceeding the limit`, logLevel);
}
} else if (w) { } else if (w) {
Logger(`Deletion history skipped: ${e}`, LOG_LEVEL_VERBOSE); Logger(`Deletion history skipped: ${e}`, LOG_LEVEL_VERBOSE);
} else { } else {
@@ -1849,8 +2056,8 @@ Or if you are sure know what had been happened, we can unlock the database from
})); }));
} }
if (!initialScan) { if (!initialScan) {
let caches: { [key: string]: { storageMtime: number; docMtime: number } } = {}; // let caches: { [key: string]: { storageMtime: number; docMtime: number } } = {};
caches = await this.kvDB.get<{ [key: string]: { storageMtime: number; docMtime: number } }>("diff-caches") || {}; // caches = await this.kvDB.get<{ [key: string]: { storageMtime: number; docMtime: number } }>("diff-caches") || {};
type FileDocPair = { file: TFile, id: DocumentID }; type FileDocPair = { file: TFile, id: DocumentID };
const processPrepareSyncFile = new QueueProcessor( const processPrepareSyncFile = new QueueProcessor(
@@ -1876,7 +2083,7 @@ Or if you are sure know what had been happened, we can unlock the database from
new QueueProcessor( new QueueProcessor(
async (loadedPairs) => { async (loadedPairs) => {
const e = loadedPairs[0]; const e = loadedPairs[0];
await this.syncFileBetweenDBandStorage(e.file, e.doc, initialScan, caches); await this.syncFileBetweenDBandStorage(e.file, e.doc, initialScan);
return; return;
}, { batchSize: 1, concurrentLimit: 5, delay: 10, suspended: false } }, { batchSize: 1, concurrentLimit: 5, delay: 10, suspended: false }
)) ))
@@ -1884,7 +2091,7 @@ Or if you are sure know what had been happened, we can unlock the database from
processPrepareSyncFile.startPipeline(); processPrepareSyncFile.startPipeline();
initProcess.push(async () => { initProcess.push(async () => {
await processPrepareSyncFile.waitForPipeline(); await processPrepareSyncFile.waitForPipeline();
await this.kvDB.set("diff-caches", caches); // await this.kvDB.set("diff-caches", caches);
}) })
} }
await Promise.all(initProcess); await Promise.all(initProcess);
@@ -1933,6 +2140,10 @@ Or if you are sure know what had been happened, we can unlock the database from
if (baseLeaf == false || leftLeaf == false || rightLeaf == false) { if (baseLeaf == false || leftLeaf == false || rightLeaf == false) {
return false; return false;
} }
if (leftLeaf.deleted && rightLeaf.deleted) {
// Both are deleted
return false;
}
// diff between base and each revision // diff between base and each revision
const dmp = new diff_match_patch(); const dmp = new diff_match_patch();
const mapLeft = dmp.diff_linesToChars_(baseLeaf.data, leftLeaf.data); const mapLeft = dmp.diff_linesToChars_(baseLeaf.data, leftLeaf.data);
@@ -2093,6 +2304,9 @@ Or if you are sure know what had been happened, we can unlock the database from
if (baseLeaf == false || leftLeaf == false || rightLeaf == false) { if (baseLeaf == false || leftLeaf == false || rightLeaf == false) {
return false; return false;
} }
if (leftLeaf.deleted && rightLeaf.deleted) {
return false;
}
const baseObj = { data: tryParseJSON(baseLeaf.data, {}) } as Record<string | number | symbol, any>; const baseObj = { data: tryParseJSON(baseLeaf.data, {}) } as Record<string | number | symbol, any>;
const leftObj = { data: tryParseJSON(leftLeaf.data, {}) } as Record<string | number | symbol, any>; const leftObj = { data: tryParseJSON(leftLeaf.data, {}) } as Record<string | number | symbol, any>;
const rightObj = { data: tryParseJSON(rightLeaf.data, {}) } as Record<string | number | symbol, any>; const rightObj = { data: tryParseJSON(rightLeaf.data, {}) } as Record<string | number | symbol, any>;
@@ -2178,7 +2392,6 @@ Or if you are sure know what had been happened, we can unlock the database from
if (p != undefined) { if (p != undefined) {
// remove conflicted revision. // remove conflicted revision.
await this.localDatabase.deleteDBEntry(path, { rev: conflictedRev }); await this.localDatabase.deleteDBEntry(path, { rev: conflictedRev });
const file = this.vaultAccess.getAbstractFileByPath(stripAllPrefixes(path)) as TFile; const file = this.vaultAccess.getAbstractFileByPath(stripAllPrefixes(path)) as TFile;
if (file) { if (file) {
if (await this.vaultAccess.vaultModify(file, p)) { if (await this.vaultAccess.vaultModify(file, p)) {
@@ -2215,10 +2428,10 @@ Or if you are sure know what had been happened, we can unlock the database from
const isBinary = !isPlainText(path); const isBinary = !isPlainText(path);
const alwaysNewer = this.settings.resolveConflictsByNewerFile; const alwaysNewer = this.settings.resolveConflictsByNewerFile;
if (isSame || isBinary || alwaysNewer) { if (isSame || isBinary || alwaysNewer) {
const lMtime = ~~(leftLeaf.mtime / 1000); const result = compareMTime(leftLeaf.mtime, rightLeaf.mtime)
const rMtime = ~~(rightLeaf.mtime / 1000);
let loser = leftLeaf; let loser = leftLeaf;
if (lMtime > rMtime) { // if (lMtime > rMtime) {
if (result != TARGET_IS_NEW) {
loser = rightLeaf; loser = rightLeaf;
} }
await this.localDatabase.deleteDBEntry(path, { rev: loser.rev }); await this.localDatabase.deleteDBEntry(path, { rev: loser.rev });
@@ -2240,7 +2453,7 @@ Or if you are sure know what had been happened, we can unlock the database from
conflictProcessQueueCount = reactiveSource(0); conflictProcessQueueCount = reactiveSource(0);
conflictResolveQueue = conflictResolveQueue =
new KeyedQueueProcessor(async (entries: { filename: FilePathWithPrefix, file: TFile }[]) => { new KeyedQueueProcessor(async (entries: { filename: FilePathWithPrefix }[]) => {
const entry = entries[0]; const entry = entries[0];
const filename = entry.filename; const filename = entry.filename;
const conflictCheckResult = await this.checkConflictAndPerformAutoMerge(filename); const conflictCheckResult = await this.checkConflictAndPerformAutoMerge(filename);
@@ -2281,11 +2494,12 @@ Or if you are sure know what had been happened, we can unlock the database from
new QueueProcessor((files: FilePathWithPrefix[]) => { new QueueProcessor((files: FilePathWithPrefix[]) => {
const filename = files[0]; const filename = files[0];
const file = this.vaultAccess.getAbstractFileByPath(filename); const file = this.vaultAccess.getAbstractFileByPath(filename);
if (!file) return; // if (!file) return;
if (!(file instanceof TFile)) return; // if (!(file instanceof TFile)) return;
if ((file instanceof TFolder)) return;
// Check again? // Check again?
return [{ key: filename, entity: { filename, file } }]; return [{ key: filename, entity: { filename } }];
// this.conflictResolveQueue.enqueueWithKey(filename, { filename, file }); // this.conflictResolveQueue.enqueueWithKey(filename, { filename, file });
}, { }, {
suspended: false, batchSize: 1, concurrentLimit: 5, delay: 10, keepResultUntilDownstreamConnected: true, pipeTo: this.conflictResolveQueue, totalRemainingReactiveSource: this.conflictProcessQueueCount suspended: false, batchSize: 1, concurrentLimit: 5, delay: 10, keepResultUntilDownstreamConnected: true, pipeTo: this.conflictResolveQueue, totalRemainingReactiveSource: this.conflictProcessQueueCount
@@ -2355,7 +2569,7 @@ Or if you are sure know what had been happened, we can unlock the database from
Logger(`${filename} Skipped`); Logger(`${filename} Skipped`);
return; return;
} }
await this.doc2storage(doc, undefined, force); await this.processEntryDoc(doc, undefined, force);
} else if (targetFile instanceof TFile) { } else if (targetFile instanceof TFile) {
//normal case //normal case
const file = targetFile; const file = targetFile;
@@ -2364,7 +2578,7 @@ Or if you are sure know what had been happened, we can unlock the database from
Logger(`${filename} Skipped`); Logger(`${filename} Skipped`);
return; return;
} }
await this.doc2storage(doc, file, force); await this.processEntryDoc(doc, file, force);
} else { } else {
Logger(`target files:${filename} is exists as the folder`); Logger(`target files:${filename} is exists as the folder`);
//something went wrong.. //something went wrong..
@@ -2372,7 +2586,7 @@ Or if you are sure know what had been happened, we can unlock the database from
//when to opened file; //when to opened file;
} }
async syncFileBetweenDBandStorage(file: TFile, doc: LoadedEntry, initialScan: boolean, caches: { [key: string]: { storageMtime: number; docMtime: number } }) { async syncFileBetweenDBandStorage(file: TFile, doc: LoadedEntry, initialScan: boolean) {
if (!doc) { if (!doc) {
throw new Error(`Missing doc:${(file as any).path}`) throw new Error(`Missing doc:${(file as any).path}`)
} }
@@ -2385,38 +2599,37 @@ Or if you are sure know what had been happened, we can unlock the database from
} }
} }
const storageMtime = ~~(file.stat.mtime / 1000); const compareResult = compareFileFreshness(file, doc);
const docMtime = ~~(doc.mtime / 1000); switch (compareResult) {
const dK = `${file.path}-diff`; case BASE_IS_NEW:
const isLastDiff = dK in caches ? caches[dK] : { storageMtime: 0, docMtime: 0 }; if (!this.isFileSizeExceeded(file.stat.size)) {
if (isLastDiff.docMtime == docMtime && isLastDiff.storageMtime == storageMtime) { Logger("STORAGE -> DB :" + file.path);
// Logger("STORAGE .. DB :" + file.path, LOG_LEVEL_VERBOSE); await this.updateIntoDB(file, initialScan);
caches[dK] = { storageMtime, docMtime }; fireAndForget(() => this.checkAndApplySettingFromMarkdown(file.path, true));
return caches; } else {
Logger(`STORAGE -> DB : ${file.path} has been skipped due to file size exceeding the limit`, LOG_LEVEL_NOTICE);
}
break;
case TARGET_IS_NEW:
if (!this.isFileSizeExceeded(doc.size)) {
Logger("STORAGE <- DB :" + file.path);
const docx = await this.localDatabase.getDBEntry(getPathFromTFile(file), null, false, false, true);
if (docx != false) {
await this.processEntryDoc(docx, file);
} else {
Logger(`STORAGE <- DB : Cloud not read ${file.path}, possibly deleted`, LOG_LEVEL_NOTICE);
}
return caches;
} else {
Logger(`STORAGE <- DB : ${file.path} has been skipped due to file size exceeding the limit`, LOG_LEVEL_NOTICE);
}
break;
case EVEN:
Logger("STORAGE == DB :" + file.path + "", LOG_LEVEL_VERBOSE);
break;
default:
Logger("STORAGE ?? DB :" + file.path + " Something got weird");
} }
if (storageMtime > docMtime) {
//newer local file.
Logger("STORAGE -> DB :" + file.path);
Logger(`${storageMtime} > ${docMtime}`);
await this.updateIntoDB(file, initialScan);
caches[dK] = { storageMtime, docMtime };
return caches;
} else if (storageMtime < docMtime) {
//newer database file.
Logger("STORAGE <- DB :" + file.path);
Logger(`${storageMtime} < ${docMtime}`);
const docx = await this.localDatabase.getDBEntry(getPathFromTFile(file), null, false, false);
if (docx != false) {
await this.doc2storage(docx, file);
} else {
Logger("STORAGE <- DB :" + file.path + " Skipped");
}
caches[dK] = { storageMtime, docMtime };
return caches;
}
Logger("STORAGE == DB :" + file.path + "", LOG_LEVEL_VERBOSE);
caches[dK] = { storageMtime, docMtime };
return caches;
} }
@@ -2487,6 +2700,7 @@ Or if you are sure know what had been happened, we can unlock the database from
if (oldData.deleted != newData.deleted) return false; if (oldData.deleted != newData.deleted) return false;
if (!await isDocContentSame(old.data, newData.data)) return false; if (!await isDocContentSame(old.data, newData.data)) return false;
Logger(msg + "Skipped (not changed) " + fullPath + ((d._deleted || d.deleted) ? " (deleted)" : ""), LOG_LEVEL_VERBOSE); Logger(msg + "Skipped (not changed) " + fullPath + ((d._deleted || d.deleted) ? " (deleted)" : ""), LOG_LEVEL_VERBOSE);
markChangesAreSame(old, d.mtime, old.mtime);
return true; return true;
// d._rev = old._rev; // d._rev = old._rev;
} }
@@ -2505,10 +2719,11 @@ Or if you are sure know what had been happened, we can unlock the database from
return true; return true;
} }
const ret = await this.localDatabase.putDBEntry(d, initialScan); const ret = await this.localDatabase.putDBEntry(d, initialScan);
if (ret !== false) {
Logger(msg + fullPath); Logger(msg + fullPath);
if (this.settings.syncOnSave && !this.suspended) { if (this.settings.syncOnSave && !this.suspended) {
await this.replicate(); scheduleTask("perform-replicate-after-save", 250, () => this.replicate());
}
} }
return ret != false; return ret != false;
} }

7
src/stores.ts Normal file
View File

@@ -0,0 +1,7 @@
import { PersistentMap } from "./lib/src/PersistentMap";
export let sameChangePairs: PersistentMap<number[]>;
export function initializeStores(vaultName: string) {
sameChangePairs = new PersistentMap<number[]>(`ls-persist-same-changes-${vaultName}`);
}

48
src/utils.ts
View File

@@ -1,4 +1,4 @@
import { normalizePath, Platform, TAbstractFile, App, type RequestUrlParam, requestUrl } from "./deps"; import { normalizePath, Platform, TAbstractFile, App, type RequestUrlParam, requestUrl, TFile } from "./deps";
import { path2id_base, id2path_base, isValidFilenameInLinux, isValidFilenameInDarwin, isValidFilenameInWidows, isValidFilenameInAndroid, stripAllPrefixes } from "./lib/src/path"; import { path2id_base, id2path_base, isValidFilenameInLinux, isValidFilenameInDarwin, isValidFilenameInWidows, isValidFilenameInAndroid, stripAllPrefixes } from "./lib/src/path";
import { Logger } from "./lib/src/logger"; import { Logger } from "./lib/src/logger";
@@ -8,6 +8,7 @@ import { InputStringDialog, PopoverSelectString } from "./dialogs";
import type ObsidianLiveSyncPlugin from "./main"; import type ObsidianLiveSyncPlugin from "./main";
import { writeString } from "./lib/src/strbin"; import { writeString } from "./lib/src/strbin";
import { fireAndForget } from "./lib/src/utils"; import { fireAndForget } from "./lib/src/utils";
import { sameChangePairs } from "./stores";
export { scheduleTask, setPeriodicTask, cancelTask, cancelAllTasks, cancelPeriodicTask, cancelAllPeriodicTask, } from "./lib/src/task"; export { scheduleTask, setPeriodicTask, cancelTask, cancelAllTasks, cancelPeriodicTask, cancelAllPeriodicTask, } from "./lib/src/task";
@@ -415,3 +416,48 @@ export async function performRebuildDB(plugin: ObsidianLiveSyncPlugin, method: "
await plugin.addOnSetup.rebuildEverything(); await plugin.addOnSetup.rebuildEverything();
} }
} }
export const BASE_IS_NEW = Symbol("base");
export const TARGET_IS_NEW = Symbol("target");
export const EVEN = Symbol("even");
// Why 2000? : ZIP FILE Does not have enough resolution.
const resolution = 2000;
export function compareMTime(baseMTime: number, targetMTime: number): typeof BASE_IS_NEW | typeof TARGET_IS_NEW | typeof EVEN {
const truncatedBaseMTime = (~~(baseMTime / resolution)) * resolution;
const truncatedTargetMTime = (~~(targetMTime / resolution)) * resolution;
// Logger(`Resolution MTime ${truncatedBaseMTime} and ${truncatedTargetMTime} `, LOG_LEVEL_VERBOSE);
if (truncatedBaseMTime == truncatedTargetMTime) return EVEN;
if (truncatedBaseMTime > truncatedTargetMTime) return BASE_IS_NEW;
if (truncatedBaseMTime < truncatedTargetMTime) return TARGET_IS_NEW;
throw new Error("Unexpected error");
}
export function markChangesAreSame(file: TFile | AnyEntry | string, mtime1: number, mtime2: number) {
if (mtime1 === mtime2) return true;
const key = typeof file == "string" ? file : file instanceof TFile ? file.path : file.path ?? file._id;
const pairs = sameChangePairs.get(key, []);
if (pairs.some(e => e == mtime1 || e == mtime2)) {
sameChangePairs.set(key, [...new Set([...pairs, mtime1, mtime2])]);
} else {
sameChangePairs.set(key, [mtime1, mtime2]);
}
}
export function isMarkedAsSameChanges(file: TFile | AnyEntry | string, mtimes: number[]) {
const key = typeof file == "string" ? file : file instanceof TFile ? file.path : file.path ?? file._id;
const pairs = sameChangePairs.get(key, []);
if (mtimes.every(e => pairs.indexOf(e) !== -1)) {
return EVEN;
}
}
export function compareFileFreshness(baseFile: TFile | AnyEntry, checkTarget: TFile | AnyEntry): typeof BASE_IS_NEW | typeof TARGET_IS_NEW | typeof EVEN {
const modifiedBase = baseFile instanceof TFile ? baseFile?.stat?.mtime ?? 0 : baseFile?.mtime ?? 0;
const modifiedTarget = checkTarget instanceof TFile ? checkTarget?.stat?.mtime ?? 0 : checkTarget?.mtime ?? 0;
if (modifiedBase && modifiedTarget && isMarkedAsSameChanges(baseFile, [modifiedBase, modifiedTarget])) {
return EVEN;
}
return compareMTime(modifiedBase, modifiedTarget);
}

21
styles.css
View File

@@ -98,8 +98,10 @@
} }
.CodeMirror-wrap::before, .CodeMirror-wrap::before,
.cm-s-obsidian > .cm-editor::before, .markdown-preview-view.cm-s-obsidian::before,
.canvas-wrapper::before { .markdown-source-view.cm-s-obsidian::before,
.canvas-wrapper::before,
.empty-state::before {
content: attr(data-log); content: attr(data-log);
text-align: right; text-align: right;
white-space: pre-wrap; white-space: pre-wrap;
@@ -115,6 +117,19 @@
filter: grayscale(100%); filter: grayscale(100%);
} }
.empty-state::before,
.markdown-preview-view.cm-s-obsidian::before,
.markdown-source-view.cm-s-obsidian::before {
top: var(--header-height);
right: 1em;
}
.is-mobile .empty-state::before,
.is-mobile .markdown-preview-view.cm-s-obsidian::before,
.is-mobile .markdown-source-view.cm-s-obsidian::before {
top: var(--view-header-height);
right: 1em;
}
.canvas-wrapper::before { .canvas-wrapper::before {
right: 48px; right: 48px;
} }
@@ -292,7 +307,7 @@ span.ls-mark-cr::after {
position: absolute; position: absolute;
top: 0; top: 0;
left: 0; left: 0;
animation: ls-blink-diff 0.5s cubic-bezier(0.4, 0, 1, 1) infinite alternate; animation: ls-blink-diff 0.5s cubic-bezier(0.4, 0, 1, 1) infinite alternate;
} }
@keyframes ls-blink-diff { @keyframes ls-blink-diff {
0% { 0% {

39
updates.md
View File

@@ -7,8 +7,47 @@ Of course, I think this would be our suffering in some cases. However, I would l
Sorry for being absent so much long. And thank you for your patience! Sorry for being absent so much long. And thank you for your patience!
Note: we got a very performance improvement. Note: we got a very performance improvement.
Note at 0.22.2: **Now, to rescue mobile devices, Maximum file size is set to 50 by default**. Please configure the limit as you need. If you do not want to limit the sizes, set zero manually, please.
#### Version history #### Version history
- 0.23.4
- Fixed:
- Now the result of conflict resolution could be surely written into the storage.
- Deleted files can be handled correctly again in the history dialogue and conflict dialogue.
- Some wrong log messages were fixed.
- Change handling now has become more stable.
- Some event handling became to be safer.
- Improved:
- Dumping document information shows conflicts and revisions.
- The timestamp-only differences can be surely cached.
- Timestamp difference detection can be rounded by two seconds.
- Refactored:
- A bit of organisation to write the test.
- 0.22.3
- Fixed:
- No longer detects storage changes which have been caused by Self-hosted LiveSync itself.
- Setting sync file will be detected only if it has been configured now.
- And its log will be shown only while the verbose log is enabled.
- Customisation file enumeration has got less blingy.
- Deletion of files is now reliably synchronised.
- Fixed and improved:
- In-editor-status is now shown in the following areas:
- Note editing pane (Source mode and live-preview mode).
- New tab pane.
- Canvas pane.
- 0.22.2
- Fixed:
- Now the results of resolving conflicts are surely synchronised.
- Modified:
- Some setting items got new clear names. (`Sync Settings` -> `Targets`).
- New feature:
- We can limit the synchronising files by their size. (`Sync Settings` -> `Targets` -> `Maximum file size`).
- It depends on the size of the newer one.
- At Obsidian 1.5.3 on mobile, we should set this to around 50MB to avoid restarting Obsidian.
- Now the settings could be stored in a specific markdown file to synchronise or switch it (`General Setting` -> `Share settings via markdown`).
- [Screwdriver](https://github.com/vrtmrz/obsidian-screwdriver) is quite good, but mostly we only need this.
- Customisation of the obsoleted device is now able to be deleted at once.
- We have to put the maintenance mode in at the Customisation sync dialogue.
- 0.22.1 - 0.22.1
- New feature: - New feature:
- We can perform automatic conflict resolution for inactive files, and postpone only manual ones by `Postpone manual resolution of inactive files`. - We can perform automatic conflict resolution for inactive files, and postpone only manual ones by `Postpone manual resolution of inactive files`.