public/obsidian-livesync
1
0
Fork 0
mirror of https://github.com/vrtmrz/obsidian-livesync.git synced 2026-05-15 03:51:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

Compare commits

base: public:0.17.34
Branches Tags
public:main public:test_real_obsidian public:v0_25_62 public:update_lib public:feat-userhashsalt public:auto_conf_2 public:auto_conf public:beta public:alpha public:refactor_services_2 public:refactor_services public:dev public:svelteui public:forprivacy
public:0.25.62 public:0.25.61 public:0.25.60 public:0.25.59 public:0.25.58 public:0.25.57 public:0.25.56+patched5 public:0.25.56+patched4 public:0.25.56+patched3 public:0.25.56+patched2 public:0.25.56+patched1 public:0.25.56 public:0.25.55 public:0.25.54 public:0.25.53 public:0.25.52-patched-3 public:0.25.52-patched-2 public:0.25.52-patched-1 public:0.25.52 public:0.25.51 public:0.25.50 public:0.25.49 public:0.25.48 public:0.25.47 public:0.25.46 public:0.25.45 public:0.25.44 public:0.25.43-patched-9 public:0.25.43-patched-8 public:0.25.43-patched-7 public:0.25.43-patched-6 public:0.25.43-patched-5 public:0.25.43-patched-4 public:0.25.43-patched-3 public:0.25.43-patched-2 public:0.25.43-patched-1 public:0.25.43 public:0.25.42 public:0.25.41-patched-3 public:0.25.41-patched-2 public:0.25.41-patched-1 public:0.25.41 public:0.25.40 public:0.25.39 public:0.25.38 public:0.25.37 public:0.25.36 public:0.25.35 public:0.25.34 public:0.25.33 public:0.25.32 public:0.25.31 public:0.25.30 public:0.25.30-dev public:0.25.29 public:0.25.28 public:0.25.27 public:0.25.26 public:0.25.25 public:0.25.24 public:0.25.24.beta3 public:0.25.24.beta2 public:0.25.24.beta1 public:0.25.23 public:0.25.23.beta1 public:0.25.22 public:0.25.21 public:0.25.21.beta2 public:0.25.21.beta1 public:0.25.20 public:0.25.19 public:0.25.18 public:0.25.17 public:0.25.16 public:0.25.15 public:0.25.14 public:0.25.13 public:0.25.12 public:0.25.11 public:0.25.10 public:0.25.9 public:0.25.8 public:0.25.7 public:0.25.6 public:0.25.5 public:0.25.4 public:0.25.3 public:0.25.2 public:0.25.1 public:0.25.0 public:0.25.0-beta1 public:0.24.31 public:0.24.30 public:0.24.29 public:0.24.28 public:0.24.27 public:0.24.26 public:0.24.25 public:0.24.24 public:0.24.23 public:0.24.22 public:0.24.21 public:0.24.20 public:0.24.19 public:0.24.18 public:0.24.17 public:0.24.16 public:0.24.15 public:0.24.14 public:0.24.13 public:0.24.12 public:0.24.11 public:0.24.10 public:0.24.9 public:0.24.8 public:0.24.7 public:0.24.6 public:0.24.5 public:0.24.4 public:0.24.3 public:0.24.2 public:0.24.1 public:0.24.0 public:0.24.0.dev-rc8 public:0.24.0.dev-rc7 public:0.24.0.dev-rc6 public:0.24.0.dev-rc5 public:0.24.0.dev-rc4 public:0.24.0.dev-rc3 public:0.24.0.dev-rc2 public:0.23.23 public:0.23.22 public:0.23.21 public:0.23.20 public:0.23.19 public:0.23.18 public:0.23.17 public:0.23.16 public:0.23.15 public:0.23.14 public:0.23.13 public:0.23.12 public:0.23.11 public:0.23.10 public:0.23.9 public:0.23.8 public:0.23.7 public:0.23.6 public:0.23.5 public:0.23.4 public:0.23.3 public:0.23.2 public:0.23.1 public:0.23.0 public:0.22.19 public:0.22.18 public:0.22.17 public:0.22.16 public:0.22.15 public:0.22.14 public:0.22.13 public:0.22.12 public:0.22.11 public:0.22.10 public:0.22.9 public:0.22.8 public:0.22.7 public:0.22.6 public:0.22.5 public:0.22.4 public:0.22.3 public:0.22.2 public:0.22.1 public:0.22.0 public:0.21.5 public:0.21.4 public:0.21.3 public:0.21.2 public:0.21.1 public:0.21.0 public:0.20.7 public:0.20.6 public:0.20.5 public:0.20.4 public:0.20.3 public:0.20.2 public:0.20.1 public:0.20.0 public:0.19.23 public:0.19.22 public:0.19.21 public:0.19.20 public:0.19.19 public:0.19.18 public:0.19.17 public:0.19.16 public:0.19.15 public:0.19.14 public:0.19.13 public:0.19.12 public:0.19.11 public:0.19.10 public:0.19.9 public:0.19.8 public:0.19.7 public:0.19.6 public:0.19.5 public:0.19.4 public:0.19.3 public:0.19.2 public:0.19.1 public:0.19.0 public:0.18.6 public:0.18.5 public:0.18.4 public:0.18.3 public:0.18.2 public:0.18.1 public:0.18.0 public:0.17.34 public:0.17.33 public:0.17.32 public:0.17.31 public:0.17.30 public:0.17.29 public:0.17.28 public:0.17.27 public:0.17.26 public:0.17.25 public:0.17.24 public:0.17.23 public:0.17.22 public:0.17.21 public:0.17.20 public:0.17.19 public:0.17.18 public:0.17.17 public:0.17.16 public:0.17.15 public:0.17.14 public:0.17.13 public:0.17.12 public:0.17.11 public:0.17.10 public:0.17.9 public:0.17.8 public:0.17.7 public:0.17.6 public:0.17.5 public:0.17.4 public:0.17.3 public:0.17.2 public:0.17.1 public:0.17.0 public:0.16.8 public:0.16.7 public:0.16.6 public:0.16.5 public:0.16.4 public:0.16.3 public:0.16.2 public:0.16.1 public:0.16.0 public:0.15.10 public:0.15.9 public:0.15.8 public:0.15.7 public:0.15.6 public:0.15.5 public:0.15.4 public:0.15.3 public:0.15.2 public:0.15.1 public:0.15.0 public:0.14.8 public:0.14.7 public:0.14.6 public:0.14.5 public:0.14.4 public:0.14.3 public:0.14.2 public:0.14.1 public:0.14.0 public:0.13.4 public:0.13.3 public:0.13.2 public:0.13.1 public:0.13.0 public:0.12.5 public:0.12.4 public:0.12.3 public:0.12.2 public:0.12.1 public:0.12.0 public:0.11.10 public:0.11.9 public:0.11.8 public:0.11.7 public:0.11.6 public:0.11.5 public:0.11.4 public:0.11.3 public:0.11.2 public:0.11.1 public:0.11.0 public:0.10.1 public:0.10.0 public:0.9.0 public:0.8.8 public:0.8.7 public:0.8.6 public:0.8.5 public:0.8.3 public:0.8.2 public:0.8.1 public:0.8.0 public:0.7.2 public:0.7.1 public:0.7.0 public:0.6.1 public:0.6.0 public:0.5.0 public:0.4.1 public:0.4.0 public:0.3.1 public:0.3.0 public:0.2.1 public:0.2.0 public:0.1.26 public:0.1.25 public:0.1.24 public:0.1.23 public:0.1.22 public:0.1.21 public:0.1.20 public:0.1.19 public:0.1.18 public:0.1.17 public:0.1.16 public:0.1.15 public:0.1.14 public:0.1.13 public:0.1.12 public:0.1.11 public:0.1.10 public:0.1.9 public:0.1.8 public:0.1.7 public:0.1.6 public:0.1.5 public:0.1.4 public:0.1.3 public:0.1.2 public:0.1.1 public:0.1.0 public:0.0.9 public:0.0.8 public:0.0.7 public:0.0.6
...
compare: public:0.19.2
Branches Tags
public:test_real_obsidian public:main public:v0_25_62 public:update_lib public:feat-userhashsalt public:auto_conf_2 public:auto_conf public:beta public:alpha public:refactor_services_2 public:refactor_services public:dev public:svelteui public:forprivacy
public:0.25.62 public:0.25.61 public:0.25.60 public:0.25.59 public:0.25.58 public:0.25.57 public:0.25.56+patched5 public:0.25.56+patched4 public:0.25.56+patched3 public:0.25.56+patched2 public:0.25.56+patched1 public:0.25.56 public:0.25.55 public:0.25.54 public:0.25.53 public:0.25.52-patched-3 public:0.25.52-patched-2 public:0.25.52-patched-1 public:0.25.52 public:0.25.51 public:0.25.50 public:0.25.49 public:0.25.48 public:0.25.47 public:0.25.46 public:0.25.45 public:0.25.44 public:0.25.43-patched-9 public:0.25.43-patched-8 public:0.25.43-patched-7 public:0.25.43-patched-6 public:0.25.43-patched-5 public:0.25.43-patched-4 public:0.25.43-patched-3 public:0.25.43-patched-2 public:0.25.43-patched-1 public:0.25.43 public:0.25.42 public:0.25.41-patched-3 public:0.25.41-patched-2 public:0.25.41-patched-1 public:0.25.41 public:0.25.40 public:0.25.39 public:0.25.38 public:0.25.37 public:0.25.36 public:0.25.35 public:0.25.34 public:0.25.33 public:0.25.32 public:0.25.31 public:0.25.30 public:0.25.30-dev public:0.25.29 public:0.25.28 public:0.25.27 public:0.25.26 public:0.25.25 public:0.25.24 public:0.25.24.beta3 public:0.25.24.beta2 public:0.25.24.beta1 public:0.25.23 public:0.25.23.beta1 public:0.25.22 public:0.25.21 public:0.25.21.beta2 public:0.25.21.beta1 public:0.25.20 public:0.25.19 public:0.25.18 public:0.25.17 public:0.25.16 public:0.25.15 public:0.25.14 public:0.25.13 public:0.25.12 public:0.25.11 public:0.25.10 public:0.25.9 public:0.25.8 public:0.25.7 public:0.25.6 public:0.25.5 public:0.25.4 public:0.25.3 public:0.25.2 public:0.25.1 public:0.25.0 public:0.25.0-beta1 public:0.24.31 public:0.24.30 public:0.24.29 public:0.24.28 public:0.24.27 public:0.24.26 public:0.24.25 public:0.24.24 public:0.24.23 public:0.24.22 public:0.24.21 public:0.24.20 public:0.24.19 public:0.24.18 public:0.24.17 public:0.24.16 public:0.24.15 public:0.24.14 public:0.24.13 public:0.24.12 public:0.24.11 public:0.24.10 public:0.24.9 public:0.24.8 public:0.24.7 public:0.24.6 public:0.24.5 public:0.24.4 public:0.24.3 public:0.24.2 public:0.24.1 public:0.24.0 public:0.24.0.dev-rc8 public:0.24.0.dev-rc7 public:0.24.0.dev-rc6 public:0.24.0.dev-rc5 public:0.24.0.dev-rc4 public:0.24.0.dev-rc3 public:0.24.0.dev-rc2 public:0.23.23 public:0.23.22 public:0.23.21 public:0.23.20 public:0.23.19 public:0.23.18 public:0.23.17 public:0.23.16 public:0.23.15 public:0.23.14 public:0.23.13 public:0.23.12 public:0.23.11 public:0.23.10 public:0.23.9 public:0.23.8 public:0.23.7 public:0.23.6 public:0.23.5 public:0.23.4 public:0.23.3 public:0.23.2 public:0.23.1 public:0.23.0 public:0.22.19 public:0.22.18 public:0.22.17 public:0.22.16 public:0.22.15 public:0.22.14 public:0.22.13 public:0.22.12 public:0.22.11 public:0.22.10 public:0.22.9 public:0.22.8 public:0.22.7 public:0.22.6 public:0.22.5 public:0.22.4 public:0.22.3 public:0.22.2 public:0.22.1 public:0.22.0 public:0.21.5 public:0.21.4 public:0.21.3 public:0.21.2 public:0.21.1 public:0.21.0 public:0.20.7 public:0.20.6 public:0.20.5 public:0.20.4 public:0.20.3 public:0.20.2 public:0.20.1 public:0.20.0 public:0.19.23 public:0.19.22 public:0.19.21 public:0.19.20 public:0.19.19 public:0.19.18 public:0.19.17 public:0.19.16 public:0.19.15 public:0.19.14 public:0.19.13 public:0.19.12 public:0.19.11 public:0.19.10 public:0.19.9 public:0.19.8 public:0.19.7 public:0.19.6 public:0.19.5 public:0.19.4 public:0.19.3 public:0.19.2 public:0.19.1 public:0.19.0 public:0.18.6 public:0.18.5 public:0.18.4 public:0.18.3 public:0.18.2 public:0.18.1 public:0.18.0 public:0.17.34 public:0.17.33 public:0.17.32 public:0.17.31 public:0.17.30 public:0.17.29 public:0.17.28 public:0.17.27 public:0.17.26 public:0.17.25 public:0.17.24 public:0.17.23 public:0.17.22 public:0.17.21 public:0.17.20 public:0.17.19 public:0.17.18 public:0.17.17 public:0.17.16 public:0.17.15 public:0.17.14 public:0.17.13 public:0.17.12 public:0.17.11 public:0.17.10 public:0.17.9 public:0.17.8 public:0.17.7 public:0.17.6 public:0.17.5 public:0.17.4 public:0.17.3 public:0.17.2 public:0.17.1 public:0.17.0 public:0.16.8 public:0.16.7 public:0.16.6 public:0.16.5 public:0.16.4 public:0.16.3 public:0.16.2 public:0.16.1 public:0.16.0 public:0.15.10 public:0.15.9 public:0.15.8 public:0.15.7 public:0.15.6 public:0.15.5 public:0.15.4 public:0.15.3 public:0.15.2 public:0.15.1 public:0.15.0 public:0.14.8 public:0.14.7 public:0.14.6 public:0.14.5 public:0.14.4 public:0.14.3 public:0.14.2 public:0.14.1 public:0.14.0 public:0.13.4 public:0.13.3 public:0.13.2 public:0.13.1 public:0.13.0 public:0.12.5 public:0.12.4 public:0.12.3 public:0.12.2 public:0.12.1 public:0.12.0 public:0.11.10 public:0.11.9 public:0.11.8 public:0.11.7 public:0.11.6 public:0.11.5 public:0.11.4 public:0.11.3 public:0.11.2 public:0.11.1 public:0.11.0 public:0.10.1 public:0.10.0 public:0.9.0 public:0.8.8 public:0.8.7 public:0.8.6 public:0.8.5 public:0.8.3 public:0.8.2 public:0.8.1 public:0.8.0 public:0.7.2 public:0.7.1 public:0.7.0 public:0.6.1 public:0.6.0 public:0.5.0 public:0.4.1 public:0.4.0 public:0.3.1 public:0.3.0 public:0.2.1 public:0.2.0 public:0.1.26 public:0.1.25 public:0.1.24 public:0.1.23 public:0.1.22 public:0.1.21 public:0.1.20 public:0.1.19 public:0.1.18 public:0.1.17 public:0.1.16 public:0.1.15 public:0.1.14 public:0.1.13 public:0.1.12 public:0.1.11 public:0.1.10 public:0.1.9 public:0.1.8 public:0.1.7 public:0.1.6 public:0.1.5 public:0.1.4 public:0.1.3 public:0.1.2 public:0.1.1 public:0.1.0 public:0.0.9 public:0.0.8 public:0.0.7 public:0.0.6

25 Commits
0.17.34 ... 0.19.2

Author SHA1 Message Date
vorotamoroz
7c06750d93 bump 2023-05-02 18:00:55 +09:00
vorotamoroz
808fdc0944 Fixed: 
- Fixed garbage collection error while unreferenced chunks exist many.
- Fixed filename validation on Linux.

Improved:
- Showing status is now thinned for performance.
- Enhance caching while collecting chunks.
 2023-05-02 17:59:58 +09:00
vorotamoroz
ce25eee74b bump 2023-04-30 11:31:09 +09:00
vorotamoroz
146c170dec Fixed: 
- Fixed hidden file handling on Linux

Improved:
- Now customization sync works more smoothly.
 2023-04-30 11:28:39 +09:00
vorotamoroz
cf06f878db bump 2023-04-28 14:24:37 +09:00
vorotamoroz
e77031f1cd Implemented: 
- New feature `Customization sync` has replaced `Plugin and their settings`
 2023-04-28 13:32:58 +09:00
vorotamoroz
3f2224c3a6 Merge pull request #203 from garlic-hub/garlic-hub-patch-1 
Update quick_setup.md
 2023-04-21 17:15:31 +09:00
garlic-hub
2322b5bc34 Update quick_setup.md 2023-04-20 21:30:56 +00:00
vorotamoroz
83ac5e7086 bump 2023-04-14 17:39:37 +09:00
vorotamoroz
09f35a2af4 New features: 
- Now remote database cleaning-up will be detected automatically.
- A solution selection dialogue will be shown if synchronisation is rejected after cleaning or rebuilding the remote database.
- During fetching or rebuilding, we can configure `Hidden file synchronisation` on the spot.
 2023-04-14 17:39:09 +09:00
vorotamoroz
fae0a9d76a bump 2023-04-13 17:33:28 +09:00
vorotamoroz
9a27c9bfe5 - Actions for maintaining databases moved to the 🎛️Maintain databases. 
- Clean-up of unreferenced chunks has been implemented on an **experimental**.
 2023-04-13 17:33:17 +09:00
vorotamoroz
5e75917b8d bump 2023-04-12 12:08:35 +09:00
vorotamoroz
3322d13b55 - Fixed: 
- `Fetch` and `Rebuild database` will work more safely.
- Case-sensitive renaming now works fine.
  Revoked the logic which was made at #130, however, looks fine now.
 2023-04-12 12:08:08 +09:00
vorotamoroz
851c9f8a71 Pop-up is now correctly shown after hidden file synchronisation. 2023-04-11 12:54:20 +09:00
vorotamoroz
b02596dfa1 bump 2023-04-11 12:45:40 +09:00
vorotamoroz
02c69b202e Improved: 
- The setting pane refined.
- We can enable `hidden files sync` with several initial behaviours: `Merge`, `Fetch` remote, and `Overwrite` remote.
- No longer `Touch hidden files`
 2023-04-11 12:45:24 +09:00
vorotamoroz
6b2c7b56a5 add note. 2023-04-10 15:18:32 +09:00
vorotamoroz
820168a5ab bump. 2023-04-10 15:15:20 +09:00
vorotamoroz
40015642e4 Fixed 
- fixed type annotation
- update lib
 2023-04-10 15:14:47 +09:00
vorotamoroz
7a5cffb6a8 bump 2023-04-10 12:07:03 +09:00
vorotamoroz
e395e53248 Implemented: 
- Explicit types
- Path obfuscation.
- ... and minor changes.
 2023-04-10 12:04:30 +09:00
vorotamoroz
97f91b1eb0 Update README.md 2023-03-30 18:23:52 +09:00
vorotamoroz
2f4159182e Update README.md 2023-03-30 18:23:30 +09:00
vorotamoroz
302a4024a8 Update README.md 2023-03-30 18:22:49 +09:00
27 changed files with 2702 additions and 1182 deletions
Expand all files Collapse all files

27
.eslintrc
View File

@@ -1,19 +1,34 @@
{ {
"root": true, "root": true,
"parser": "@typescript-eslint/parser", "parser": "@typescript-eslint/parser",
"plugins": ["@typescript-eslint"], "plugins": [
"extends": ["eslint:recommended", "plugin:@typescript-eslint/eslint-recommended", "plugin:@typescript-eslint/recommended"], "@typescript-eslint"
],
"extends": [
"eslint:recommended",
"plugin:@typescript-eslint/eslint-recommended",
"plugin:@typescript-eslint/recommended"
],
"parserOptions": { "parserOptions": {
"sourceType": "module" "sourceType": "module",
"project": [
"tsconfig.json"
]
}, },
"rules": { "rules": {
"no-unused-vars": "off", "no-unused-vars": "off",
"@typescript-eslint/no-unused-vars": ["error", { "args": "none" }], "@typescript-eslint/no-unused-vars": [
"error",
{
"args": "none"
}
],
"@typescript-eslint/ban-ts-comment": "off", "@typescript-eslint/ban-ts-comment": "off",
"no-prototype-builtins": "off", "no-prototype-builtins": "off",
"@typescript-eslint/no-empty-function": "off", "@typescript-eslint/no-empty-function": "off",
"require-await": "warn", "require-await": "warn",
"no-async-promise-executor": "off", "no-async-promise-executor": "off",
"@typescript-eslint/no-explicit-any": "off" "@typescript-eslint/no-explicit-any": "off",
"@typescript-eslint/no-unnecessary-type-assertion": "error"
} }
} }

3
README.md
View File

@@ -56,7 +56,8 @@ If you answered `No` to both, your databases will be rebuilt by the content on y
## Test Server ## Test Server
Setting up an instance of Cloudant or local CouchDB is a little complicated, so I set up a [Tasting server for self-hosted-livesync](https://olstaste.vrtmrz.net/). Try it out for free! ~~Setting up an instance of Cloudant or local CouchDB is a little complicated, so I set up a [Tasting server for self-hosted-livesync](https://olstaste.vrtmrz.net/). Try it out for free!~~
Now (30 May 2023) suspending while the server transfer.
Note: Please read "Limitations" carefully. Do not send your private vault. Note: Please read "Limitations" carefully. Do not send your private vault.
## Information in StatusBar ## Information in StatusBar

30
docs/quick_setup.md
View File

@@ -1,32 +1,32 @@
# Quick setup # Quick setup
The Setup wizard has been implemented since v0.15.0. This simplifies the initial setup. The Setup wizard has been implemented since v0.15.0 simplifying the initial setup.
Note: The subsequent devices should be set up using the `Copy setup URI` and `Open setup URI`. Note: Subsequent devices should be set up using the `Copy setup URI` and `Open setup URI` functionality.
## How to open and use wizard ## How to open and use the wizard
Open from `🪄 Setup wizard` in the setting dialogue. If there is no configuration or no synchronisation settings have been activated, it should already be open. Open the `🪄 Setup wizard` in the settings dialogue. If the plugin has not been configured before, it should already be open.
![](../images/quick_setup_1.png) ![](../images/quick_setup_1.png)
### Discard the existing configuration and set up ### Discard the existing configuration and set up
If you have made any settings, this button allows you to discard them all before setting up. If you have changed any settings, this button allows you to discard all changes before setting up.
### Do not discard the existing configuration and set up ### Do not discard the existing configuration and set up
Simply reconfigure. Be careful. In wizard mode, you cannot see all configuration items, even if they have been configured. Simply reconfigure. Be careful. In wizard mode, you cannot see all configuration items, even if they have been configured.
Pressing `Next` on any of these will put the configuration dialog into wizard mode. Pressing `Next` on one of the above options will put the configuration dialog into wizard mode.
### Wizard mode ### Wizard mode
![](../images/quick_setup_2.png) ![](../images/quick_setup_2.png)
We can set it up step by step. Let's see how to use it step-by-step.
## Remote Database configuration ## Remote Database configuration
### Remote database configuration ### Remote database configuration
Enter the information in the database we have set up. Enter the information for the database we have set up.
![](../images/quick_setup_3.png) ![](../images/quick_setup_3.png)
@@ -34,7 +34,7 @@ Enter the information in the database we have set up.
![](../images/quick_setup_4.png) ![](../images/quick_setup_4.png)
If End to End encryption is enabled, the possibility of a third party who does not know the Passphrase being able to read the contents of the Remote database if they are leaked is reduced. So we strongly recommend enabling it. When End to End encryption is enabled, a third party will be less likely to be able to read your Remote database in the event of a data breach/leak (assuming they do not know the Passphrase). We strongly recommend enabling it.
Encryption is based on 256-bit AES-GCM. Encryption is based on 256-bit AES-GCM.
This setting can be disabled if you are inside a closed network and it is clear that you will not be accessed by third parties. This setting can be disabled if you are inside a closed network and it is clear that you will not be accessed by third parties.
@@ -45,7 +45,7 @@ Here we can check the status of the connection to the database and the database
![](../images/quick_setup_5.png) ![](../images/quick_setup_5.png)
#### Test Database Connection #### Test Database Connection
Check whether we can connect to the database. If it fails, there are several reasons, but once you have done the `Check database configuration`, check if it fails there too. Check whether we can connect to the database. If it fails, there are several possible reasons, but first attempt the `Check database configuration` check to see if it fails there too.
#### Check database configuration #### Check database configuration
@@ -61,21 +61,21 @@ If the Fix buttons disappear and all become check marks, we are done.
### Next ### Next
Go to the Local Database configuration. Go to the Local Database configuration.
### Discard exist database and proceed ### Discard existing database and proceed
Discard the contents of the Remote database and go to the Local Database configuration. Discard the contents of the Remote database and go to the Local Database configuration.
## Local Database configuration ## Local Database configuration
![](../images/quick_setup_8.png) ![](../images/quick_setup_8.png)
Configure the local database. If we already have a Vaults with Self-hosted LiveSync installed and having the same directory name as currently we are setting up, please specify a different suffix than the Vault you have already set up here. Configure the local database. If we already have a Vault with Self-hosted LiveSync installed which has the same directory name as the one we are currently setting up, please specify a different suffix than the Vault you have already set up here.
## Miscellaneous ## Miscellaneous
Finally, finish the miscellaneous configurations and select a preset for synchronisation. Finally, finish the miscellaneous configurations and select a preset for synchronisation.
![](../images/quick_setup_9_1.png) ![](../images/quick_setup_9_1.png)
The `Show status inside editor` can be enabled to your liking. If enabled, the status is displayed in the top right-hand corner of the editor. The `Show status inside editor` can be enabled to your liking. If enabled, the status is displayed in the top right-hand corner of the editor. Learn how to read the status bar [here](/README.md#information-in-statusbar).
![](../images/quick_setup_9_2.png) ![](../images/quick_setup_9_2.png)
@@ -88,10 +88,10 @@ Set the passphrase as you like.
The Setup URI will be copied to the clipboard, which you can then transfer to the second and subsequent devices in some way. The Setup URI will be copied to the clipboard, which you can then transfer to the second and subsequent devices in some way.
# How to set up the second and subsequent units # How to set up the second and subsequent units
After installing Self-hosted LiveSync on the device, select `Open setup URI` from the command palette and enter the setup URI you transferred. Afterwards, enter your passphrase and a setup wizard will open. After installing Self-hosted LiveSync on the first device, select `Open setup URI` from the command palette and enter the setup URI you transferred. Afterwards, enter your passphrase and a setup wizard will open.
Answer the following. Answer the following.
- `Yes` to `Importing LiveSync's conf, OK?` - `Yes` to `Importing LiveSync's conf, OK?`
- `Set it up as secondary or subsequent device` to `How would you like to set it up?`. - `Set it up as secondary or subsequent device` to `How would you like to set it up?`.
Then, The configuration will now take effect and replication will start. Your files will be synchronised soon! Then, The configuration will take effect and replication will start. Your files will be synchronised soon! You may need to close the settings dialog and reopen it to see the settings fields populated properly, but they will be set.

2
manifest.json
View File

@@ -1,7 +1,7 @@
{ {
"id": "obsidian-livesync", "id": "obsidian-livesync",
"name": "Self-hosted LiveSync", "name": "Self-hosted LiveSync",
"version": "0.17.34", "version": "0.19.2",
"minAppVersion": "0.9.12", "minAppVersion": "0.9.12",
"description": "Community implementation of self-hosted livesync. Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.", "description": "Community implementation of self-hosted livesync. Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.",
"author": "vorotamoroz", "author": "vorotamoroz",

4
package-lock.json generated
View File

@@ -1,12 +1,12 @@
{ {
"name": "obsidian-livesync", "name": "obsidian-livesync",
"version": "0.17.34", "version": "0.19.2",
"lockfileVersion": 2, "lockfileVersion": 2,
"requires": true, "requires": true,
"packages": { "packages": {
"": { "": {
"name": "obsidian-livesync", "name": "obsidian-livesync",
"version": "0.17.34", "version": "0.19.2",
"license": "MIT", "license": "MIT",
"dependencies": { "dependencies": {
"diff-match-patch": "^1.0.5", "diff-match-patch": "^1.0.5",

2
package.json
View File

@@ -1,6 +1,6 @@
{ {
"name": "obsidian-livesync", "name": "obsidian-livesync",
"version": "0.17.34", "version": "0.19.2",
"description": "Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.", "description": "Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.",
"main": "main.js", "main": "main.js",
"type": "module", "type": "module",

668
src/CmdConfigSync.ts Normal file
View File

@@ -0,0 +1,668 @@
import { writable } from 'svelte/store';
import { Notice, PluginManifest, stringifyYaml, parseYaml } from "./deps";
import { EntryDoc, LoadedEntry, LOG_LEVEL, InternalFileEntry, FilePathWithPrefix, FilePath, DocumentID } from "./lib/src/types";
import { ICXHeader, PERIODIC_PLUGIN_SWEEP, } from "./types";
import { delay, getDocData } from "./lib/src/utils";
import { Logger } from "./lib/src/logger";
import { PouchDB } from "./lib/src/pouchdb-browser.js";
import { WrappedNotice } from "./lib/src/wrapper";
import { base64ToArrayBuffer, arrayBufferToBase64, readString, writeString, uint8ArrayToHexString } from "./lib/src/strbin";
import { runWithLock } from "./lib/src/lock";
import { LiveSyncCommands } from "./LiveSyncCommands";
import { stripAllPrefixes } from "./lib/src/path";
import { PeriodicProcessor, askYesNo, disposeMemoObject, memoIfNotExist, memoObject, retrieveMemoObject, scheduleTask } from "./utils";
import { Semaphore } from "./lib/src/semaphore";
import { PluginDialogModal } from "./dialogs";
import { JsonResolveModal } from "./JsonResolveModal";
export const pluginList = writable([] as PluginDataExDisplay[]);
export const pluginIsEnumerating = writable(false);
const hashString = (async (key: string) => {
const buff = writeString(key);
const digest = await crypto.subtle.digest('SHA-256', buff);
return uint8ArrayToHexString(new Uint8Array(digest));
})
export type PluginDataExFile = {
filename: string,
data?: string[],
mtime: number,
size: number,
version?: string,
displayName?: string,
}
export type PluginDataExDisplay = {
documentPath: FilePathWithPrefix,
category: string,
name: string,
term: string,
displayName?: string,
files: PluginDataExFile[],
version?: string,
mtime: number,
}
export type PluginDataEx = {
documentPath?: FilePathWithPrefix,
category: string,
name: string,
displayName?: string,
term: string,
files: PluginDataExFile[],
version?: string,
mtime: number,
};
export class ConfigSync extends LiveSyncCommands {
confirmPopup: WrappedNotice = null;
get kvDB() {
return this.plugin.kvDB;
}
ensureDirectoryEx(fullPath: string) {
return this.plugin.ensureDirectoryEx(fullPath);
}
pluginDialog: PluginDialogModal = null;
periodicPluginSweepProcessor = new PeriodicProcessor(this.plugin, async () => await this.scanAllConfigFiles(false));
pluginList: PluginDataExDisplay[] = [];
showPluginSyncModal() {
if (!this.settings.usePluginSync) {
return;
}
if (this.pluginDialog != null) {
this.pluginDialog.open();
} else {
this.pluginDialog = new PluginDialogModal(this.app, this.plugin);
this.pluginDialog.open();
}
}
hidePluginSyncModal() {
if (this.pluginDialog != null) {
this.pluginDialog.close();
this.pluginDialog = null;
}
}
onunload() {
this.hidePluginSyncModal();
this.periodicPluginSweepProcessor?.disable();
}
onload() {
this.plugin.addCommand({
id: "livesync-plugin-dialog-ex",
name: "Show customization sync dialog",
callback: () => {
this.showPluginSyncModal();
},
});
}
getFileCategory(filePath: string): "CONFIG" | "THEME" | "SNIPPET" | "PLUGIN_MAIN" | "PLUGIN_ETC" | "PLUGIN_DATA" | "" {
if (filePath.split("/").length == 2 && filePath.endsWith(".json")) return "CONFIG";
if (filePath.split("/").length == 4 && filePath.startsWith(`${this.app.vault.configDir}/themes/`)) return "THEME";
if (filePath.startsWith(`${this.app.vault.configDir}/snippets/`) && filePath.endsWith(".css")) return "SNIPPET";
if (filePath.startsWith(`${this.app.vault.configDir}/plugins/`)) {
if (filePath.endsWith("/styles.css") || filePath.endsWith("/manifest.json") || filePath.endsWith("/main.js")) {
return "PLUGIN_MAIN";
} else if (filePath.endsWith("/data.json")) {
return "PLUGIN_DATA";
} else {
//TODO: to be configurable.
// With algorithm which implemented at v0.19.0, is too heavy.
return "";
// return "PLUGIN_ETC";
}
// return "PLUGIN";
}
return "";
}
isTargetPath(filePath: string): boolean {
if (!filePath.startsWith(this.app.vault.configDir)) return false;
// Idea non-filter option?
return this.getFileCategory(filePath) != "";
}
async onInitializeDatabase(showNotice: boolean) {
if (this.settings.usePluginSync) {
try {
Logger("Scanning customizations...");
await this.scanAllConfigFiles(showNotice);
Logger("Scanning customizations : done");
} catch (ex) {
Logger("Scanning customizations : failed");
Logger(ex, LOG_LEVEL.VERBOSE);
}
}
}
async beforeReplicate(showNotice: boolean) {
if (this.settings.autoSweepPlugins && this.settings.usePluginSync) {
await this.scanAllConfigFiles(showNotice);
}
}
async onResume() {
if (this.plugin.suspended) {
return;
}
if (this.settings.autoSweepPlugins) {
await this.scanAllConfigFiles(false);
}
this.periodicPluginSweepProcessor.enable(this.settings.autoSweepPluginsPeriodic && !this.settings.watchInternalFileChanges ? (PERIODIC_PLUGIN_SWEEP * 1000) : 0);
}
async reloadPluginList(showMessage: boolean) {
this.pluginList = [];
pluginList.set(this.pluginList)
await this.updatePluginList(showMessage);
}
async updatePluginList(showMessage: boolean, updatedDocumentPath?: FilePathWithPrefix): Promise<void> {
const logLevel = showMessage ? LOG_LEVEL.NOTICE : LOG_LEVEL.INFO;
// pluginList.set([]);
if (!this.settings.usePluginSync) {
this.pluginList = [];
pluginList.set(this.pluginList)
return;
}
await runWithLock("update-plugin-list", false, async () => {
// if (updatedDocumentPath != "") pluginList.update(e => e.filter(ee => ee.documentPath != updatedDocumentPath));
// const work: Record<string, Record<string, Record<string, Record<string, PluginDataEntryEx>>>> = {};
const entries = [] as PluginDataExDisplay[]
const plugins = this.localDatabase.findEntries(ICXHeader + "", `${ICXHeader}\u{10ffff}`, { include_docs: true });
const semaphore = Semaphore(4);
const processes = [] as Promise<void>[];
let count = 0;
pluginIsEnumerating.set(true);
let processed = false;
try {
for await (const plugin of plugins) {
const path = plugin.path || this.getPath(plugin);
if (updatedDocumentPath && updatedDocumentPath != path) {
continue;
}
processed = true;
const oldEntry = (this.pluginList.find(e => e.documentPath == path));
if (oldEntry && oldEntry.mtime == plugin.mtime) continue;
processes.push((async (v) => {
const release = await semaphore.acquire(1);
try {
Logger(`Enumerating files... ${count++}`, logLevel, "get-plugins");
Logger(`plugin-${path}`, LOG_LEVEL.VERBOSE);
const wx = await this.localDatabase.getDBEntry(path, null, false, false);
if (wx) {
const data = parseYaml(getDocData(wx.data)) as PluginDataEx;
const xFiles = [] as PluginDataExFile[];
for (const file of data.files) {
const work = { ...file };
const tempStr = getDocData(work.data);
work.data = [await hashString(tempStr)];
xFiles.push(work);
}
entries.push({
...data,
documentPath: this.getPath(wx),
files: xFiles
});
}
} catch (ex) {
//TODO
Logger(`Something happened at enumerating customization :${v.path}`, LOG_LEVEL.NOTICE);
console.warn(ex);
} finally {
release();
}
}
)(plugin));
}
await Promise.all(processes);
let newList = [...this.pluginList];
for (const item of entries) {
newList = newList.filter(x => x.documentPath != item.documentPath);
newList.push(item)
}
if (updatedDocumentPath != "" && !processed) newList = newList.filter(e => e.documentPath != updatedDocumentPath);
this.pluginList = newList;
pluginList.set(newList);
Logger(`All files enumerated`, logLevel, "get-plugins");
} finally {
pluginIsEnumerating.set(false);
}
});
// return entries;
}
async compareUsingDisplayData(dataA: PluginDataExDisplay, dataB: PluginDataExDisplay) {
const docA = await this.localDatabase.getDBEntry(dataA.documentPath);
const docB = await this.localDatabase.getDBEntry(dataB.documentPath);
if (docA && docB) {
const pluginDataA = parseYaml(getDocData(docA.data)) as PluginDataEx;
pluginDataA.documentPath = dataA.documentPath;
const pluginDataB = parseYaml(getDocData(docB.data)) as PluginDataEx;
pluginDataB.documentPath = dataB.documentPath;
// Use outer structure to wrap each data.
return await this.showJSONMergeDialogAndMerge(docA, docB, pluginDataA, pluginDataB);
}
return false;
}
showJSONMergeDialogAndMerge(docA: LoadedEntry, docB: LoadedEntry, pluginDataA: PluginDataEx, pluginDataB: PluginDataEx): Promise<boolean> {
const fileA = { ...pluginDataA.files[0], ctime: pluginDataA.files[0].mtime, _id: `${pluginDataA.documentPath}` as DocumentID };
const fileB = pluginDataB.files[0];
const docAx = { ...docA, ...fileA } as LoadedEntry, docBx = { ...docB, ...fileB } as LoadedEntry
return runWithLock("config:merge-data", false, () => new Promise((res) => {
Logger("Opening data-merging dialog", LOG_LEVEL.VERBOSE);
// const docs = [docA, docB];
const path = stripAllPrefixes(docAx.path.split("/").slice(-1).join("/") as FilePath);
const modal = new JsonResolveModal(this.app, path, [docAx, docBx], async (keep, result) => {
if (result == null) return res(false);
try {
res(await this.applyData(pluginDataA, result));
} catch (ex) {
Logger("Could not apply merged file");
Logger(ex, LOG_LEVEL.VERBOSE);
res(false);
}
}, "📡", "🛰️", "B");
modal.open();
}));
}
async applyData(data: PluginDataEx, content?: string): Promise<boolean> {
Logger(`Applying ${data.displayName || data.name}..`);
const baseDir = this.app.vault.configDir;
try {
if (!data.documentPath) throw "InternalError: Document path not exist";
const dx = await this.localDatabase.getDBEntry(data.documentPath);
if (dx == false) {
throw "Not found on database"
}
const loadedData = parseYaml(getDocData(dx.data)) as PluginDataEx;
for (const f of loadedData.files) {
Logger(`Applying ${f.filename} of ${data.displayName || data.name}..`);
try {
// console.dir(f);
const path = `${baseDir}/${f.filename}`;
await this.ensureDirectoryEx(path);
if (!content) {
const dt = base64ToArrayBuffer(f.data);
await this.app.vault.adapter.writeBinary(path, dt);
} else {
await this.app.vault.adapter.write(path, content);
}
Logger(`Applying ${f.filename} of ${data.displayName || data.name}.. Done`);
} catch (ex) {
Logger(`Applying ${f.filename} of ${data.displayName || data.name}.. Failed`);
Logger(ex, LOG_LEVEL.VERBOSE);
}
}
const uPath = `${baseDir}/${loadedData.files[0].filename}` as FilePath;
await this.storeCustomizationFiles(uPath);
await this.updatePluginList(true, uPath);
await delay(100);
Logger(`Config ${data.displayName || data.name} has been applied`, LOG_LEVEL.NOTICE);
if (data.category == "PLUGIN_DATA" || data.category == "PLUGIN_MAIN") {
//@ts-ignore
const manifests = Object.values(this.app.plugins.manifests) as any as PluginManifest[];
//@ts-ignore
const enabledPlugins = this.app.plugins.enabledPlugins as Set<string>;
const pluginManifest = manifests.find((manifest) => enabledPlugins.has(manifest.id) && manifest.dir == `${baseDir}/plugins/${data.name}`);
if (pluginManifest) {
Logger(`Unloading plugin: ${pluginManifest.name}`, LOG_LEVEL.NOTICE, "plugin-reload-" + pluginManifest.id);
// @ts-ignore
await this.app.plugins.unloadPlugin(pluginManifest.id);
// @ts-ignore
await this.app.plugins.loadPlugin(pluginManifest.id);
Logger(`Plugin reloaded: ${pluginManifest.name}`, LOG_LEVEL.NOTICE, "plugin-reload-" + pluginManifest.id);
}
} else if (data.category == "CONFIG") {
scheduleTask("configReload", 250, async () => {
if (await askYesNo(this.app, "Do you want to restart and reload Obsidian now?") == "yes") {
// @ts-ignore
this.app.commands.executeCommandById("app:reload")
}
})
}
return true;
} catch (ex) {
Logger(`Applying ${data.displayName || data.name}.. Failed`);
Logger(ex, LOG_LEVEL.VERBOSE);
return false;
}
}
async deleteData(data: PluginDataEx): Promise<boolean> {
try {
if (data.documentPath) {
await this.deleteConfigOnDatabase(data.documentPath);
await this.updatePluginList(false, data.documentPath);
Logger(`Delete: ${data.documentPath}`, LOG_LEVEL.NOTICE);
}
return true;
} catch (ex) {
Logger(`Failed to delete: ${data.documentPath}`, LOG_LEVEL.NOTICE);
return false;
}
}
async parseReplicationResultItem(docs: PouchDB.Core.ExistingDocument<EntryDoc>) {
if (docs._id.startsWith(ICXHeader)) {
if (this.plugin.settings.usePluginSync) {
await this.updatePluginList(false, docs.path ? docs.path : this.getPath(docs));
}
if (this.plugin.settings.usePluginSync && this.plugin.settings.notifyPluginOrSettingUpdated) {
if (!this.pluginDialog || (this.pluginDialog && !this.pluginDialog.isOpened())) {
const fragment = createFragment((doc) => {
doc.createEl("span", null, (a) => {
a.appendText(`Some configuration has been arrived, Press `);
a.appendChild(a.createEl("a", null, (anchor) => {
anchor.text = "HERE";
anchor.addEventListener("click", () => {
this.showPluginSyncModal();
});
}));
a.appendText(` to open the config sync dialog , or press elsewhere to dismiss this message.`);
});
});
const updatedPluginKey = "popupUpdated-plugins";
scheduleTask(updatedPluginKey, 1000, async () => {
const popup = await memoIfNotExist(updatedPluginKey, () => new Notice(fragment, 0));
//@ts-ignore
const isShown = popup?.noticeEl?.isShown();
if (!isShown) {
memoObject(updatedPluginKey, new Notice(fragment, 0));
}
scheduleTask(updatedPluginKey + "-close", 20000, () => {
const popup = retrieveMemoObject<Notice>(updatedPluginKey);
if (!popup)
return;
//@ts-ignore
if (popup?.noticeEl?.isShown()) {
popup.hide();
}
disposeMemoObject(updatedPluginKey);
});
});
}
}
return true;
}
return false;
}
async realizeSettingSyncMode(): Promise<void> {
this.periodicPluginSweepProcessor?.disable();
if (this.plugin.suspended)
return;
if (!this.settings.usePluginSync) {
return;
}
if (this.settings.autoSweepPlugins) {
await this.scanAllConfigFiles(false);
}
this.periodicPluginSweepProcessor.enable(this.settings.autoSweepPluginsPeriodic && !this.settings.watchInternalFileChanges ? (PERIODIC_PLUGIN_SWEEP * 1000) : 0);
return;
}
recentProcessedInternalFiles = [] as string[];
async makeEntryFromFile(path: FilePath): Promise<false | PluginDataExFile> {
const stat = await this.app.vault.adapter.stat(path);
let version: string | undefined;
let displayName: string | undefined;
if (!stat) {
return false;
}
const contentBin = await this.app.vault.adapter.readBinary(path);
let content: string[];
try {
content = await arrayBufferToBase64(contentBin);
if (path.toLowerCase().endsWith("/manifest.json")) {
const v = readString(new Uint8Array(contentBin));
try {
const json = JSON.parse(v);
if ("version" in json) {
version = `${json.version}`;
}
if ("name" in json) {
displayName = `${json.name}`;
}
} catch (ex) {
Logger(`Configuration sync data: ${path} looks like manifest, but could not read the version`, LOG_LEVEL.INFO);
}
}
} catch (ex) {
Logger(`The file ${path} could not be encoded`);
Logger(ex, LOG_LEVEL.VERBOSE);
return false;
}
const mtime = stat.mtime;
return {
filename: path.substring(this.app.vault.configDir.length + 1),
data: content,
mtime,
size: stat.size,
version,
displayName: displayName,
}
}
filenameToUnifiedKey(path: string, termOverRide?: string) {
const term = termOverRide || this.plugin.deviceAndVaultName;
const category = this.getFileCategory(path);
const name = (category == "CONFIG" || category == "SNIPPET") ?
(path.split("/").slice(-1)[0]) :
(category == "PLUGIN_ETC" ?
path.split("/").slice(-2).join("/") :
path.split("/").slice(-2)[0]);
return `${ICXHeader}${term}/${category}/${name}.md` as FilePathWithPrefix
}
async storeCustomizationFiles(path: FilePath, termOverRide?: string) {
const term = termOverRide || this.plugin.deviceAndVaultName;
const vf = this.filenameToUnifiedKey(path, term);
return await runWithLock(`plugin-${vf}`, false, async () => {
const category = this.getFileCategory(path);
let mtime = 0;
let fileTargets = [] as FilePath[];
// let savePath = "";
const name = (category == "CONFIG" || category == "SNIPPET") ?
(path.split("/").reverse()[0]) :
(path.split("/").reverse()[1]);
const parentPath = path.split("/").slice(0, -1).join("/");
const prefixedFileName = this.filenameToUnifiedKey(path, term);
const id = await this.path2id(prefixedFileName);
const dt: PluginDataEx = {
category: category,
files: [],
name: name,
mtime: 0,
term: term
}
// let scheduleKey = "";
if (category == "CONFIG" || category == "SNIPPET" || category == "PLUGIN_ETC" || category == "PLUGIN_DATA") {
fileTargets = [path];
if (category == "PLUGIN_ETC") {
dt.displayName = path.split("/").slice(-1).join("/");
}
} else if (category == "PLUGIN_MAIN") {
fileTargets = ["manifest.json", "main.js", "styles.css"].map(e => `${parentPath}/${e}` as FilePath);
} else if (category == "THEME") {
fileTargets = ["manifest.json", "theme.css"].map(e => `${parentPath}/${e}` as FilePath);
}
for (const target of fileTargets) {
const data = await this.makeEntryFromFile(target);
if (data == false) {
Logger(`Config: skipped: ${target} `, LOG_LEVEL.VERBOSE);
continue;
}
if (data.version) {
dt.version = data.version;
}
if (data.displayName) {
dt.displayName = data.displayName;
}
// Use average for total modified time.
mtime = mtime == 0 ? data.mtime : ((data.mtime + mtime) / 2);
dt.files.push(data);
}
dt.mtime = mtime;
// Logger(`Configuration saving: ${prefixedFileName}`);
if (dt.files.length == 0) {
Logger(`Nothing left: deleting.. ${path}`);
await this.deleteConfigOnDatabase(prefixedFileName);
await this.updatePluginList(false, prefixedFileName);
return
}
const content = stringifyYaml(dt);
try {
const old = await this.localDatabase.getDBEntryMeta(prefixedFileName, null, false);
let saveData: LoadedEntry;
if (old === false) {
saveData = {
_id: id,
path: prefixedFileName,
data: content,
mtime,
ctime: mtime,
datatype: "newnote",
size: content.length,
children: [],
deleted: false,
type: "newnote",
};
} else {
if (old.mtime == mtime) {
// Logger(`STORAGE --> DB:${file.path}: (hidden) Not changed`, LOG_LEVEL.VERBOSE);
return true;
}
saveData =
{
...old,
data: content,
mtime,
size: content.length,
datatype: "newnote",
children: [],
deleted: false,
type: "newnote",
};
}
const ret = await this.localDatabase.putDBEntry(saveData);
await this.updatePluginList(false, saveData.path);
Logger(`STORAGE --> DB:${prefixedFileName}: (config) Done`);
return ret;
} catch (ex) {
Logger(`STORAGE --> DB:${prefixedFileName}: (config) Failed`);
Logger(ex, LOG_LEVEL.VERBOSE);
return false;
}
})
}
async watchVaultRawEventsAsync(path: FilePath) {
if (!this.isTargetPath(path)) return false;
const stat = await this.app.vault.adapter.stat(path);
// Make sure that target is a file.
if (stat && stat.type != "file")
return false;
const storageMTime = ~~((stat && stat.mtime || 0) / 1000);
const key = `${path}-${storageMTime}`;
if (this.recentProcessedInternalFiles.contains(key)) {
// If recently processed, it may caused by self.
return true;
}
this.recentProcessedInternalFiles = [key, ...this.recentProcessedInternalFiles].slice(0, 100);
this.storeCustomizationFiles(path).then(() => {/* Fire and forget */ });
}
async scanAllConfigFiles(showMessage: boolean) {
const logLevel = showMessage ? LOG_LEVEL.NOTICE : LOG_LEVEL.INFO;
Logger("Scanning customizing files.", logLevel, "scan-all-config");
const term = this.plugin.deviceAndVaultName;
if (term == "") {
Logger("We have to configure the device name", LOG_LEVEL.NOTICE);
return;
}
const filesAll = await this.scanInternalFiles();
const files = filesAll.filter(e => this.isTargetPath(e)).map(e => ({ key: this.filenameToUnifiedKey(e), file: e }));
const virtualPathsOfLocalFiles = [...new Set(files.map(e => e.key))];
const filesOnDB = ((await this.localDatabase.allDocsRaw({ startkey: ICXHeader + "", endkey: `${ICXHeader}\u{10ffff}`, include_docs: true })).rows.map(e => e.doc) as InternalFileEntry[]).filter(e => !e.deleted);
let deleteCandidate = filesOnDB.map(e => this.getPath(e)).filter(e => e.startsWith(`${ICXHeader}${term}/`));
for (const vp of virtualPathsOfLocalFiles) {
const p = files.find(e => e.key == vp).file;
await this.storeCustomizationFiles(p);
deleteCandidate = deleteCandidate.filter(e => e != vp);
}
for (const vp of deleteCandidate) {
await this.deleteConfigOnDatabase(vp);
}
this.updatePluginList(false).then(/* fire and forget */);
}
async deleteConfigOnDatabase(prefixedFileName: FilePathWithPrefix, forceWrite = false) {
// const id = await this.path2id(prefixedFileName);
const mtime = new Date().getTime();
await runWithLock("file-x-" + prefixedFileName, false, async () => {
try {
const old = await this.localDatabase.getDBEntryMeta(prefixedFileName, null, false) as InternalFileEntry | false;
let saveData: InternalFileEntry;
if (old === false) {
Logger(`STORAGE -x> DB:${prefixedFileName}: (config) already deleted (Not found on database)`);
} else {
if (old.deleted) {
Logger(`STORAGE -x> DB:${prefixedFileName}: (config) already deleted`);
return;
}
saveData =
{
...old,
mtime,
size: 0,
children: [],
deleted: true,
type: "newnote",
};
}
await this.localDatabase.putRaw(saveData);
await this.updatePluginList(false, prefixedFileName);
Logger(`STORAGE -x> DB:${prefixedFileName}: (config) Done`);
} catch (ex) {
Logger(`STORAGE -x> DB:${prefixedFileName}: (config) Failed`);
Logger(ex, LOG_LEVEL.VERBOSE);
return false;
}
});
}
async scanInternalFiles(): Promise<FilePath[]> {
const filenames = (await this.getFiles(this.app.vault.configDir, 2)).filter(e => e.startsWith(".")).filter(e => !e.startsWith(".trash"));
return filenames as FilePath[];
}
async getFiles(
path: string,
lastDepth: number
) {
if (lastDepth == -1) return [];
const w = await this.app.vault.adapter.list(path);
let files = [
...w.files
];
for (const v of w.folders) {
files = files.concat(await this.getFiles(v, lastDepth - 1));
}
return files;
}
}

171
src/CmdHiddenFileSync.ts
View File

@@ -1,19 +1,20 @@
import { Notice, normalizePath, PluginManifest } from "./deps"; import { Notice, normalizePath, PluginManifest } from "./deps";
import { EntryDoc, LoadedEntry, LOG_LEVEL, InternalFileEntry } from "./lib/src/types"; import { EntryDoc, LoadedEntry, LOG_LEVEL, InternalFileEntry, FilePathWithPrefix, FilePath } from "./lib/src/types";
import { InternalFileInfo, ICHeader, ICHeaderEnd } from "./types"; import { InternalFileInfo, ICHeader, ICHeaderEnd } from "./types";
import { delay, isDocContentSame } from "./lib/src/utils"; import { delay, isDocContentSame } from "./lib/src/utils";
import { Logger } from "./lib/src/logger"; import { Logger } from "./lib/src/logger";
import { PouchDB } from "./lib/src/pouchdb-browser.js"; import { PouchDB } from "./lib/src/pouchdb-browser.js";
import { disposeMemoObject, id2path, memoIfNotExist, memoObject, path2id, retrieveMemoObject, scheduleTask, trimPrefix, isInternalMetadata, filename2idInternalMetadata, id2filenameInternalMetadata, PeriodicProcessor } from "./utils"; import { disposeMemoObject, memoIfNotExist, memoObject, retrieveMemoObject, scheduleTask, isInternalMetadata, PeriodicProcessor } from "./utils";
import { WrappedNotice } from "./lib/src/wrapper"; import { WrappedNotice } from "./lib/src/wrapper";
import { base64ToArrayBuffer, arrayBufferToBase64 } from "./lib/src/strbin"; import { base64ToArrayBuffer, arrayBufferToBase64 } from "./lib/src/strbin";
import { runWithLock } from "./lib/src/lock"; import { runWithLock } from "./lib/src/lock";
import { Semaphore } from "./lib/src/semaphore"; import { Semaphore } from "./lib/src/semaphore";
import { JsonResolveModal } from "./JsonResolveModal"; import { JsonResolveModal } from "./JsonResolveModal";
import { LiveSyncCommands } from "./LiveSyncCommands"; import { LiveSyncCommands } from "./LiveSyncCommands";
import { addPrefix, stripAllPrefixes } from "./lib/src/path";
export class HiddenFileSync extends LiveSyncCommands { export class HiddenFileSync extends LiveSyncCommands {
periodicInternalFileScanProcessor: PeriodicProcessor = new PeriodicProcessor(this.plugin, async () => await this.syncInternalFilesAndDatabase("push", false)); periodicInternalFileScanProcessor: PeriodicProcessor = new PeriodicProcessor(this.plugin, async () => this.settings.syncInternalFiles && this.localDatabase.isReady && await this.syncInternalFilesAndDatabase("push", false));
confirmPopup: WrappedNotice = null; confirmPopup: WrappedNotice = null;
get kvDB() { get kvDB() {
return this.plugin.kvDB; return this.plugin.kvDB;
@@ -21,13 +22,13 @@ export class HiddenFileSync extends LiveSyncCommands {
ensureDirectoryEx(fullPath: string) { ensureDirectoryEx(fullPath: string) {
return this.plugin.ensureDirectoryEx(fullPath); return this.plugin.ensureDirectoryEx(fullPath);
} }
getConflictedDoc(path: string, rev: string) { getConflictedDoc(path: FilePathWithPrefix, rev: string) {
return this.plugin.getConflictedDoc(path, rev); return this.plugin.getConflictedDoc(path, rev);
} }
onunload() { onunload() {
this.periodicInternalFileScanProcessor?.disable(); this.periodicInternalFileScanProcessor?.disable();
} }
onload(): void | Promise<void> { onload() {
this.plugin.addCommand({ this.plugin.addCommand({
id: "livesync-scaninternal", id: "livesync-scaninternal",
name: "Sync hidden files", name: "Sync hidden files",
@@ -49,11 +50,12 @@ export class HiddenFileSync extends LiveSyncCommands {
} }
} }
async beforeReplicate(showNotice: boolean) { async beforeReplicate(showNotice: boolean) {
if (this.settings.syncInternalFiles && this.settings.syncInternalFilesBeforeReplication && !this.settings.watchInternalFileChanges) { if (this.localDatabase.isReady && this.settings.syncInternalFiles && this.settings.syncInternalFilesBeforeReplication && !this.settings.watchInternalFileChanges) {
await this.syncInternalFilesAndDatabase("push", showNotice); await this.syncInternalFilesAndDatabase("push", showNotice);
} }
} }
async onResume() { async onResume() {
this.periodicInternalFileScanProcessor?.disable();
if (this.plugin.suspended) if (this.plugin.suspended)
return; return;
if (this.settings.syncInternalFiles) { if (this.settings.syncInternalFiles) {
@@ -66,13 +68,17 @@ export class HiddenFileSync extends LiveSyncCommands {
} }
realizeSettingSyncMode(): Promise<void> { realizeSettingSyncMode(): Promise<void> {
this.periodicInternalFileScanProcessor?.disable(); this.periodicInternalFileScanProcessor?.disable();
if (this.plugin.suspended)
return;
if (!this.plugin.isReady)
return;
this.periodicInternalFileScanProcessor.enable(this.settings.syncInternalFiles && this.settings.syncInternalFilesInterval ? (this.settings.syncInternalFilesInterval * 1000) : 0); this.periodicInternalFileScanProcessor.enable(this.settings.syncInternalFiles && this.settings.syncInternalFilesInterval ? (this.settings.syncInternalFilesInterval * 1000) : 0);
return; return;
} }
procInternalFiles: string[] = []; procInternalFiles: string[] = [];
async execInternalFile() { async execInternalFile() {
await runWithLock("execinternal", false, async () => { await runWithLock("execInternal", false, async () => {
const w = [...this.procInternalFiles]; const w = [...this.procInternalFiles];
this.procInternalFiles = []; this.procInternalFiles = [];
Logger(`Applying hidden ${w.length} files change...`); Logger(`Applying hidden ${w.length} files change...`);
@@ -88,7 +94,8 @@ export class HiddenFileSync extends LiveSyncCommands {
} }
recentProcessedInternalFiles = [] as string[]; recentProcessedInternalFiles = [] as string[];
async watchVaultRawEventsAsync(path: string) { async watchVaultRawEventsAsync(path: FilePath) {
if (!this.settings.syncInternalFiles) return;
const stat = await this.app.vault.adapter.stat(path); const stat = await this.app.vault.adapter.stat(path);
// sometimes folder is coming. // sometimes folder is coming.
if (stat && stat.type != "file") if (stat && stat.type != "file")
@@ -100,8 +107,9 @@ export class HiddenFileSync extends LiveSyncCommands {
return; return;
} }
this.recentProcessedInternalFiles = [key, ...this.recentProcessedInternalFiles].slice(0, 100); this.recentProcessedInternalFiles = [key, ...this.recentProcessedInternalFiles].slice(0, 100);
const id = filename2idInternalMetadata(path); // const id = await this.path2id(path, ICHeader);
const filesOnDB = await this.localDatabase.getDBEntryMeta(id); const prefixedFileName = addPrefix(path, ICHeader);
const filesOnDB = await this.localDatabase.getDBEntryMeta(prefixedFileName);
const dbMTime = ~~((filesOnDB && filesOnDB.mtime || 0) / 1000); const dbMTime = ~~((filesOnDB && filesOnDB.mtime || 0) / 1000);
// Skip unchanged file. // Skip unchanged file.
@@ -115,12 +123,6 @@ export class HiddenFileSync extends LiveSyncCommands {
await this.deleteInternalFileOnDatabase(path); await this.deleteInternalFileOnDatabase(path);
} else { } else {
await this.storeInternalFileToDatabase({ path: path, ...stat }); await this.storeInternalFileToDatabase({ path: path, ...stat });
const pluginDir = this.app.vault.configDir + "/plugins/";
const pluginFiles = ["manifest.json", "data.json", "style.css", "main.js"];
if (path.startsWith(pluginDir) && pluginFiles.some(e => path.endsWith(e)) && this.settings.usePluginSync) {
const pluginName = trimPrefix(path, pluginDir).split("/")[0];
await this.plugin.addOnPluginAndTheirSettings.sweepPlugin(false, pluginName);
}
} }
} }
@@ -132,35 +134,36 @@ export class HiddenFileSync extends LiveSyncCommands {
if (!("_conflicts" in doc)) if (!("_conflicts" in doc))
continue; continue;
if (isInternalMetadata(doc._id)) { if (isInternalMetadata(doc._id)) {
await this.resolveConflictOnInternalFile(doc._id); await this.resolveConflictOnInternalFile(doc.path);
} }
} }
} }
async resolveConflictOnInternalFile(id: string): Promise<boolean> { async resolveConflictOnInternalFile(path: FilePathWithPrefix): Promise<boolean> {
try { try {
// Retrieve data // Retrieve data
const doc = await this.localDatabase.localDatabase.get(id, { conflicts: true }); const id = await this.path2id(path, ICHeader);
const doc = await this.localDatabase.getRaw(id, { conflicts: true });
// If there is no conflict, return with false. // If there is no conflict, return with false.
if (!("_conflicts" in doc)) if (!("_conflicts" in doc))
return false; return false;
if (doc._conflicts.length == 0) if (doc._conflicts.length == 0)
return false; return false;
Logger(`Hidden file conflicted:${id2filenameInternalMetadata(id)}`); Logger(`Hidden file conflicted:${path}`);
const conflicts = doc._conflicts.sort((a, b) => Number(a.split("-")[0]) - Number(b.split("-")[0])); const conflicts = doc._conflicts.sort((a, b) => Number(a.split("-")[0]) - Number(b.split("-")[0]));
const revA = doc._rev; const revA = doc._rev;
const revB = conflicts[0]; const revB = conflicts[0];
if (doc._id.endsWith(".json")) { if (path.endsWith(".json")) {
const conflictedRev = conflicts[0]; const conflictedRev = conflicts[0];
const conflictedRevNo = Number(conflictedRev.split("-")[0]); const conflictedRevNo = Number(conflictedRev.split("-")[0]);
//Search //Search
const revFrom = (await this.localDatabase.localDatabase.get<EntryDoc>(id, { revs_info: true })); const revFrom = (await this.localDatabase.getRaw<EntryDoc>(id, { revs_info: true }));
const commonBase = revFrom._revs_info.filter(e => e.status == "available" && Number(e.rev.split("-")[0]) < conflictedRevNo).first()?.rev ?? ""; const commonBase = revFrom._revs_info.filter(e => e.status == "available" && Number(e.rev.split("-")[0]) < conflictedRevNo).first()?.rev ?? "";
const result = await this.plugin.mergeObject(id, commonBase, doc._rev, conflictedRev); const result = await this.plugin.mergeObject(path, commonBase, doc._rev, conflictedRev);
if (result) { if (result) {
Logger(`Object merge:${id}`, LOG_LEVEL.INFO); Logger(`Object merge:${path}`, LOG_LEVEL.INFO);
const filename = id2filenameInternalMetadata(id); const filename = stripAllPrefixes(path);
const isExists = await this.app.vault.adapter.exists(filename); const isExists = await this.app.vault.adapter.exists(filename);
if (!isExists) { if (!isExists) {
await this.ensureDirectoryEx(filename); await this.ensureDirectoryEx(filename);
@@ -169,24 +172,24 @@ export class HiddenFileSync extends LiveSyncCommands {
const stat = await this.app.vault.adapter.stat(filename); const stat = await this.app.vault.adapter.stat(filename);
await this.storeInternalFileToDatabase({ path: filename, ...stat }); await this.storeInternalFileToDatabase({ path: filename, ...stat });
await this.extractInternalFileFromDatabase(filename); await this.extractInternalFileFromDatabase(filename);
await this.localDatabase.localDatabase.remove(id, revB); await this.localDatabase.removeRaw(id, revB);
return this.resolveConflictOnInternalFile(id); return this.resolveConflictOnInternalFile(path);
} else { } else {
Logger(`Object merge is not applicable.`, LOG_LEVEL.VERBOSE); Logger(`Object merge is not applicable.`, LOG_LEVEL.VERBOSE);
} }
const docAMerge = await this.localDatabase.getDBEntry(id, { rev: revA }); const docAMerge = await this.localDatabase.getDBEntry(path, { rev: revA });
const docBMerge = await this.localDatabase.getDBEntry(id, { rev: revB }); const docBMerge = await this.localDatabase.getDBEntry(path, { rev: revB });
if (docAMerge != false && docBMerge != false) { if (docAMerge != false && docBMerge != false) {
if (await this.showJSONMergeDialogAndMerge(docAMerge, docBMerge)) { if (await this.showJSONMergeDialogAndMerge(docAMerge, docBMerge)) {
await delay(200); await delay(200);
// Again for other conflicted revisions. // Again for other conflicted revisions.
return this.resolveConflictOnInternalFile(id); return this.resolveConflictOnInternalFile(path);
} }
return false; return false;
} }
} }
const revBDoc = await this.localDatabase.localDatabase.get(id, { rev: revB }); const revBDoc = await this.localDatabase.getRaw(id, { rev: revB });
// determine which revision should been deleted. // determine which revision should been deleted.
// simply check modified time // simply check modified time
const mtimeA = ("mtime" in doc && doc.mtime) || 0; const mtimeA = ("mtime" in doc && doc.mtime) || 0;
@@ -195,19 +198,19 @@ export class HiddenFileSync extends LiveSyncCommands {
// console.log(`mtime:${mtimeA} - ${mtimeB}`); // console.log(`mtime:${mtimeA} - ${mtimeB}`);
const delRev = mtimeA < mtimeB ? revA : revB; const delRev = mtimeA < mtimeB ? revA : revB;
// delete older one. // delete older one.
await this.localDatabase.localDatabase.remove(id, delRev); await this.localDatabase.removeRaw(id, delRev);
Logger(`Older one has been deleted:${id2filenameInternalMetadata(id)}`); Logger(`Older one has been deleted:${path}`);
// check the file again // check the file again
return this.resolveConflictOnInternalFile(id); return this.resolveConflictOnInternalFile(path);
} catch (ex) { } catch (ex) {
Logger("Failed to resolve conflict (Hidden)"); Logger(`Failed to resolve conflict (Hidden): ${path}`);
Logger(ex, LOG_LEVEL.VERBOSE); Logger(ex, LOG_LEVEL.VERBOSE);
return false; return false;
} }
} }
//TODO: Tidy up. Even though it is experimental feature, So dirty... //TODO: Tidy up. Even though it is experimental feature, So dirty...
async syncInternalFilesAndDatabase(direction: "push" | "pull" | "safe", showMessage: boolean, files: InternalFileInfo[] | false = false, targetFiles: string[] | false = false) { async syncInternalFilesAndDatabase(direction: "push" | "pull" | "safe" | "pullForce" | "pushForce", showMessage: boolean, files: InternalFileInfo[] | false = false, targetFiles: string[] | false = false) {
await this.resolveConflictOnInternalFiles(); await this.resolveConflictOnInternalFiles();
const logLevel = showMessage ? LOG_LEVEL.NOTICE : LOG_LEVEL.INFO; const logLevel = showMessage ? LOG_LEVEL.NOTICE : LOG_LEVEL.INFO;
Logger("Scanning hidden files.", logLevel, "sync_internal"); Logger("Scanning hidden files.", logLevel, "sync_internal");
@@ -216,9 +219,8 @@ export class HiddenFileSync extends LiveSyncCommands {
.split(",").filter(e => e).map(e => new RegExp(e, "i")); .split(",").filter(e => e).map(e => new RegExp(e, "i"));
if (!files) if (!files)
files = await this.scanInternalFiles(); files = await this.scanInternalFiles();
const filesOnDB = ((await this.localDatabase.localDatabase.allDocs({ startkey: ICHeader, endkey: ICHeaderEnd, include_docs: true })).rows.map(e => e.doc) as InternalFileEntry[]).filter(e => !e.deleted); const filesOnDB = ((await this.localDatabase.allDocsRaw({ startkey: ICHeader, endkey: ICHeaderEnd, include_docs: true })).rows.map(e => e.doc) as InternalFileEntry[]).filter(e => !e.deleted);
const allFileNamesSrc = [...new Set([...files.map(e => normalizePath(e.path)), ...filesOnDB.map(e => stripAllPrefixes(this.getPath(e)))])];
const allFileNamesSrc = [...new Set([...files.map(e => normalizePath(e.path)), ...filesOnDB.map(e => normalizePath(id2path(id2filenameInternalMetadata(e._id))))])];
const allFileNames = allFileNamesSrc.filter(filename => !targetFiles || (targetFiles && targetFiles.indexOf(filename) !== -1)); const allFileNames = allFileNamesSrc.filter(filename => !targetFiles || (targetFiles && targetFiles.indexOf(filename) !== -1));
function compareMTime(a: number, b: number) { function compareMTime(a: number, b: number) {
const wa = ~~(a / 1000); const wa = ~~(a / 1000);
@@ -258,6 +260,7 @@ export class HiddenFileSync extends LiveSyncCommands {
let caches: { [key: string]: { storageMtime: number; docMtime: number; }; } = {}; let caches: { [key: string]: { storageMtime: number; docMtime: number; }; } = {};
caches = await this.kvDB.get<{ [key: string]: { storageMtime: number; docMtime: number; }; }>("diff-caches-internal") || {}; caches = await this.kvDB.get<{ [key: string]: { storageMtime: number; docMtime: number; }; }>("diff-caches-internal") || {};
for (const filename of allFileNames) { for (const filename of allFileNames) {
if (!filename) continue;
processed++; processed++;
if (processed % 100 == 0) if (processed % 100 == 0)
Logger(`Hidden file: ${processed}/${fileCount}`, logLevel, "sync_internal"); Logger(`Hidden file: ${processed}/${fileCount}`, logLevel, "sync_internal");
@@ -265,7 +268,7 @@ export class HiddenFileSync extends LiveSyncCommands {
continue; continue;
const fileOnStorage = files.find(e => e.path == filename); const fileOnStorage = files.find(e => e.path == filename);
const fileOnDatabase = filesOnDB.find(e => e._id == filename2idInternalMetadata(id2path(filename))); const fileOnDatabase = filesOnDB.find(e => stripAllPrefixes(this.getPath(e)) == filename);
const addProc = async (p: () => Promise<void>): Promise<void> => { const addProc = async (p: () => Promise<void>): Promise<void> => {
const releaser = await semaphore.acquire(1); const releaser = await semaphore.acquire(1);
try { try {
@@ -280,43 +283,45 @@ export class HiddenFileSync extends LiveSyncCommands {
const cache = filename in caches ? caches[filename] : { storageMtime: 0, docMtime: 0 }; const cache = filename in caches ? caches[filename] : { storageMtime: 0, docMtime: 0 };
p.push(addProc(async () => { p.push(addProc(async () => {
if (fileOnStorage && fileOnDatabase) { const xFileOnStorage = fileOnStorage;
const xFileOnDatabase = fileOnDatabase;
if (xFileOnStorage && xFileOnDatabase) {
// Both => Synchronize // Both => Synchronize
if (fileOnDatabase.mtime == cache.docMtime && fileOnStorage.mtime == cache.storageMtime) { if ((direction != "pullForce" && direction != "pushForce") && xFileOnDatabase.mtime == cache.docMtime && xFileOnStorage.mtime == cache.storageMtime) {
return; return;
} }
const nw = compareMTime(fileOnStorage.mtime, fileOnDatabase.mtime); const nw = compareMTime(xFileOnStorage.mtime, xFileOnDatabase.mtime);
if (nw > 0) { if (nw > 0 || direction == "pushForce") {
await this.storeInternalFileToDatabase(fileOnStorage); await this.storeInternalFileToDatabase(xFileOnStorage);
} }
if (nw < 0) { if (nw < 0 || direction == "pullForce") {
// skip if not extraction performed. // skip if not extraction performed.
if (!await this.extractInternalFileFromDatabase(filename)) if (!await this.extractInternalFileFromDatabase(filename))
return; return;
} }
// If process successfully updated or file contents are same, update cache. // If process successfully updated or file contents are same, update cache.
cache.docMtime = fileOnDatabase.mtime; cache.docMtime = xFileOnDatabase.mtime;
cache.storageMtime = fileOnStorage.mtime; cache.storageMtime = xFileOnStorage.mtime;
caches[filename] = cache; caches[filename] = cache;
countUpdatedFolder(filename); countUpdatedFolder(filename);
} else if (!fileOnStorage && fileOnDatabase) { } else if (!xFileOnStorage && xFileOnDatabase) {
if (direction == "push") { if (direction == "push" || direction == "pushForce") {
if (fileOnDatabase.deleted) if (xFileOnDatabase.deleted)
return; return;
await this.deleteInternalFileOnDatabase(filename, false); await this.deleteInternalFileOnDatabase(filename, false);
} else if (direction == "pull") { } else if (direction == "pull" || direction == "pullForce") {
if (await this.extractInternalFileFromDatabase(filename)) { if (await this.extractInternalFileFromDatabase(filename)) {
countUpdatedFolder(filename); countUpdatedFolder(filename);
} }
} else if (direction == "safe") { } else if (direction == "safe") {
if (fileOnDatabase.deleted) if (xFileOnDatabase.deleted)
return; return;
if (await this.extractInternalFileFromDatabase(filename)) { if (await this.extractInternalFileFromDatabase(filename)) {
countUpdatedFolder(filename); countUpdatedFolder(filename);
} }
} }
} else if (fileOnStorage && !fileOnDatabase) { } else if (xFileOnStorage && !xFileOnDatabase) {
await this.storeInternalFileToDatabase(fileOnStorage); await this.storeInternalFileToDatabase(xFileOnStorage);
} else { } else {
throw new Error("Invalid state on hidden file sync"); throw new Error("Invalid state on hidden file sync");
// Something corrupted? // Something corrupted?
@@ -327,7 +332,7 @@ export class HiddenFileSync extends LiveSyncCommands {
await this.kvDB.set("diff-caches-internal", caches); await this.kvDB.set("diff-caches-internal", caches);
// When files has been retrieved from the database. they must be reloaded. // When files has been retrieved from the database. they must be reloaded.
if (direction == "pull" && filesChanged != 0) { if ((direction == "pull" || direction == "pullForce") && filesChanged != 0) {
const configDir = normalizePath(this.app.vault.configDir); const configDir = normalizePath(this.app.vault.configDir);
// Show notification to restart obsidian when something has been changed in configDir. // Show notification to restart obsidian when something has been changed in configDir.
if (configDir in updatedFolders) { if (configDir in updatedFolders) {
@@ -335,7 +340,7 @@ export class HiddenFileSync extends LiveSyncCommands {
let updatedCount = updatedFolders[configDir]; let updatedCount = updatedFolders[configDir];
try { try {
//@ts-ignore //@ts-ignore
const manifests = Object.values(this.app.plugins.manifests) as PluginManifest[]; const manifests = Object.values(this.app.plugins.manifests) as any as PluginManifest[];
//@ts-ignore //@ts-ignore
const enabledPlugins = this.app.plugins.enabledPlugins as Set<string>; const enabledPlugins = this.app.plugins.enabledPlugins as Set<string>;
const enabledPluginManifests = manifests.filter(e => enabledPlugins.has(e.id)); const enabledPluginManifests = manifests.filter(e => enabledPlugins.has(e.id));
@@ -427,7 +432,8 @@ export class HiddenFileSync extends LiveSyncCommands {
} }
async storeInternalFileToDatabase(file: InternalFileInfo, forceWrite = false) { async storeInternalFileToDatabase(file: InternalFileInfo, forceWrite = false) {
const id = filename2idInternalMetadata(path2id(file.path)); const id = await this.path2id(file.path, ICHeader);
const prefixedFileName = addPrefix(file.path, ICHeader);
const contentBin = await this.app.vault.adapter.readBinary(file.path); const contentBin = await this.app.vault.adapter.readBinary(file.path);
let content: string[]; let content: string[];
try { try {
@@ -438,13 +444,14 @@ export class HiddenFileSync extends LiveSyncCommands {
return false; return false;
} }
const mtime = file.mtime; const mtime = file.mtime;
return await runWithLock("file-" + id, false, async () => { return await runWithLock("file-" + prefixedFileName, false, async () => {
try { try {
const old = await this.localDatabase.getDBEntry(id, null, false, false); const old = await this.localDatabase.getDBEntry(prefixedFileName, null, false, false);
let saveData: LoadedEntry; let saveData: LoadedEntry;
if (old === false) { if (old === false) {
saveData = { saveData = {
_id: id, _id: id,
path: prefixedFileName,
data: content, data: content,
mtime, mtime,
ctime: mtime, ctime: mtime,
@@ -456,7 +463,7 @@ export class HiddenFileSync extends LiveSyncCommands {
}; };
} else { } else {
if (isDocContentSame(old.data, content) && !forceWrite) { if (isDocContentSame(old.data, content) && !forceWrite) {
// Logger(`internal files STORAGE --> DB:${file.path}: Not changed`); // Logger(`STORAGE --> DB:${file.path}: (hidden) Not changed`, LOG_LEVEL.VERBOSE);
return; return;
} }
saveData = saveData =
@@ -471,7 +478,6 @@ export class HiddenFileSync extends LiveSyncCommands {
type: "newnote", type: "newnote",
}; };
} }
const ret = await this.localDatabase.putDBEntry(saveData, true); const ret = await this.localDatabase.putDBEntry(saveData, true);
Logger(`STORAGE --> DB:${file.path}: (hidden) Done`); Logger(`STORAGE --> DB:${file.path}: (hidden) Done`);
return ret; return ret;
@@ -483,16 +489,18 @@ export class HiddenFileSync extends LiveSyncCommands {
}); });
} }
async deleteInternalFileOnDatabase(filename: string, forceWrite = false) { async deleteInternalFileOnDatabase(filename: FilePath, forceWrite = false) {
const id = filename2idInternalMetadata(path2id(filename)); const id = await this.path2id(filename, ICHeader);
const prefixedFileName = addPrefix(filename, ICHeader);
const mtime = new Date().getTime(); const mtime = new Date().getTime();
await runWithLock("file-" + id, false, async () => { await runWithLock("file-" + prefixedFileName, false, async () => {
try { try {
const old = await this.localDatabase.getDBEntry(id, null, false, false) as InternalFileEntry | false; const old = await this.localDatabase.getDBEntry(prefixedFileName, null, false, false) as InternalFileEntry | false;
let saveData: InternalFileEntry; let saveData: InternalFileEntry;
if (old === false) { if (old === false) {
saveData = { saveData = {
_id: id, _id: id,
path: prefixedFileName,
mtime, mtime,
ctime: mtime, ctime: mtime,
size: 0, size: 0,
@@ -515,7 +523,7 @@ export class HiddenFileSync extends LiveSyncCommands {
type: "newnote", type: "newnote",
}; };
} }
await this.localDatabase.localDatabase.put(saveData); await this.localDatabase.putRaw(saveData);
Logger(`STORAGE -x> DB:${filename}: (hidden) Done`); Logger(`STORAGE -x> DB:${filename}: (hidden) Done`);
} catch (ex) { } catch (ex) {
Logger(`STORAGE -x> DB:${filename}: (hidden) Failed`); Logger(`STORAGE -x> DB:${filename}: (hidden) Failed`);
@@ -525,20 +533,20 @@ export class HiddenFileSync extends LiveSyncCommands {
}); });
} }
async extractInternalFileFromDatabase(filename: string, force = false) { async extractInternalFileFromDatabase(filename: FilePath, force = false) {
const isExists = await this.app.vault.adapter.exists(filename); const isExists = await this.app.vault.adapter.exists(filename);
const id = filename2idInternalMetadata(path2id(filename)); const prefixedFileName = addPrefix(filename, ICHeader);
return await runWithLock("file-" + id, false, async () => { return await runWithLock("file-" + prefixedFileName, false, async () => {
try { try {
// Check conflicted status // Check conflicted status
//TODO option //TODO option
const fileOnDB = await this.localDatabase.getDBEntry(id, { conflicts: true }, false, false) as false | LoadedEntry; const fileOnDB = await this.localDatabase.getDBEntry(prefixedFileName, { conflicts: true }, false, false);
if (fileOnDB === false) if (fileOnDB === false)
throw new Error(`File not found on database.:${id}`); throw new Error(`File not found on database.:${filename}`);
// Prevent overrite for Prevent overwriting while some conflicted revision exists. // Prevent overwrite for Prevent overwriting while some conflicted revision exists.
if (fileOnDB?._conflicts?.length) { if (fileOnDB?._conflicts?.length) {
Logger(`Hidden file ${id} has conflicted revisions, to keep in safe, writing to storage has been prevented`, LOG_LEVEL.INFO); Logger(`Hidden file ${filename} has conflicted revisions, to keep in safe, writing to storage has been prevented`, LOG_LEVEL.INFO);
return; return;
} }
const deleted = "deleted" in fileOnDB ? fileOnDB.deleted : false; const deleted = "deleted" in fileOnDB ? fileOnDB.deleted : false;
@@ -600,22 +608,25 @@ export class HiddenFileSync extends LiveSyncCommands {
showJSONMergeDialogAndMerge(docA: LoadedEntry, docB: LoadedEntry): Promise<boolean> { showJSONMergeDialogAndMerge(docA: LoadedEntry, docB: LoadedEntry): Promise<boolean> {
return new Promise((res) => { return runWithLock("conflict:merge-data", false, () => new Promise((res) => {
Logger("Opening data-merging dialog", LOG_LEVEL.VERBOSE); Logger("Opening data-merging dialog", LOG_LEVEL.VERBOSE);
const docs = [docA, docB]; const docs = [docA, docB];
const modal = new JsonResolveModal(this.app, id2path(docA._id), [docA, docB], async (keep, result) => { const path = stripAllPrefixes(docA.path);
const modal = new JsonResolveModal(this.app, path, [docA, docB], async (keep, result) => {
// modal.close(); // modal.close();
try { try {
const filename = id2filenameInternalMetadata(docA._id); const filename = path;
let needFlush = false; let needFlush = false;
if (!result && !keep) { if (!result && !keep) {
Logger(`Skipped merging: ${filename}`); Logger(`Skipped merging: ${filename}`);
res(false);
return;
} }
//Delete old revisions //Delete old revisions
if (result || keep) { if (result || keep) {
for (const doc of docs) { for (const doc of docs) {
if (doc._rev != keep) { if (doc._rev != keep) {
if (await this.localDatabase.deleteDBEntry(doc._id, { rev: doc._rev })) { if (await this.localDatabase.deleteDBEntry(this.getPath(doc), { rev: doc._rev })) {
Logger(`Conflicted revision has been deleted: ${filename}`); Logger(`Conflicted revision has been deleted: ${filename}`);
needFlush = true; needFlush = true;
} }
@@ -651,7 +662,7 @@ export class HiddenFileSync extends LiveSyncCommands {
} }
}); });
modal.open(); modal.open();
}); }));
} }
async scanInternalFiles(): Promise<InternalFileInfo[]> { async scanInternalFiles(): Promise<InternalFileInfo[]> {
@@ -663,7 +674,7 @@ export class HiddenFileSync extends LiveSyncCommands {
const filenames = (await this.getFiles(findRoot, [], null, ignoreFilter)).filter(e => e.startsWith(".")).filter(e => !e.startsWith(".trash")); const filenames = (await this.getFiles(findRoot, [], null, ignoreFilter)).filter(e => e.startsWith(".")).filter(e => !e.startsWith(".trash"));
const files = filenames.map(async (e) => { const files = filenames.map(async (e) => {
return { return {
path: e, path: e as FilePath,
stat: await this.app.vault.adapter.stat(e) stat: await this.app.vault.adapter.stat(e)
}; };
}); });

20
src/CmdPluginAndTheirSettings.ts
View File

@@ -1,5 +1,5 @@
import { normalizePath, PluginManifest } from "./deps"; import { normalizePath, PluginManifest } from "./deps";
import { EntryDoc, LoadedEntry, LOG_LEVEL } from "./lib/src/types"; import { DocumentID, EntryDoc, FilePathWithPrefix, LoadedEntry, LOG_LEVEL } from "./lib/src/types";
import { PluginDataEntry, PERIODIC_PLUGIN_SWEEP, PluginList, DevicePluginList, PSCHeader, PSCHeaderEnd } from "./types"; import { PluginDataEntry, PERIODIC_PLUGIN_SWEEP, PluginList, DevicePluginList, PSCHeader, PSCHeaderEnd } from "./types";
import { getDocData, isDocContentSame } from "./lib/src/utils"; import { getDocData, isDocContentSame } from "./lib/src/utils";
import { Logger } from "./lib/src/logger"; import { Logger } from "./lib/src/logger";
@@ -42,6 +42,7 @@ export class PluginAndTheirSettings extends LiveSyncCommands {
this.showPluginSyncModal(); this.showPluginSyncModal();
}, },
}); });
this.showPluginSyncModal();
} }
onunload() { onunload() {
this.hidePluginSyncModal(); this.hidePluginSyncModal();
@@ -99,9 +100,8 @@ export class PluginAndTheirSettings extends LiveSyncCommands {
async getPluginList(): Promise<{ plugins: PluginList; allPlugins: DevicePluginList; thisDevicePlugins: DevicePluginList; }> { async getPluginList(): Promise<{ plugins: PluginList; allPlugins: DevicePluginList; thisDevicePlugins: DevicePluginList; }> {
const db = this.localDatabase.localDatabase; const docList = await this.localDatabase.allDocsRaw<PluginDataEntry>({ startkey: PSCHeader, endkey: PSCHeaderEnd, include_docs: false });
const docList = await db.allDocs<PluginDataEntry>({ startkey: PSCHeader, endkey: PSCHeaderEnd, include_docs: false }); const oldDocs: PluginDataEntry[] = ((await Promise.all(docList.rows.map(async (e) => await this.localDatabase.getDBEntry(e.id as FilePathWithPrefix /* WARN!! THIS SHOULD BE WRAPPED */)))).filter((e) => e !== false) as LoadedEntry[]).map((e) => JSON.parse(getDocData(e.data)));
const oldDocs: PluginDataEntry[] = ((await Promise.all(docList.rows.map(async (e) => await this.localDatabase.getDBEntry(e.id)))).filter((e) => e !== false) as LoadedEntry[]).map((e) => JSON.parse(getDocData(e.data)));
const plugins: { [key: string]: PluginDataEntry[]; } = {}; const plugins: { [key: string]: PluginDataEntry[]; } = {};
const allPlugins: { [key: string]: PluginDataEntry; } = {}; const allPlugins: { [key: string]: PluginDataEntry; } = {};
const thisDevicePlugins: { [key: string]: PluginDataEntry; } = {}; const thisDevicePlugins: { [key: string]: PluginDataEntry; } = {};
@@ -166,12 +166,11 @@ export class PluginAndTheirSettings extends LiveSyncCommands {
await runWithLock("sweepplugin", true, async () => { await runWithLock("sweepplugin", true, async () => {
const logLevel = showMessage ? LOG_LEVEL.NOTICE : LOG_LEVEL.INFO; const logLevel = showMessage ? LOG_LEVEL.NOTICE : LOG_LEVEL.INFO;
if (!this.deviceAndVaultName) { if (!this.deviceAndVaultName) {
Logger("You have to set your device and vault name.", LOG_LEVEL.NOTICE); Logger("You have to set your device name.", LOG_LEVEL.NOTICE);
return; return;
} }
Logger("Scanning plugins", logLevel); Logger("Scanning plugins", logLevel);
const db = this.localDatabase.localDatabase; const oldDocs = await this.localDatabase.allDocsRaw<EntryDoc>({
const oldDocs = await db.allDocs({
startkey: `ps:${this.deviceAndVaultName}-${specificPlugin}`, startkey: `ps:${this.deviceAndVaultName}-${specificPlugin}`,
endkey: `ps:${this.deviceAndVaultName}-${specificPlugin}\u{10ffff}`, endkey: `ps:${this.deviceAndVaultName}-${specificPlugin}\u{10ffff}`,
include_docs: true, include_docs: true,
@@ -179,7 +178,7 @@ export class PluginAndTheirSettings extends LiveSyncCommands {
// Logger("OLD DOCS.", LOG_LEVEL.VERBOSE); // Logger("OLD DOCS.", LOG_LEVEL.VERBOSE);
// sweep current plugin. // sweep current plugin.
const procs = manifests.map(async (m) => { const procs = manifests.map(async (m) => {
const pluginDataEntryID = `ps:${this.deviceAndVaultName}-${m.id}`; const pluginDataEntryID = `ps:${this.deviceAndVaultName}-${m.id}` as DocumentID;
try { try {
if (specificPlugin && m.id != specificPlugin) { if (specificPlugin && m.id != specificPlugin) {
return; return;
@@ -213,6 +212,7 @@ export class PluginAndTheirSettings extends LiveSyncCommands {
}; };
const d: LoadedEntry = { const d: LoadedEntry = {
_id: p._id, _id: p._id,
path: p._id as string as FilePathWithPrefix,
data: JSON.stringify(p), data: JSON.stringify(p),
ctime: mtime, ctime: mtime,
mtime: mtime, mtime: mtime,
@@ -223,7 +223,7 @@ export class PluginAndTheirSettings extends LiveSyncCommands {
}; };
Logger(`check diff:${m.name}(${m.id})`, LOG_LEVEL.VERBOSE); Logger(`check diff:${m.name}(${m.id})`, LOG_LEVEL.VERBOSE);
await runWithLock("plugin-" + m.id, false, async () => { await runWithLock("plugin-" + m.id, false, async () => {
const old = await this.localDatabase.getDBEntry(p._id, null, false, false); const old = await this.localDatabase.getDBEntry(p._id as string as FilePathWithPrefix /* This also should be explained */, null, false, false);
if (old !== false) { if (old !== false) {
const oldData = { data: old.data, deleted: old._deleted }; const oldData = { data: old.data, deleted: old._deleted };
const newData = { data: d.data, deleted: d._deleted }; const newData = { data: d.data, deleted: d._deleted };
@@ -259,7 +259,7 @@ export class PluginAndTheirSettings extends LiveSyncCommands {
return e.doc; return e.doc;
}); });
Logger(`Deleting old plugin:(${delDocs.length})`, LOG_LEVEL.VERBOSE); Logger(`Deleting old plugin:(${delDocs.length})`, LOG_LEVEL.VERBOSE);
await db.bulkDocs(delDocs); await this.localDatabase.bulkDocsRaw(delDocs);
Logger(`Scan plugin done.`, logLevel); Logger(`Scan plugin done.`, logLevel);
}); });
} }

169
src/CmdSetupLiveSync.ts
View File

@@ -5,6 +5,9 @@ import { PouchDB } from "./lib/src/pouchdb-browser.js";
import { askSelectString, askYesNo, askString } from "./utils"; import { askSelectString, askYesNo, askString } from "./utils";
import { decrypt, encrypt } from "./lib/src/e2ee_v2"; import { decrypt, encrypt } from "./lib/src/e2ee_v2";
import { LiveSyncCommands } from "./LiveSyncCommands"; import { LiveSyncCommands } from "./LiveSyncCommands";
import { delay } from "./lib/src/utils";
import { confirmWithMessage } from "./dialogs";
import { Platform } from "./deps";
export class SetupLiveSync extends LiveSyncCommands { export class SetupLiveSync extends LiveSyncCommands {
onunload() { } onunload() { }
@@ -97,7 +100,8 @@ export class SetupLiveSync extends LiveSyncCommands {
const setupAsNew = "Set it up as secondary or subsequent device"; const setupAsNew = "Set it up as secondary or subsequent device";
const setupAgain = "Reconfigure and reconstitute the data"; const setupAgain = "Reconfigure and reconstitute the data";
const setupManually = "Leave everything to me"; const setupManually = "Leave everything to me";
newSettingW.syncInternalFiles = false;
newSettingW.usePluginSync = false;
const setupType = await askSelectString(this.app, "How would you like to set it up?", [setupAsNew, setupAgain, setupJustImport, setupManually]); const setupType = await askSelectString(this.app, "How would you like to set it up?", [setupAsNew, setupAgain, setupJustImport, setupManually]);
if (setupType == setupJustImport) { if (setupType == setupJustImport) {
this.plugin.settings = newSettingW; this.plugin.settings = newSettingW;
@@ -106,11 +110,7 @@ export class SetupLiveSync extends LiveSyncCommands {
} else if (setupType == setupAsNew) { } else if (setupType == setupAsNew) {
this.plugin.settings = newSettingW; this.plugin.settings = newSettingW;
this.plugin.usedPassphrase = ""; this.plugin.usedPassphrase = "";
await this.plugin.saveSettings(); await this.fetchLocal();
await this.plugin.resetLocalDatabase();
await this.plugin.localDatabase.initializeDatabase();
await this.plugin.markRemoteResolved();
await this.plugin.replicate(true);
} else if (setupType == setupAgain) { } else if (setupType == setupAgain) {
const confirm = "I know this operation will rebuild all my databases with files on this device, and files that are on the remote database and I didn't synchronize to any other devices will be lost and want to proceed indeed."; const confirm = "I know this operation will rebuild all my databases with files on this device, and files that are on the remote database and I didn't synchronize to any other devices will be lost and want to proceed indeed.";
if (await askSelectString(this.app, "Do you really want to do this?", ["Cancel", confirm]) != confirm) { if (await askSelectString(this.app, "Do you really want to do this?", ["Cancel", confirm]) != confirm) {
@@ -118,15 +118,7 @@ export class SetupLiveSync extends LiveSyncCommands {
} }
this.plugin.settings = newSettingW; this.plugin.settings = newSettingW;
this.plugin.usedPassphrase = ""; this.plugin.usedPassphrase = "";
await this.plugin.saveSettings(); await this.rebuildEverything();
await this.plugin.resetLocalDatabase();
await this.plugin.localDatabase.initializeDatabase();
await this.plugin.initializeDatabase(true);
await this.plugin.tryResetRemoteDatabase();
await this.plugin.markRemoteLocked();
await this.plugin.markRemoteResolved();
await this.plugin.replicate(true);
} else if (setupType == setupManually) { } else if (setupType == setupManually) {
const keepLocalDB = await askYesNo(this.app, "Keep local DB?"); const keepLocalDB = await askYesNo(this.app, "Keep local DB?");
const keepRemoteDB = await askYesNo(this.app, "Keep remote DB?"); const keepRemoteDB = await askYesNo(this.app, "Keep remote DB?");
@@ -134,6 +126,8 @@ export class SetupLiveSync extends LiveSyncCommands {
// nothing to do. so peaceful. // nothing to do. so peaceful.
this.plugin.settings = newSettingW; this.plugin.settings = newSettingW;
this.plugin.usedPassphrase = ""; this.plugin.usedPassphrase = "";
this.suspendAllSync();
this.suspendExtraSync();
await this.plugin.saveSettings(); await this.plugin.saveSettings();
const replicate = await askYesNo(this.app, "Unlock and replicate?"); const replicate = await askYesNo(this.app, "Unlock and replicate?");
if (replicate == "yes") { if (replicate == "yes") {
@@ -189,4 +183,149 @@ export class SetupLiveSync extends LiveSyncCommands {
Logger("Couldn't parse or decrypt configuration uri.", LOG_LEVEL.NOTICE); Logger("Couldn't parse or decrypt configuration uri.", LOG_LEVEL.NOTICE);
} }
} }
suspendExtraSync() {
Logger("Hidden files and plugin synchronization have been temporarily disabled. Please enable them after the fetching, if you need them.", LOG_LEVEL.NOTICE)
this.plugin.settings.syncInternalFiles = false;
this.plugin.settings.usePluginSync = false;
this.plugin.settings.autoSweepPlugins = false;
}
async askHiddenFileConfiguration(opt: { enableFetch?: boolean, enableOverwrite?: boolean }) {
this.plugin.addOnSetup.suspendExtraSync();
const message = `Would you like to enable \`Hidden File Synchronization\` or \`Customization sync\`?
${opt.enableFetch ? " - Fetch: Use files stored from other devices. \n" : ""}${opt.enableOverwrite ? "- Overwrite: Use files from this device. \n" : ""}- Custom: Synchronize only customization files with a dedicated interface.
- Keep them disabled: Do not use hidden file synchronization.
Of course, we are able to disable these features.`
const CHOICE_FETCH = "Fetch";
const CHOICE_OVERWRITE = "Overwrite";
const CHOICE_CUSTOMIZE = "Custom";
const CHOICE_DISMISS = "keep them disabled";
const choices = [];
if (opt?.enableFetch) {
choices.push(CHOICE_FETCH);
}
if (opt?.enableOverwrite) {
choices.push(CHOICE_OVERWRITE);
}
choices.push(CHOICE_CUSTOMIZE);
choices.push(CHOICE_DISMISS);
const ret = await confirmWithMessage(this.plugin, "Hidden file sync", message, choices, CHOICE_DISMISS, 40);
if (ret == CHOICE_FETCH) {
await this.configureHiddenFileSync("FETCH");
} else if (ret == CHOICE_OVERWRITE) {
await this.configureHiddenFileSync("OVERWRITE");
} else if (ret == CHOICE_DISMISS) {
await this.configureHiddenFileSync("DISABLE");
} else if (ret == CHOICE_CUSTOMIZE) {
await this.configureHiddenFileSync("CUSTOMIZE");
}
}
async configureHiddenFileSync(mode: "FETCH" | "OVERWRITE" | "MERGE" | "DISABLE" | "CUSTOMIZE") {
this.plugin.addOnSetup.suspendExtraSync();
if (mode == "DISABLE") {
this.plugin.settings.syncInternalFiles = false;
this.plugin.settings.usePluginSync = false;
await this.plugin.saveSettings();
return;
}
if (mode != "CUSTOMIZE") {
Logger("Gathering files for enabling Hidden File Sync", LOG_LEVEL.NOTICE);
if (mode == "FETCH") {
await this.plugin.addOnHiddenFileSync.syncInternalFilesAndDatabase("pullForce", true);
} else if (mode == "OVERWRITE") {
await this.plugin.addOnHiddenFileSync.syncInternalFilesAndDatabase("pushForce", true);
} else if (mode == "MERGE") {
await this.plugin.addOnHiddenFileSync.syncInternalFilesAndDatabase("safe", true);
}
this.plugin.settings.syncInternalFiles = true;
await this.plugin.saveSettings();
Logger(`Done! Restarting the app is strongly recommended!`, LOG_LEVEL.NOTICE);
} else if (mode == "CUSTOMIZE") {
if (!this.plugin.deviceAndVaultName) {
let name = await askString(this.app, "Device name", "Please set this device name", `desktop`);
if (!name) {
if (Platform.isAndroidApp) {
name = "android-app"
} else if (Platform.isIosApp) {
name = "ios"
} else if (Platform.isMacOS) {
name = "macos"
} else if (Platform.isMobileApp) {
name = "mobile-app"
} else if (Platform.isMobile) {
name = "mobile"
} else if (Platform.isSafari) {
name = "safari"
} else if (Platform.isDesktop) {
name = "desktop"
} else if (Platform.isDesktopApp) {
name = "desktop-app"
} else {
name = "unknown"
}
name = name + Math.random().toString(36).slice(-4);
}
this.plugin.deviceAndVaultName = name;
}
this.plugin.settings.usePluginSync = true;
await this.plugin.saveSettings();
await this.plugin.addOnConfigSync.scanAllConfigFiles(true);
}
}
suspendAllSync() {
this.plugin.settings.liveSync = false;
this.plugin.settings.periodicReplication = false;
this.plugin.settings.syncOnSave = false;
this.plugin.settings.syncOnStart = false;
this.plugin.settings.syncOnFileOpen = false;
this.plugin.settings.syncAfterMerge = false;
//this.suspendExtraSync();
}
async fetchLocal() {
this.suspendExtraSync();
await this.plugin.realizeSettingSyncMode();
await this.plugin.resetLocalDatabase();
await delay(1000);
await this.plugin.markRemoteResolved();
await this.plugin.openDatabase();
this.plugin.isReady = true;
await delay(500);
await this.plugin.replicateAllFromServer(true);
await delay(1000);
await this.plugin.replicateAllFromServer(true);
await this.askHiddenFileConfiguration({ enableFetch: true });
}
async rebuildRemote() {
this.suspendExtraSync();
await this.plugin.realizeSettingSyncMode();
await this.plugin.markRemoteLocked();
await this.plugin.tryResetRemoteDatabase();
await this.plugin.markRemoteLocked();
await delay(500);
await this.askHiddenFileConfiguration({ enableOverwrite: true });
await delay(1000);
await this.plugin.replicateAllToServer(true);
await delay(1000);
await this.plugin.replicateAllToServer(true);
}
async rebuildEverything() {
this.suspendExtraSync();
await this.plugin.realizeSettingSyncMode();
await this.plugin.resetLocalDatabase();
await delay(1000);
await this.plugin.initializeDatabase(true);
await this.plugin.markRemoteLocked();
await this.plugin.tryResetRemoteDatabase();
await this.plugin.markRemoteLocked();
await delay(500);
await this.askHiddenFileConfiguration({ enableOverwrite: true });
await delay(1000);
await this.plugin.replicateAllToServer(true);
await delay(1000);
await this.plugin.replicateAllToServer(true);
}
} }

33
src/DocumentHistoryModal.ts
View File

@@ -1,12 +1,13 @@
import { TFile, Modal, App } from "./deps"; import { TFile, Modal, App } from "./deps";
import { isValidPath, path2id } from "./utils"; import { getPathFromTFile, isValidPath } from "./utils";
import { base64ToArrayBuffer, base64ToString, escapeStringToHTML } from "./lib/src/strbin"; import { base64ToArrayBuffer, base64ToString, escapeStringToHTML } from "./lib/src/strbin";
import ObsidianLiveSyncPlugin from "./main"; import ObsidianLiveSyncPlugin from "./main";
import { DIFF_DELETE, DIFF_EQUAL, DIFF_INSERT, diff_match_patch } from "diff-match-patch"; import { DIFF_DELETE, DIFF_EQUAL, DIFF_INSERT, diff_match_patch } from "diff-match-patch";
import { LoadedEntry, LOG_LEVEL } from "./lib/src/types"; import { DocumentID, FilePathWithPrefix, LoadedEntry, LOG_LEVEL } from "./lib/src/types";
import { Logger } from "./lib/src/logger"; import { Logger } from "./lib/src/logger";
import { isErrorOfMissingDoc } from "./lib/src/utils_couchdb"; import { isErrorOfMissingDoc } from "./lib/src/utils_couchdb";
import { getDocData } from "./lib/src/utils"; import { getDocData } from "./lib/src/utils";
import { stripPrefix } from "./lib/src/path";
export class DocumentHistoryModal extends Modal { export class DocumentHistoryModal extends Modal {
plugin: ObsidianLiveSyncPlugin; plugin: ObsidianLiveSyncPlugin;
@@ -15,26 +16,35 @@ export class DocumentHistoryModal extends Modal {
info: HTMLDivElement; info: HTMLDivElement;
fileInfo: HTMLDivElement; fileInfo: HTMLDivElement;
showDiff = false; showDiff = false;
id: DocumentID;
file: string; file: FilePathWithPrefix;
revs_info: PouchDB.Core.RevisionInfo[] = []; revs_info: PouchDB.Core.RevisionInfo[] = [];
currentDoc: LoadedEntry; currentDoc: LoadedEntry;
currentText = ""; currentText = "";
currentDeleted = false; currentDeleted = false;
constructor(app: App, plugin: ObsidianLiveSyncPlugin, file: TFile | string) { constructor(app: App, plugin: ObsidianLiveSyncPlugin, file: TFile | FilePathWithPrefix, id: DocumentID) {
super(app); super(app);
this.plugin = plugin; this.plugin = plugin;
this.file = (file instanceof TFile) ? file.path : file; this.file = (file instanceof TFile) ? getPathFromTFile(file) : file;
this.id = id;
if (!file) {
this.file = this.plugin.id2path(id, null);
}
if (localStorage.getItem("ols-history-highlightdiff") == "1") { if (localStorage.getItem("ols-history-highlightdiff") == "1") {
this.showDiff = true; this.showDiff = true;
} }
} }
async loadFile() { async loadFile() {
if (!this.id) {
this.id = await this.plugin.path2id(this.file);
}
const db = this.plugin.localDatabase; const db = this.plugin.localDatabase;
try { try {
const w = await db.localDatabase.get(path2id(this.file), { revs_info: true }); const w = await db.localDatabase.get(this.id, { revs_info: true });
this.revs_info = w._revs_info.filter((e) => e?.status == "available"); this.revs_info = w._revs_info.filter((e) => e?.status == "available");
this.range.max = `${this.revs_info.length - 1}`; this.range.max = `${this.revs_info.length - 1}`;
this.range.value = this.range.max; this.range.value = this.range.max;
@@ -47,6 +57,9 @@ export class DocumentHistoryModal extends Modal {
this.range.disabled = true; this.range.disabled = true;
this.showDiff this.showDiff
this.contentView.setText(`History of this file was not recorded.`); this.contentView.setText(`History of this file was not recorded.`);
} else {
this.contentView.setText(`Error occurred.`);
Logger(ex, LOG_LEVEL.VERBOSE);
} }
} }
} }
@@ -55,7 +68,7 @@ export class DocumentHistoryModal extends Modal {
const db = this.plugin.localDatabase; const db = this.plugin.localDatabase;
const index = this.revs_info.length - 1 - (this.range.value as any) / 1; const index = this.revs_info.length - 1 - (this.range.value as any) / 1;
const rev = this.revs_info[index]; const rev = this.revs_info[index];
const w = await db.getDBEntry(path2id(this.file), { rev: rev.rev }, false, false, true); const w = await db.getDBEntry(this.file, { rev: rev.rev }, false, false, true);
this.currentText = ""; this.currentText = "";
this.currentDeleted = false; this.currentDeleted = false;
if (w === false) { if (w === false) {
@@ -73,7 +86,7 @@ export class DocumentHistoryModal extends Modal {
const prevRevIdx = this.revs_info.length - 1 - ((this.range.value as any) / 1 - 1); const prevRevIdx = this.revs_info.length - 1 - ((this.range.value as any) / 1 - 1);
if (prevRevIdx >= 0 && prevRevIdx < this.revs_info.length) { if (prevRevIdx >= 0 && prevRevIdx < this.revs_info.length) {
const oldRev = this.revs_info[prevRevIdx].rev; const oldRev = this.revs_info[prevRevIdx].rev;
const w2 = await db.getDBEntry(path2id(this.file), { rev: oldRev }, false, false, true); const w2 = await db.getDBEntry(this.file, { rev: oldRev }, false, false, true);
if (w2 != false) { if (w2 != false) {
const dmp = new diff_match_patch(); const dmp = new diff_match_patch();
const w2data = w2.datatype == "plain" ? getDocData(w2.data) : base64ToString(w2.data); const w2data = w2.datatype == "plain" ? getDocData(w2.data) : base64ToString(w2.data);
@@ -102,7 +115,6 @@ export class DocumentHistoryModal extends Modal {
result = escapeStringToHTML(w1data); result = escapeStringToHTML(w1data);
} }
this.contentView.innerHTML = (this.currentDeleted ? "(At this revision, the file has been deleted)\n" : "") + result; this.contentView.innerHTML = (this.currentDeleted ? "(At this revision, the file has been deleted)\n" : "") + result;
} }
} }
@@ -173,7 +185,8 @@ export class DocumentHistoryModal extends Modal {
buttons.createEl("button", { text: "Back to this revision" }, (e) => { buttons.createEl("button", { text: "Back to this revision" }, (e) => {
e.addClass("mod-cta"); e.addClass("mod-cta");
e.addEventListener("click", async () => { e.addEventListener("click", async () => {
const pathToWrite = this.file.startsWith("i:") ? this.file.substring("i:".length) : this.file; // const pathToWrite = this.plugin.id2path(this.id, true);
const pathToWrite = stripPrefix(this.file);
if (!isValidPath(pathToWrite)) { if (!isValidPath(pathToWrite)) {
Logger("Path is not valid to write content.", LOG_LEVEL.INFO); Logger("Path is not valid to write content.", LOG_LEVEL.INFO);
} }

16
src/JsonResolveModal.ts
View File

@@ -1,19 +1,25 @@
import { App, Modal } from "./deps"; import { App, Modal } from "./deps";
import { LoadedEntry } from "./lib/src/types"; import { FilePath, LoadedEntry } from "./lib/src/types";
import JsonResolvePane from "./JsonResolvePane.svelte"; import JsonResolvePane from "./JsonResolvePane.svelte";
export class JsonResolveModal extends Modal { export class JsonResolveModal extends Modal {
// result: Array<[number, string]>; // result: Array<[number, string]>;
filename: string; filename: FilePath;
callback: (keepRev: string, mergedStr?: string) => Promise<void>; callback: (keepRev: string, mergedStr?: string) => Promise<void>;
docs: LoadedEntry[]; docs: LoadedEntry[];
component: JsonResolvePane; component: JsonResolvePane;
nameA: string;
nameB: string;
defaultSelect: string;
constructor(app: App, filename: string, docs: LoadedEntry[], callback: (keepRev: string, mergedStr?: string) => Promise<void>) { constructor(app: App, filename: FilePath, docs: LoadedEntry[], callback: (keepRev: string, mergedStr?: string) => Promise<void>, nameA?: string, nameB?: string, defaultSelect?: string) {
super(app); super(app);
this.callback = callback; this.callback = callback;
this.filename = filename; this.filename = filename;
this.docs = docs; this.docs = docs;
this.nameA = nameA;
this.nameB = nameB;
this.defaultSelect = defaultSelect;
} }
async UICallback(keepRev: string, mergedStr?: string) { async UICallback(keepRev: string, mergedStr?: string) {
this.close(); this.close();
@@ -31,6 +37,10 @@ export class JsonResolveModal extends Modal {
target: contentEl, target: contentEl,
props: { props: {
docs: this.docs, docs: this.docs,
filename: this.filename,
nameA: this.nameA,
nameB: this.nameB,
defaultSelect: this.defaultSelect,
callback: (keepRev, mergedStr) => this.UICallback(keepRev, mergedStr), callback: (keepRev, mergedStr) => this.UICallback(keepRev, mergedStr),
}, },
}); });

46
src/JsonResolvePane.svelte
View File

@@ -1,15 +1,18 @@
<script lang="ts"> <script lang="ts">
import { Diff, DIFF_DELETE, DIFF_INSERT, diff_match_patch } from "diff-match-patch"; import { Diff, DIFF_DELETE, DIFF_INSERT, diff_match_patch } from "diff-match-patch";
import type { LoadedEntry } from "./lib/src/types"; import type { FilePath, LoadedEntry } from "./lib/src/types";
import { base64ToString } from "./lib/src/strbin"; import { base64ToString } from "./lib/src/strbin";
import { getDocData } from "./lib/src/utils"; import { getDocData } from "./lib/src/utils";
import { id2path, mergeObject } from "./utils"; import { mergeObject } from "./utils";
export let docs: LoadedEntry[] = []; export let docs: LoadedEntry[] = [];
export let callback: (keepRev: string, mergedStr?: string) => Promise<void> = async (_, __) => { export let callback: (keepRev: string, mergedStr?: string) => Promise<void> = async (_, __) => {
Promise.resolve(); Promise.resolve();
}; };
export let filename: FilePath = "" as FilePath;
export let nameA: string = "A";
export let nameB: string = "B";
export let defaultSelect: string = "";
let docA: LoadedEntry = undefined; let docA: LoadedEntry = undefined;
let docB: LoadedEntry = undefined; let docB: LoadedEntry = undefined;
let docAContent = ""; let docAContent = "";
@@ -19,14 +22,8 @@
let objAB: any = {}; let objAB: any = {};
let objBA: any = {}; let objBA: any = {};
let diffs: Diff[]; let diffs: Diff[];
const modes = [ type SelectModes = "" | "A" | "B" | "AB" | "BA";
["", "Not now"], let mode: SelectModes = defaultSelect as SelectModes;
["A", "A"],
["B", "B"],
["AB", "A + B"],
["BA", "B + A"],
] as ["" | "A" | "B" | "AB" | "BA", string][];
let mode: "" | "A" | "B" | "AB" | "BA" = "";
function docToString(doc: LoadedEntry) { function docToString(doc: LoadedEntry) {
return doc.datatype == "plain" ? getDocData(doc.data) : base64ToString(doc.data); return doc.datatype == "plain" ? getDocData(doc.data) : base64ToString(doc.data);
@@ -46,8 +43,13 @@
return getDiff(JSON.stringify(a, null, 2), JSON.stringify(b, null, 2)); return getDiff(JSON.stringify(a, null, 2), JSON.stringify(b, null, 2));
} }
function apply() { function apply() {
if (mode == "A") return callback(docA._rev, null); if (docA._id == docB._id) {
if (mode == "B") return callback(docB._rev, null); if (mode == "A") return callback(docA._rev, null);
if (mode == "B") return callback(docB._rev, null);
} else {
if (mode == "A") return callback(null, docToString(docA));
if (mode == "B") return callback(null, docToString(docB));
}
if (mode == "BA") return callback(null, JSON.stringify(objBA, null, 2)); if (mode == "BA") return callback(null, JSON.stringify(objBA, null, 2));
if (mode == "AB") return callback(null, JSON.stringify(objAB, null, 2)); if (mode == "AB") return callback(null, JSON.stringify(objAB, null, 2));
callback(null, null); callback(null, null);
@@ -91,13 +93,19 @@
$: selectedObj = mode in mergedObjs ? mergedObjs[mode] : {}; $: selectedObj = mode in mergedObjs ? mergedObjs[mode] : {};
$: { $: {
diffs = getJsonDiff(objA, selectedObj); diffs = getJsonDiff(objA, selectedObj);
console.dir(selectedObj);
} }
$: filename = id2path(docA?._id ?? "");
$: modes = [
["", "Not now"],
["A", nameA || "A"],
["B", nameB || "B"],
["AB", `${nameA || "A"} + ${nameB || "B"}`],
["BA", `${nameB || "B"} + ${nameA || "A"}`],
] as ["" | "A" | "B" | "AB" | "BA", string][];
</script> </script>
<h1>Conflicted settings</h1> <h1>Conflicted settings</h1>
<div><span>{filename}</span></div> <h2>{filename}</h2>
{#if !docA || !docB} {#if !docA || !docB}
<div class="message">Just for a minute, please!</div> <div class="message">Just for a minute, please!</div>
<div class="buttons"> <div class="buttons">
@@ -125,12 +133,14 @@
NO PREVIEW NO PREVIEW
{/if} {/if}
<div> <div>
A Rev:{revStringToRevNumber(docA._rev)} ,{new Date(docA.mtime).toLocaleString()} {nameA}
{#if docA._id == docB._id} Rev:{revStringToRevNumber(docA._rev)} {/if} ,{new Date(docA.mtime).toLocaleString()}
{docAContent.length} letters {docAContent.length} letters
</div> </div>
<div> <div>
B Rev:{revStringToRevNumber(docB._rev)} ,{new Date(docB.mtime).toLocaleString()} {nameB}
{#if docA._id == docB._id} Rev:{revStringToRevNumber(docB._rev)} {/if} ,{new Date(docB.mtime).toLocaleString()}
{docBContent.length} letters {docBContent.length} letters
</div> </div>

12
src/LiveSyncCommands.ts
View File

@@ -1,4 +1,4 @@
import { EntryDoc } from "./lib/src/types"; import { AnyEntry, DocumentID, EntryDoc, EntryHasPath, FilePath, FilePathWithPrefix } from "./lib/src/types";
import { PouchDB } from "./lib/src/pouchdb-browser.js"; import { PouchDB } from "./lib/src/pouchdb-browser.js";
import type ObsidianLiveSyncPlugin from "./main"; import type ObsidianLiveSyncPlugin from "./main";
@@ -14,6 +14,16 @@ export abstract class LiveSyncCommands {
get localDatabase() { get localDatabase() {
return this.plugin.localDatabase; return this.plugin.localDatabase;
} }
id2path(id: DocumentID, entry?: EntryHasPath, stripPrefix?: boolean): FilePathWithPrefix {
return this.plugin.id2path(id, entry, stripPrefix);
}
async path2id(filename: FilePathWithPrefix | FilePath, prefix?: string): Promise<DocumentID> {
return await this.plugin.path2id(filename, prefix);
}
getPath(entry: AnyEntry): FilePathWithPrefix {
return this.plugin.getPath(entry);
}
constructor(plugin: ObsidianLiveSyncPlugin) { constructor(plugin: ObsidianLiveSyncPlugin) {
this.plugin = plugin; this.plugin = plugin;
} }

178
src/LocalPouchDB.ts
View File

@@ -1,178 +0,0 @@
import { requestUrl, RequestUrlParam, RequestUrlResponse } from "./deps";
import { KeyValueDatabase, OpenKeyValueDatabase } from "./KeyValueDB.js";
import { LocalPouchDBBase } from "./lib/src/LocalPouchDBBase.js";
import { Logger } from "./lib/src/logger.js";
import { PouchDB } from "./lib/src/pouchdb-browser.js";
import { EntryDoc, LOG_LEVEL, ObsidianLiveSyncSettings } from "./lib/src/types.js";
import { enableEncryption } from "./lib/src/utils_couchdb.js";
import { isCloudantURI, isValidRemoteCouchDBURI } from "./lib/src/utils_couchdb.js";
import { id2path, path2id } from "./utils.js";
export class LocalPouchDB extends LocalPouchDBBase {
kvDB: KeyValueDatabase;
settings: ObsidianLiveSyncSettings;
id2path(filename: string): string {
return id2path(filename);
}
path2id(filename: string): string {
return path2id(filename);
}
CreatePouchDBInstance<T>(name?: string, options?: PouchDB.Configuration.DatabaseConfiguration): PouchDB.Database<T> {
if (this.settings.useIndexedDBAdapter) {
options.adapter = "indexeddb";
return new PouchDB(name + "-indexeddb", options);
}
return new PouchDB(name, options);
}
beforeOnUnload(): void {
this.kvDB.close();
}
onClose(): void {
this.kvDB.close();
}
async onInitializeDatabase(): Promise<void> {
this.kvDB = await OpenKeyValueDatabase(this.dbname + "-livesync-kv");
}
async onResetDatabase(): Promise<void> {
await this.kvDB.destroy();
}
last_successful_post = false;
getLastPostFailedBySize() {
return !this.last_successful_post;
}
async fetchByAPI(request: RequestUrlParam): Promise<RequestUrlResponse> {
const ret = await requestUrl(request);
if (ret.status - (ret.status % 100) !== 200) {
const er: Error & { status?: number } = new Error(`Request Error:${ret.status}`);
if (ret.json) {
er.message = ret.json.reason;
er.name = `${ret.json.error ?? ""}:${ret.json.message ?? ""}`;
}
er.status = ret.status;
throw er;
}
return ret;
}
async connectRemoteCouchDB(uri: string, auth: { username: string; password: string }, disableRequestURI: boolean, passphrase: string | false, useDynamicIterationCount: boolean): Promise<string | { db: PouchDB.Database<EntryDoc>; info: PouchDB.Core.DatabaseInfo }> {
if (!isValidRemoteCouchDBURI(uri)) return "Remote URI is not valid";
if (uri.toLowerCase() != uri) return "Remote URI and database name could not contain capital letters.";
if (uri.indexOf(" ") !== -1) return "Remote URI and database name could not contain spaces.";
let authHeader = "";
if (auth.username && auth.password) {
const utf8str = String.fromCharCode.apply(null, new TextEncoder().encode(`${auth.username}:${auth.password}`));
const encoded = window.btoa(utf8str);
authHeader = "Basic " + encoded;
} else {
authHeader = "";
}
// const _this = this;
const conf: PouchDB.HttpAdapter.HttpAdapterConfiguration = {
adapter: "http",
auth,
fetch: async (url: string | Request, opts: RequestInit) => {
let size = "";
const localURL = url.toString().substring(uri.length);
const method = opts.method ?? "GET";
if (opts.body) {
const opts_length = opts.body.toString().length;
if (opts_length > 1000 * 1000 * 10) {
// over 10MB
if (isCloudantURI(uri)) {
this.last_successful_post = false;
Logger("This request should fail on IBM Cloudant.", LOG_LEVEL.VERBOSE);
throw new Error("This request should fail on IBM Cloudant.");
}
}
size = ` (${opts_length})`;
}
if (!disableRequestURI && typeof url == "string" && typeof (opts.body ?? "") == "string") {
const body = opts.body as string;
const transformedHeaders = { ...(opts.headers as Record<string, string>) };
if (authHeader != "") transformedHeaders["authorization"] = authHeader;
delete transformedHeaders["host"];
delete transformedHeaders["Host"];
delete transformedHeaders["content-length"];
delete transformedHeaders["Content-Length"];
const requestParam: RequestUrlParam = {
url: url as string,
method: opts.method,
body: body,
headers: transformedHeaders,
contentType: "application/json",
// contentType: opts.headers,
};
try {
const r = await this.fetchByAPI(requestParam);
if (method == "POST" || method == "PUT") {
this.last_successful_post = r.status - (r.status % 100) == 200;
} else {
this.last_successful_post = true;
}
Logger(`HTTP:${method}${size} to:${localURL} -> ${r.status}`, LOG_LEVEL.DEBUG);
return new Response(r.arrayBuffer, {
headers: r.headers,
status: r.status,
statusText: `${r.status}`,
});
} catch (ex) {
Logger(`HTTP:${method}${size} to:${localURL} -> failed`, LOG_LEVEL.VERBOSE);
// limit only in bulk_docs.
if (url.toString().indexOf("_bulk_docs") !== -1) {
this.last_successful_post = false;
}
Logger(ex);
throw ex;
}
}
// -old implementation
try {
const response: Response = await fetch(url, opts);
if (method == "POST" || method == "PUT") {
this.last_successful_post = response.ok;
} else {
this.last_successful_post = true;
}
Logger(`HTTP:${method}${size} to:${localURL} -> ${response.status}`, LOG_LEVEL.DEBUG);
return response;
} catch (ex) {
Logger(`HTTP:${method}${size} to:${localURL} -> failed`, LOG_LEVEL.VERBOSE);
// limit only in bulk_docs.
if (url.toString().indexOf("_bulk_docs") !== -1) {
this.last_successful_post = false;
}
Logger(ex);
throw ex;
}
// return await fetch(url, opts);
},
};
const db: PouchDB.Database<EntryDoc> = new PouchDB<EntryDoc>(uri, conf);
if (passphrase !== "false" && typeof passphrase === "string") {
enableEncryption(db, passphrase, useDynamicIterationCount);
}
try {
const info = await db.info();
return { db: db, info: info };
} catch (ex) {
let msg = `${ex.name}:${ex.message}`;
if (ex.name == "TypeError" && ex.message == "Failed to fetch") {
msg += "\n**Note** This error caused by many reasons. The only sure thing is you didn't touch the server.\nTo check details, open inspector.";
}
Logger(ex, LOG_LEVEL.VERBOSE);
return msg;
}
}
}

619
src/ObsidianLiveSyncSettingTab.ts
View File

@@ -1,6 +1,5 @@
import { App, PluginSettingTab, Setting, sanitizeHTMLToDom, RequestUrlParam, requestUrl, TextAreaComponent, MarkdownRenderer, stringifyYaml } from "./deps"; import { App, PluginSettingTab, Setting, sanitizeHTMLToDom, TextAreaComponent, MarkdownRenderer, stringifyYaml } from "./deps";
import { DEFAULT_SETTINGS, LOG_LEVEL, ObsidianLiveSyncSettings, ConfigPassphraseStore, RemoteDBSettings } from "./lib/src/types"; import { DEFAULT_SETTINGS, LOG_LEVEL, ObsidianLiveSyncSettings, ConfigPassphraseStore, RemoteDBSettings } from "./lib/src/types";
import { path2id, id2path } from "./utils";
import { delay } from "./lib/src/utils"; import { delay } from "./lib/src/utils";
import { Semaphore } from "./lib/src/semaphore"; import { Semaphore } from "./lib/src/semaphore";
import { versionNumberString2Number } from "./lib/src/strbin"; import { versionNumberString2Number } from "./lib/src/strbin";
@@ -8,24 +7,9 @@ import { Logger } from "./lib/src/logger";
import { checkSyncInfo, isCloudantURI } from "./lib/src/utils_couchdb.js"; import { checkSyncInfo, isCloudantURI } from "./lib/src/utils_couchdb.js";
import { testCrypt } from "./lib/src/e2ee_v2"; import { testCrypt } from "./lib/src/e2ee_v2";
import ObsidianLiveSyncPlugin from "./main"; import ObsidianLiveSyncPlugin from "./main";
import { balanceChunks, localDatabaseCleanUp, performRebuildDB, remoteDatabaseCleanup, requestToCouchDB } from "./utils";
const requestToCouchDB = async (baseUri: string, username: string, password: string, origin: string, key?: string, body?: string) => {
const utf8str = String.fromCharCode.apply(null, new TextEncoder().encode(`${username}:${password}`));
const encoded = window.btoa(utf8str);
const authHeader = "Basic " + encoded;
// const origin = "capacitor://localhost";
const transformedHeaders: Record<string, string> = { authorization: authHeader, origin: origin };
const uri = `${baseUri}/_node/_local/_config${key ? "/" + key : ""}`;
const requestParam: RequestUrlParam = {
url: uri,
method: body ? "PUT" : "GET",
headers: transformedHeaders,
contentType: "application/json",
body: body ? JSON.stringify(body) : undefined,
};
return await requestUrl(requestParam);
};
export class ObsidianLiveSyncSettingTab extends PluginSettingTab { export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
plugin: ObsidianLiveSyncPlugin; plugin: ObsidianLiveSyncPlugin;
selectedScreen = ""; selectedScreen = "";
@@ -75,7 +59,7 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
<label class='sls-setting-label c-40'><input type='radio' name='disp' value='40' class='sls-setting-tab' ><div class='sls-setting-menu-btn'>🔧</div></label> <label class='sls-setting-label c-40'><input type='radio' name='disp' value='40' class='sls-setting-tab' ><div class='sls-setting-menu-btn'>🔧</div></label>
<label class='sls-setting-label c-50 wizardHidden'><input type='radio' name='disp' value='50' class='sls-setting-tab' ><div class='sls-setting-menu-btn'>🧰</div></label> <label class='sls-setting-label c-50 wizardHidden'><input type='radio' name='disp' value='50' class='sls-setting-tab' ><div class='sls-setting-menu-btn'>🧰</div></label>
<label class='sls-setting-label c-60 wizardHidden'><input type='radio' name='disp' value='60' class='sls-setting-tab' ><div class='sls-setting-menu-btn'>🔌</div></label> <label class='sls-setting-label c-60 wizardHidden'><input type='radio' name='disp' value='60' class='sls-setting-tab' ><div class='sls-setting-menu-btn'>🔌</div></label>
<label class='sls-setting-label c-70 wizardHidden'><input type='radio' name='disp' value='70' class='sls-setting-tab' ><div class='sls-setting-menu-btn'>🚑</div></label> <label class='sls-setting-label c-70 wizardHidden'><input type='radio' name='disp' value='70' class='sls-setting-tab' ><div class='sls-setting-menu-btn'>🎛️</div></label>
`; `;
const menuTabs = w.querySelectorAll(".sls-setting-label"); const menuTabs = w.querySelectorAll(".sls-setting-label");
const changeDisplay = (screen: string) => { const changeDisplay = (screen: string) => {
@@ -88,11 +72,11 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
} }
w.querySelectorAll(`.sls-setting-label`).forEach((element) => { w.querySelectorAll(`.sls-setting-label`).forEach((element) => {
element.removeClass("selected"); element.removeClass("selected");
(element.querySelector("input[type=radio]") as HTMLInputElement).checked = false; (element.querySelector<HTMLInputElement>("input[type=radio]")).checked = false;
}); });
w.querySelectorAll(`.sls-setting-label.c-${screen}`).forEach((element) => { w.querySelectorAll(`.sls-setting-label.c-${screen}`).forEach((element) => {
element.addClass("selected"); element.addClass("selected");
(element.querySelector("input[type=radio]") as HTMLInputElement).checked = true; (element.querySelector<HTMLInputElement>("input[type=radio]")).checked = true;
}); });
this.selectedScreen = screen; this.selectedScreen = screen;
}; };
@@ -307,11 +291,13 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
this.plugin.settings.encrypt = value; this.plugin.settings.encrypt = value;
passphraseSetting.setDisabled(!value); passphraseSetting.setDisabled(!value);
dynamicIteration.setDisabled(!value); dynamicIteration.setDisabled(!value);
usePathObfuscationEl.setDisabled(!value);
await this.plugin.saveSettings(); await this.plugin.saveSettings();
} else { } else {
encrypt = value; encrypt = value;
passphraseSetting.setDisabled(!value); passphraseSetting.setDisabled(!value);
dynamicIteration.setDisabled(!value); dynamicIteration.setDisabled(!value);
usePathObfuscationEl.setDisabled(!value);
await this.plugin.saveSettings(); await this.plugin.saveSettings();
markDirtyControl(); markDirtyControl();
} }
@@ -322,7 +308,8 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
const markDirtyControl = () => { const markDirtyControl = () => {
passphraseSetting.controlEl.toggleClass("sls-item-dirty", passphrase != this.plugin.settings.passphrase); passphraseSetting.controlEl.toggleClass("sls-item-dirty", passphrase != this.plugin.settings.passphrase);
e2e.controlEl.toggleClass("sls-item-dirty", encrypt != this.plugin.settings.encrypt); e2e.controlEl.toggleClass("sls-item-dirty", encrypt != this.plugin.settings.encrypt);
dynamicIteration.controlEl.toggleClass("sls-item-dirty", useDynamicIterationCount != this.plugin.settings.useDynamicIterationCount) dynamicIteration.controlEl.toggleClass("sls-item-dirty", useDynamicIterationCount != this.plugin.settings.useDynamicIterationCount);
usePathObfuscationEl.controlEl.toggleClass("sls-item-dirty", usePathObfuscation != this.plugin.settings.usePathObfuscation);
} }
const passphraseSetting = new Setting(containerRemoteDatabaseEl) const passphraseSetting = new Setting(containerRemoteDatabaseEl)
@@ -345,6 +332,23 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
}); });
passphraseSetting.setDisabled(!encrypt); passphraseSetting.setDisabled(!encrypt);
let usePathObfuscation = this.plugin.settings.usePathObfuscation;
const usePathObfuscationEl = new Setting(containerRemoteDatabaseEl)
.setName("Path Obfuscation")
.setDesc("(Experimental) Obfuscate paths of files. If we configured, we should rebuild the database.")
.addToggle((toggle) =>
toggle.setValue(usePathObfuscation).onChange(async (value) => {
if (inWizard) {
this.plugin.settings.usePathObfuscation = value;
await this.plugin.saveSettings();
} else {
usePathObfuscation = value;
await this.plugin.saveSettings();
markDirtyControl();
}
})
);
const dynamicIteration = new Setting(containerRemoteDatabaseEl) const dynamicIteration = new Setting(containerRemoteDatabaseEl)
.setName("Use dynamic iteration count (experimental)") .setName("Use dynamic iteration count (experimental)")
.setDesc("Balancing the encryption/decryption load against the length of the passphrase if toggled. (v0.17.5 or higher required)") .setDesc("Balancing the encryption/decryption load against the length of the passphrase if toggled. (v0.17.5 or higher required)")
@@ -401,24 +405,16 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
if (!encrypt) { if (!encrypt) {
passphrase = ""; passphrase = "";
} }
this.plugin.settings.liveSync = false; this.plugin.addOnSetup.suspendAllSync();
this.plugin.settings.periodicReplication = false; this.plugin.addOnSetup.suspendExtraSync();
this.plugin.settings.syncOnSave = false;
this.plugin.settings.syncOnStart = false;
this.plugin.settings.syncOnFileOpen = false;
this.plugin.settings.syncAfterMerge = false;
this.plugin.settings.encrypt = encrypt; this.plugin.settings.encrypt = encrypt;
this.plugin.settings.passphrase = passphrase; this.plugin.settings.passphrase = passphrase;
this.plugin.settings.useDynamicIterationCount = useDynamicIterationCount; this.plugin.settings.useDynamicIterationCount = useDynamicIterationCount;
this.plugin.settings.usePathObfuscation = usePathObfuscation;
await this.plugin.saveSettings(); await this.plugin.saveSettings();
markDirtyControl(); markDirtyControl();
if (sendToServer) { if (sendToServer) {
await this.plugin.initializeDatabase(true); await this.plugin.addOnSetup.rebuildRemote()
await this.plugin.markRemoteLocked();
await this.plugin.tryResetRemoteDatabase();
await this.plugin.markRemoteLocked();
await this.plugin.replicateAllToServer(true);
} else { } else {
await this.plugin.markRemoteResolved(); await this.plugin.markRemoteResolved();
await this.plugin.replicate(true); await this.plugin.replicate(true);
@@ -430,90 +426,60 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
.setClass("wizardHidden") .setClass("wizardHidden")
.addButton((button) => .addButton((button) =>
button button
.setButtonText("Apply") .setButtonText("Just apply")
.setWarning()
.setDisabled(false)
.onClick(async () => {
await applyEncryption(true);
})
)
.addButton((button) =>
button
.setButtonText("Apply w/o rebuilding")
.setWarning() .setWarning()
.setDisabled(false) .setDisabled(false)
.onClick(async () => { .onClick(async () => {
await applyEncryption(false); await applyEncryption(false);
}) })
); )
const rebuildDB = async (method: "localOnly" | "remoteOnly" | "rebuildBothByThisDevice") => {
this.plugin.settings.liveSync = false;
this.plugin.settings.periodicReplication = false;
this.plugin.settings.syncOnSave = false;
this.plugin.settings.syncOnStart = false;
this.plugin.settings.syncOnFileOpen = false;
this.plugin.settings.syncAfterMerge = false;
this.plugin.settings.syncInternalFiles = false;
this.plugin.settings.usePluginSync = false;
Logger("Hidden files and plugin synchronization have been temporarily disabled. Please enable them after the fetching, if you need them.", LOG_LEVEL.NOTICE)
await this.plugin.saveSettings();
applyDisplayEnabled();
await delay(2000);
if (method == "localOnly") {
await this.plugin.resetLocalDatabase();
await delay(1000);
await this.plugin.markRemoteResolved();
await this.plugin.openDatabase();
this.plugin.isReady = true;
await this.plugin.replicateAllFromServer(true);
}
if (method == "remoteOnly") {
await this.plugin.markRemoteLocked();
await this.plugin.tryResetRemoteDatabase();
await this.plugin.markRemoteLocked();
await this.plugin.replicateAllToServer(true);
}
if (method == "rebuildBothByThisDevice") {
await this.plugin.resetLocalDatabase();
await delay(1000);
await this.plugin.initializeDatabase(true);
await this.plugin.markRemoteLocked();
await this.plugin.tryResetRemoteDatabase();
await this.plugin.markRemoteLocked();
await this.plugin.replicateAllToServer(true);
}
}
new Setting(containerRemoteDatabaseEl)
.setName("Overwrite remote database")
.setDesc("Overwrite remote database with local DB and passphrase.")
.setClass("wizardHidden")
.addButton((button) => .addButton((button) =>
button button
.setButtonText("Send") .setButtonText("Apply and Fetch")
.setWarning() .setWarning()
.setDisabled(false) .setDisabled(false)
.onClick(async () => { .onClick(async () => {
await rebuildDB("remoteOnly"); await rebuildDB("localOnly");
}) })
) )
new Setting(containerRemoteDatabaseEl)
.setName("Rebuild everything")
.setDesc("Rebuild local and remote database with local files.")
.setClass("wizardHidden")
.addButton((button) => .addButton((button) =>
button button
.setButtonText("Rebuild") .setButtonText("Apply and Rebuild")
.setWarning() .setWarning()
.setDisabled(false) .setDisabled(false)
.onClick(async () => { .onClick(async () => {
await rebuildDB("rebuildBothByThisDevice"); await rebuildDB("rebuildBothByThisDevice");
}) })
) );
const rebuildDB = async (method: "localOnly" | "remoteOnly" | "rebuildBothByThisDevice") => {
if (encrypt && passphrase == "") {
Logger("If you enable encryption, you have to set the passphrase", LOG_LEVEL.NOTICE);
return;
}
if (encrypt && !(await testCrypt())) {
Logger("WARNING! Your device would not support encryption.", LOG_LEVEL.NOTICE);
return;
}
if (!encrypt) {
passphrase = "";
}
this.plugin.addOnSetup.suspendAllSync();
this.plugin.addOnSetup.suspendExtraSync();
this.plugin.settings.encrypt = encrypt;
this.plugin.settings.passphrase = passphrase;
this.plugin.settings.useDynamicIterationCount = useDynamicIterationCount;
this.plugin.settings.usePathObfuscation = usePathObfuscation;
Logger("All synchronization have been temporarily disabled. Please enable them after the fetching, if you need them.", LOG_LEVEL.NOTICE)
await this.plugin.saveSettings();
markDirtyControl();
applyDisplayEnabled();
// @ts-ignore
this.plugin.app.setting.close();
await delay(2000);
await performRebuildDB(this.plugin, method);
}
new Setting(containerRemoteDatabaseEl) new Setting(containerRemoteDatabaseEl)
.setName("Test Database Connection") .setName("Test Database Connection")
@@ -698,19 +664,6 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
text: "", text: "",
}); });
new Setting(containerRemoteDatabaseEl)
.setName("Lock remote database")
.setDesc("Lock remote database to prevent synchronization with other devices.")
.setClass("wizardHidden")
.addButton((button) =>
button
.setButtonText("Lock")
.setDisabled(false)
.setWarning()
.onClick(async () => {
await this.plugin.markRemoteLocked();
})
);
let rebuildRemote = false; let rebuildRemote = false;
new Setting(containerRemoteDatabaseEl) new Setting(containerRemoteDatabaseEl)
@@ -775,21 +728,6 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
}) })
); );
new Setting(containerLocalDatabaseEl)
.setName("Fetch rebuilt DB")
.setDesc("Restore or reconstruct local database from remote database.")
.setClass("wizardHidden")
.addButton((button) =>
button
.setButtonText("Fetch")
.setWarning()
.setDisabled(false)
.onClick(async () => {
await rebuildDB("localOnly");
})
)
let newDatabaseName = this.plugin.settings.additionalSuffixOfDatabaseName + ""; let newDatabaseName = this.plugin.settings.additionalSuffixOfDatabaseName + "";
new Setting(containerLocalDatabaseEl) new Setting(containerLocalDatabaseEl)
.setName("Database suffix") .setName("Database suffix")
@@ -920,7 +858,7 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
}); });
c.addClass("op-warn"); c.addClass("op-warn");
} }
containerSyncSettingEl.createEl("h3", { text: "Synchronization Methods" });
const syncLive: Setting[] = []; const syncLive: Setting[] = [];
const syncNonLive: Setting[] = []; const syncNonLive: Setting[] = [];
syncLive.push( syncLive.push(
@@ -973,6 +911,7 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
text.inputEl.setAttribute("type", "number"); text.inputEl.setAttribute("type", "number");
}), }),
new Setting(containerSyncSettingEl) new Setting(containerSyncSettingEl)
.setName("Sync on Save") .setName("Sync on Save")
.setDesc("When you save file, sync automatically") .setDesc("When you save file, sync automatically")
@@ -1014,7 +953,7 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
}) })
), ),
); );
containerSyncSettingEl.createEl("h3", { text: "File deletion" });
new Setting(containerSyncSettingEl) new Setting(containerSyncSettingEl)
.setName("Use Trash for deleted files") .setName("Use Trash for deleted files")
.setDesc("Do not delete files that are deleted in remote, just move to trash.") .setDesc("Do not delete files that are deleted in remote, just move to trash.")
@@ -1035,6 +974,7 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
}) })
); );
containerSyncSettingEl.createEl("h3", { text: "Conflict resolution" });
new Setting(containerSyncSettingEl) new Setting(containerSyncSettingEl)
.setName("Use newer file if conflicted (beta)") .setName("Use newer file if conflicted (beta)")
.setDesc("Resolve conflicts by newer files automatically.") .setDesc("Resolve conflicts by newer files automatically.")
@@ -1073,14 +1013,56 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
}) })
); );
new Setting(containerSyncSettingEl) containerSyncSettingEl.createEl("h3", { text: "Hidden files" });
.setName("Sync hidden files") const LABEL_ENABLED = "🔁 : Enabled";
.addToggle((toggle) => const LABEL_DISABLED = "⏹️ : Disabled"
toggle.setValue(this.plugin.settings.syncInternalFiles).onChange(async (value) => {
this.plugin.settings.syncInternalFiles = value; const hiddenFileSyncSetting = new Setting(containerSyncSettingEl)
await this.plugin.saveSettings(); .setName("Hidden file synchronization")
const hiddenFileSyncSettingEl = hiddenFileSyncSetting.settingEl
const hiddenFileSyncSettingDiv = hiddenFileSyncSettingEl.createDiv("");
hiddenFileSyncSettingDiv.innerText = this.plugin.settings.syncInternalFiles ? LABEL_ENABLED : LABEL_DISABLED;
if (this.plugin.settings.syncInternalFiles) {
new Setting(containerSyncSettingEl)
.setName("Disable Hidden files sync")
.addButton((button) => {
button.setButtonText("Disable")
.onClick(async () => {
this.plugin.settings.syncInternalFiles = false;
await this.plugin.saveSettings();
this.display();
})
}) })
); } else {
new Setting(containerSyncSettingEl)
.setName("Enable Hidden files sync")
.addButton((button) => {
button.setButtonText("Merge")
.onClick(async () => {
// @ts-ignore
this.plugin.app.setting.close()
await this.plugin.addOnSetup.configureHiddenFileSync("MERGE");
})
})
.addButton((button) => {
button.setButtonText("Fetch")
.onClick(async () => {
// @ts-ignore
this.plugin.app.setting.close()
await this.plugin.addOnSetup.configureHiddenFileSync("FETCH");
})
})
.addButton((button) => {
button.setButtonText("Overwrite")
.onClick(async () => {
// @ts-ignore
this.plugin.app.setting.close()
await this.plugin.addOnSetup.configureHiddenFileSync("OVERWRITE");
})
});
}
new Setting(containerSyncSettingEl) new Setting(containerSyncSettingEl)
.setName("Scan for hidden files before replication") .setName("Scan for hidden files before replication")
@@ -1145,31 +1127,9 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
}) })
}) })
new Setting(containerSyncSettingEl)
.setName("Touch hidden files")
.setDesc("Update the modified time of all hidden files to the current time.")
.addButton((button) =>
button
.setButtonText("Touch")
.setWarning()
.setDisabled(false)
.onClick(async () => {
const filesAll = await this.plugin.addOnHiddenFileSync.scanInternalFiles();
const targetFiles = await this.plugin.filterTargetFiles(filesAll);
const now = Date.now();
const newFiles = targetFiles.map(e => ({ ...e, mtime: now }));
let i = 0;
const maxFiles = newFiles.length;
for (const file of newFiles) {
i++;
Logger(`Touched:${file.path} (${i}/${maxFiles})`, LOG_LEVEL.NOTICE, "touch-files");
await this.plugin.applyMTimeToFile(file);
}
})
)
containerSyncSettingEl.createEl("h3", { containerSyncSettingEl.createEl("h3", {
text: sanitizeHTMLToDom(`Experimental`), text: sanitizeHTMLToDom(`Synchronization filters`),
}); });
new Setting(containerSyncSettingEl) new Setting(containerSyncSettingEl)
.setName("Regular expression to ignore files") .setName("Regular expression to ignore files")
@@ -1217,7 +1177,7 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
return text; return text;
} }
); );
containerSyncSettingEl.createEl("h3", { text: "Efficiency" });
new Setting(containerSyncSettingEl) new Setting(containerSyncSettingEl)
.setName("Chunk size") .setName("Chunk size")
.setDesc("Customize chunk size for binary files (0.1MBytes). This cannot be increased when using IBM Cloudant.") .setDesc("Customize chunk size for binary files (0.1MBytes). This cannot be increased when using IBM Cloudant.")
@@ -1236,7 +1196,7 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
}); });
new Setting(containerSyncSettingEl) new Setting(containerSyncSettingEl)
.setName("Read chunks online.") .setName("Read chunks online")
.setDesc("If this option is enabled, LiveSync reads chunks online directly instead of replicating them locally. Increasing Custom chunk size is recommended.") .setDesc("If this option is enabled, LiveSync reads chunks online directly instead of replicating them locally. Increasing Custom chunk size is recommended.")
.addToggle((toggle) => { .addToggle((toggle) => {
toggle toggle
@@ -1246,8 +1206,7 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
await this.plugin.saveSettings(); await this.plugin.saveSettings();
}) })
return toggle; return toggle;
} });
);
containerSyncSettingEl.createEl("h3", { containerSyncSettingEl.createEl("h3", {
text: sanitizeHTMLToDom(`Advanced settings`), text: sanitizeHTMLToDom(`Advanced settings`),
}); });
@@ -1441,8 +1400,8 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
} }
} }
this.plugin.saveSettings(); this.plugin.saveSettings();
await this.plugin.realizeSettingSyncMode();
this.display(); this.display();
await this.plugin.realizeSettingSyncMode();
if (inWizard) { if (inWizard) {
// @ts-ignore // @ts-ignore
this.plugin.app.setting.close() this.plugin.app.setting.close()
@@ -1640,7 +1599,174 @@ ${stringifyYaml(pluginConfig)}`;
}) })
); );
new Setting(containerHatchEl)
addScreenElement("50", containerHatchEl);
// With great respect, thank you TfTHacker!
// Refer: https://github.com/TfTHacker/obsidian42-brat/blob/main/src/features/BetaPlugins.ts
const containerPluginSettings = containerEl.createDiv();
containerPluginSettings.createEl("h3", { text: "Customization sync (beta)" });
const updateDisabledOfDeviceAndVaultName = () => {
vaultName.setDisabled(this.plugin.settings.autoSweepPlugins || this.plugin.settings.autoSweepPluginsPeriodic);
vaultName.setTooltip(this.plugin.settings.autoSweepPlugins || this.plugin.settings.autoSweepPluginsPeriodic ? "You could not change when you enabling auto scan." : "");
};
new Setting(containerPluginSettings).setName("Enable customization sync").addToggle((toggle) =>
toggle.setValue(this.plugin.settings.usePluginSync).onChange(async (value) => {
this.plugin.settings.usePluginSync = value;
await this.plugin.saveSettings();
})
);
new Setting(containerPluginSettings)
.setName("Scan customization automatically")
.setDesc("Scan customization before replicating.")
.addToggle((toggle) =>
toggle.setValue(this.plugin.settings.autoSweepPlugins).onChange(async (value) => {
this.plugin.settings.autoSweepPlugins = value;
updateDisabledOfDeviceAndVaultName();
await this.plugin.saveSettings();
})
);
new Setting(containerPluginSettings)
.setName("Scan customization periodically")
.setDesc("Scan customization every 1 minute. This configuration will be ignored if monitoring changes of hidden files has been enabled.")
.addToggle((toggle) =>
toggle.setValue(this.plugin.settings.autoSweepPluginsPeriodic).onChange(async (value) => {
this.plugin.settings.autoSweepPluginsPeriodic = value;
updateDisabledOfDeviceAndVaultName();
await this.plugin.saveSettings();
})
);
new Setting(containerPluginSettings)
.setName("Notify customized")
.setDesc("Notify when other device has newly customized.")
.addToggle((toggle) =>
toggle.setValue(this.plugin.settings.notifyPluginOrSettingUpdated).onChange(async (value) => {
this.plugin.settings.notifyPluginOrSettingUpdated = value;
await this.plugin.saveSettings();
})
);
const vaultName = new Setting(containerPluginSettings)
.setName("Device name")
.setDesc("")
.addText((text) => {
text.setPlaceholder("desktop")
.setValue(this.plugin.deviceAndVaultName)
.onChange(async (value) => {
this.plugin.deviceAndVaultName = value;
await this.plugin.saveSettings();
});
// text.inputEl.setAttribute("type", "password");
});
new Setting(containerPluginSettings)
.setName("Open")
.setDesc("Open the dialog")
.addButton((button) => {
button
.setButtonText("Open")
.setDisabled(false)
.onClick(() => {
this.plugin.addOnConfigSync.showPluginSyncModal();
});
});
updateDisabledOfDeviceAndVaultName();
addScreenElement("60", containerPluginSettings);
const containerMaintenanceEl = containerEl.createDiv();
containerMaintenanceEl.createEl("h3", { text: "Maintain databases" });
containerMaintenanceEl.createEl("h4", { text: "The remote database" });
new Setting(containerMaintenanceEl)
.setName("Lock remote database")
.setDesc("Lock remote database to prevent synchronization with other devices.")
.addButton((button) =>
button
.setButtonText("Lock")
.setDisabled(false)
.setWarning()
.onClick(async () => {
await this.plugin.markRemoteLocked();
})
);
new Setting(containerMaintenanceEl)
.setName("Overwrite remote database")
.setDesc("Overwrite remote database with local DB and passphrase.")
.addButton((button) =>
button
.setButtonText("Send")
.setWarning()
.setDisabled(false)
.onClick(async () => {
await rebuildDB("remoteOnly");
})
)
new Setting(containerMaintenanceEl)
.setName("(Beta) Clean the remote database")
.setDesc("")
.addButton((button) =>
button.setButtonText("Count")
.setDisabled(false)
.onClick(async () => {
await remoteDatabaseCleanup(this.plugin, true);
})
).addButton((button) =>
button.setButtonText("Perform cleaning")
.setDisabled(false)
.setWarning()
.onClick(async () => {
// @ts-ignore
this.plugin.app.setting.close()
await remoteDatabaseCleanup(this.plugin, false);
await balanceChunks(this.plugin, false);
})
);
containerMaintenanceEl.createEl("h4", { text: "The local database" });
new Setting(containerMaintenanceEl)
.setName("Fetch rebuilt DB")
.setDesc("Restore or reconstruct local database from remote database.")
.addButton((button) =>
button
.setButtonText("Fetch")
.setWarning()
.setDisabled(false)
.onClick(async () => {
await rebuildDB("localOnly");
})
)
new Setting(containerMaintenanceEl)
.setName("(Beta) Clean the local database")
.setDesc("This feature requires enabling 'Use new Adapter'")
.addButton((button) =>
button.setButtonText("Count")
.setDisabled(false)
.onClick(async () => {
await localDatabaseCleanUp(this.plugin, false, true);
})
).addButton((button) =>
button.setButtonText("Perform cleaning")
.setDisabled(false)
.setWarning()
.onClick(async () => {
// @ts-ignore
this.plugin.app.setting.close()
await localDatabaseCleanUp(this.plugin, false, false);
})
);
new Setting(containerMaintenanceEl)
.setName("Discard local database to reset or uninstall Self-hosted LiveSync") .setName("Discard local database to reset or uninstall Self-hosted LiveSync")
.addButton((button) => .addButton((button) =>
button button
@@ -1653,148 +1779,37 @@ ${stringifyYaml(pluginConfig)}`;
}) })
); );
addScreenElement("50", containerHatchEl); containerMaintenanceEl.createEl("h4", { text: "Both databases" });
// With great respect, thank you TfTHacker!
// Refer: https://github.com/TfTHacker/obsidian42-brat/blob/main/src/features/BetaPlugins.ts
const containerPluginSettings = containerEl.createDiv();
containerPluginSettings.createEl("h3", { text: "Plugins and settings (beta)" });
const updateDisabledOfDeviceAndVaultName = () => { new Setting(containerMaintenanceEl)
vaultName.setDisabled(this.plugin.settings.autoSweepPlugins || this.plugin.settings.autoSweepPluginsPeriodic); .setName("Rebuild everything")
vaultName.setTooltip(this.plugin.settings.autoSweepPlugins || this.plugin.settings.autoSweepPluginsPeriodic ? "You could not change when you enabling auto scan." : ""); .setDesc("Rebuild local and remote database with local files.")
}; .addButton((button) =>
new Setting(containerPluginSettings).setName("Enable plugin synchronization").addToggle((toggle) =>
toggle.setValue(this.plugin.settings.usePluginSync).onChange(async (value) => {
this.plugin.settings.usePluginSync = value;
await this.plugin.saveSettings();
})
);
new Setting(containerPluginSettings)
.setName("Scan plugins automatically")
.setDesc("Scan plugins before replicating.")
.addToggle((toggle) =>
toggle.setValue(this.plugin.settings.autoSweepPlugins).onChange(async (value) => {
this.plugin.settings.autoSweepPlugins = value;
updateDisabledOfDeviceAndVaultName();
await this.plugin.saveSettings();
})
);
new Setting(containerPluginSettings)
.setName("Scan plugins periodically")
.setDesc("Scan plugins every 1 minute. This configuration will be ignored if monitoring changes of hidden files has been enabled.")
.addToggle((toggle) =>
toggle.setValue(this.plugin.settings.autoSweepPluginsPeriodic).onChange(async (value) => {
this.plugin.settings.autoSweepPluginsPeriodic = value;
updateDisabledOfDeviceAndVaultName();
await this.plugin.saveSettings();
})
);
new Setting(containerPluginSettings)
.setName("Notify updates")
.setDesc("Notify when any device has a newer plugin or its setting.")
.addToggle((toggle) =>
toggle.setValue(this.plugin.settings.notifyPluginOrSettingUpdated).onChange(async (value) => {
this.plugin.settings.notifyPluginOrSettingUpdated = value;
await this.plugin.saveSettings();
})
);
const vaultName = new Setting(containerPluginSettings)
.setName("Device and Vault name")
.setDesc("")
.addText((text) => {
text.setPlaceholder("desktop-main")
.setValue(this.plugin.deviceAndVaultName)
.onChange(async (value) => {
this.plugin.deviceAndVaultName = value;
await this.plugin.saveSettings();
});
// text.inputEl.setAttribute("type", "password");
});
new Setting(containerPluginSettings)
.setName("Open")
.setDesc("Open the plugin dialog")
.addButton((button) => {
button button
.setButtonText("Open") .setButtonText("Rebuild")
.setWarning()
.setDisabled(false) .setDisabled(false)
.onClick(() => { .onClick(async () => {
this.plugin.addOnPluginAndTheirSettings.showPluginSyncModal(); await rebuildDB("rebuildBothByThisDevice");
}); })
}); )
new Setting(containerMaintenanceEl)
.setName("(Beta) Complement each other with possible missing chunks.")
.setDesc("")
.addButton((button) =>
button
.setButtonText("Balance")
.setWarning()
.setDisabled(false)
.onClick(async () => {
await balanceChunks(this.plugin, false);
})
)
applyDisplayEnabled();
addScreenElement("70", containerMaintenanceEl);
updateDisabledOfDeviceAndVaultName();
addScreenElement("60", containerPluginSettings);
const containerCorruptedDataEl = containerEl.createDiv();
containerCorruptedDataEl.createEl("h3", { text: "Corrupted or missing data" });
containerCorruptedDataEl.createEl("h4", { text: "Corrupted" });
if (Object.keys(this.plugin.localDatabase.corruptedEntries).length > 0) {
const cx = containerCorruptedDataEl.createEl("div", { text: "If you have a copy of these files on any device, simply edit them once and sync. If not, there's nothing we can do except deleting them. sorry.." });
for (const k in this.plugin.localDatabase.corruptedEntries) {
const xx = cx.createEl("div", { text: `${k}` });
const ba = xx.createEl("button", { text: `Delete this` }, (e) => {
e.addEventListener("click", async () => {
await this.plugin.localDatabase.deleteDBEntry(k);
xx.remove();
});
});
ba.addClass("mod-warning");
xx.createEl("button", { text: `Restore from file` }, (e) => {
e.addEventListener("click", async () => {
const f = await this.app.vault.getFiles().filter((e) => path2id(e.path) == k);
if (f.length == 0) {
Logger("Not found in vault", LOG_LEVEL.NOTICE);
return;
}
await this.plugin.updateIntoDB(f[0]);
xx.remove();
});
});
xx.addClass("mod-warning");
}
} else {
containerCorruptedDataEl.createEl("div", { text: "There is no corrupted data." });
}
containerCorruptedDataEl.createEl("h4", { text: "Missing or waiting" });
if (Object.keys(this.plugin.queuedFiles).length > 0) {
const cx = containerCorruptedDataEl.createEl("div", {
text: "These files have missing or waiting chunks. Perhaps these chunks will arrive in a while after replication. But if they don't, you have to restore it's database entry from a existing local file by hitting the button below.",
});
const files = [...new Set([...this.plugin.queuedFiles.map((e) => e.entry._id)])];
for (const k of files) {
const xx = cx.createEl("div", { text: `${id2path(k)}` });
const ba = xx.createEl("button", { text: `Delete this` }, (e) => {
e.addEventListener("click", async () => {
await this.plugin.localDatabase.deleteDBEntry(k);
xx.remove();
});
});
ba.addClass("mod-warning");
xx.createEl("button", { text: `Restore from file` }, (e) => {
e.addEventListener("click", async () => {
const f = await this.app.vault.getFiles().filter((e) => path2id(e.path) == k);
if (f.length == 0) {
Logger("Not found in vault", LOG_LEVEL.NOTICE);
return;
}
await this.plugin.updateIntoDB(f[0]);
xx.remove();
});
});
xx.addClass("mod-warning");
}
} else {
containerCorruptedDataEl.createEl("div", { text: "There is no missing or waiting chunk." });
}
applyDisplayEnabled(); applyDisplayEnabled();
addScreenElement("70", containerCorruptedDataEl);
if (this.selectedScreen == "") { if (this.selectedScreen == "") {
if (lastVersion != this.plugin.settings.lastReadUpdates) { if (lastVersion != this.plugin.settings.lastReadUpdates) {
if (JSON.stringify(this.plugin.settings) != JSON.stringify(DEFAULT_SETTINGS)) { if (JSON.stringify(this.plugin.settings) != JSON.stringify(DEFAULT_SETTINGS)) {

318
src/PluginCombo.svelte Normal file
View File

@@ -0,0 +1,318 @@
<script lang="ts">
import type { PluginDataExDisplay } from "./CmdConfigSync";
import { Logger } from "./lib/src/logger";
import { versionNumberString2Number } from "./lib/src/strbin";
import { FilePath, LOG_LEVEL } from "./lib/src/types";
import { getDocData } from "./lib/src/utils";
import type ObsidianLiveSyncPlugin from "./main";
import { askString, scheduleTask } from "./utils";
export let list: PluginDataExDisplay[] = [];
export let thisTerm = "";
export let hideNotApplicable = false;
export let selectNewest = 0;
export let applyAllPluse = 0;
export let applyData: (data: PluginDataExDisplay) => Promise<boolean>;
export let compareData: (dataA: PluginDataExDisplay, dataB: PluginDataExDisplay) => Promise<boolean>;
export let deleteData: (data: PluginDataExDisplay) => Promise<boolean>;
export let hidden: boolean;
export let plugin: ObsidianLiveSyncPlugin;
export let isMaintenanceMode: boolean = false;
const addOn = plugin.addOnConfigSync;
let selected = "";
let freshness = "";
let equivalency = "";
let version = "";
let canApply: boolean = false;
let canCompare: boolean = false;
let currentSelectNewest = 0;
let currentApplyAll = 0;
// Selectable terminals
let terms = [] as string[];
async function comparePlugin(local: PluginDataExDisplay, remote: PluginDataExDisplay) {
let freshness = "";
let equivalency = "";
let version = "";
let contentCheck = false;
let canApply: boolean = false;
let canCompare = false;
if (!local && !remote) {
// NO OP. whats happened?
freshness = "";
} else if (local && !remote) {
freshness = "⚠ Local only";
} else if (remote && !local) {
freshness = "✓ Remote only";
canApply = true;
} else {
const dtDiff = (local?.mtime ?? 0) - (remote?.mtime ?? 0);
if (dtDiff / 1000 < -10) {
freshness = "✓ Newer";
canApply = true;
contentCheck = true;
} else if (dtDiff / 1000 > 10) {
freshness = "⚠ Older";
canApply = true;
contentCheck = true;
} else {
freshness = "⚖️ Same old";
canApply = false;
contentCheck = true;
}
}
const localVersionStr = local?.version || "0.0.0";
const remoteVersionStr = remote?.version || "0.0.0";
if (local?.version || remote?.version) {
const localVersion = versionNumberString2Number(localVersionStr);
const remoteVersion = versionNumberString2Number(remoteVersionStr);
if (localVersion == remoteVersion) {
version = "⚖️ Same ver.";
} else if (localVersion > remoteVersion) {
version = `⚠ Lower ${localVersionStr} > ${remoteVersionStr}`;
} else if (localVersion < remoteVersion) {
version = `✓ Higher ${localVersionStr} < ${remoteVersionStr}`;
}
}
if (contentCheck) {
const { canApply, equivalency, canCompare } = await checkEquivalency(local, remote);
return { canApply, freshness, equivalency, version, canCompare };
}
return { canApply, freshness, equivalency, version, canCompare };
}
async function checkEquivalency(local: PluginDataExDisplay, remote: PluginDataExDisplay) {
let equivalency = "";
let canApply = false;
let canCompare = false;
const filenames = [...new Set([...local.files.map((e) => e.filename), ...remote.files.map((e) => e.filename)])];
const matchingStatus = filenames
.map((filename) => {
const localFile = local.files.find((e) => e.filename == filename);
const remoteFile = remote.files.find((e) => e.filename == filename);
if (!localFile && !remoteFile) {
return 0b0000000;
} else if (localFile && !remoteFile) {
return 0b0000010; //"LOCAL_ONLY";
} else if (!localFile && remoteFile) {
return 0b0001000; //"REMOTE ONLY"
} else {
if (getDocData(localFile.data) == getDocData(remoteFile.data)) {
return 0b0000100; //"EVEN"
} else {
return 0b0010000; //"DIFFERENT";
}
}
})
.reduce((p, c) => p | c, 0);
if (matchingStatus == 0b0000100) {
equivalency = "⚖️ Same";
canApply = false;
} else if (matchingStatus <= 0b0000100) {
equivalency = "Same or local only";
canApply = false;
} else if (matchingStatus == 0b0010000) {
canApply = true;
canCompare = true;
equivalency = "≠ Different";
} else {
canApply = true;
canCompare = true;
equivalency = "≠ Different";
}
return { equivalency, canApply, canCompare };
}
async function performCompare(local: PluginDataExDisplay, remote: PluginDataExDisplay) {
const result = await comparePlugin(local, remote);
canApply = result.canApply;
freshness = result.freshness;
equivalency = result.equivalency;
version = result.version;
canCompare = result.canCompare;
if (local?.files.length != 1 || !local?.files?.first()?.filename?.endsWith(".json")) {
canCompare = false;
}
}
async function updateTerms(list: PluginDataExDisplay[], selectNewest: boolean, isMaintenanceMode: boolean) {
const local = list.find((e) => e.term == thisTerm);
selected = "";
if (isMaintenanceMode) {
terms = [...new Set(list.map((e) => e.term))];
} else if (hideNotApplicable) {
const termsTmp = [];
const wk = [...new Set(list.map((e) => e.term))];
for (const termName of wk) {
const remote = list.find((e) => e.term == termName);
if ((await comparePlugin(local, remote)).canApply) {
termsTmp.push(termName);
}
}
terms = [...termsTmp];
} else {
terms = [...new Set(list.map((e) => e.term))].filter((e) => e != thisTerm);
}
let newest: PluginDataExDisplay = local;
if (selectNewest) {
for (const term of terms) {
const remote = list.find((e) => e.term == term);
if (remote && remote.mtime && (newest?.mtime || 0) < remote.mtime) {
newest = remote;
}
}
if (newest && newest.term != thisTerm) {
selected = newest.term;
}
// selectNewest = false;
}
}
$: {
// React pulse and select
const doSelectNewest = selectNewest != currentSelectNewest;
currentSelectNewest = selectNewest;
updateTerms(list, doSelectNewest, isMaintenanceMode);
}
$: {
// React pulse and apply
const doApply = applyAllPluse != currentApplyAll;
currentApplyAll = applyAllPluse;
if (doApply && selected) {
if (!hidden) {
applySelected();
}
}
}
$: {
freshness = "";
equivalency = "";
version = "";
canApply = false;
if (selected == "") {
// NO OP.
} else if (selected == thisTerm) {
freshness = "This device";
canApply = false;
} else {
const local = list.find((e) => e.term == thisTerm);
const remote = list.find((e) => e.term == selected);
performCompare(local, remote);
}
}
async function applySelected() {
const local = list.find((e) => e.term == thisTerm);
const selectedItem = list.find((e) => e.term == selected);
if (selectedItem && (await applyData(selectedItem))) {
scheduleTask("update-plugin-list", 250, () => addOn.updatePluginList(true, local.documentPath));
}
}
async function compareSelected() {
const local = list.find((e) => e.term == thisTerm);
const selectedItem = list.find((e) => e.term == selected);
if (local && selectedItem && (await compareData(local, selectedItem))) {
scheduleTask("update-plugin-list", 250, () => addOn.updatePluginList(true, local.documentPath));
}
}
async function deleteSelected() {
const selectedItem = list.find((e) => e.term == selected);
// const deletedPath = selectedItem.documentPath;
if (selectedItem && (await deleteData(selectedItem))) {
scheduleTask("update-plugin-list", 250, () => addOn.reloadPluginList(true));
}
}
async function duplicateItem() {
const local = list.find((e) => e.term == thisTerm);
const duplicateTermName = await askString(plugin.app, "Duplicate", "device name", "");
if (duplicateTermName) {
if (duplicateTermName.contains("/")) {
Logger(`We can not use "/" to the device name`, LOG_LEVEL.NOTICE);
return;
}
const key = `${plugin.app.vault.configDir}/${local.files[0].filename}`;
await addOn.storeCustomizationFiles(key as FilePath, duplicateTermName);
await addOn.updatePluginList(false, addOn.filenameToUnifiedKey(key, duplicateTermName));
}
}
</script>
{#if terms.length > 0}
<span class="spacer" />
{#if !hidden}
<span class="messages">
<span class="message">{freshness}</span>
<span class="message">{equivalency}</span>
<span class="message">{version}</span>
</span>
<select bind:value={selected}>
<option value={""}>-</option>
{#each terms as term}
<option value={term}>{term}</option>
{/each}
</select>
{#if canApply || (isMaintenanceMode && selected != "")}
{#if canCompare}
<button on:click={compareSelected}>🔍</button>
{:else}
<button disabled />
{/if}
<button on:click={applySelected}>✓</button>
{:else}
<button disabled />
<button disabled />
{/if}
{#if isMaintenanceMode}
{#if selected != ""}
<button on:click={deleteSelected}>🗑️</button>
{:else}
<button on:click={duplicateItem}>📑</button>
{/if}
{/if}
{/if}
{:else}
<span class="spacer" />
<span class="message even">All devices are even</span>
<button disabled />
<button disabled />
{/if}
<style>
.spacer {
min-width: 1px;
flex-grow: 1;
}
button {
margin: 2px 4px;
min-width: 3em;
max-width: 4em;
}
button:disabled {
border: none;
box-shadow: none;
background-color: transparent;
visibility: collapse;
}
button:disabled:hover {
border: none;
box-shadow: none;
background-color: transparent;
visibility: collapse;
}
span.message {
color: var(--text-muted);
font-size: var(--font-ui-smaller);
padding: 0 1em;
line-height: var(--line-height-tight);
}
span.messages {
display: flex;
flex-direction: column;
align-items: center;
}
:global(.is-mobile) .spacer {
margin-left: auto;
}
</style>

461
src/PluginPane.svelte
View File

@@ -1,309 +1,230 @@
<script lang="ts"> <script lang="ts">
import { onMount } from "svelte"; import { onMount } from "svelte";
import { DevicePluginList, PluginDataEntry } from "./types";
import { versionNumberString2Number } from "./lib/src/strbin";
import ObsidianLiveSyncPlugin from "./main"; import ObsidianLiveSyncPlugin from "./main";
import { PluginAndTheirSettings } from "./CmdPluginAndTheirSettings"; import { PluginDataExDisplay, pluginIsEnumerating, pluginList } from "./CmdConfigSync";
import PluginCombo from "./PluginCombo.svelte";
type JudgeResult = "" | "NEWER" | "EVEN" | "EVEN_BUT_DIFFERENT" | "OLDER" | "REMOTE_ONLY";
interface PluginDataEntryDisp extends PluginDataEntry {
versionInfo: string;
mtimeInfo: string;
mtimeFlag: JudgeResult;
versionFlag: JudgeResult;
}
export let plugin: ObsidianLiveSyncPlugin; export let plugin: ObsidianLiveSyncPlugin;
let plugins: PluginDataEntry[] = [];
let deviceAndPlugins: { [key: string]: PluginDataEntryDisp[] } = {};
let devicePluginList: [string, PluginDataEntryDisp[]][] = null;
let ownPlugins: DevicePluginList = null;
let showOwnPlugins = false;
let targetList: { [key: string]: boolean } = {};
let addOn: PluginAndTheirSettings; $: hideNotApplicable = true;
$: { $: thisTerm = plugin.deviceAndVaultName;
const f = plugin.addOns.filter((e) => e instanceof PluginAndTheirSettings);
if (f && f.length > 0) { const addOn = plugin.addOnConfigSync;
addOn = f[0] as PluginAndTheirSettings;
} let list: PluginDataExDisplay[] = [];
let selectNewestPulse = 0;
let hideEven = true;
let loading = false;
let applyAllPluse = 0;
let isMaintenanceMode = false;
async function requestUpdate() {
await addOn.updatePluginList(true);
} }
function saveTargetList() { async function requestReload() {
window.localStorage.setItem("ols-plugin-targetlist", JSON.stringify(targetList)); await addOn.reloadPluginList(true);
} }
pluginList.subscribe((e) => {
function loadTargetList() { list = e;
let e = window.localStorage.getItem("ols-plugin-targetlist") || "{}"; });
try { pluginIsEnumerating.subscribe((e) => {
targetList = JSON.parse(e); loading = e;
} catch (_) { });
// NO OP.
}
}
function clearSelection() {
targetList = {};
}
async function updateList() {
let x = await addOn.getPluginList();
ownPlugins = x.thisDevicePlugins;
plugins = Object.values(x.allPlugins);
let targetListItems = Array.from(new Set(plugins.map((e) => e.deviceVaultName + "---" + e.manifest.id)));
let newTargetList: { [key: string]: boolean } = {};
for (const id of targetListItems) {
for (const tag of ["---plugin", "---setting"]) {
newTargetList[id + tag] = id + tag in targetList && targetList[id + tag];
}
}
targetList = newTargetList;
saveTargetList();
}
$: {
deviceAndPlugins = {};
for (const p of plugins) {
if (p.deviceVaultName == plugin.deviceAndVaultName && !showOwnPlugins) {
continue;
}
if (!(p.deviceVaultName in deviceAndPlugins)) {
deviceAndPlugins[p.deviceVaultName] = [];
}
let dispInfo: PluginDataEntryDisp = {
...p,
versionInfo: "",
mtimeInfo: "",
versionFlag: "",
mtimeFlag: "",
};
dispInfo.versionInfo = p.manifest.version;
let x = new Date().getTime() / 1000;
let mtime = p.mtime / 1000;
let diff = (x - mtime) / 60;
if (p.mtime == 0) {
dispInfo.mtimeInfo = `-`;
} else if (diff < 60) {
dispInfo.mtimeInfo = `${diff | 0} Mins ago`;
} else if (diff < 60 * 24) {
dispInfo.mtimeInfo = `${(diff / 60) | 0} Hours ago`;
} else if (diff < 60 * 24 * 10) {
dispInfo.mtimeInfo = `${(diff / (60 * 24)) | 0} Days ago`;
} else {
dispInfo.mtimeInfo = new Date(dispInfo.mtime).toLocaleString();
}
// compare with own plugin
let id = p.manifest.id;
if (id in ownPlugins) {
// Which we have.
const ownPlugin = ownPlugins[id];
let localVer = versionNumberString2Number(ownPlugin.manifest.version);
let pluginVer = versionNumberString2Number(p.manifest.version);
if (localVer > pluginVer) {
dispInfo.versionFlag = "OLDER";
} else if (localVer == pluginVer) {
if (ownPlugin.manifestJson + (ownPlugin.styleCss ?? "") + ownPlugin.mainJs != p.manifestJson + (p.styleCss ?? "") + p.mainJs) {
dispInfo.versionFlag = "EVEN_BUT_DIFFERENT";
} else {
dispInfo.versionFlag = "EVEN";
}
} else if (localVer < pluginVer) {
dispInfo.versionFlag = "NEWER";
}
if ((ownPlugin.dataJson ?? "") == (p.dataJson ?? "")) {
if (ownPlugin.mtime == 0 && p.mtime == 0) {
dispInfo.mtimeFlag = "";
} else {
dispInfo.mtimeFlag = "EVEN";
}
} else {
if (((ownPlugin.mtime / 1000) | 0) > ((p.mtime / 1000) | 0)) {
dispInfo.mtimeFlag = "OLDER";
} else if (((ownPlugin.mtime / 1000) | 0) == ((p.mtime / 1000) | 0)) {
dispInfo.mtimeFlag = "EVEN_BUT_DIFFERENT";
} else if (((ownPlugin.mtime / 1000) | 0) < ((p.mtime / 1000) | 0)) {
dispInfo.mtimeFlag = "NEWER";
}
}
} else {
dispInfo.versionFlag = "REMOTE_ONLY";
dispInfo.mtimeFlag = "REMOTE_ONLY";
}
deviceAndPlugins[p.deviceVaultName].push(dispInfo);
}
devicePluginList = Object.entries(deviceAndPlugins);
}
function getDispString(stat: JudgeResult): string {
if (stat == "") return "";
if (stat == "NEWER") return " (Newer)";
if (stat == "OLDER") return " (Older)";
if (stat == "EVEN") return " (Even)";
if (stat == "EVEN_BUT_DIFFERENT") return " (Even but different)";
if (stat == "REMOTE_ONLY") return " (Remote Only)";
return "";
}
onMount(async () => { onMount(async () => {
loadTargetList(); requestUpdate();
await updateList();
}); });
function toggleShowOwnPlugins() { function filterList(list: PluginDataExDisplay[], categories: string[]) {
showOwnPlugins = !showOwnPlugins; const w = list.filter((e) => categories.indexOf(e.category) !== -1);
return w.sort((a, b) => `${a.category}-${a.name}`.localeCompare(`${b.category}-${b.name}`));
} }
function toggleTarget(key: string) { function groupBy(items: PluginDataExDisplay[], key: string) {
targetList[key] = !targetList[key]; let ret = {} as Record<string, PluginDataExDisplay[]>;
saveTargetList(); for (const v of items) {
} //@ts-ignore
const k = (key in v ? v[key] : "") as string;
function toggleAll(devicename: string) { ret[k] = ret[k] || [];
for (const c in targetList) { ret[k].push(v);
if (c.startsWith(devicename)) {
targetList[c] = true;
}
} }
} for (const k in ret) {
ret[k] = ret[k].sort((a, b) => `${a.category}-${a.name}`.localeCompare(`${b.category}-${b.name}`));
async function sweepPlugins() {
//@ts-ignore
await plugin.app.plugins.loadManifests();
await addOn.sweepPlugin(true);
updateList();
}
async function applyPlugins() {
for (const c in targetList) {
if (targetList[c] == true) {
const [deviceAndVault, id, opt] = c.split("---");
if (deviceAndVault in deviceAndPlugins) {
const entry = deviceAndPlugins[deviceAndVault].find((e) => e.manifest.id == id);
if (entry) {
if (opt == "plugin") {
if (entry.versionFlag != "EVEN") await addOn.applyPlugin(entry);
} else if (opt == "setting") {
if (entry.mtimeFlag != "EVEN") await addOn.applyPluginData(entry);
}
}
}
}
} }
//@ts-ignore const w = Object.entries(ret);
await plugin.app.plugins.loadManifests(); return w.sort(([a], [b]) => `${a}`.localeCompare(`${b}`));
await addOn.sweepPlugin(true);
updateList();
} }
async function checkUpdates() { const displays = {
await addOn.checkPluginUpdate(); CONFIG: "Configuration",
THEME: "Themes",
SNIPPET: "Snippets",
};
async function scanAgain() {
await addOn.scanAllConfigFiles(true);
await requestUpdate();
} }
async function replicateAndRefresh() { async function replicate() {
await plugin.replicate(true); await plugin.replicate(true);
updateList();
} }
function selectAllNewest() {
selectNewestPulse++;
}
function applyAll() {
applyAllPluse++;
}
async function applyData(data: PluginDataExDisplay): Promise<boolean> {
return await addOn.applyData(data);
}
async function compareData(docA: PluginDataExDisplay, docB: PluginDataExDisplay): Promise<boolean> {
return await addOn.compareUsingDisplayData(docA, docB);
}
async function deleteData(data: PluginDataExDisplay): Promise<boolean> {
return await addOn.deleteData(data);
}
$: options = {
thisTerm,
hideNotApplicable,
selectNewest: selectNewestPulse,
applyAllPluse,
applyData,
compareData,
deleteData,
plugin,
isMaintenanceMode,
};
</script> </script>
<div> <div>
<h1>Plugins and their settings</h1> <div>
<div class="ols-plugins-div-buttons"> <h1>Customization sync</h1>
Show own items <div class="buttons">
<div class="checkbox-container" class:is-enabled={showOwnPlugins} on:click={toggleShowOwnPlugins} /> <button on:click={() => scanAgain()}>Scan changes</button>
</div> <button on:click={() => replicate()}>Sync once</button>
<div class="sls-plugins-wrap"> <button on:click={() => requestUpdate()}>Refresh</button>
<table class="sls-plugins-tbl"> {#if isMaintenanceMode}
<tr style="position:sticky"> <button on:click={() => requestReload()}>Reload</button>
<th class="sls-plugins-tbl-device-head">Name</th>
<th class="sls-plugins-tbl-device-head">Info</th>
<th class="sls-plugins-tbl-device-head">Target</th>
</tr>
{#if !devicePluginList}
<tr>
<td colspan="3" class="sls-table-tail tcenter"> Retrieving... </td>
</tr>
{:else if devicePluginList.length == 0}
<tr>
<td colspan="3" class="sls-table-tail tcenter"> No plugins found. </td>
</tr>
{:else}
{#each devicePluginList as [deviceName, devicePlugins]}
<tr>
<th colspan="2" class="sls-plugins-tbl-device-head">{deviceName}</th>
<th class="sls-plugins-tbl-device-head">
<button class="mod-cta" on:click={() => toggleAll(deviceName)}>✔</button>
</th>
</tr>
{#each devicePlugins as plugin}
<tr>
<td class="sls-table-head">{plugin.manifest.name}</td>
<td class="sls-table-tail tcenter">{plugin.versionInfo}{getDispString(plugin.versionFlag)}</td>
<td class="sls-table-tail tcenter">
{#if plugin.versionFlag === "EVEN" || plugin.versionFlag === ""}
-
{:else}
<div class="wrapToggle">
<div
class="checkbox-container"
class:is-enabled={targetList[plugin.deviceVaultName + "---" + plugin.manifest.id + "---plugin"]}
on:click={() => toggleTarget(plugin.deviceVaultName + "---" + plugin.manifest.id + "---plugin")}
/>
</div>
{/if}
</td>
</tr>
<tr>
<td class="sls-table-head">Settings</td>
<td class="sls-table-tail tcenter">{plugin.mtimeInfo}{getDispString(plugin.mtimeFlag)}</td>
<td class="sls-table-tail tcenter">
{#if plugin.mtimeFlag === "EVEN" || plugin.mtimeFlag === ""}
-
{:else}
<div class="wrapToggle">
<div
class="checkbox-container"
class:is-enabled={targetList[plugin.deviceVaultName + "---" + plugin.manifest.id + "---setting"]}
on:click={() => toggleTarget(plugin.deviceVaultName + "---" + plugin.manifest.id + "---setting")}
/>
</div>
{/if}
</td>
</tr>
<tr class="divider">
<th colspan="3" />
</tr>
{/each}
{/each}
{/if} {/if}
</table> <button on:click={() => selectAllNewest()}>Select All Shiny</button>
</div>
<div class="buttons">
<button on:click={() => applyAll()}>Apply All</button>
</div>
</div> </div>
<div class="ols-plugins-div-buttons"> {#if loading}
<button class="" on:click={replicateAndRefresh}>Replicate and refresh</button> <div>
<button class="" on:click={clearSelection}>Clear Selection</button> <span>Updating list...</span>
</div>
{/if}
<div class="list">
{#if list.length == 0}
<div class="center">No Items.</div>
{:else}
{#each Object.entries(displays) as [key, label]}
<div>
<h3>{label}</h3>
{#each groupBy(filterList(list, [key]), "name") as [name, listX]}
<div class="labelrow {hideEven ? 'hideeven' : ''}">
<div class="title">
{name}
</div>
<PluginCombo {...options} list={listX} hidden={false} />
</div>
{/each}
</div>
{/each}
<div>
<h3>Plugins</h3>
{#each groupBy(filterList(list, ["PLUGIN_MAIN", "PLUGIN_DATA", "PLUGIN_ETC"]), "name") as [name, listX]}
<div class="labelrow {hideEven ? 'hideeven' : ''}">
<div class="title">
{name}
</div>
<PluginCombo {...options} list={listX} hidden={true} />
</div>
<div class="filerow {hideEven ? 'hideeven' : ''}">
<div class="filetitle">Main</div>
<PluginCombo {...options} list={filterList(listX, ["PLUGIN_MAIN"])} hidden={false} />
</div>
<div class="filerow {hideEven ? 'hideeven' : ''}">
<div class="filetitle">Data</div>
<PluginCombo {...options} list={filterList(listX, ["PLUGIN_DATA"])} hidden={false} />
</div>
{/each}
</div>
{/if}
</div> </div>
<div class="buttons">
<div class="ols-plugins-div-buttons"> <label><span>Hide not applicable items</span><input type="checkbox" bind:checked={hideEven} /></label>
<button class="mod-cta" on:click={checkUpdates}>Check Updates</button> </div>
<button class="mod-cta" on:click={sweepPlugins}>Scan installed</button> <div class="buttons">
<button class="mod-cta" on:click={applyPlugins}>Apply all</button> <label><span>Maintenance mode</span><input type="checkbox" bind:checked={isMaintenanceMode} /></label>
</div> </div>
<!-- <div class="ols-plugins-div-buttons">-->
<!-- <button class="mod-warning" on:click={applyPlugins}>Delete all selected</button>-->
<!-- </div>-->
</div> </div>
<style> <style>
.ols-plugins-div-buttons { .labelrow {
margin-left: 0.4em;
display: flex;
justify-content: flex-start;
align-items: center;
border-top: 1px solid var(--background-modifier-border);
padding: 4px;
flex-wrap: wrap;
}
.filerow {
margin-left: 1.25em;
display: flex;
justify-content: flex-start;
align-items: center;
padding-right: 4px;
flex-wrap: wrap;
}
.filerow.hideeven:has(.even),
.labelrow.hideeven:has(.even) {
display: none;
}
.title {
color: var(--text-normal);
font-size: var(--font-ui-medium);
line-height: var(--line-height-tight);
margin-right: auto;
}
.filetitle {
color: var(--text-normal);
font-size: var(--font-ui-medium);
line-height: var(--line-height-tight);
margin-right: auto;
}
.buttons {
display: flex; display: flex;
flex-direction: row; flex-direction: row;
justify-content: flex-end; justify-content: flex-end;
margin-top: 8px; margin-top: 8px;
flex-wrap: wrap;
}
.buttons > button {
margin-left: 4px;
width: auto;
} }
.wrapToggle { label {
display: flex; display: flex;
justify-content: center; justify-content: center;
align-content: center; align-items: center;
}
label > span {
margin-right: 0.25em;
}
:global(.is-mobile) .title,
:global(.is-mobile) .filetitle {
width: 100%;
}
.center {
display: flex;
justify-content: center;
align-items: center;
min-height: 3em;
} }
</style> </style>

8
src/StorageEventManager.ts
View File

@@ -1,7 +1,7 @@
import { Plugin_2, TAbstractFile, TFile, TFolder } from "./deps"; import { Plugin_2, TAbstractFile, TFile, TFolder } from "./deps";
import { isPlainText, shouldBeIgnored } from "./lib/src/path"; import { isPlainText, shouldBeIgnored } from "./lib/src/path";
import { getGlobalStore } from "./lib/src/store"; import { getGlobalStore } from "./lib/src/store";
import { ObsidianLiveSyncSettings } from "./lib/src/types"; import { FilePath, ObsidianLiveSyncSettings } from "./lib/src/types";
import { FileEventItem, FileEventType, FileInfo, InternalFileInfo, queueItem } from "./types"; import { FileEventItem, FileEventType, FileInfo, InternalFileInfo, queueItem } from "./types";
import { recentlyTouched } from "./utils"; import { recentlyTouched } from "./utils";
@@ -57,14 +57,14 @@ export class StorageEventManagerObsidian extends StorageEventManager {
watchVaultRename(file: TAbstractFile, oldFile: string, ctx?: any) { watchVaultRename(file: TAbstractFile, oldFile: string, ctx?: any) {
if (file instanceof TFile) { if (file instanceof TFile) {
this.appendWatchEvent([ this.appendWatchEvent([
{ type: "DELETE", file: { path: oldFile as FilePath, mtime: file.stat.mtime, ctime: file.stat.ctime, size: file.stat.size, deleted: true } },
{ type: "CREATE", file }, { type: "CREATE", file },
{ type: "DELETE", file: { path: oldFile, mtime: file.stat.mtime, ctime: file.stat.ctime, size: file.stat.size, deleted: true } }
], ctx); ], ctx);
} }
} }
// Watch raw events (Internal API) // Watch raw events (Internal API)
watchVaultRawEvents(path: string) { watchVaultRawEvents(path: FilePath) {
if (!this.plugin.settings.syncInternalFiles) return; if (!this.plugin.settings.syncInternalFiles && !this.plugin.settings.usePluginSync) return;
if (!this.plugin.settings.watchInternalFileChanges) return; if (!this.plugin.settings.watchInternalFileChanges) return;
if (!path.startsWith(app.vault.configDir)) return; if (!path.startsWith(app.vault.configDir)) return;
const ignorePatterns = this.plugin.settings.syncInternalFilesIgnorePatterns const ignorePatterns = this.plugin.settings.syncInternalFilesIgnorePatterns

12
src/deps.ts
View File

@@ -1,4 +1,12 @@
import { FilePath } from "./lib/src/types";
export { export {
addIcon, App, DataWriteOptions, debounce, Editor, FuzzySuggestModal, MarkdownRenderer, MarkdownView, Modal, normalizePath, Notice, Platform, Plugin, PluginManifest, addIcon, App, DataWriteOptions, debounce, Editor, FuzzySuggestModal, MarkdownRenderer, MarkdownView, Modal, Notice, Platform, Plugin, PluginManifest,
PluginSettingTab, Plugin_2, requestUrl, RequestUrlParam, RequestUrlResponse, sanitizeHTMLToDom, Setting, stringifyYaml, TAbstractFile, TextAreaComponent, TFile, TFolder PluginSettingTab, Plugin_2, requestUrl, RequestUrlParam, RequestUrlResponse, sanitizeHTMLToDom, Setting, stringifyYaml, TAbstractFile, TextAreaComponent, TFile, TFolder,
parseYaml
} from "obsidian"; } from "obsidian";
import {
normalizePath as normalizePath_
} from "obsidian";
const normalizePath = normalizePath_ as <T extends string | FilePath>(from: T) => T;
export { normalizePath }

107
src/dialogs.ts
View File

@@ -1,4 +1,5 @@
import { App, FuzzySuggestModal, Modal, Setting } from "./deps"; import { ButtonComponent } from "obsidian";
import { App, FuzzySuggestModal, MarkdownRenderer, Modal, Plugin, Setting } from "./deps";
import ObsidianLiveSyncPlugin from "./main"; import ObsidianLiveSyncPlugin from "./main";
//@ts-ignore //@ts-ignore
@@ -8,6 +9,9 @@ export class PluginDialogModal extends Modal {
plugin: ObsidianLiveSyncPlugin; plugin: ObsidianLiveSyncPlugin;
logEl: HTMLDivElement; logEl: HTMLDivElement;
component: PluginPane = null; component: PluginPane = null;
isOpened() {
return this.component != null;
}
constructor(app: App, plugin: ObsidianLiveSyncPlugin) { constructor(app: App, plugin: ObsidianLiveSyncPlugin) {
super(app); super(app);
@@ -123,4 +127,103 @@ export class PopoverSelectString extends FuzzySuggestModal<string> {
} }
}, 100); }, 100);
} }
} }
export class MessageBox extends Modal {
plugin: Plugin;
title: string;
contentMd: string;
buttons: string[];
result: string;
isManuallyClosed = false;
defaultAction: string | undefined;
timeout: number | undefined;
timer: ReturnType<typeof setInterval> = undefined;
defaultButtonComponent: ButtonComponent | undefined;
onSubmit: (result: string | boolean) => void;
constructor(plugin: Plugin, title: string, contentMd: string, buttons: string[], defaultAction: (typeof buttons)[number], timeout: number, onSubmit: (result: (typeof buttons)[number] | false) => void) {
super(plugin.app);
this.plugin = plugin;
this.title = title;
this.contentMd = contentMd;
this.buttons = buttons;
this.onSubmit = onSubmit;
this.defaultAction = defaultAction;
this.timeout = timeout;
if (this.timeout) {
this.timer = setInterval(() => {
this.timeout--;
if (this.timeout < 0) {
if (this.timer) {
clearInterval(this.timer);
this.timer = undefined;
}
this.result = defaultAction;
this.isManuallyClosed = true;
this.close();
} else {
this.defaultButtonComponent.setButtonText(`( ${this.timeout} ) ${defaultAction}`);
}
}, 1000);
}
}
onOpen() {
const { contentEl } = this;
contentEl.addEventListener("click", () => {
if (this.timer) {
clearInterval(this.timer);
this.timer = undefined;
}
})
contentEl.createEl("h1", { text: this.title });
const div = contentEl.createDiv();
MarkdownRenderer.renderMarkdown(this.contentMd, div, "/", null);
const buttonSetting = new Setting(contentEl);
for (const button of this.buttons) {
buttonSetting.addButton((btn) => {
btn
.setButtonText(button)
.onClick(() => {
this.isManuallyClosed = true;
this.result = button;
if (this.timer) {
clearInterval(this.timer);
this.timer = undefined;
}
this.close();
})
if (button == this.defaultAction) {
this.defaultButtonComponent = btn;
}
return btn;
}
)
}
}
onClose() {
const { contentEl } = this;
contentEl.empty();
if (this.timer) {
clearInterval(this.timer);
this.timer = undefined;
}
if (this.isManuallyClosed) {
this.onSubmit(this.result);
} else {
this.onSubmit(false);
}
}
}
export function confirmWithMessage(plugin: Plugin, title: string, contentMd: string, buttons: string[], defaultAction?: (typeof buttons)[number], timeout?: number): Promise<(typeof buttons)[number] | false> {
return new Promise((res) => {
const dialog = new MessageBox(plugin, title, contentMd, buttons, defaultAction, timeout, (result) => res(result));
dialog.open();
});
}

2
src/lib

Submodule src/lib updated: d103106931...fb3070851f

453
src/main.ts
View File

@@ -2,23 +2,23 @@ const isDebug = false;
import { Diff, DIFF_DELETE, DIFF_EQUAL, DIFF_INSERT, diff_match_patch } from "diff-match-patch"; import { Diff, DIFF_DELETE, DIFF_EQUAL, DIFF_INSERT, diff_match_patch } from "diff-match-patch";
import { debounce, Notice, Plugin, TFile, addIcon, TFolder, normalizePath, TAbstractFile, Editor, MarkdownView, RequestUrlParam, RequestUrlResponse, requestUrl } from "./deps"; import { debounce, Notice, Plugin, TFile, addIcon, TFolder, normalizePath, TAbstractFile, Editor, MarkdownView, RequestUrlParam, RequestUrlResponse, requestUrl } from "./deps";
import { EntryDoc, LoadedEntry, ObsidianLiveSyncSettings, diff_check_result, diff_result_leaf, EntryBody, LOG_LEVEL, VER, DEFAULT_SETTINGS, diff_result, FLAGMD_REDFLAG, SYNCINFO_ID, SALT_OF_PASSPHRASE, ConfigPassphraseStore, CouchDBConnection, FLAGMD_REDFLAG2, FLAGMD_REDFLAG3, PREFIXMD_LOGFILE, DatabaseConnectingStatus } from "./lib/src/types"; import { EntryDoc, LoadedEntry, ObsidianLiveSyncSettings, diff_check_result, diff_result_leaf, EntryBody, LOG_LEVEL, VER, DEFAULT_SETTINGS, diff_result, FLAGMD_REDFLAG, SYNCINFO_ID, SALT_OF_PASSPHRASE, ConfigPassphraseStore, CouchDBConnection, FLAGMD_REDFLAG2, FLAGMD_REDFLAG3, PREFIXMD_LOGFILE, DatabaseConnectingStatus, EntryHasPath, DocumentID, FilePathWithPrefix, FilePath, AnyEntry } from "./lib/src/types";
import { InternalFileInfo, queueItem, CacheData, FileEventItem, FileWatchEventQueueMax } from "./types"; import { InternalFileInfo, queueItem, CacheData, FileEventItem, FileWatchEventQueueMax } from "./types";
import { delay, getDocData, isDocContentSame } from "./lib/src/utils"; import { getDocData, isDocContentSame } from "./lib/src/utils";
import { Logger } from "./lib/src/logger"; import { Logger } from "./lib/src/logger";
import { PouchDB } from "./lib/src/pouchdb-browser.js"; import { PouchDB } from "./lib/src/pouchdb-browser.js";
import { LogDisplayModal } from "./LogDisplayModal"; import { LogDisplayModal } from "./LogDisplayModal";
import { ConflictResolveModal } from "./ConflictResolveModal"; import { ConflictResolveModal } from "./ConflictResolveModal";
import { ObsidianLiveSyncSettingTab } from "./ObsidianLiveSyncSettingTab"; import { ObsidianLiveSyncSettingTab } from "./ObsidianLiveSyncSettingTab";
import { DocumentHistoryModal } from "./DocumentHistoryModal"; import { DocumentHistoryModal } from "./DocumentHistoryModal";
import { applyPatch, cancelAllPeriodicTask, cancelAllTasks, cancelTask, generatePatchObj, id2path, isObjectMargeApplicable, isSensibleMargeApplicable, flattenObject, path2id, scheduleTask, tryParseJSON, createFile, modifyFile, isValidPath, getAbstractFileByPath, touch, recentlyTouched, isInternalMetadata, isPluginMetadata, id2filenameInternalMetadata, isChunk, askSelectString, askYesNo, askString, PeriodicProcessor, clearTouched } from "./utils"; import { applyPatch, cancelAllPeriodicTask, cancelAllTasks, cancelTask, generatePatchObj, id2path, isObjectMargeApplicable, isSensibleMargeApplicable, flattenObject, path2id, scheduleTask, tryParseJSON, createFile, modifyFile, isValidPath, getAbstractFileByPath, touch, recentlyTouched, isInternalMetadata, isPluginMetadata, stripInternalMetadataPrefix, isChunk, askSelectString, askYesNo, askString, PeriodicProcessor, clearTouched, getPath, getPathWithoutPrefix, getPathFromTFile, localDatabaseCleanUp, balanceChunks, performRebuildDB } from "./utils";
import { encrypt, tryDecrypt } from "./lib/src/e2ee_v2"; import { encrypt, tryDecrypt } from "./lib/src/e2ee_v2";
import { enableEncryption, isCloudantURI, isErrorOfMissingDoc, isValidRemoteCouchDBURI } from "./lib/src/utils_couchdb"; import { enableEncryption, isCloudantURI, isErrorOfMissingDoc, isValidRemoteCouchDBURI } from "./lib/src/utils_couchdb";
import { getGlobalStore, ObservableStore, observeStores } from "./lib/src/store"; import { getGlobalStore, ObservableStore, observeStores } from "./lib/src/store";
import { lockStore, logMessageStore, logStore } from "./lib/src/stores"; import { lockStore, logMessageStore, logStore } from "./lib/src/stores";
import { setNoticeClass } from "./lib/src/wrapper"; import { setNoticeClass } from "./lib/src/wrapper";
import { base64ToString, versionNumberString2Number, base64ToArrayBuffer, arrayBufferToBase64 } from "./lib/src/strbin"; import { base64ToString, versionNumberString2Number, base64ToArrayBuffer, arrayBufferToBase64 } from "./lib/src/strbin";
import { isPlainText, shouldBeIgnored } from "./lib/src/path"; import { addPrefix, isPlainText, shouldBeIgnored, stripAllPrefixes } from "./lib/src/path";
import { runWithLock } from "./lib/src/lock"; import { runWithLock } from "./lib/src/lock";
import { Semaphore } from "./lib/src/semaphore"; import { Semaphore } from "./lib/src/semaphore";
import { StorageEventManager, StorageEventManagerObsidian } from "./StorageEventManager"; import { StorageEventManager, StorageEventManagerObsidian } from "./StorageEventManager";
@@ -26,14 +26,13 @@ import { LiveSyncLocalDB, LiveSyncLocalDBEnv } from "./lib/src/LiveSyncLocalDB";
import { LiveSyncDBReplicator, LiveSyncReplicatorEnv } from "./lib/src/LiveSyncReplicator"; import { LiveSyncDBReplicator, LiveSyncReplicatorEnv } from "./lib/src/LiveSyncReplicator";
import { KeyValueDatabase, OpenKeyValueDatabase } from "./KeyValueDB"; import { KeyValueDatabase, OpenKeyValueDatabase } from "./KeyValueDB";
import { LiveSyncCommands } from "./LiveSyncCommands"; import { LiveSyncCommands } from "./LiveSyncCommands";
import { PluginAndTheirSettings } from "./CmdPluginAndTheirSettings";
import { HiddenFileSync } from "./CmdHiddenFileSync"; import { HiddenFileSync } from "./CmdHiddenFileSync";
import { SetupLiveSync } from "./CmdSetupLiveSync"; import { SetupLiveSync } from "./CmdSetupLiveSync";
import { ConfigSync } from "./CmdConfigSync";
import { confirmWithMessage } from "./dialogs";
setNoticeClass(Notice); setNoticeClass(Notice);
export default class ObsidianLiveSyncPlugin extends Plugin export default class ObsidianLiveSyncPlugin extends Plugin
implements LiveSyncLocalDBEnv, LiveSyncReplicatorEnv { implements LiveSyncLocalDBEnv, LiveSyncReplicatorEnv {
@@ -49,10 +48,11 @@ export default class ObsidianLiveSyncPlugin extends Plugin
packageVersion = ""; packageVersion = "";
manifestVersion = ""; manifestVersion = "";
addOnPluginAndTheirSettings = new PluginAndTheirSettings(this); // addOnPluginAndTheirSettings = new PluginAndTheirSettings(this);
addOnHiddenFileSync = new HiddenFileSync(this); addOnHiddenFileSync = new HiddenFileSync(this);
addOnSetup = new SetupLiveSync(this); addOnSetup = new SetupLiveSync(this);
addOns = [this.addOnPluginAndTheirSettings, this.addOnHiddenFileSync, this.addOnSetup] as LiveSyncCommands[]; addOnConfigSync = new ConfigSync(this);
addOns = [this.addOnHiddenFileSync, this.addOnSetup, this.addOnConfigSync] as LiveSyncCommands[];
periodicSyncProcessor = new PeriodicProcessor(this, async () => await this.replicate()); periodicSyncProcessor = new PeriodicProcessor(this, async () => await this.replicate());
@@ -131,7 +131,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin
delete transformedHeaders["content-length"]; delete transformedHeaders["content-length"];
delete transformedHeaders["Content-Length"]; delete transformedHeaders["Content-Length"];
const requestParam: RequestUrlParam = { const requestParam: RequestUrlParam = {
url: url as string, url,
method: opts.method, method: opts.method,
body: body, body: body,
headers: transformedHeaders, headers: transformedHeaders,
@@ -205,12 +205,25 @@ export default class ObsidianLiveSyncPlugin extends Plugin
} }
} }
id2path(filename: string): string { id2path(id: DocumentID, entry: EntryHasPath, stripPrefix?: boolean): FilePathWithPrefix {
return id2path(filename); const tempId = id2path(id, entry);
if (stripPrefix && isInternalMetadata(tempId)) {
const out = stripInternalMetadataPrefix(tempId);
return out;
}
return tempId;
} }
path2id(filename: string): string { getPath(entry: AnyEntry) {
return path2id(filename); return getPath(entry);
} }
getPathWithoutPrefix(entry: AnyEntry) {
return getPathWithoutPrefix(entry);
}
async path2id(filename: FilePathWithPrefix | FilePath, prefix?: string): Promise<DocumentID> {
const destPath = addPrefix(filename, prefix);
return await path2id(destPath, this.settings.usePathObfuscation ? this.settings.passphrase : "");
}
createPouchDBInstance<T>(name?: string, options?: PouchDB.Configuration.DatabaseConfiguration): PouchDB.Database<T> { createPouchDBInstance<T>(name?: string, options?: PouchDB.Configuration.DatabaseConfiguration): PouchDB.Database<T> {
if (this.settings.useIndexedDBAdapter) { if (this.settings.useIndexedDBAdapter) {
options.adapter = "indexeddb"; options.adapter = "indexeddb";
@@ -291,43 +304,45 @@ export default class ObsidianLiveSyncPlugin extends Plugin
} }
} }
showHistory(file: TFile | string) { showHistory(file: TFile | FilePathWithPrefix, id: DocumentID) {
new DocumentHistoryModal(this.app, this, file).open(); new DocumentHistoryModal(this.app, this, file, id).open();
} }
async fileHistory() { async fileHistory() {
const notes: { path: string, mtime: number }[] = []; const notes: { id: DocumentID, path: FilePathWithPrefix, dispPath: string, mtime: number }[] = [];
for await (const doc of this.localDatabase.findAllDocs()) { for await (const doc of this.localDatabase.findAllDocs()) {
notes.push({ path: id2path(doc._id), mtime: doc.mtime }); notes.push({ id: doc._id, path: this.getPath(doc), dispPath: this.getPathWithoutPrefix(doc), mtime: doc.mtime });
} }
notes.sort((a, b) => b.mtime - a.mtime); notes.sort((a, b) => b.mtime - a.mtime);
const notesList = notes.map(e => e.path); const notesList = notes.map(e => e.dispPath);
const target = await askSelectString(this.app, "File to view History", notesList); const target = await askSelectString(this.app, "File to view History", notesList);
if (target) { if (target) {
this.showHistory(target); const targetId = notes.find(e => e.dispPath == target);
this.showHistory(targetId.path, undefined);
} }
} }
async pickFileForResolve() { async pickFileForResolve() {
const notes: { path: string, mtime: number }[] = []; const notes: { id: DocumentID, path: FilePathWithPrefix, dispPath: string, mtime: number }[] = [];
for await (const doc of this.localDatabase.findAllDocs({ conflicts: true })) { for await (const doc of this.localDatabase.findAllDocs({ conflicts: true })) {
if (!("_conflicts" in doc)) continue; if (!("_conflicts" in doc)) continue;
notes.push({ path: id2path(doc._id), mtime: doc.mtime }); notes.push({ id: doc._id, path: this.getPath(doc), dispPath: this.getPathWithoutPrefix(doc), mtime: doc.mtime });
} }
notes.sort((a, b) => b.mtime - a.mtime); notes.sort((a, b) => b.mtime - a.mtime);
const notesList = notes.map(e => e.path); const notesList = notes.map(e => e.dispPath);
if (notesList.length == 0) { if (notesList.length == 0) {
Logger("There are no conflicted documents", LOG_LEVEL.NOTICE); Logger("There are no conflicted documents", LOG_LEVEL.NOTICE);
return false; return false;
} }
const target = await askSelectString(this.app, "File to view History", notesList); const target = await askSelectString(this.app, "File to view History", notesList);
if (target) { if (target) {
await this.resolveConflicted(target); const targetItem = notes.find(e => e.dispPath == target);
await this.resolveConflicted(targetItem.path);
return true; return true;
} }
return false; return false;
} }
async resolveConflicted(target: string) { async resolveConflicted(target: FilePathWithPrefix) {
if (isInternalMetadata(target)) { if (isInternalMetadata(target)) {
await this.addOnHiddenFileSync.resolveConflictOnInternalFile(target); await this.addOnHiddenFileSync.resolveConflictOnInternalFile(target);
} else if (isPluginMetadata(target)) { } else if (isPluginMetadata(target)) {
@@ -346,7 +361,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin
for await (const doc of this.localDatabase.findAllDocs({ conflicts: true })) { for await (const doc of this.localDatabase.findAllDocs({ conflicts: true })) {
if (doc.type == "newnote" || doc.type == "plain") { if (doc.type == "newnote" || doc.type == "plain") {
if (doc.deleted && (doc.mtime - limit) < 0) { if (doc.deleted && (doc.mtime - limit) < 0) {
notes.push({ path: id2path(doc._id), mtime: doc.mtime, ttl: (doc.mtime - limit) / 1000 / 86400, doc: doc }); notes.push({ path: this.getPath(doc), mtime: doc.mtime, ttl: (doc.mtime - limit) / 1000 / 86400, doc: doc });
} }
} }
} }
@@ -360,8 +375,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin
Logger(`Deletion history expired: ${v.path}`); Logger(`Deletion history expired: ${v.path}`);
const delDoc = v.doc; const delDoc = v.doc;
delDoc._deleted = true; delDoc._deleted = true;
// console.dir(delDoc); await this.localDatabase.putRaw(delDoc);
await this.localDatabase.localDatabase.put(delDoc);
} }
Logger(`Checking expired file history done`); Logger(`Checking expired file history done`);
} }
@@ -375,25 +389,13 @@ export default class ObsidianLiveSyncPlugin extends Plugin
try { try {
if (this.isRedFlagRaised() || this.isRedFlag2Raised() || this.isRedFlag3Raised()) { if (this.isRedFlagRaised() || this.isRedFlag2Raised() || this.isRedFlag3Raised()) {
this.settings.batchSave = false; this.settings.batchSave = false;
this.settings.liveSync = false; this.addOnSetup.suspendAllSync();
this.settings.periodicReplication = false; this.addOnSetup.suspendExtraSync();
this.settings.syncOnSave = false;
this.settings.syncOnStart = false;
this.settings.syncOnFileOpen = false;
this.settings.syncAfterMerge = false;
this.settings.autoSweepPlugins = false;
this.settings.usePluginSync = false;
this.settings.suspendFileWatching = true; this.settings.suspendFileWatching = true;
this.settings.syncInternalFiles = false;
await this.saveSettings(); await this.saveSettings();
if (this.isRedFlag2Raised()) { if (this.isRedFlag2Raised()) {
Logger(`${FLAGMD_REDFLAG2} has been detected! Self-hosted LiveSync suspends all sync and rebuild everything.`, LOG_LEVEL.NOTICE); Logger(`${FLAGMD_REDFLAG2} has been detected! Self-hosted LiveSync suspends all sync and rebuild everything.`, LOG_LEVEL.NOTICE);
await this.resetLocalDatabase(); await this.addOnSetup.rebuildEverything();
await this.initializeDatabase(true);
await this.markRemoteLocked();
await this.tryResetRemoteDatabase();
await this.markRemoteLocked();
await this.replicateAllToServer(true);
await this.deleteRedFlag2(); await this.deleteRedFlag2();
if (await askYesNo(this.app, "Do you want to disable Suspend file watching and restart obsidian now?") == "yes") { if (await askYesNo(this.app, "Do you want to disable Suspend file watching and restart obsidian now?") == "yes") {
this.settings.suspendFileWatching = false; this.settings.suspendFileWatching = false;
@@ -403,12 +405,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin
} }
} else if (this.isRedFlag3Raised()) { } else if (this.isRedFlag3Raised()) {
Logger(`${FLAGMD_REDFLAG3} has been detected! Self-hosted LiveSync will discard the local database and fetch everything from the remote once again.`, LOG_LEVEL.NOTICE); Logger(`${FLAGMD_REDFLAG3} has been detected! Self-hosted LiveSync will discard the local database and fetch everything from the remote once again.`, LOG_LEVEL.NOTICE);
await this.resetLocalDatabase(); await this.addOnSetup.fetchLocal();
await delay(1000);
await this.markRemoteResolved();
await this.openDatabase();
this.isReady = true;
await this.replicateAllFromServer(true);
await this.deleteRedFlag3(); await this.deleteRedFlag3();
if (await askYesNo(this.app, "Do you want to disable Suspend file watching and restart obsidian now?") == "yes") { if (await askYesNo(this.app, "Do you want to disable Suspend file watching and restart obsidian now?") == "yes") {
this.settings.suspendFileWatching = false; this.settings.suspendFileWatching = false;
@@ -453,7 +450,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin
Logger(`Additional safety scan..`, LOG_LEVEL.VERBOSE); Logger(`Additional safety scan..`, LOG_LEVEL.VERBOSE);
for await (const doc of this.localDatabase.findAllDocs({ conflicts: true })) { for await (const doc of this.localDatabase.findAllDocs({ conflicts: true })) {
if (!("_conflicts" in doc)) continue; if (!("_conflicts" in doc)) continue;
notes.push({ path: id2path(doc._id), mtime: doc.mtime }); notes.push({ path: this.getPath(doc), mtime: doc.mtime });
} }
if (notes.length > 0) { if (notes.length > 0) {
Logger(`Some files have been left conflicted! Please resolve them by "Pick a file to resolve conflict". The list is written in the log.`, LOG_LEVEL.NOTICE); Logger(`Some files have been left conflicted! Please resolve them by "Pick a file to resolve conflict". The list is written in the log.`, LOG_LEVEL.NOTICE);
@@ -555,14 +552,14 @@ export default class ObsidianLiveSyncPlugin extends Plugin
id: "livesync-dump", id: "livesync-dump",
name: "Dump information of this doc ", name: "Dump information of this doc ",
editorCallback: (editor: Editor, view: MarkdownView) => { editorCallback: (editor: Editor, view: MarkdownView) => {
this.localDatabase.getDBEntry(view.file.path, {}, true, false); this.localDatabase.getDBEntry(getPathFromTFile(view.file), {}, true, false);
}, },
}); });
this.addCommand({ this.addCommand({
id: "livesync-checkdoc-conflicted", id: "livesync-checkdoc-conflicted",
name: "Resolve if conflicted.", name: "Resolve if conflicted.",
editorCallback: async (editor: Editor, view: MarkdownView) => { editorCallback: async (editor: Editor, view: MarkdownView) => {
await this.showIfConflicted(view.file.path); await this.showIfConflicted(getPathFromTFile(view.file));
}, },
}); });
@@ -600,7 +597,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin
id: "livesync-history", id: "livesync-history",
name: "Show history", name: "Show history",
editorCallback: (editor: Editor, view: MarkdownView) => { editorCallback: (editor: Editor, view: MarkdownView) => {
this.showHistory(view.file); this.showHistory(view.file, null);
}, },
}); });
this.addCommand({ this.addCommand({
@@ -766,6 +763,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin
} }
this.settings = settings; this.settings = settings;
if ("workingEncrypt" in this.settings) delete this.settings.workingEncrypt; if ("workingEncrypt" in this.settings) delete this.settings.workingEncrypt;
if ("workingPassphrase" in this.settings) delete this.settings.workingPassphrase; if ("workingPassphrase" in this.settings) delete this.settings.workingPassphrase;
@@ -899,10 +897,17 @@ export default class ObsidianLiveSyncPlugin extends Plugin
const file = queue.args.file; const file = queue.args.file;
const key = `file-last-proc-${queue.type}-${file.path}`; const key = `file-last-proc-${queue.type}-${file.path}`;
const last = Number(await this.kvDB.get(key) || 0); const last = Number(await this.kvDB.get(key) || 0);
let mtime = file.mtime;
if (queue.type == "DELETE") { if (queue.type == "DELETE") {
await this.deleteFromDBbyPath(file.path); await this.deleteFromDBbyPath(file.path);
mtime = file.mtime - 1;
const keyD1 = `file-last-proc-CREATE-${file.path}`;
const keyD2 = `file-last-proc-CHANGED-${file.path}`;
await this.kvDB.set(keyD1, mtime);
await this.kvDB.set(keyD2, mtime);
} else if (queue.type == "INTERNAL") { } else if (queue.type == "INTERNAL") {
await this.addOnHiddenFileSync.watchVaultRawEventsAsync(file.path); await this.addOnHiddenFileSync.watchVaultRawEventsAsync(file.path);
await this.addOnConfigSync.watchVaultRawEventsAsync(file.path);
} else { } else {
const targetFile = this.app.vault.getAbstractFileByPath(file.path); const targetFile = this.app.vault.getAbstractFileByPath(file.path);
if (!(targetFile instanceof TFile)) { if (!(targetFile instanceof TFile)) {
@@ -917,6 +922,8 @@ export default class ObsidianLiveSyncPlugin extends Plugin
const cache = queue.args.cache; const cache = queue.args.cache;
if (queue.type == "CREATE" || queue.type == "CHANGED") { if (queue.type == "CREATE" || queue.type == "CHANGED") {
const keyD1 = `file-last-proc-DELETED-${file.path}`;
await this.kvDB.set(keyD1, mtime);
if (!await this.updateIntoDB(targetFile, false, cache)) { if (!await this.updateIntoDB(targetFile, false, cache)) {
Logger(`DB -> STORAGE: failed, cancel the relative operations: ${targetFile.path}`, LOG_LEVEL.INFO); Logger(`DB -> STORAGE: failed, cancel the relative operations: ${targetFile.path}`, LOG_LEVEL.INFO);
// cancel running queues and remove one of atomic operation // cancel running queues and remove one of atomic operation
@@ -929,7 +936,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin
await this.watchVaultRenameAsync(targetFile, queue.args.oldPath); await this.watchVaultRenameAsync(targetFile, queue.args.oldPath);
} }
} }
await this.kvDB.set(key, file.mtime); await this.kvDB.set(key, mtime);
} while (this.vaultManager.getQueueLength() > 0); } while (this.vaultManager.getQueueLength() > 0);
return true; return true;
}) })
@@ -953,7 +960,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin
if (this.settings.syncOnFileOpen && !this.suspended) { if (this.settings.syncOnFileOpen && !this.suspended) {
await this.replicate(); await this.replicate();
} }
await this.showIfConflicted(file.path); await this.showIfConflicted(getPathFromTFile(file));
} }
async applyBatchChange() { async applyBatchChange() {
@@ -1012,6 +1019,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin
const messageContent = typeof message == "string" ? message : message instanceof Error ? `${message.name}:${message.message}` : JSON.stringify(message, null, 2); const messageContent = typeof message == "string" ? message : message instanceof Error ? `${message.name}:${message.message}` : JSON.stringify(message, null, 2);
if (message instanceof Error) { if (message instanceof Error) {
// debugger; // debugger;
console.dir(message.stack);
} }
const newMessage = timestamp + "->" + messageContent; const newMessage = timestamp + "->" + messageContent;
@@ -1091,31 +1099,33 @@ export default class ObsidianLiveSyncPlugin extends Plugin
async doc2storage(docEntry: EntryBody, file?: TFile, force?: boolean) { async doc2storage(docEntry: EntryBody, file?: TFile, force?: boolean) {
const mode = file == undefined ? "create" : "modify"; const mode = file == undefined ? "create" : "modify";
const pathSrc = id2path(docEntry._id); const path = this.getPath(docEntry);
if (shouldBeIgnored(pathSrc)) { if (shouldBeIgnored(path)) {
return; return;
} }
if (!this.isTargetFile(pathSrc)) return; if (!this.isTargetFile(path)) return;
if (docEntry._deleted || docEntry.deleted) { if (docEntry._deleted || docEntry.deleted) {
// This occurs not only when files are deleted, but also when conflicts are resolved. // This occurs not only when files are deleted, but also when conflicts are resolved.
// We have to check no other revisions are left. // We have to check no other revisions are left.
const lastDocs = await this.localDatabase.getDBEntry(pathSrc); const lastDocs = await this.localDatabase.getDBEntry(path);
if (path != file.path) {
Logger(`delete skipped: ${file.path} :Not exactly matched`, LOG_LEVEL.VERBOSE);
}
if (lastDocs === false) { if (lastDocs === false) {
await this.deleteVaultItem(file); await this.deleteVaultItem(file);
} else { } else {
// it perhaps delete some revisions. // it perhaps delete some revisions.
// may be we have to reload this // may be we have to reload this
await this.pullFile(pathSrc, null, true); await this.pullFile(path, null, true);
Logger(`delete skipped:${lastDocs._id}`, LOG_LEVEL.VERBOSE); Logger(`delete skipped:${file.path}`, LOG_LEVEL.VERBOSE);
} }
return; return;
} }
const localMtime = ~~((file?.stat?.mtime || 0) / 1000); const localMtime = ~~((file?.stat?.mtime || 0) / 1000);
const docMtime = ~~(docEntry.mtime / 1000); const docMtime = ~~(docEntry.mtime / 1000);
const doc = await this.localDatabase.getDBEntry(pathSrc, { rev: docEntry._rev }); const doc = await this.localDatabase.getDBEntry(path, { rev: docEntry._rev });
if (doc === false) return; if (doc === false) return;
const path = id2path(doc._id);
const msg = `DB -> STORAGE (${mode}${force ? ",force" : ""},${doc.datatype}) `; const msg = `DB -> STORAGE (${mode}${force ? ",force" : ""},${doc.datatype}) `;
if (doc.datatype != "newnote" && doc.datatype != "plain") { if (doc.datatype != "newnote" && doc.datatype != "plain") {
Logger(msg + "ERROR, Invalid datatype: " + path + "(" + doc.datatype + ")", LOG_LEVEL.NOTICE); Logger(msg + "ERROR, Invalid datatype: " + path + "(" + doc.datatype + ")", LOG_LEVEL.NOTICE);
@@ -1134,7 +1144,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin
outFile = await createFile(normalizePath(path), writeData, { ctime: doc.ctime, mtime: doc.mtime, }); outFile = await createFile(normalizePath(path), writeData, { ctime: doc.ctime, mtime: doc.mtime, });
} else { } else {
await modifyFile(file, writeData, { ctime: doc.ctime, mtime: doc.mtime }); await modifyFile(file, writeData, { ctime: doc.ctime, mtime: doc.mtime });
outFile = getAbstractFileByPath(file.path) as TFile; outFile = getAbstractFileByPath(getPathFromTFile(file)) as TFile;
} }
Logger(msg + path); Logger(msg + path);
touch(outFile); touch(outFile);
@@ -1172,7 +1182,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin
handleDBChanged(change: EntryBody) { handleDBChanged(change: EntryBody) {
// If the file is opened, we have to apply immediately // If the file is opened, we have to apply immediately
const af = app.workspace.getActiveFile(); const af = app.workspace.getActiveFile();
if (af && af.path == id2path(change._id)) { if (af && af.path == this.getPath(change)) {
this.queuedEntries = this.queuedEntries.filter(e => e._id != change._id); this.queuedEntries = this.queuedEntries.filter(e => e._id != change._id);
return this.handleDBChangedAsync(change); return this.handleDBChangedAsync(change);
} }
@@ -1188,15 +1198,16 @@ export default class ObsidianLiveSyncPlugin extends Plugin
const entry = this.queuedEntries.shift(); const entry = this.queuedEntries.shift();
// If the same file is to be manipulated, leave it to the last process. // If the same file is to be manipulated, leave it to the last process.
if (this.queuedEntries.some(e => e._id == entry._id)) continue; if (this.queuedEntries.some(e => e._id == entry._id)) continue;
const path = getPath(entry);
try { try {
const releaser = await semaphore.acquire(1); const releaser = await semaphore.acquire(1);
runWithLock(`dbchanged-${entry._id}`, false, async () => { runWithLock(`dbchanged-${path}`, false, async () => {
Logger(`Applying ${entry._id} (${entry._rev}) change...`, LOG_LEVEL.VERBOSE); Logger(`Applying ${path} (${entry._id}: ${entry._rev}) change...`, LOG_LEVEL.VERBOSE);
await this.handleDBChangedAsync(entry); await this.handleDBChangedAsync(entry);
Logger(`Applied ${entry._id} (${entry._rev}) change...`); Logger(`Applied ${path} (${entry._id}:${entry._rev}) change...`);
}).finally(() => { releaser(); }); }).finally(() => { releaser(); });
} catch (ex) { } catch (ex) {
Logger(`Failed to apply the change of ${entry._id} (${entry._rev})`); Logger(`Failed to apply the change of ${path} (${entry._id}:${entry._rev})`);
} }
} while (this.queuedEntries.length > 0); } while (this.queuedEntries.length > 0);
} finally { } finally {
@@ -1205,7 +1216,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin
} }
async handleDBChangedAsync(change: EntryBody) { async handleDBChangedAsync(change: EntryBody) {
const targetFile = getAbstractFileByPath(id2path(change._id)); const targetFile = getAbstractFileByPath(this.getPathWithoutPrefix(change));
if (targetFile == null) { if (targetFile == null) {
if (change._deleted || change.deleted) { if (change._deleted || change.deleted) {
return; return;
@@ -1232,17 +1243,17 @@ export default class ObsidianLiveSyncPlugin extends Plugin
await this.doc2storage(doc, file); await this.doc2storage(doc, file);
queueConflictCheck(); queueConflictCheck();
} else { } else {
const d = await this.localDatabase.getDBEntryMeta(id2path(change._id), { conflicts: true }, true); const d = await this.localDatabase.getDBEntryMeta(this.getPath(change), { conflicts: true }, true);
if (d && !d._conflicts) { if (d && !d._conflicts) {
await this.doc2storage(doc, file); await this.doc2storage(doc, file);
} else { } else {
if (!queueConflictCheck()) { if (!queueConflictCheck()) {
Logger(`${id2path(change._id)} is conflicted, write to the storage has been pended.`, LOG_LEVEL.NOTICE); Logger(`${this.getPath(change)} is conflicted, write to the storage has been pended.`, LOG_LEVEL.NOTICE);
} }
} }
} }
} else { } else {
Logger(`${id2path(change._id)} is already exist as the folder`); Logger(`${this.getPath(change)} is already exist as the folder`);
} }
} }
@@ -1258,7 +1269,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin
async loadQueuedFiles() { async loadQueuedFiles() {
const lsKey = "obsidian-livesync-queuefiles-" + this.getVaultName(); const lsKey = "obsidian-livesync-queuefiles-" + this.getVaultName();
const ids = JSON.parse(localStorage.getItem(lsKey) || "[]") as string[]; const ids = JSON.parse(localStorage.getItem(lsKey) || "[]") as string[];
const ret = await this.localDatabase.localDatabase.allDocs({ keys: ids, include_docs: true }); const ret = await this.localDatabase.allDocsRaw<EntryDoc>({ keys: ids, include_docs: true });
for (const doc of ret.rows) { for (const doc of ret.rows) {
if (doc.doc && !this.queuedFiles.some((e) => e.entry._id == doc.doc._id)) { if (doc.doc && !this.queuedFiles.some((e) => e.entry._id == doc.doc._id)) {
await this.parseIncomingDoc(doc.doc as PouchDB.Core.ExistingDocument<EntryBody & PouchDB.Core.AllDocsMeta>); await this.parseIncomingDoc(doc.doc as PouchDB.Core.ExistingDocument<EntryBody & PouchDB.Core.AllDocsMeta>);
@@ -1276,9 +1287,9 @@ export default class ObsidianLiveSyncPlugin extends Plugin
queue.done = true; queue.done = true;
if (isInternalMetadata(queue.entry._id)) { if (isInternalMetadata(queue.entry._id)) {
//system file //system file
const filename = id2path(id2filenameInternalMetadata(queue.entry._id)); const filename = this.getPathWithoutPrefix(queue.entry);
this.addOnHiddenFileSync.procInternalFile(filename); this.addOnHiddenFileSync.procInternalFile(filename);
} else if (isValidPath(id2path(queue.entry._id))) { } else if (isValidPath(this.getPath(queue.entry))) {
this.handleDBChanged(queue.entry); this.handleDBChanged(queue.entry);
} else { } else {
Logger(`Skipped: ${queue.entry._id}`, LOG_LEVEL.VERBOSE); Logger(`Skipped: ${queue.entry._id}`, LOG_LEVEL.VERBOSE);
@@ -1315,7 +1326,8 @@ export default class ObsidianLiveSyncPlugin extends Plugin
if (isNewFileCompleted) this.procQueuedFiles(); if (isNewFileCompleted) this.procQueuedFiles();
} }
async parseIncomingDoc(doc: PouchDB.Core.ExistingDocument<EntryBody>) { async parseIncomingDoc(doc: PouchDB.Core.ExistingDocument<EntryBody>) {
if (!this.isTargetFile(id2path(doc._id))) return; const path = this.getPath(doc);
if (!this.isTargetFile(path)) return;
const skipOldFile = this.settings.skipOlderFilesOnSync && false; //patched temporary. const skipOldFile = this.settings.skipOlderFilesOnSync && false; //patched temporary.
// Do not handle internal files if the feature has not been enabled. // Do not handle internal files if the feature has not been enabled.
if (isInternalMetadata(doc._id) && !this.settings.syncInternalFiles) return; if (isInternalMetadata(doc._id) && !this.settings.syncInternalFiles) return;
@@ -1326,19 +1338,19 @@ export default class ObsidianLiveSyncPlugin extends Plugin
FLAGMD_REDFLAG2, FLAGMD_REDFLAG2,
FLAGMD_REDFLAG3 FLAGMD_REDFLAG3
]; ];
if (!isInternalMetadata(doc._id) && ignoreFiles.contains(id2path(doc._id))) { if (!isInternalMetadata(doc._id) && ignoreFiles.contains(path)) {
return; return;
} }
if ((!isInternalMetadata(doc._id)) && skipOldFile) { if ((!isInternalMetadata(doc._id)) && skipOldFile) {
const info = getAbstractFileByPath(id2path(doc._id)); const info = getAbstractFileByPath(stripAllPrefixes(path));
if (info && info instanceof TFile) { if (info && info instanceof TFile) {
const localMtime = ~~((info as TFile).stat.mtime / 1000); const localMtime = ~~(info.stat.mtime / 1000);
const docMtime = ~~(doc.mtime / 1000); const docMtime = ~~(doc.mtime / 1000);
//TODO: some margin required. //TODO: some margin required.
if (localMtime >= docMtime) { if (localMtime >= docMtime) {
Logger(`${doc._id} Skipped, older than storage.`, LOG_LEVEL.VERBOSE); Logger(`${path} (${doc._id}, ${doc._rev}) Skipped, older than storage.`, LOG_LEVEL.VERBOSE);
return; return;
} }
} }
@@ -1349,12 +1361,12 @@ export default class ObsidianLiveSyncPlugin extends Plugin
missingChildren: [] as string[], missingChildren: [] as string[],
timeout: now + this.chunkWaitTimeout, timeout: now + this.chunkWaitTimeout,
}; };
// If `Read chunks online` is enabled, retrieve chunks from the remote CouchDB directly. // If `Read chunks online` is disabled, chunks should be transferred before here.
// However, in some cases, chunks are after that. So, if missing chunks exist, we have to wait for them.
if ((!this.settings.readChunksOnline) && "children" in doc) { if ((!this.settings.readChunksOnline) && "children" in doc) {
const c = await this.localDatabase.localDatabase.allDocs({ keys: doc.children, include_docs: false }); const c = await this.localDatabase.collectChunksWithCache(doc.children)
const missing = c.rows.filter((e) => "error" in e).map((e) => e.key); const missing = c.filter((e) => !e.chunk).map((e) => e.id);
// fetch from remote if (missing.length > 0) Logger(`${path} (${doc._id}, ${doc._rev}) Queued (waiting ${missing.length} items)`, LOG_LEVEL.VERBOSE);
if (missing.length > 0) Logger(`${doc._id}(${doc._rev}) Queued (waiting ${missing.length} items)`, LOG_LEVEL.VERBOSE);
newQueue.missingChildren = missing; newQueue.missingChildren = missing;
this.queuedFiles.push(newQueue); this.queuedFiles.push(newQueue);
} else { } else {
@@ -1366,8 +1378,9 @@ export default class ObsidianLiveSyncPlugin extends Plugin
//---> Sync //---> Sync
async parseReplicationResult(docs: Array<PouchDB.Core.ExistingDocument<EntryDoc>>): Promise<void> { async parseReplicationResult(docs: Array<PouchDB.Core.ExistingDocument<EntryDoc>>): Promise<void> {
const docsSorted = docs.sort((a, b) => b.mtime - a.mtime);
L1: L1:
for (const change of docs) { for (const change of docsSorted) {
for (const proc of this.addOns) { for (const proc of this.addOns) {
if (await proc.parseReplicationResultItem(change)) { if (await proc.parseReplicationResultItem(change)) {
continue L1; continue L1;
@@ -1496,27 +1509,25 @@ export default class ObsidianLiveSyncPlugin extends Plugin
return; return;
} }
logHideTimer: NodeJS.Timeout = null;
setStatusBarText(message: string = null, log: string = null) { setStatusBarText(message: string = null, log: string = null) {
if (!this.statusBar) return; if (!this.statusBar) return;
const newMsg = typeof message == "string" ? message : this.lastMessage; const newMsg = typeof message == "string" ? message : this.lastMessage;
const newLog = typeof log == "string" ? log : this.lastLog; const newLog = typeof log == "string" ? log : this.lastLog;
if (`${this.lastMessage}-${this.lastLog}` != `${newMsg}-${newLog}`) { if (`${this.lastMessage}-${this.lastLog}` != `${newMsg}-${newLog}`) {
this.statusBar.setText(newMsg.split("\n")[0]); scheduleTask("update-display", 50, () => {
this.statusBar.setText(newMsg.split("\n")[0]);
if (this.settings.showStatusOnEditor) { if (this.settings.showStatusOnEditor) {
const root = activeDocument.documentElement; const root = activeDocument.documentElement;
const q = root.querySelectorAll(`.CodeMirror-wrap,.cm-s-obsidian>.cm-editor,.canvas-wrapper`); const q = root.querySelectorAll(`.CodeMirror-wrap,.cm-s-obsidian>.cm-editor,.canvas-wrapper`);
q.forEach(e => e.setAttr("data-log", '' + (newMsg + "\n" + newLog) + '')) q.forEach(e => e.setAttr("data-log", '' + (newMsg + "\n" + newLog) + ''))
} else { } else {
const root = activeDocument.documentElement; const root = activeDocument.documentElement;
const q = root.querySelectorAll(`.CodeMirror-wrap,.cm-s-obsidian>.cm-editor,.canvas-wrapper`); const q = root.querySelectorAll(`.CodeMirror-wrap,.cm-s-obsidian>.cm-editor,.canvas-wrapper`);
q.forEach(e => e.setAttr("data-log", '')) q.forEach(e => e.setAttr("data-log", ''))
} }
if (this.logHideTimer != null) { }, true);
clearTimeout(this.logHideTimer); scheduleTask("log-hide", 3000, () => this.setStatusBarText(null, ""));
}
this.logHideTimer = setTimeout(() => this.setStatusBarText(null, ""), 3000);
this.lastMessage = newMsg; this.lastMessage = newMsg;
this.lastLog = newLog; this.lastLog = newLog;
} }
@@ -1532,7 +1543,43 @@ export default class ObsidianLiveSyncPlugin extends Plugin
await this.applyBatchChange(); await this.applyBatchChange();
await Promise.all(this.addOns.map(e => e.beforeReplicate(showMessage))); await Promise.all(this.addOns.map(e => e.beforeReplicate(showMessage)));
await this.loadQueuedFiles(); await this.loadQueuedFiles();
return await this.replicator.openReplication(this.settings, false, showMessage); const ret = await this.replicator.openReplication(this.settings, false, showMessage);
if (!ret) {
if (this.replicator.remoteLockedAndDeviceNotAccepted) {
if (this.replicator.remoteCleaned) {
const message = `
The remote database has been cleaned up.
To synchronize, this device must also be cleaned up or fetch everything again once.
Fetching may takes some time. Cleaning up is not stable yet but fast.
`
const CHOICE_CLEANUP = "Clean up";
const CHOICE_FETCH = "Fetch again";
const CHOICE_DISMISS = "Dismiss";
const ret = await confirmWithMessage(this, "Locked", message, [CHOICE_CLEANUP, CHOICE_FETCH, CHOICE_DISMISS], CHOICE_DISMISS, 10);
if (ret == CHOICE_CLEANUP) {
await localDatabaseCleanUp(this, true, false);
await balanceChunks(this, false);
}
if (ret == CHOICE_FETCH) {
await performRebuildDB(this, "localOnly");
}
} else {
const message = `
The remote database has been rebuilt.
To synchronize, this device must fetch everything again once.
Or if you are sure know what had been happened, we can unlock the database from the setting dialog.
`
const CHOICE_FETCH = "Fetch again";
const CHOICE_DISMISS = "Dismiss";
const ret = await confirmWithMessage(this, "Locked", message, [CHOICE_FETCH, CHOICE_DISMISS], CHOICE_DISMISS, 10);
if (ret == CHOICE_FETCH) {
await performRebuildDB(this, "localOnly");
}
}
}
}
return ret;
} }
async initializeDatabase(showingNotice?: boolean, reopenDatabase = true) { async initializeDatabase(showingNotice?: boolean, reopenDatabase = true) {
@@ -1563,12 +1610,12 @@ export default class ObsidianLiveSyncPlugin extends Plugin
return await this.replicator.replicateAllFromServer(this.settings, showingNotice); return await this.replicator.replicateAllFromServer(this.settings, showingNotice);
} }
async markRemoteLocked() { async markRemoteLocked(lockByClean?: boolean) {
return await this.replicator.markRemoteLocked(this.settings, true); return await this.replicator.markRemoteLocked(this.settings, true, lockByClean);
} }
async markRemoteUnlocked() { async markRemoteUnlocked() {
return await this.replicator.markRemoteLocked(this.settings, false); return await this.replicator.markRemoteLocked(this.settings, false, false);
} }
async markRemoteResolved() { async markRemoteResolved() {
@@ -1590,10 +1637,10 @@ export default class ObsidianLiveSyncPlugin extends Plugin
const filesStorage = this.app.vault.getFiles().filter(e => this.isTargetFile(e)); const filesStorage = this.app.vault.getFiles().filter(e => this.isTargetFile(e));
const filesStorageName = filesStorage.map((e) => e.path); const filesStorageName = filesStorage.map((e) => e.path);
Logger("Collecting local files on the DB", LOG_LEVEL.VERBOSE); Logger("Collecting local files on the DB", LOG_LEVEL.VERBOSE);
const filesDatabase = [] as string[] const filesDatabase = [] as FilePathWithPrefix[]
for await (const docId of this.localDatabase.findAllDocNames()) { for await (const doc of this.localDatabase.findAllNormalDocs()) {
const path = id2path(docId); const path = getPath(doc);
if (isValidPath(docId) && this.isTargetFile(path)) { if (isValidPath(path) && this.isTargetFile(path)) {
filesDatabase.push(path); filesDatabase.push(path);
} }
} }
@@ -1605,7 +1652,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin
initialScan = true; initialScan = true;
Logger("Database looks empty, save files as initial sync data"); Logger("Database looks empty, save files as initial sync data");
} }
const onlyInStorage = filesStorage.filter((e) => filesDatabase.indexOf(e.path) == -1); const onlyInStorage = filesStorage.filter((e) => filesDatabase.indexOf(getPathFromTFile(e)) == -1);
const onlyInDatabase = filesDatabase.filter((e) => filesStorageName.indexOf(e) == -1); const onlyInDatabase = filesDatabase.filter((e) => filesStorageName.indexOf(e) == -1);
const onlyInStorageNames = onlyInStorage.map((e) => e.path); const onlyInStorageNames = onlyInStorage.map((e) => e.path);
@@ -1659,9 +1706,16 @@ export default class ObsidianLiveSyncPlugin extends Plugin
caches = await this.kvDB.get<{ [key: string]: { storageMtime: number; docMtime: number } }>("diff-caches") || {}; caches = await this.kvDB.get<{ [key: string]: { storageMtime: number; docMtime: number } }>("diff-caches") || {};
const docsCount = syncFiles.length; const docsCount = syncFiles.length;
do { do {
const syncFilesX = syncFiles.splice(0, 100); const syncFilesXSrc = syncFiles.splice(0, 100);
const docs = await this.localDatabase.localDatabase.allDocs({ keys: syncFilesX.map(e => path2id(e.path)), include_docs: true }) const syncFilesX = [] as { file: TFile, id: DocumentID }[];
const syncFilesToSync = syncFilesX.map((e) => ({ file: e, doc: docs.rows.find(ee => ee.id == path2id(e.path)).doc as LoadedEntry })); for (const file of syncFilesXSrc) {
const id = await this.path2id(getPathFromTFile(file));
syncFilesX.push({ file: file, id: id });
}
const docs = await this.localDatabase.allDocsRaw<EntryDoc>({ keys: syncFilesX.map(e => e.id), include_docs: true })
const docsMap = docs.rows.reduce((p, c) => ({ ...p, [c.id]: c.doc }), {} as Record<DocumentID, EntryDoc>)
const syncFilesToSync = syncFilesX.map((e) => ({ file: e.file, doc: docsMap[e.id] as LoadedEntry }));
await runAll(`CHECK FILE STATUS:${syncFiles.length}/${docsCount}`, syncFilesToSync, async (e) => { await runAll(`CHECK FILE STATUS:${syncFiles.length}/${docsCount}`, syncFilesToSync, async (e) => {
caches = await this.syncFileBetweenDBandStorage(e.file, e.doc, initialScan, caches); caches = await this.syncFileBetweenDBandStorage(e.file, e.doc, initialScan, caches);
}); });
@@ -1680,7 +1734,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin
} }
// --> conflict resolving // --> conflict resolving
async getConflictedDoc(path: string, rev: string): Promise<false | diff_result_leaf> { async getConflictedDoc(path: FilePathWithPrefix, rev: string): Promise<false | diff_result_leaf> {
try { try {
const doc = await this.localDatabase.getDBEntry(path, { rev: rev }, false, false, true); const doc = await this.localDatabase.getDBEntry(path, { rev: rev }, false, false, true);
if (doc === false) return false; if (doc === false) return false;
@@ -1705,7 +1759,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin
return false; return false;
} }
//TODO: TIDY UP //TODO: TIDY UP
async mergeSensibly(path: string, baseRev: string, currentRev: string, conflictedRev: string): Promise<Diff[] | false> { async mergeSensibly(path: FilePathWithPrefix, baseRev: string, currentRev: string, conflictedRev: string): Promise<Diff[] | false> {
const baseLeaf = await this.getConflictedDoc(path, baseRev); const baseLeaf = await this.getConflictedDoc(path, baseRev);
const leftLeaf = await this.getConflictedDoc(path, currentRev); const leftLeaf = await this.getConflictedDoc(path, currentRev);
const rightLeaf = await this.getConflictedDoc(path, conflictedRev); const rightLeaf = await this.getConflictedDoc(path, conflictedRev);
@@ -1862,7 +1916,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin
} }
} }
async mergeObject(path: string, baseRev: string, currentRev: string, conflictedRev: string): Promise<string | false> { async mergeObject(path: FilePathWithPrefix, baseRev: string, currentRev: string, conflictedRev: string): Promise<string | false> {
try { try {
const baseLeaf = await this.getConflictedDoc(path, baseRev); const baseLeaf = await this.getConflictedDoc(path, baseRev);
const leftLeaf = await this.getConflictedDoc(path, currentRev); const leftLeaf = await this.getConflictedDoc(path, currentRev);
@@ -1917,7 +1971,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin
* @param path the file location * @param path the file location
* @returns true -> resolved, false -> nothing to do, or check result. * @returns true -> resolved, false -> nothing to do, or check result.
*/ */
async getConflictedStatus(path: string): Promise<diff_check_result> { async getConflictedStatus(path: FilePathWithPrefix): Promise<diff_check_result> {
const test = await this.localDatabase.getDBEntry(path, { conflicts: true, revs_info: true }, false, false, true); const test = await this.localDatabase.getDBEntry(path, { conflicts: true, revs_info: true }, false, false, true);
if (test === false) return false; if (test === false) return false;
if (test == null) return false; if (test == null) return false;
@@ -1928,7 +1982,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin
const conflictedRev = conflicts[0]; const conflictedRev = conflicts[0];
const conflictedRevNo = Number(conflictedRev.split("-")[0]); const conflictedRevNo = Number(conflictedRev.split("-")[0]);
//Search //Search
const revFrom = (await this.localDatabase.localDatabase.get<EntryDoc>(path2id(path), { revs_info: true })); const revFrom = (await this.localDatabase.getRaw<EntryDoc>(await this.path2id(path), { revs_info: true }));
const commonBase = revFrom._revs_info.filter(e => e.status == "available" && Number(e.rev.split("-")[0]) < conflictedRevNo).first()?.rev ?? ""; const commonBase = revFrom._revs_info.filter(e => e.status == "available" && Number(e.rev.split("-")[0]) < conflictedRevNo).first()?.rev ?? "";
let p = undefined; let p = undefined;
if (commonBase) { if (commonBase) {
@@ -1956,7 +2010,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin
// remove conflicted revision. // remove conflicted revision.
await this.localDatabase.deleteDBEntry(path, { rev: conflictedRev }); await this.localDatabase.deleteDBEntry(path, { rev: conflictedRev });
const file = getAbstractFileByPath(path) as TFile; const file = getAbstractFileByPath(stripAllPrefixes(path)) as TFile;
if (file) { if (file) {
await this.app.vault.modify(file, p); await this.app.vault.modify(file, p);
await this.updateIntoDB(file); await this.updateIntoDB(file);
@@ -2020,90 +2074,86 @@ export default class ObsidianLiveSyncPlugin extends Plugin
}; };
} }
showMergeDialog(filename: string, conflictCheckResult: diff_result): Promise<boolean> { showMergeDialog(filename: FilePathWithPrefix, conflictCheckResult: diff_result): Promise<boolean> {
return new Promise((res, rej) => { return runWithLock("resolve-conflict:" + filename, false, () =>
Logger("open conflict dialog", LOG_LEVEL.VERBOSE); new Promise((res, rej) => {
new ConflictResolveModal(this.app, filename, conflictCheckResult, async (selected) => { Logger("open conflict dialog", LOG_LEVEL.VERBOSE);
const testDoc = await this.localDatabase.getDBEntry(filename, { conflicts: true }, false, false, true); new ConflictResolveModal(this.app, filename, conflictCheckResult, async (selected) => {
if (testDoc === false) { const testDoc = await this.localDatabase.getDBEntry(filename, { conflicts: true }, false, false, true);
Logger("Missing file..", LOG_LEVEL.VERBOSE); if (testDoc === false) {
return res(true); Logger("Missing file..", LOG_LEVEL.VERBOSE);
} return res(true);
if (!testDoc._conflicts) { }
Logger("Nothing have to do with this conflict", LOG_LEVEL.VERBOSE); if (!testDoc._conflicts) {
return res(true); Logger("Nothing have to do with this conflict", LOG_LEVEL.VERBOSE);
} return res(true);
const toDelete = selected; }
const toKeep = conflictCheckResult.left.rev != toDelete ? conflictCheckResult.left.rev : conflictCheckResult.right.rev; const toDelete = selected;
if (toDelete == "") { const toKeep = conflictCheckResult.left.rev != toDelete ? conflictCheckResult.left.rev : conflictCheckResult.right.rev;
// concat both, if (toDelete == "") {
// delete conflicted revision and write a new file, store it again. // concat both,
const p = conflictCheckResult.diff.map((e) => e[1]).join(""); // delete conflicted revision and write a new file, store it again.
await this.localDatabase.deleteDBEntry(filename, { rev: testDoc._conflicts[0] }); const p = conflictCheckResult.diff.map((e) => e[1]).join("");
const file = getAbstractFileByPath(filename) as TFile; await this.localDatabase.deleteDBEntry(filename, { rev: testDoc._conflicts[0] });
if (file) { const file = getAbstractFileByPath(stripAllPrefixes(filename)) as TFile;
await this.app.vault.modify(file, p); if (file) {
await this.updateIntoDB(file); await this.app.vault.modify(file, p);
await this.updateIntoDB(file);
} else {
const newFile = await this.app.vault.create(filename, p);
await this.updateIntoDB(newFile);
}
await this.pullFile(filename);
Logger("concat both file");
if (this.settings.syncAfterMerge && !this.suspended) {
await this.replicate();
}
setTimeout(() => {
//resolved, check again.
this.showIfConflicted(filename);
}, 500);
} else if (toDelete == null) {
Logger("Leave it still conflicted");
} else { } else {
const newFile = await this.app.vault.create(filename, p); await this.localDatabase.deleteDBEntry(filename, { rev: toDelete });
await this.updateIntoDB(newFile); await this.pullFile(filename, null, true, toKeep);
Logger(`Conflict resolved:${filename}`);
if (this.settings.syncAfterMerge && !this.suspended) {
await this.replicate();
}
setTimeout(() => {
//resolved, check again.
this.showIfConflicted(filename);
}, 500);
} }
await this.pullFile(filename);
Logger("concat both file");
if (this.settings.syncAfterMerge && !this.suspended) {
await this.replicate();
}
setTimeout(() => {
//resolved, check again.
this.showIfConflicted(filename);
}, 500);
} else if (toDelete == null) {
Logger("Leave it still conflicted");
} else {
await this.localDatabase.deleteDBEntry(filename, { rev: toDelete });
await this.pullFile(filename, null, true, toKeep);
Logger(`Conflict resolved:${filename}`);
if (this.settings.syncAfterMerge && !this.suspended) {
await this.replicate();
}
setTimeout(() => {
//resolved, check again.
this.showIfConflicted(filename);
}, 500);
}
return res(true); return res(true);
}).open(); }).open();
}); })
);
} }
conflictedCheckFiles: string[] = []; conflictedCheckFiles: FilePath[] = [];
// queueing the conflicted file check // queueing the conflicted file check
conflictedCheckTimer: number;
queueConflictedCheck(file: TFile) { queueConflictedCheck(file: TFile) {
this.conflictedCheckFiles = this.conflictedCheckFiles.filter((e) => e != file.path); this.conflictedCheckFiles = this.conflictedCheckFiles.filter((e) => e != file.path);
this.conflictedCheckFiles.push(file.path); this.conflictedCheckFiles.push(getPathFromTFile(file));
if (this.conflictedCheckTimer != null) { scheduleTask("check-conflict", 100, async () => {
window.clearTimeout(this.conflictedCheckTimer); const checkFiles = JSON.parse(JSON.stringify(this.conflictedCheckFiles)) as FilePath[];
}
this.conflictedCheckTimer = window.setTimeout(async () => {
this.conflictedCheckTimer = null;
const checkFiles = JSON.parse(JSON.stringify(this.conflictedCheckFiles)) as string[];
for (const filename of checkFiles) { for (const filename of checkFiles) {
try { try {
const file = getAbstractFileByPath(filename); const file = getAbstractFileByPath(filename);
if (file != null && file instanceof TFile) { if (file != null && file instanceof TFile) {
await this.showIfConflicted(file.path); await this.showIfConflicted(getPathFromTFile(file));
} }
} catch (ex) { } catch (ex) {
Logger(ex); Logger(ex);
} }
} }
}, 100); });
} }
async showIfConflicted(filename: string) { async showIfConflicted(filename: FilePathWithPrefix) {
await runWithLock("conflicted", false, async () => { await runWithLock("conflicted", false, async () => {
const conflictCheckResult = await this.getConflictedStatus(filename); const conflictCheckResult = await this.getConflictedStatus(filename);
if (conflictCheckResult === false) { if (conflictCheckResult === false) {
@@ -2126,9 +2176,9 @@ export default class ObsidianLiveSyncPlugin extends Plugin
}); });
} }
async pullFile(filename: string, fileList?: TFile[], force?: boolean, rev?: string, waitForReady = true) { async pullFile(filename: FilePathWithPrefix, fileList?: TFile[], force?: boolean, rev?: string, waitForReady = true) {
const targetFile = getAbstractFileByPath(id2path(filename)); const targetFile = getAbstractFileByPath(stripAllPrefixes(filename));
if (!this.isTargetFile(id2path(filename))) return; if (!this.isTargetFile(filename)) return;
if (targetFile == null) { if (targetFile == null) {
//have to create; //have to create;
const doc = await this.localDatabase.getDBEntry(filename, rev ? { rev: rev } : null, false, waitForReady); const doc = await this.localDatabase.getDBEntry(filename, rev ? { rev: rev } : null, false, waitForReady);
@@ -2171,7 +2221,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin
const dK = `${file.path}-diff`; const dK = `${file.path}-diff`;
const isLastDiff = dK in caches ? caches[dK] : { storageMtime: 0, docMtime: 0 }; const isLastDiff = dK in caches ? caches[dK] : { storageMtime: 0, docMtime: 0 };
if (isLastDiff.docMtime == docMtime && isLastDiff.storageMtime == storageMtime) { if (isLastDiff.docMtime == docMtime && isLastDiff.storageMtime == storageMtime) {
Logger("STORAGE .. DB :" + file.path, LOG_LEVEL.VERBOSE); // Logger("STORAGE .. DB :" + file.path, LOG_LEVEL.VERBOSE);
caches[dK] = { storageMtime, docMtime }; caches[dK] = { storageMtime, docMtime };
return caches; return caches;
} }
@@ -2186,7 +2236,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin
//newer database file. //newer database file.
Logger("STORAGE <- DB :" + file.path); Logger("STORAGE <- DB :" + file.path);
Logger(`${storageMtime} < ${docMtime}`); Logger(`${storageMtime} < ${docMtime}`);
const docx = await this.localDatabase.getDBEntry(file.path, null, false, false); const docx = await this.localDatabase.getDBEntry(getPathFromTFile(file), null, false, false);
if (docx != false) { if (docx != false) {
await this.doc2storage(docx, file); await this.doc2storage(docx, file);
} else { } else {
@@ -2241,9 +2291,11 @@ export default class ObsidianLiveSyncPlugin extends Plugin
datatype = "plain"; datatype = "plain";
} }
} }
const fullPath = path2id(file.path); const fullPath = getPathFromTFile(file);
const id = await this.path2id(fullPath);
const d: LoadedEntry = { const d: LoadedEntry = {
_id: fullPath, _id: id,
path: getPathFromTFile(file),
data: content, data: content,
ctime: file.stat.ctime, ctime: file.stat.ctime,
mtime: file.stat.mtime, mtime: file.stat.mtime,
@@ -2292,7 +2344,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin
async deleteFromDB(file: TFile) { async deleteFromDB(file: TFile) {
if (!this.isTargetFile(file)) return; if (!this.isTargetFile(file)) return;
const fullPath = file.path; const fullPath = getPathFromTFile(file);
Logger(`deleteDB By path:${fullPath}`); Logger(`deleteDB By path:${fullPath}`);
await this.deleteFromDBbyPath(fullPath); await this.deleteFromDBbyPath(fullPath);
if (this.settings.syncOnSave && !this.suspended) { if (this.settings.syncOnSave && !this.suspended) {
@@ -2300,7 +2352,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin
} }
} }
async deleteFromDBbyPath(fullPath: string) { async deleteFromDBbyPath(fullPath: FilePath) {
await this.localDatabase.deleteDBEntry(fullPath); await this.localDatabase.deleteDBEntry(fullPath);
if (this.settings.syncOnSave && !this.suspended) { if (this.settings.syncOnSave && !this.suspended) {
await this.replicate(); await this.replicate();
@@ -2352,24 +2404,25 @@ export default class ObsidianLiveSyncPlugin extends Plugin
await this.app.vault.adapter.append(file.path, "", { ctime: file.ctime, mtime: file.mtime }); await this.app.vault.adapter.append(file.path, "", { ctime: file.ctime, mtime: file.mtime });
} }
async resolveConflictByNewerEntry(id: string) { async resolveConflictByNewerEntry(path: FilePathWithPrefix) {
const doc = await this.localDatabase.localDatabase.get(id, { conflicts: true }); const id = await this.path2id(path);
const doc = await this.localDatabase.getRaw<AnyEntry>(id, { conflicts: true });
// If there is no conflict, return with false. // If there is no conflict, return with false.
if (!("_conflicts" in doc)) return false; if (!("_conflicts" in doc)) return false;
if (doc._conflicts.length == 0) return false; if (doc._conflicts.length == 0) return false;
Logger(`Hidden file conflicted:${id2filenameInternalMetadata(id)}`); Logger(`Hidden file conflicted:${this.getPath(doc)}`);
const conflicts = doc._conflicts.sort((a, b) => Number(a.split("-")[0]) - Number(b.split("-")[0])); const conflicts = doc._conflicts.sort((a, b) => Number(a.split("-")[0]) - Number(b.split("-")[0]));
const revA = doc._rev; const revA = doc._rev;
const revB = conflicts[0]; const revB = conflicts[0];
const revBDoc = await this.localDatabase.localDatabase.get(id, { rev: revB }); const revBDoc = await this.localDatabase.getRaw<EntryDoc>(id, { rev: revB });
// determine which revision should been deleted. // determine which revision should been deleted.
// simply check modified time // simply check modified time
const mtimeA = ("mtime" in doc && doc.mtime) || 0; const mtimeA = ("mtime" in doc && doc.mtime) || 0;
const mtimeB = ("mtime" in revBDoc && revBDoc.mtime) || 0; const mtimeB = ("mtime" in revBDoc && revBDoc.mtime) || 0;
const delRev = mtimeA < mtimeB ? revA : revB; const delRev = mtimeA < mtimeB ? revA : revB;
// delete older one. // delete older one.
await this.localDatabase.localDatabase.remove(id, delRev); await this.localDatabase.removeRaw(id, delRev);
Logger(`Older one has been deleted:${id2filenameInternalMetadata(id)}`); Logger(`Older one has been deleted:${this.getPath(doc)}`);
return true; return true;
} }

17
src/types.ts
View File

@@ -1,5 +1,5 @@
import { PluginManifest, TFile } from "./deps"; import { PluginManifest, TFile } from "./deps";
import { DatabaseEntry, EntryBody } from "./lib/src/types"; import { DatabaseEntry, EntryBody, FilePath } from "./lib/src/types";
export interface PluginDataEntry extends DatabaseEntry { export interface PluginDataEntry extends DatabaseEntry {
deviceVaultName: string; deviceVaultName: string;
@@ -24,7 +24,7 @@ export interface DevicePluginList {
export const PERIODIC_PLUGIN_SWEEP = 60; export const PERIODIC_PLUGIN_SWEEP = 60;
export interface InternalFileInfo { export interface InternalFileInfo {
path: string; path: FilePath;
mtime: number; mtime: number;
ctime: number; ctime: number;
size: number; size: number;
@@ -32,7 +32,7 @@ export interface InternalFileInfo {
} }
export interface FileInfo { export interface FileInfo {
path: string; path: FilePath;
mtime: number; mtime: number;
ctime: number; ctime: number;
size: number; size: number;
@@ -62,12 +62,21 @@ export type FileEventItem = {
key: string, key: string,
} }
// Hidden items (Now means `chunk`)
export const CHeader = "h:"; export const CHeader = "h:";
// Plug-in Stored Container (Obsolete)
export const PSCHeader = "ps:"; export const PSCHeader = "ps:";
export const PSCHeaderEnd = "ps;"; export const PSCHeaderEnd = "ps;";
// Internal data Container
export const ICHeader = "i:"; export const ICHeader = "i:";
export const ICHeaderEnd = "i;"; export const ICHeaderEnd = "i;";
export const ICHeaderLength = ICHeader.length; export const ICHeaderLength = ICHeader.length;
// Internal data Container (eXtended)
export const ICXHeader = "ix:";
export const FileWatchEventQueueMax = 10; export const FileWatchEventQueueMax = 10;
export const configURIBase = "obsidian://setuplivesync?settings="; export const configURIBase = "obsidian://setuplivesync?settings=";

392
src/utils.ts
View File

@@ -1,24 +1,53 @@
import { DataWriteOptions, normalizePath, TFile, Platform, TAbstractFile, App, Plugin_2 } from "./deps"; import { DataWriteOptions, normalizePath, TFile, Platform, TAbstractFile, App, Plugin_2, RequestUrlParam, requestUrl } from "./deps";
import { path2id_base, id2path_base, isValidFilenameInLinux, isValidFilenameInDarwin, isValidFilenameInWidows, isValidFilenameInAndroid } from "./lib/src/path"; import { path2id_base, id2path_base, isValidFilenameInLinux, isValidFilenameInDarwin, isValidFilenameInWidows, isValidFilenameInAndroid, stripAllPrefixes } from "./lib/src/path";
import { Logger } from "./lib/src/logger"; import { Logger } from "./lib/src/logger";
import { LOG_LEVEL } from "./lib/src/types"; import { AnyEntry, DocumentID, EntryDoc, EntryHasPath, FilePath, FilePathWithPrefix, LOG_LEVEL, NewEntry } from "./lib/src/types";
import { CHeader, ICHeader, ICHeaderLength, PSCHeader } from "./types"; import { CHeader, ICHeader, ICHeaderLength, PSCHeader } from "./types";
import { InputStringDialog, PopoverSelectString } from "./dialogs"; import { InputStringDialog, PopoverSelectString } from "./dialogs";
import ObsidianLiveSyncPlugin from "./main";
import { runWithLock } from "./lib/src/lock";
// For backward compatibility, using the path for determining id. // For backward compatibility, using the path for determining id.
// Only CouchDB unacceptable ID (that starts with an underscore) has been prefixed with "/". // Only CouchDB unacceptable ID (that starts with an underscore) has been prefixed with "/".
// The first slash will be deleted when the path is normalized. // The first slash will be deleted when the path is normalized.
export function path2id(filename: string): string { export async function path2id(filename: FilePathWithPrefix | FilePath, obfuscatePassphrase: string | false): Promise<DocumentID> {
const x = normalizePath(filename); const temp = filename.split(":");
return path2id_base(x); const path = temp.pop();
const normalizedPath = normalizePath(path as FilePath);
temp.push(normalizedPath);
const fixedPath = temp.join(":") as FilePathWithPrefix;
const out = await path2id_base(fixedPath, obfuscatePassphrase);
return out;
} }
export function id2path(filename: string): string { export function id2path(id: DocumentID, entry?: EntryHasPath): FilePathWithPrefix {
return id2path_base(normalizePath(filename)); const filename = id2path_base(id, entry);
const temp = filename.split(":");
const path = temp.pop();
const normalizedPath = normalizePath(path as FilePath);
temp.push(normalizedPath);
const fixedPath = temp.join(":") as FilePathWithPrefix;
return fixedPath;
}
export function getPath(entry: AnyEntry) {
return id2path(entry._id, entry);
}
export function getPathWithoutPrefix(entry: AnyEntry) {
const f = getPath(entry);
return stripAllPrefixes(f);
}
export function getPathFromTFile(file: TAbstractFile) {
return file.path as FilePath;
} }
const tasks: { [key: string]: ReturnType<typeof setTimeout> } = {}; const tasks: { [key: string]: ReturnType<typeof setTimeout> } = {};
export function scheduleTask(key: string, timeout: number, proc: (() => Promise<any> | void)) { export function scheduleTask(key: string, timeout: number, proc: (() => Promise<any> | void), skipIfTaskExist?: boolean) {
if (skipIfTaskExist && key in tasks) {
return;
}
cancelTask(key); cancelTask(key);
tasks[key] = setTimeout(async () => { tasks[key] = setTimeout(async () => {
delete tasks[key]; delete tasks[key];
@@ -208,8 +237,8 @@ export function applyPatch(from: Record<string | number | symbol, any>, patch: R
} }
export function mergeObject( export function mergeObject(
objA: Record<string | number | symbol, any>, objA: Record<string | number | symbol, any> | [any],
objB: Record<string | number | symbol, any> objB: Record<string | number | symbol, any> | [any]
) { ) {
const newEntries = Object.entries(objB); const newEntries = Object.entries(objB);
const ret: any = { ...objA }; const ret: any = { ...objA };
@@ -252,6 +281,11 @@ export function mergeObject(
ret[key] = v; ret[key] = v;
} }
} }
if (Array.isArray(objA) && Array.isArray(objB)) {
return Object.values(Object.entries(ret)
.sort()
.reduce((p, [key, value]) => ({ ...p, [key]: value }), {}));
}
return Object.entries(ret) return Object.entries(ret)
.sort() .sort()
.reduce((p, [key, value]) => ({ ...p, [key]: value }), {}); .reduce((p, [key, value]) => ({ ...p, [key]: value }), {});
@@ -300,21 +334,23 @@ export function isValidPath(filename: string) {
let touchedFiles: string[] = []; let touchedFiles: string[] = [];
export function getAbstractFileByPath(path: string): TAbstractFile | null { export function getAbstractFileByPath(path: FilePath): TAbstractFile | null {
// Hidden API but so useful. // Disabled temporary.
// @ts-ignore return app.vault.getAbstractFileByPath(path);
if ("getAbstractFileByPathInsensitive" in app.vault && (app.vault.adapter?.insensitive ?? false)) { // // Hidden API but so useful.
// @ts-ignore // // @ts-ignore
return app.vault.getAbstractFileByPathInsensitive(path); // if ("getAbstractFileByPathInsensitive" in app.vault && (app.vault.adapter?.insensitive ?? false)) {
} else { // // @ts-ignore
return app.vault.getAbstractFileByPath(path); // return app.vault.getAbstractFileByPathInsensitive(path);
} // } else {
// return app.vault.getAbstractFileByPath(path);
// }
} }
export function trimPrefix(target: string, prefix: string) { export function trimPrefix(target: string, prefix: string) {
return target.startsWith(prefix) ? target.substring(prefix.length) : target; return target.startsWith(prefix) ? target.substring(prefix.length) : target;
} }
export function touch(file: TFile | string) { export function touch(file: TFile | FilePath) {
const f = file instanceof TFile ? file : getAbstractFileByPath(file) as TFile; const f = file instanceof TFile ? file : getAbstractFileByPath(file) as TFile;
const key = `${f.path}-${f.stat.mtime}-${f.stat.size}`; const key = `${f.path}-${f.stat.mtime}-${f.stat.size}`;
touchedFiles.unshift(key); touchedFiles.unshift(key);
@@ -331,17 +367,17 @@ export function clearTouched() {
/** /**
* returns is internal chunk of file * returns is internal chunk of file
* @param str ID * @param id ID
* @returns * @returns
*/ */
export function isInternalMetadata(str: string): boolean { export function isInternalMetadata(id: FilePath | FilePathWithPrefix | DocumentID): boolean {
return str.startsWith(ICHeader); return id.startsWith(ICHeader);
} }
export function id2filenameInternalMetadata(str: string): string { export function stripInternalMetadataPrefix<T extends FilePath | FilePathWithPrefix | DocumentID>(id: T): T {
return str.substring(ICHeaderLength); return id.substring(ICHeaderLength) as T;
} }
export function filename2idInternalMetadata(str: string): string { export function id2InternalMetadataId(id: DocumentID): DocumentID {
return ICHeader + str; return ICHeader + id as DocumentID;
} }
// const CHeaderLength = CHeader.length; // const CHeaderLength = CHeader.length;
@@ -402,3 +438,303 @@ export class PeriodicProcessor {
if (this._timer) clearInterval(this._timer); if (this._timer) clearInterval(this._timer);
} }
} }
function sizeToHumanReadable(size: number | undefined) {
if (!size) return "-";
const i = Math.floor(Math.log(size) / Math.log(1024));
return Number.parseInt((size / Math.pow(1024, i)).toFixed(2)) + ' ' + ['B', 'kB', 'MB', 'GB', 'TB'][i];
}
export const _requestToCouchDBFetch = async (baseUri: string, username: string, password: string, path?: string, body?: string | any, method?: string) => {
const utf8str = String.fromCharCode.apply(null, new TextEncoder().encode(`${username}:${password}`));
const encoded = window.btoa(utf8str);
const authHeader = "Basic " + encoded;
const transformedHeaders: Record<string, string> = { authorization: authHeader, "content-type": "application/json" };
const uri = `${baseUri}/${path}`;
const requestParam = {
url: uri,
method: method || (body ? "PUT" : "GET"),
headers: new Headers(transformedHeaders),
contentType: "application/json",
body: JSON.stringify(body),
};
return await fetch(uri, requestParam);
}
export const _requestToCouchDB = async (baseUri: string, username: string, password: string, origin: string, path?: string, body?: any, method?: string) => {
const utf8str = String.fromCharCode.apply(null, new TextEncoder().encode(`${username}:${password}`));
const encoded = window.btoa(utf8str);
const authHeader = "Basic " + encoded;
const transformedHeaders: Record<string, string> = { authorization: authHeader, origin: origin };
const uri = `${baseUri}/${path}`;
const requestParam: RequestUrlParam = {
url: uri,
method: method || (body ? "PUT" : "GET"),
headers: transformedHeaders,
contentType: "application/json",
body: body ? JSON.stringify(body) : undefined,
};
return await requestUrl(requestParam);
}
export const requestToCouchDB = async (baseUri: string, username: string, password: string, origin: string, key?: string, body?: string, method?: string) => {
const uri = `_node/_local/_config${key ? "/" + key : ""}`;
return await _requestToCouchDB(baseUri, username, password, origin, uri, body, method);
};
export async function performRebuildDB(plugin: ObsidianLiveSyncPlugin, method: "localOnly" | "remoteOnly" | "rebuildBothByThisDevice") {
if (method == "localOnly") {
await plugin.addOnSetup.fetchLocal();
}
if (method == "remoteOnly") {
await plugin.addOnSetup.rebuildRemote();
}
if (method == "rebuildBothByThisDevice") {
await plugin.addOnSetup.rebuildEverything();
}
}
export const gatherChunkUsage = async (db: PouchDB.Database<EntryDoc>) => {
const used = new Map();
const unreferenced = new Map();
const removed = new Map();
const missing = new Map();
const xx = await db.allDocs({ startkey: "h:", endkey: `h:\u{10ffff}` });
for (const xxd of xx.rows) {
const chunk = xxd.id
unreferenced.set(chunk, xxd.value.rev);
}
const x = await db.find({ limit: 999999999, selector: { children: { $exists: true, $type: "array" } }, fields: ["_id", "path", "mtime", "children"] });
for (const temp of x.docs) {
for (const chunk of (temp as NewEntry).children) {
used.set(chunk, (used.has(chunk) ? used.get(chunk) : 0) + 1);
if (unreferenced.has(chunk)) {
removed.set(chunk, unreferenced.get(chunk));
unreferenced.delete(chunk);
} else {
if (!removed.has(chunk)) {
if (!missing.has(temp._id)) {
missing.set(temp._id, []);
}
missing.get(temp._id).push(chunk);
}
}
}
}
return { used, unreferenced, missing };
}
export const localDatabaseCleanUp = async (plugin: ObsidianLiveSyncPlugin, force: boolean, dryRun: boolean) => {
await runWithLock("clean-up:local", true, async () => {
const db = plugin.localDatabase.localDatabase;
if ((db as any)?.adapter != "indexeddb") {
if (force && !dryRun) {
Logger("Fetch from the remote database", LOG_LEVEL.NOTICE, "clean-up-db");
await performRebuildDB(plugin, "localOnly");
return;
} else {
Logger("This feature requires enabling `Use new adapter`. Please enable it", LOG_LEVEL.NOTICE, "clean-up-db");
return;
}
}
Logger(`The remote database has been locked for garbage collection`, LOG_LEVEL.NOTICE, "clean-up-db");
Logger(`Gathering chunk usage information`, LOG_LEVEL.NOTICE, "clean-up-db");
const { unreferenced, missing } = await gatherChunkUsage(db);
if (missing.size != 0) {
Logger(`Some chunks are not found! We have to rescue`, LOG_LEVEL.NOTICE);
Logger(missing, LOG_LEVEL.VERBOSE);
} else {
Logger(`All chunks are OK`, LOG_LEVEL.NOTICE);
}
const payload = {} as Record<string, string[]>;
for (const [id, rev] of unreferenced) {
payload[id] = [rev];
}
const removeItems = Object.keys(payload).length;
if (removeItems == 0) {
Logger(`No unreferenced chunks found (Local)`, LOG_LEVEL.NOTICE);
await plugin.markRemoteResolved();
}
if (dryRun) {
Logger(`There are ${removeItems} unreferenced chunks (Local)`, LOG_LEVEL.NOTICE);
return;
}
Logger(`Deleting unreferenced chunks: ${removeItems}`, LOG_LEVEL.NOTICE, "clean-up-db");
for (const [id, rev] of unreferenced) {
//@ts-ignore
const ret = await db.purge(id, rev);
Logger(ret, LOG_LEVEL.VERBOSE);
}
plugin.localDatabase.refreshSettings();
Logger(`Compacting local database...`, LOG_LEVEL.NOTICE, "clean-up-db");
await db.compact();
await plugin.markRemoteResolved();
Logger("Done!", LOG_LEVEL.NOTICE, "clean-up-db");
})
}
export const balanceChunks = async (plugin: ObsidianLiveSyncPlugin, dryRun: boolean) => {
await runWithLock("clean-up:balance", true, async () => {
const localDB = plugin.localDatabase.localDatabase;
Logger(`Gathering chunk usage information`, LOG_LEVEL.NOTICE, "clean-up-db");
const ret = await plugin.replicator.connectRemoteCouchDBWithSetting(plugin.settings, plugin.isMobile);
if (typeof ret === "string") {
Logger(`Connect error: ${ret}`, LOG_LEVEL.NOTICE, "clean-up-db");
return;
}
const localChunks = new Map<string, string>();
const xx = await localDB.allDocs({ startkey: "h:", endkey: `h:\u{10ffff}` });
for (const xxd of xx.rows) {
const chunk = xxd.id
localChunks.set(chunk, xxd.value.rev);
}
// const info = ret.info;
const remoteDB = ret.db;
const remoteChunks = new Map<string, string>();
const xxr = await remoteDB.allDocs({ startkey: "h:", endkey: `h:\u{10ffff}` });
for (const xxd of xxr.rows) {
const chunk = xxd.id
remoteChunks.set(chunk, xxd.value.rev);
}
const localToRemote = new Map<string, string>([...localChunks]);
const remoteToLocal = new Map<string, string>([...remoteChunks]);
for (const id of new Set([...localChunks.keys(), ...remoteChunks.keys()])) {
if (remoteChunks.has(id)) {
localToRemote.delete(id);
}
if (localChunks.has(id)) {
remoteToLocal.delete(id);
}
}
function arrayToChunkedArray<T>(src: T[], size = 25) {
const ret = [] as T[][];
let i = 0;
while (i < src.length) {
ret.push(src.slice(i, i += size));
}
return ret;
}
if (localToRemote.size == 0) {
Logger(`No chunks need to be sent`, LOG_LEVEL.NOTICE);
} else {
Logger(`${localToRemote.size} chunks need to be sent`, LOG_LEVEL.NOTICE);
if (!dryRun) {
const w = arrayToChunkedArray([...localToRemote]);
for (const chunk of w) {
for (const [id,] of chunk) {
const queryRet = await localDB.allDocs({ keys: [id], include_docs: true });
const docs = queryRet.rows.filter(e => !("error" in e)).map(x => x.doc);
const ret = await remoteDB.bulkDocs(docs, { new_edits: false });
Logger(ret, LOG_LEVEL.VERBOSE);
}
}
Logger(`Done! ${remoteToLocal.size} chunks have been sent`, LOG_LEVEL.NOTICE);
}
}
if (remoteToLocal.size == 0) {
Logger(`No chunks need to be retrieved`, LOG_LEVEL.NOTICE);
} else {
Logger(`${remoteToLocal.size} chunks need to be retrieved`, LOG_LEVEL.NOTICE);
if (!dryRun) {
const w = arrayToChunkedArray([...remoteToLocal]);
for (const chunk of w) {
for (const [id,] of chunk) {
const queryRet = await remoteDB.allDocs({ keys: [id], include_docs: true });
const docs = queryRet.rows.filter(e => !("error" in e)).map(x => x.doc);
const ret = await localDB.bulkDocs(docs, { new_edits: false });
Logger(ret, LOG_LEVEL.VERBOSE);
}
}
Logger(`Done! ${remoteToLocal.size} chunks have been retrieved`, LOG_LEVEL.NOTICE);
}
}
})
}
export const remoteDatabaseCleanup = async (plugin: ObsidianLiveSyncPlugin, dryRun: boolean) => {
const getSize = function (info: PouchDB.Core.DatabaseInfo, key: "active" | "external" | "file") {
return Number.parseInt((info as any)?.sizes?.[key] ?? 0);
}
await runWithLock("clean-up:remote", true, async () => {
const CHUNK_SIZE = 100;
function makeChunkedArrayFromArray<T>(items: T[]): T[][] {
const chunked = [];
for (let i = 0; i < items.length; i += CHUNK_SIZE) {
chunked.push(items.slice(i, i + CHUNK_SIZE));
}
return chunked;
}
try {
const ret = await plugin.replicator.connectRemoteCouchDBWithSetting(plugin.settings, plugin.isMobile);
if (typeof ret === "string") {
Logger(`Connect error: ${ret}`, LOG_LEVEL.NOTICE, "clean-up-db");
return;
}
const info = ret.info;
Logger(JSON.stringify(info), LOG_LEVEL.VERBOSE, "clean-up-db");
Logger(`Database active-size: ${sizeToHumanReadable(getSize(info, "active"))}, external-size:${sizeToHumanReadable(getSize(info, "external"))}, file-size: ${sizeToHumanReadable(getSize(info, "file"))}`, LOG_LEVEL.NOTICE);
if (!dryRun) {
Logger(`The remote database has been locked for garbage collection`, LOG_LEVEL.NOTICE, "clean-up-db");
await plugin.markRemoteLocked(true);
}
Logger(`Gathering chunk usage information`, LOG_LEVEL.NOTICE, "clean-up-db");
const db = ret.db;
const { unreferenced, missing } = await gatherChunkUsage(db);
if (missing.size != 0) {
Logger(`Some chunks are not found! We have to rescue`, LOG_LEVEL.NOTICE);
Logger(missing, LOG_LEVEL.VERBOSE);
} else {
Logger(`All chunks are OK`, LOG_LEVEL.NOTICE);
}
const payload = {} as Record<string, string[]>;
for (const [id, rev] of unreferenced) {
payload[id] = [rev];
}
const removeItems = Object.keys(payload).length;
if (removeItems == 0) {
Logger(`No unreferenced chunk found (Remote)`, LOG_LEVEL.NOTICE);
return;
}
if (dryRun) {
Logger(`There are ${removeItems} unreferenced chunks (Remote)`, LOG_LEVEL.NOTICE);
return;
}
Logger(`Deleting unreferenced chunks: ${removeItems}`, LOG_LEVEL.NOTICE, "clean-up-db");
const buffer = makeChunkedArrayFromArray(Object.entries(payload));
for (const chunkedPayload of buffer) {
const rets = await _requestToCouchDBFetch(
`${plugin.settings.couchDB_URI}/${plugin.settings.couchDB_DBNAME}`,
plugin.settings.couchDB_USER,
plugin.settings.couchDB_PASSWORD,
"_purge",
chunkedPayload.reduce((p, c) => ({ ...p, [c[0]]: c[1] }), {}), "POST");
// const result = await rets();
Logger(JSON.stringify(await rets.json()), LOG_LEVEL.VERBOSE);
}
Logger(`Compacting database...`, LOG_LEVEL.NOTICE, "clean-up-db");
await db.compact();
const endInfo = await db.info();
Logger(`Processed database active-size: ${sizeToHumanReadable(getSize(endInfo, "active"))}, external-size:${sizeToHumanReadable(getSize(endInfo, "external"))}, file-size: ${sizeToHumanReadable(getSize(endInfo, "file"))}`, LOG_LEVEL.NOTICE);
Logger(`Reduced sizes: active-size: ${sizeToHumanReadable(getSize(info, "active") - getSize(endInfo, "active"))}, external-size:${sizeToHumanReadable(getSize(info, "external") - getSize(endInfo, "external"))}, file-size: ${sizeToHumanReadable(getSize(info, "file") - getSize(endInfo, "file"))}`, LOG_LEVEL.NOTICE);
Logger(JSON.stringify(endInfo), LOG_LEVEL.VERBOSE, "clean-up-db");
Logger(`Local database cleaning up...`);
await localDatabaseCleanUp(plugin, true, false);
} catch (ex) {
Logger("Failed to clean up db.")
Logger(ex, LOG_LEVEL.VERBOSE);
}
});
}

51
updates.md
View File

@@ -1,33 +1,28 @@
### 0.17.0 ### 0.19.0
- 0.17.0 has no surfaced changes but the design of saving chunks has been changed. They have compatibility but changing files after upgrading makes different chunks than before 0.16.x.
Please rebuild databases once if you have been worried about storage usage.
- Improved: #### Customization sync
- Splitting markdown
- Saving chunks Since `Plugin and their settings` have been broken, so I tried to fix it, not just fix it, but fix it the way it should be.
Now, we have `Customization sync`.
It is a real shame that the compatibility between these features has been broken. However, this new feature is surely useful and I believe that worth getting over the pain.
We can use the new feature with the same configuration. Only the menu on the command palette has been changed. The dialog can be opened by `Show customization sync dialog`.
I hope you will give it a try.
- Changed:
- Chunk ID numbering rules
#### Minors #### Minors
- __0.17.1 to 0.17.30 has been moved into `update_old.md`__
- 0.17.31 - 0.19.1
- Fixed: Fixed hidden file handling on Linux
- Improved: Now customization sync works more smoothly.
- 0.19.2
- Fixed: - Fixed:
- Now `redflag3` can be run surely. - Fixed garbage collection error while unreferenced chunks exist many.
- Synchronisation can now be aborted. - Fixed filename validation on Linux.
- Note: The synchronisation flow has been rewritten drastically. Please do not haste to inform me if you have noticed anything. - Improved:
- 0.17.32 - Showing status is now thinned for performance.
- Fixed: - Enhance caching while collecting chunks.
- Now periodic internal file scanning works well.
- The handler of Window-visibility-changed has been fixed. ... To continue on to `updates_old.md`.
- And minor fixes possibly included.
- Refactored:
- Unused logic has been removed.
- Some utility functions have been moved into suitable files.
- Function names have been renamed.
- 0.17.33
- Maintenance update: Refactored; the responsibilities that `LocalDatabase` had were shared. (Hoping) No changes in behaviour.
- 0.17.34
- Fixed: The `Fetch` that was broken at 0.17.33 has been fixed.
- Refactored again: Internal file sync, plug-in sync and Set up URI have been moved into each file.
... To continue on to `updates_old.md`.

63
updates_old.md
View File

@@ -1,3 +1,46 @@
### 0.18.0
#### Now, paths of files in the database can now be obfuscated. (Experimental Feature)
At before v0.18.0, Self-hosted LiveSync used the path of files, to detect and resolve conflicts. In naive. The ID of the document stored in the CouchDB was naturally the filename.
However, it means a sort of lacking confidentiality. If the credentials of the database have been leaked, the attacker (or an innocent bystander) can read the path of files. So we could not use confidential things in the filename in some environments.
Since v0.18.0, they can be obfuscated. so it is no longer possible to decipher the path from the ID. Instead of that, it costs a bit CPU load than before, and the data structure has been changed a bit.
We can configure the `Path Obfuscation` in the `Remote database configuration` pane.
Note: **When changing this configuration, we need to rebuild both of the local and the remote databases**.
#### Minors
- 0.18.1
- Fixed:
- Some messages are fixed (Typo)
- File type detection now works fine!
- 0.18.2
- Improved:
- The setting pane has been refined.
- We can enable `hidden files sync` with several initial behaviours; `Merge`, `Fetch` remote, and `Overwrite` remote.
- No longer `Touch hidden files`.
- 0.18.3
- Fixed Pop-up is now correctly shown after hidden file synchronisation.
- 0.18.4
- Fixed:
- `Fetch` and `Rebuild database` will work more safely.
- Case-sensitive renaming now works fine.
Revoked the logic which was made at #130, however, looks fine now.
- 0.18.5
- Improved:
- Actions for maintaining databases moved to the `🎛️Maintain databases`.
- Clean-up of unreferenced chunks has been implemented on an **experimental**.
- This feature requires enabling `Use new adapter`.
- Be sure to fully all devices synchronised before perform it.
- After cleaning up the remote, all devices will be locked out. If we are sure had it be synchronised, we can perform only cleaning-up locally. If not, we have to perform `Fetch`.
- 0.18.6
- New features:
- Now remote database cleaning-up will be detected automatically.
- A solution selection dialogue will be shown if synchronisation is rejected after cleaning or rebuilding the remote database.
- During fetching or rebuilding, we can configure `Hidden file synchronisation` on the spot.
- It let us free from conflict resolution on initial synchronising.
### 0.17.0 ### 0.17.0
- 0.17.0 has no surfaced changes but the design of saving chunks has been changed. They have compatibility but changing files after upgrading makes different chunks than before 0.16.x. - 0.17.0 has no surfaced changes but the design of saving chunks has been changed. They have compatibility but changing files after upgrading makes different chunks than before 0.16.x.
Please rebuild databases once if you have been worried about storage usage. Please rebuild databases once if you have been worried about storage usage.
@@ -155,6 +198,26 @@
- Fixed a problem about reading chunks online when a file has more chunks than the concurrency limit. - Fixed a problem about reading chunks online when a file has more chunks than the concurrency limit.
- Rollbacked: - Rollbacked:
- Logs are kept only for 100 lines, again. - Logs are kept only for 100 lines, again.
- 0.17.31
- Fixed:
- Now `redflag3` can be run surely.
- Synchronisation can now be aborted.
- Note: The synchronisation flow has been rewritten drastically. Please do not haste to inform me if you have noticed anything.
- 0.17.32
- Fixed:
- Now periodic internal file scanning works well.
- The handler of Window-visibility-changed has been fixed.
- And minor fixes possibly included.
- Refactored:
- Unused logic has been removed.
- Some utility functions have been moved into suitable files.
- Function names have been renamed.
- 0.17.33
- Maintenance update: Refactored; the responsibilities that `LocalDatabase` had were shared. (Hoping) No changes in behaviour.
- 0.17.34
- Fixed: The `Fetch` that was broken at 0.17.33 has been fixed.
- Refactored again: Internal file sync, plug-in sync and Set up URI have been moved into each file.
### 0.16.0 ### 0.16.0
- Now hidden files need not be scanned. Changes will be detected automatically. - Now hidden files need not be scanned. Changes will be detected automatically.
- If you want it to back to its previous behaviour, please disable `Monitor changes to internal files`. - If you want it to back to its previous behaviour, please disable `Monitor changes to internal files`.