public/obsidian-livesync
1
0
Fork 0
mirror of https://github.com/vrtmrz/obsidian-livesync.git synced 2026-02-22 20:18:48 +00:00
Code Issues Packages Projects Releases Wiki Activity

Compare commits

base: public:0.23.3
Branches Tags
public:main public:beta public:snyk-upgrade-7fd31e6f474568014502e7205847f257 public:snyk-fix-90f3b71450f7543a1c61724d8afa0e71 public:refactor_services_2 public:refactor_services public:dev public:snyk-upgrade-fe9625385aa4be849c7157033fd2abb0 public:snyk-upgrade-99a31e70c822894595544b0c9813b11a public:svelteui public:disenchant public:snyk-fix-61f76ef0bce2a276df9552c2fc80a14f public:snyk-upgrade-2baa7938ba318464c3bad5c3fd5cbda5 public:snyk-upgrade-7af62a575eab20e8fb4e9565bf1adcfc public:snyk-upgrade-b32f281b31a81483f126b8fb6c54a957 public:snyk-upgrade-bd61b1103442d3faecc9fc726c34e9a4 public:snyk-upgrade-5b2d6584430c061b5187611a5d7fb64a public:snyk-fix-0e9fbbcb374fcb8d5e8cb027d8414fb7 public:snyk-upgrade-5242daf91e706d1ae500c9ea6545636e public:snyk-upgrade-70434a6822fbad83424135dfad9df909 public:snyk-upgrade-8fa01ccad0352125f1866d41e57d35e1 public:snyk-upgrade-39eb97b4655e70091f718a88fccd57c1 public:snyk-upgrade-2b99d54d19e207887e36c3071b796925 public:forprivacy public:snyk-upgrade-dad9088d0a333677e0157e3deef7fc38 public:snyk-upgrade-b9a76d74b3e5723759af5b3319aabb80 public:snyk-upgrade-7da6459d8ff5d293d80de9d80061d912 public:snyk-upgrade-6a3786fc4cbe394542e9d1b6058c46e7 public:snyk-upgrade-8fc9e62771dc41b0f23dc0efefb5c949
public:0.25.43-patched-9 public:0.25.43-patched-8 public:0.25.43-patched-7 public:0.25.43-patched-6 public:0.25.43-patched-5 public:0.25.43-patched-4 public:0.25.43-patched-3 public:0.25.43-patched-2 public:0.25.43-patched-1 public:0.25.43 public:0.25.42 public:0.25.41-patched-3 public:0.25.41-patched-2 public:0.25.41-patched-1 public:0.25.41 public:0.25.40 public:0.25.39 public:0.25.38 public:0.25.37 public:0.25.36 public:0.25.35 public:0.25.34 public:0.25.33 public:0.25.32 public:0.25.31 public:0.25.30 public:0.25.30-dev public:0.25.29 public:0.25.28 public:0.25.27 public:0.25.26 public:0.25.25 public:0.25.24 public:0.25.24.beta3 public:0.25.24.beta2 public:0.25.24.beta1 public:0.25.23 public:0.25.23.beta1 public:0.25.22 public:0.25.21 public:0.25.21.beta2 public:0.25.21.beta1 public:0.25.20 public:0.25.19 public:0.25.18 public:0.25.17 public:0.25.16 public:0.25.15 public:0.25.14 public:0.25.13 public:0.25.12 public:0.25.11 public:0.25.10 public:0.25.9 public:0.25.8 public:0.25.7 public:0.25.6 public:0.25.5 public:0.25.4 public:0.25.3 public:0.25.2 public:0.25.1 public:0.25.0 public:0.25.0-beta1 public:0.24.31 public:0.24.30 public:0.24.29 public:0.24.28 public:0.24.27 public:0.24.26 public:0.24.25 public:0.24.24 public:0.24.23 public:0.24.22 public:0.24.21 public:0.24.20 public:0.24.19 public:0.24.18 public:0.24.17 public:0.24.16 public:0.24.15 public:0.24.14 public:0.24.13 public:0.24.12 public:0.24.11 public:0.24.10 public:0.24.9 public:0.24.8 public:0.24.7 public:0.24.6 public:0.24.5 public:0.24.4 public:0.24.3 public:0.24.2 public:0.24.1 public:0.24.0 public:0.24.0.dev-rc8 public:0.24.0.dev-rc7 public:0.24.0.dev-rc6 public:0.24.0.dev-rc5 public:0.24.0.dev-rc4 public:0.24.0.dev-rc3 public:0.24.0.dev-rc2 public:0.23.23 public:0.23.22 public:0.23.21 public:0.23.20 public:0.23.19 public:0.23.18 public:0.23.17 public:0.23.16 public:0.23.15 public:0.23.14 public:0.23.13 public:0.23.12 public:0.23.11 public:0.23.10 public:0.23.9 public:0.23.8 public:0.23.7 public:0.23.6 public:0.23.5 public:0.23.4 public:0.23.3 public:0.23.2 public:0.23.1 public:0.23.0 public:0.22.19 public:0.22.18 public:0.22.17 public:0.22.16 public:0.22.15 public:0.22.14 public:0.22.13 public:0.22.12 public:0.22.11 public:0.22.10 public:0.22.9 public:0.22.8 public:0.22.7 public:0.22.6 public:0.22.5 public:0.22.4 public:0.22.3 public:0.22.2 public:0.22.1 public:0.22.0 public:0.21.5 public:0.21.4 public:0.21.3 public:0.21.2 public:0.21.1 public:0.21.0 public:0.20.7 public:0.20.6 public:0.20.5 public:0.20.4 public:0.20.3 public:0.20.2 public:0.20.1 public:0.20.0 public:0.19.23 public:0.19.22 public:0.19.21 public:0.19.20 public:0.19.19 public:0.19.18 public:0.19.17 public:0.19.16 public:0.19.15 public:0.19.14 public:0.19.13 public:0.19.12 public:0.19.11 public:0.19.10 public:0.19.9 public:0.19.8 public:0.19.7 public:0.19.6 public:0.19.5 public:0.19.4 public:0.19.3 public:0.19.2 public:0.19.1 public:0.19.0 public:0.18.6 public:0.18.5 public:0.18.4 public:0.18.3 public:0.18.2 public:0.18.1 public:0.18.0 public:0.17.34 public:0.17.33 public:0.17.32 public:0.17.31 public:0.17.30 public:0.17.29 public:0.17.28 public:0.17.27 public:0.17.26 public:0.17.25 public:0.17.24 public:0.17.23 public:0.17.22 public:0.17.21 public:0.17.20 public:0.17.19 public:0.17.18 public:0.17.17 public:0.17.16 public:0.17.15 public:0.17.14 public:0.17.13 public:0.17.12 public:0.17.11 public:0.17.10 public:0.17.9 public:0.17.8 public:0.17.7 public:0.17.6 public:0.17.5 public:0.17.4 public:0.17.3 public:0.17.2 public:0.17.1 public:0.17.0 public:0.16.8 public:0.16.7 public:0.16.6 public:0.16.5 public:0.16.4 public:0.16.3 public:0.16.2 public:0.16.1 public:0.16.0 public:0.15.10 public:0.15.9 public:0.15.8 public:0.15.7 public:0.15.6 public:0.15.5 public:0.15.4 public:0.15.3 public:0.15.2 public:0.15.1 public:0.15.0 public:0.14.8 public:0.14.7 public:0.14.6 public:0.14.5 public:0.14.4 public:0.14.3 public:0.14.2 public:0.14.1 public:0.14.0 public:0.13.4 public:0.13.3 public:0.13.2 public:0.13.1 public:0.13.0 public:0.12.5 public:0.12.4 public:0.12.3 public:0.12.2 public:0.12.1 public:0.12.0 public:0.11.10 public:0.11.9 public:0.11.8 public:0.11.7 public:0.11.6 public:0.11.5 public:0.11.4 public:0.11.3 public:0.11.2 public:0.11.1 public:0.11.0 public:0.10.1 public:0.10.0 public:0.9.0 public:0.8.8 public:0.8.7 public:0.8.6 public:0.8.5 public:0.8.3 public:0.8.2 public:0.8.1 public:0.8.0 public:0.7.2 public:0.7.1 public:0.7.0 public:0.6.1 public:0.6.0 public:0.5.0 public:0.4.1 public:0.4.0 public:0.3.1 public:0.3.0 public:0.2.1 public:0.2.0 public:0.1.26 public:0.1.25 public:0.1.24 public:0.1.23 public:0.1.22 public:0.1.21 public:0.1.20 public:0.1.19 public:0.1.18 public:0.1.17 public:0.1.16 public:0.1.15 public:0.1.14 public:0.1.13 public:0.1.12 public:0.1.11 public:0.1.10 public:0.1.9 public:0.1.8 public:0.1.7 public:0.1.6 public:0.1.5 public:0.1.4 public:0.1.3 public:0.1.2 public:0.1.1 public:0.1.0 public:0.0.9 public:0.0.8 public:0.0.7 public:0.0.6
...
compare: public:0.23.9
Branches Tags
public:beta public:main public:snyk-upgrade-7fd31e6f474568014502e7205847f257 public:snyk-fix-90f3b71450f7543a1c61724d8afa0e71 public:refactor_services_2 public:refactor_services public:dev public:snyk-upgrade-fe9625385aa4be849c7157033fd2abb0 public:snyk-upgrade-99a31e70c822894595544b0c9813b11a public:svelteui public:disenchant public:snyk-fix-61f76ef0bce2a276df9552c2fc80a14f public:snyk-upgrade-2baa7938ba318464c3bad5c3fd5cbda5 public:snyk-upgrade-7af62a575eab20e8fb4e9565bf1adcfc public:snyk-upgrade-b32f281b31a81483f126b8fb6c54a957 public:snyk-upgrade-bd61b1103442d3faecc9fc726c34e9a4 public:snyk-upgrade-5b2d6584430c061b5187611a5d7fb64a public:snyk-fix-0e9fbbcb374fcb8d5e8cb027d8414fb7 public:snyk-upgrade-5242daf91e706d1ae500c9ea6545636e public:snyk-upgrade-70434a6822fbad83424135dfad9df909 public:snyk-upgrade-8fa01ccad0352125f1866d41e57d35e1 public:snyk-upgrade-39eb97b4655e70091f718a88fccd57c1 public:snyk-upgrade-2b99d54d19e207887e36c3071b796925 public:forprivacy public:snyk-upgrade-dad9088d0a333677e0157e3deef7fc38 public:snyk-upgrade-b9a76d74b3e5723759af5b3319aabb80 public:snyk-upgrade-7da6459d8ff5d293d80de9d80061d912 public:snyk-upgrade-6a3786fc4cbe394542e9d1b6058c46e7 public:snyk-upgrade-8fc9e62771dc41b0f23dc0efefb5c949
public:0.25.43-patched-9 public:0.25.43-patched-8 public:0.25.43-patched-7 public:0.25.43-patched-6 public:0.25.43-patched-5 public:0.25.43-patched-4 public:0.25.43-patched-3 public:0.25.43-patched-2 public:0.25.43-patched-1 public:0.25.43 public:0.25.42 public:0.25.41-patched-3 public:0.25.41-patched-2 public:0.25.41-patched-1 public:0.25.41 public:0.25.40 public:0.25.39 public:0.25.38 public:0.25.37 public:0.25.36 public:0.25.35 public:0.25.34 public:0.25.33 public:0.25.32 public:0.25.31 public:0.25.30 public:0.25.30-dev public:0.25.29 public:0.25.28 public:0.25.27 public:0.25.26 public:0.25.25 public:0.25.24 public:0.25.24.beta3 public:0.25.24.beta2 public:0.25.24.beta1 public:0.25.23 public:0.25.23.beta1 public:0.25.22 public:0.25.21 public:0.25.21.beta2 public:0.25.21.beta1 public:0.25.20 public:0.25.19 public:0.25.18 public:0.25.17 public:0.25.16 public:0.25.15 public:0.25.14 public:0.25.13 public:0.25.12 public:0.25.11 public:0.25.10 public:0.25.9 public:0.25.8 public:0.25.7 public:0.25.6 public:0.25.5 public:0.25.4 public:0.25.3 public:0.25.2 public:0.25.1 public:0.25.0 public:0.25.0-beta1 public:0.24.31 public:0.24.30 public:0.24.29 public:0.24.28 public:0.24.27 public:0.24.26 public:0.24.25 public:0.24.24 public:0.24.23 public:0.24.22 public:0.24.21 public:0.24.20 public:0.24.19 public:0.24.18 public:0.24.17 public:0.24.16 public:0.24.15 public:0.24.14 public:0.24.13 public:0.24.12 public:0.24.11 public:0.24.10 public:0.24.9 public:0.24.8 public:0.24.7 public:0.24.6 public:0.24.5 public:0.24.4 public:0.24.3 public:0.24.2 public:0.24.1 public:0.24.0 public:0.24.0.dev-rc8 public:0.24.0.dev-rc7 public:0.24.0.dev-rc6 public:0.24.0.dev-rc5 public:0.24.0.dev-rc4 public:0.24.0.dev-rc3 public:0.24.0.dev-rc2 public:0.23.23 public:0.23.22 public:0.23.21 public:0.23.20 public:0.23.19 public:0.23.18 public:0.23.17 public:0.23.16 public:0.23.15 public:0.23.14 public:0.23.13 public:0.23.12 public:0.23.11 public:0.23.10 public:0.23.9 public:0.23.8 public:0.23.7 public:0.23.6 public:0.23.5 public:0.23.4 public:0.23.3 public:0.23.2 public:0.23.1 public:0.23.0 public:0.22.19 public:0.22.18 public:0.22.17 public:0.22.16 public:0.22.15 public:0.22.14 public:0.22.13 public:0.22.12 public:0.22.11 public:0.22.10 public:0.22.9 public:0.22.8 public:0.22.7 public:0.22.6 public:0.22.5 public:0.22.4 public:0.22.3 public:0.22.2 public:0.22.1 public:0.22.0 public:0.21.5 public:0.21.4 public:0.21.3 public:0.21.2 public:0.21.1 public:0.21.0 public:0.20.7 public:0.20.6 public:0.20.5 public:0.20.4 public:0.20.3 public:0.20.2 public:0.20.1 public:0.20.0 public:0.19.23 public:0.19.22 public:0.19.21 public:0.19.20 public:0.19.19 public:0.19.18 public:0.19.17 public:0.19.16 public:0.19.15 public:0.19.14 public:0.19.13 public:0.19.12 public:0.19.11 public:0.19.10 public:0.19.9 public:0.19.8 public:0.19.7 public:0.19.6 public:0.19.5 public:0.19.4 public:0.19.3 public:0.19.2 public:0.19.1 public:0.19.0 public:0.18.6 public:0.18.5 public:0.18.4 public:0.18.3 public:0.18.2 public:0.18.1 public:0.18.0 public:0.17.34 public:0.17.33 public:0.17.32 public:0.17.31 public:0.17.30 public:0.17.29 public:0.17.28 public:0.17.27 public:0.17.26 public:0.17.25 public:0.17.24 public:0.17.23 public:0.17.22 public:0.17.21 public:0.17.20 public:0.17.19 public:0.17.18 public:0.17.17 public:0.17.16 public:0.17.15 public:0.17.14 public:0.17.13 public:0.17.12 public:0.17.11 public:0.17.10 public:0.17.9 public:0.17.8 public:0.17.7 public:0.17.6 public:0.17.5 public:0.17.4 public:0.17.3 public:0.17.2 public:0.17.1 public:0.17.0 public:0.16.8 public:0.16.7 public:0.16.6 public:0.16.5 public:0.16.4 public:0.16.3 public:0.16.2 public:0.16.1 public:0.16.0 public:0.15.10 public:0.15.9 public:0.15.8 public:0.15.7 public:0.15.6 public:0.15.5 public:0.15.4 public:0.15.3 public:0.15.2 public:0.15.1 public:0.15.0 public:0.14.8 public:0.14.7 public:0.14.6 public:0.14.5 public:0.14.4 public:0.14.3 public:0.14.2 public:0.14.1 public:0.14.0 public:0.13.4 public:0.13.3 public:0.13.2 public:0.13.1 public:0.13.0 public:0.12.5 public:0.12.4 public:0.12.3 public:0.12.2 public:0.12.1 public:0.12.0 public:0.11.10 public:0.11.9 public:0.11.8 public:0.11.7 public:0.11.6 public:0.11.5 public:0.11.4 public:0.11.3 public:0.11.2 public:0.11.1 public:0.11.0 public:0.10.1 public:0.10.0 public:0.9.0 public:0.8.8 public:0.8.7 public:0.8.6 public:0.8.5 public:0.8.3 public:0.8.2 public:0.8.1 public:0.8.0 public:0.7.2 public:0.7.1 public:0.7.0 public:0.6.1 public:0.6.0 public:0.5.0 public:0.4.1 public:0.4.0 public:0.3.1 public:0.3.0 public:0.2.1 public:0.2.0 public:0.1.26 public:0.1.25 public:0.1.24 public:0.1.23 public:0.1.22 public:0.1.21 public:0.1.20 public:0.1.19 public:0.1.18 public:0.1.17 public:0.1.16 public:0.1.15 public:0.1.14 public:0.1.13 public:0.1.12 public:0.1.11 public:0.1.10 public:0.1.9 public:0.1.8 public:0.1.7 public:0.1.6 public:0.1.5 public:0.1.4 public:0.1.3 public:0.1.2 public:0.1.1 public:0.1.0 public:0.0.9 public:0.0.8 public:0.0.7 public:0.0.6

20 Commits
0.23.3 ... 0.23.9

Author SHA1 Message Date
vorotamoroz
fd722b1fe5 bump 2024-05-27 12:05:41 +01:00
vorotamoroz
0bf087dba0 Fixed: 
- No longer unexpected parallel replication is performed.
- Now we can set the device name and enable customised synchronisation again.
 2024-05-27 12:04:19 +01:00
vorotamoroz
3a4b59b998 Update troubleshooting.md 2024-05-27 12:12:37 +09:00
vorotamoroz
8fc9d51c45 Add Note. 2024-05-27 04:11:44 +01:00
vorotamoroz
35feb5bf93 bump 2024-05-22 14:05:15 +01:00
vorotamoroz
b3a85c5462 New feature: 
- Now we are ready for i18n.
- The setting dialogue has been refined. Very controllable, clearly displayed disabled items, and ready to i18n.
Fixed:
- Many memory leaks have been rescued.
- Chunk caches now work well.
- Many trivial but potential bugs are fixed.
- No longer error messages will be shown on retrieving checkpoint or server information.
- Now we can check and correct tweak mismatch during the setup
Improved:
- Customisation synchronisation has got more smoother.
Tidied
- Practically unused functions have been removed or are being prepared for removal.
- Many of the type-errors and lint errors have been corrected.
- Unused files have been removed.
Note:
- From this version, some test files have been included. However, they are not enabled and released in the release build.
 2024-05-22 14:04:22 +01:00
vorotamoroz
7b0ac22c3b Create terms.md 2024-05-13 14:04:02 +09:00
vorotamoroz
dca8e4b2a4 bump 2024-05-10 11:38:03 +01:00
vorotamoroz
89de2dcc37 Fixed: 
- No longer missing tasks which have queued as the same key (e.g., for the same operation to the same file).
- Some trivial issues have been fixed.
New feature:
- Reloading Obsidian can be scheduled until that file and database operations are stable.
 2024-05-10 11:33:59 +01:00
vorotamoroz
172b08dbb3 bump 2024-05-08 23:57:19 +09:00
vorotamoroz
d518a3fc1b Fixed: 
- Now the remote chunks could be decrypted even if we are using `Incubate chunks in Document`. (The note of 0.23.6 has been fixed).
- Chunk retrieving with `Incubate chunks in document` got more efficiently.
- No longer task processor misses the completed tasks.
- Replication is no longer started automatically during changes in window visibility (e.g., task switching on the desktop) when off-focused.
 2024-05-08 23:56:29 +09:00
vorotamoroz
c6ed867498 bump 2024-05-07 12:59:55 +01:00
vorotamoroz
4f4923e977 New feature: 
- Now we can check configuration mismatching between clients before synchronisation.
- Now we can perform remote database compaction from the `Maintenance` pane.
Fixed:
- We can detect the bucket could not be reachable.
 2024-05-07 12:55:48 +01:00
vorotamoroz
a5ebf29b3d Merge pull request #417 from MichaelBrunn3r/translation 
fix: Grammar issues in settings page
 2024-05-07 20:26:59 +09:00
vorotamoroz
ee465184c8 bump 2024-05-05 23:46:16 +09:00
vorotamoroz
d7d4f1e6f2 New feature: 
- We can now use `Incubate Chunks in Document` to reduce non-well-formed chunks.
Fixed:
- No longer experimental configuration is shown on the Minimal Setup.
 2024-05-05 23:45:45 +09:00
Michael Brunner
cbf5023593 fix: Grammar issues in settings page 2024-05-04 12:34:53 +02:00
vorotamoroz
3925052f92 Add the design document of planned improving. 2024-05-04 02:59:24 +09:00
vorotamoroz
1934418258 Add the design document of planned improving. 2024-05-04 01:57:03 +09:00
vorotamoroz
2ae018b2bd Refactor: 
- Files have been categorised for clarity. The deliverables are not affected.
 2024-05-02 04:07:36 +01:00
46 changed files with 3870 additions and 2953 deletions
Expand all files Collapse all files

81
docs/design_docs_of_keep_newborn_chunks.md Normal file
View File

@@ -0,0 +1,81 @@
# Keep newborn chunks in Eden.
NOTE: This is the planned feature design document. This is planned, but not be implemented now (v0.23.3). This has not reached the design freeze and will be added to from time to time.
## Goal
Reduce the number of chunks which in volatile, and reduce the usage of storage of the remote database in middle or long term.
## Motivation
- In the current implementation, Self-hosted LiveSync splits documents into metadata and multiple chunks. In particular, chunks are split so that they do not exceed a certain length.
- This is to optimise the transfer and take advantage of the properties of CouchDB. This also complies with the restriction of IBM Cloudant on the size of a single document.
- However, creating chunks halfway through each editing operation increases the number of unnecessary chunks.
- Chunks are shared by several documents. For this reason, it is not clear whether these chunks are needed or not unless all revisions of all documents are checked. This makes it difficult to remove unnecessary data.
- On the other hand, chunks are done in units that can be neatly divided as markdown to ensure relatively accurate de-duplication, even if they are created simultaneously on multiple terminals. Therefore, it is unlikely that the data in the editing process will be reused.
- For this reason, we have made features such as Batch save available, but they are not a fundamental solution.
- As a result, there is a large amount of data that cannot be erased and is probably unused. Therefore, `Fetch chunks on demand` is currently performed for optimal communication.
- If the generation of unnecessary chunks is sufficiently reduced, this function will become unnecessary.
- The problem is that this unnecessary chunking slows down both local and remote operations.
## Prerequisite
- The implementation must be able to control the size of the document appropriately so that it does not become non-transferable (1).
- The implementation must be such that data corruption can be avoided even if forward compatibility is not maintained; due to the nature of Self-hosted LiveSync, backward version connexions are expected.
- Viewed as a feature:
- This feature should be disabled for migration users.
- This feature should be enabled for new users and after rebuilds of migrated users.
- Therefore, back into the implementation view, Ideally, the implementation should be such that data recovery can be achieved by immediately upgrading after replication.
## Outlined methods and implementation plans
### Abstract
To store and transfer only stable chunks independently and share them from multiple documents after stabilisation, new chunks, i.e. chunks that are considered non-stable, are modified to be stored in the document and transferred with the document. In this case, care should be taken not to exceed prerequisite (1).
If this is achieved, the non-leaf document will not be transferred, and even if it is, the chunk will be stored in the document, so that the size can be reduced by the compaction.
Details are given below.
1. The document will henceforth have the property eden.
```typescript
// Paritally Type
type EntryWithEden = {
eden: {
[key: DocumentID]: {
data: string,
epoch: number, // The document revision which this chunk has been born.
}
}
}
```
2. The following configuration items are added:
Note: These configurations should be shared as `Tweaks value` between each client.
- useEden : boolean
- Max chunks in eden : number
- Max Total chunk lengths in eden: number
- Max age while in eden: number
3. In the document saving operation, chunks are added to Eden within each document, having the revision number of the existing document. And if some chunks in eden are not used in the operating revision, they would be removed.
Then after being so chosen, a few chunks are also chosen to be graduated as an independent `chunk` in following rules, and they would be left the eden:
- Those that have already been confirmed to exist as independent chunks.
- This confirmation of existence may ideally be determined by a fast first-order determination, e.g. by a Bloom filter.
- Those whose length exceeds the configured maximum length.
- Those have aged over the configured value, since epoch at the operating revision.
- Those whose total length, when added up when they are arranged in reverse order of the revision in which they were generated, is after the point at which they exceed the max length in the configuration. Or, those after the configured maximum items.
4. In the document loading operation, chunks are firstly read from these eden.
5. In End-to-End Encryption, property `eden` of documents will also be encrypted.
### Note
- When this feature has been enabled, forward compatibility is temporarily lost. However, it is detected as missing chunks, and this data is not reflected in the storage in the old version. Therefore, no data loss will occur.
## Test strategy
1. Confirm that synchronisation with the previous version is possible with this feature disabled.
2. With this feature enabled, connect from the previous version and confirm that errors are detected in the previous version but the files are not corrupted.
3. Ensure that the two versions with this feature enabled can withstand normal use.
## Documentation strategy
- This document is published, and will be referred from the release note.
- Indeed, we lack a fulfilled configuration table. Efforts will be made and, if they can be produced, this document will then be referenced. But not required while in the experimental or beta feature.
- However, this might be an essential feature. Further efforts are desired.
### Consideration and Conclusion
To be described after implemented, tested, and, released.

55
docs/design_docs_of_sharing_tweak_value.md Normal file
View File

@@ -0,0 +1,55 @@
# Sharing `Tweak values`
NOTE: This is the planned feature design document. This is planned, but not be implemented now (v0.23.3). This has not reached the design freeze and will be added to from time to time.
## Goal
Share `Tweak values` between clients to match the chunk lengths, and match per-server configurations for better performance.
## Motivation
- In the current implementation, Self-hosted LiveSync splits documents into metadata and multiple chunks. In particular, chunks are split so that they do not exceed a certain length.
- This is to optimise the transfer and take advantage of the properties of CouchDB. This also complies with the restriction of IBM Cloudant on the size of a single document.
- The length of this chunk is adjusted according to a configured factor. Therefore, if this is inconsistent between clients, de-duplication will not work. This is because, in fact, they point to the same content in total, but are split in different places. This results in unnecessary transfers or storage consumption.
- The same applies to hash algorithms.
- There are more configurations which `preferred to be matched`, even if it is not required. such as the maximum size of files to be handled and the interval between requests to the remote database, unless there are specific circumstances.
- To avoid the tragedy of "Too many toggles", "Unexpected transfer amount", or "Poor performance" at once, the plug-in should know these problems or potential problems and be able to let us know.
## Prerequisite
- We must be informed of a discrepancy in a configured value that is required to be absolutely consistent and be able to make a decision on the spot.
- We should be able to see on the configuration dialogue, that there is a discrepancy between configured values that should be matched, and it should be possible to adjust them to a specific one of them (or default).
- We must not be exposed to unexpected; such as leaking credentials or their secrets.
## Outlined methods and implementation plans
### Abstract
- In the current implementation, each client checks the remote database for the existence of their node information, to detect whether the remote database accepts them.
- This is what 'Lock' is all about.
- To achieve this feature, the client will also send each configuration value. However, the configuration contains credentials and/or secret values. Hence we cannot send all of them.
- With a favourable prediction, Self-hosted LiveSync will continue to increase in feature. Each time this happens, the number of configuration values to be kept secret will also increase. Therefore, they must be handled by an allow-list.
- This allow-listed configuration are the `Tweak values`.
- If the plug-in detects mismatched `Tweak values` on checking the remote database, the plug-in will ask us to decide which is win (Mine, or theirs).
- Node information is one of the documents. Therefore, it will be replicated and saved locally. While showing dialogue, show the notice on each `Match preferred` configuration.
## Note
This feature should be mostly harmless. We will not be able to disable this.
## Test strategy
A: During synchronisation.
1. No message shall be displayed with all settings matched.
2. Message shall be displayed when there are mismatched, required match items.
1. The setting values can be changed according to the message.
2. The message can be ignored.
3. The message shall not be displayed even if there are mismatched items which is recommended to be matched.
B: On the setting dialogue.
1. All mismatched items shall be highlighted in some way.
## Documentation strategy
- This document is published, and will be referred from the release note.
- Indeed, we lack a fulfilled configuration table. Efforts will be made and, if they can be produced, this document will then be referenced. But not required while in the experimental or beta feature.
- However, this might be an essential feature. Further efforts are desired.
### Consideration and Conclusion
To be described after implemented, tested, and, released.

10
docs/terms.md Normal file
View File

@@ -0,0 +1,10 @@
# Terms used in this project
## Terms
### Chunks
<!-- TBW, sorry for the draft! -->
<!-- Please feel free to write any terms that should be mentioned. And please make pull request. I would love to fill the rest. -->
<!-- ### Chunks -->

14
docs/troubleshooting.md
View File

@@ -14,6 +14,8 @@
- [Why are the logs volatile and ephemeral?](#why-are-the-logs-volatile-and-ephemeral)
- [Some network logs are not written into the file.](#some-network-logs-are-not-written-into-the-file)
- [If a file were deleted or trimmed, the capacity of the database should be reduced, right?](#if-a-file-were-deleted-or-trimmed-the-capacity-of-the-database-should-be-reduced-right)
- [How can I use the DevTools?](#how-can-i-use-the-devtools)
- [Checking the network log](#checking-the-network-log)
- [Troubleshooting](#troubleshooting)
- [On the mobile device, cannot synchronise on the local network!](#on-the-mobile-device-cannot-synchronise-on-the-local-network)
- [I think that something bad happening on the vault...](#i-think-that-something-bad-happening-on-the-vault)
@@ -77,7 +79,7 @@ However, the logs would not be kept so long and cleared when restarted. If you w
To avoid unexpected exposure to our confidential things.
### Some network logs are not written into the file.
Especially the CORS error will be reported as a general error to the plug-in for security reasons. So we cannot detect and log it.
Especially the CORS error will be reported as a general error to the plug-in for security reasons. So we cannot detect and log it. We are only able to investigate them by [Checking the network log](#checking-the-network-log).
### If a file were deleted or trimmed, the capacity of the database should be reduced, right?
No, even though if files were deleted, chunks were not deleted.
@@ -87,7 +89,15 @@ And one more thing, we can handle the conflicts on any device even though it has
To shrink the database size, `Rebuild everything` only reliably and effectively. But do not worry, if we have synchronised well. We have the actual and real files. Only it takes a bit of time and traffics.
<!-- Add here -->
### How can I use the DevTools?
#### Checking the network log
1. Open the network pane.
2. Find the requests marked in red.
![Errored](../images/devtools1.png)
3. Capture the `Headers`, `Payload`, and, `Response`. **Please be sure to keep important information confidential**. If the `Response` contains secrets, you can omitted that.
Note: Headers contains a some credentials. **The path of the request URL, Remote Address, authority, and authorization must be concealed.**
![Concealed sample](../images/devtools2.png)
## Troubleshooting
<!-- Add here -->

6
esbuild.config.mjs
View File

@@ -16,6 +16,7 @@ if you want to view the source, please visit the github repository of this plugi
const prod = process.argv[2] === "production";
const keepTest = !prod;
const terserOpt = {
sourceMap: (!prod ? {
@@ -142,16 +143,17 @@ const context = await esbuild.context({
sourcemap: prod ? false : "inline",
treeShaking: true,
outfile: "main_org.js",
mainFields: ["browser", "module", "main"],
minifyWhitespace: false,
minifySyntax: false,
minifyIdentifiers: false,
minify: false,
dropLabels: prod && !keepTest ? ["TEST", "DEV"] : [],
// keepNames: true,
plugins: [
sveltePlugin({
preprocess: sveltePreprocess(),
compilerOptions: { css: true, preserveComments: true },
compilerOptions: { css: "injected", preserveComments: false },
}),
...plugins
],

BIN
images/devtools1.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 11 KiB

BIN
images/devtools2.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 41 KiB

2
manifest.json
View File

@@ -1,7 +1,7 @@
{
"id": "obsidian-livesync",
"name": "Self-hosted LiveSync",
"version": "0.23.3",
"version": "0.23.9",
"minAppVersion": "0.9.12",
"description": "Community implementation of self-hosted livesync. Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.",
"author": "vorotamoroz",

4
package-lock.json generated
View File

@@ -1,12 +1,12 @@
{
"name": "obsidian-livesync",
"version": "0.23.3",
"version": "0.23.9",
"lockfileVersion": 2,
"requires": true,
"packages": {
"": {
"name": "obsidian-livesync",
"version": "0.23.3",
"version": "0.23.9",
"license": "MIT",
"dependencies": {
"@aws-sdk/client-s3": "^3.556.0",

2
package.json
View File

@@ -1,6 +1,6 @@
{
"name": "obsidian-livesync",
"version": "0.23.3",
"version": "0.23.9",
"description": "Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.",
"main": "main.js",
"type": "module",

41
src/LogDisplayModal.ts
View File

@@ -1,41 +0,0 @@
import { App, Modal } from "./deps";
import type { ReactiveInstance, } from "./lib/src/reactive";
import { logMessages } from "./lib/src/stores";
import { escapeStringToHTML } from "./lib/src/strbin";
import ObsidianLiveSyncPlugin from "./main";
export class LogDisplayModal extends Modal {
plugin: ObsidianLiveSyncPlugin;
logEl: HTMLDivElement;
unsubscribe: () => void;
constructor(app: App, plugin: ObsidianLiveSyncPlugin) {
super(app);
this.plugin = plugin;
}
onOpen() {
const { contentEl } = this;
this.titleEl.setText("Sync status");
contentEl.empty();
const div = contentEl.createDiv("");
div.addClass("op-scrollable");
div.addClass("op-pre");
this.logEl = div;
function updateLog(logs: ReactiveInstance<string[]>) {
const e = logs.value;
let msg = "";
for (const v of e) {
msg += escapeStringToHTML(v) + "<br>";
}
this.logEl.innerHTML = msg;
}
logMessages.onChanged(updateLog);
this.unsubscribe = () => logMessages.offChanged(updateLog);
}
onClose() {
const { contentEl } = this;
contentEl.empty();
if (this.unsubscribe) this.unsubscribe();
}
}

2512
src/ObsidianLiveSyncSettingTab.ts
View File

File diff suppressed because it is too large Load Diff

0
src/KeyValueDB.ts → src/common/KeyValueDB.ts
View File

3
src/ObsHttpHandler.ts → src/common/ObsHttpHandler.ts
View File

@@ -11,7 +11,7 @@ import { HttpRequest, HttpResponse, type HttpHandlerOptions } from "@smithy/prot
//@ts-ignore
import { requestTimeout } from "@smithy/fetch-http-handler/dist-es/request-timeout";
import { buildQueryString } from "@smithy/querystring-builder";
import { requestUrl, type RequestUrlParam } from "./deps";
import { requestUrl, type RequestUrlParam } from "../deps.ts";
////////////////////////////////////////////////////////////////////////////////
// special handler using Obsidian requestUrl
////////////////////////////////////////////////////////////////////////////////
@@ -34,6 +34,7 @@ export class ObsHttpHandler extends FetchHttpHandler {
options === undefined ? undefined : options.requestTimeout;
this.reverseProxyNoSignUrl = reverseProxyNoSignUrl;
}
// eslint-disable-next-line require-await
async handle(
request: HttpRequest,
{ abortSignal }: HttpHandlerOptions = {}

6
src/dialogs.ts → src/common/dialogs.ts
View File

@@ -1,9 +1,9 @@
import { ButtonComponent } from "obsidian";
import { App, FuzzySuggestModal, MarkdownRenderer, Modal, Plugin, Setting } from "./deps";
import ObsidianLiveSyncPlugin from "./main";
import { App, FuzzySuggestModal, MarkdownRenderer, Modal, Plugin, Setting } from "../deps.ts";
import ObsidianLiveSyncPlugin from "../main.ts";
//@ts-ignore
import PluginPane from "./PluginPane.svelte";
import PluginPane from "../ui/PluginPane.svelte";
export class PluginDialogModal extends Modal {
plugin: ObsidianLiveSyncPlugin;

2
src/stores.ts → src/common/stores.ts
View File

@@ -1,4 +1,4 @@
import { PersistentMap } from "./lib/src/PersistentMap";
import { PersistentMap } from "../lib/src/dataobject/PersistentMap.ts";
export let sameChangePairs: PersistentMap<number[]>;

4
src/types.ts → src/common/types.ts
View File

@@ -1,5 +1,5 @@
import { type PluginManifest, TFile } from "./deps";
import { type DatabaseEntry, type EntryBody, type FilePath } from "./lib/src/types";
import { type PluginManifest, TFile } from "../deps.ts";
import { type DatabaseEntry, type EntryBody, type FilePath } from "../lib/src/common/types.ts";
export interface PluginDataEntry extends DatabaseEntry {
deviceVaultName: string;

22
src/utils.ts → src/common/utils.ts
View File

@@ -1,16 +1,16 @@
import { normalizePath, Platform, TAbstractFile, App, type RequestUrlParam, requestUrl, TFile } from "./deps";
import { path2id_base, id2path_base, isValidFilenameInLinux, isValidFilenameInDarwin, isValidFilenameInWidows, isValidFilenameInAndroid, stripAllPrefixes } from "./lib/src/path";
import { normalizePath, Platform, TAbstractFile, App, type RequestUrlParam, requestUrl, TFile } from "../deps.ts";
import { path2id_base, id2path_base, isValidFilenameInLinux, isValidFilenameInDarwin, isValidFilenameInWidows, isValidFilenameInAndroid, stripAllPrefixes } from "../lib/src/string_and_binary/path.ts";
import { Logger } from "./lib/src/logger";
import { LOG_LEVEL_VERBOSE, type AnyEntry, type DocumentID, type EntryHasPath, type FilePath, type FilePathWithPrefix } from "./lib/src/types";
import { CHeader, ICHeader, ICHeaderLength, ICXHeader, PSCHeader } from "./types";
import { InputStringDialog, PopoverSelectString } from "./dialogs";
import type ObsidianLiveSyncPlugin from "./main";
import { writeString } from "./lib/src/strbin";
import { fireAndForget } from "./lib/src/utils";
import { sameChangePairs } from "./stores";
import { Logger } from "../lib/src/common/logger.ts";
import { LOG_LEVEL_VERBOSE, type AnyEntry, type DocumentID, type EntryHasPath, type FilePath, type FilePathWithPrefix } from "../lib/src/common/types.ts";
import { CHeader, ICHeader, ICHeaderLength, ICXHeader, PSCHeader } from "./types.ts";
import { InputStringDialog, PopoverSelectString } from "./dialogs.ts";
import type ObsidianLiveSyncPlugin from "../main.ts";
import { writeString } from "../lib/src/string_and_binary/strbin.ts";
import { fireAndForget } from "../lib/src/common/utils.ts";
import { sameChangePairs } from "./stores.ts";
export { scheduleTask, setPeriodicTask, cancelTask, cancelAllTasks, cancelPeriodicTask, cancelAllPeriodicTask, } from "./lib/src/task";
export { scheduleTask, setPeriodicTask, cancelTask, cancelAllTasks, cancelPeriodicTask, cancelAllPeriodicTask, } from "../lib/src/concurrency/task.ts";
// For backward compatibility, using the path for determining id.
// Only CouchDB unacceptable ID (that starts with an underscore) has been prefixed with "/".

2
src/deps.ts
View File

@@ -1,4 +1,4 @@
import { type FilePath } from "./lib/src/types";
import { type FilePath } from "./lib/src/common/types.ts";
export {
addIcon, App, debounce, Editor, FuzzySuggestModal, MarkdownRenderer, MarkdownView, Modal, Notice, Platform, Plugin, PluginSettingTab, requestUrl, sanitizeHTMLToDom, Setting, stringifyYaml, TAbstractFile, TextAreaComponent, TFile, TFolder,

176
src/CmdConfigSync.ts → src/features/CmdConfigSync.ts
View File

@@ -1,24 +1,26 @@
import { writable } from 'svelte/store';
import { Notice, type PluginManifest, parseYaml, normalizePath, type ListedFiles } from "./deps";
import { Notice, type PluginManifest, parseYaml, normalizePath, type ListedFiles } from "../deps.ts";
import type { EntryDoc, LoadedEntry, InternalFileEntry, FilePathWithPrefix, FilePath, DocumentID, AnyEntry, SavingEntry } from "./lib/src/types";
import { LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE, MODE_SELECTIVE } from "./lib/src/types";
import { ICXHeader, PERIODIC_PLUGIN_SWEEP, } from "./types";
import { createSavingEntryFromLoadedEntry, createTextBlob, delay, fireAndForget, getDocData, isDocContentSame, throttle } from "./lib/src/utils";
import { Logger } from "./lib/src/logger";
import { readString, decodeBinary, arrayBufferToBase64, digestHash } from "./lib/src/strbin";
import { serialized } from "./lib/src/lock";
import { LiveSyncCommands } from "./LiveSyncCommands";
import { stripAllPrefixes } from "./lib/src/path";
import { PeriodicProcessor, askYesNo, disposeMemoObject, memoIfNotExist, memoObject, retrieveMemoObject, scheduleTask } from "./utils";
import { PluginDialogModal } from "./dialogs";
import { JsonResolveModal } from "./JsonResolveModal";
import { QueueProcessor } from './lib/src/processor';
import { pluginScanningCount } from './lib/src/stores';
import type ObsidianLiveSyncPlugin from './main';
import type { EntryDoc, LoadedEntry, InternalFileEntry, FilePathWithPrefix, FilePath, DocumentID, AnyEntry, SavingEntry } from "../lib/src/common/types.ts";
import { LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE, MODE_SELECTIVE } from "../lib/src/common/types.ts";
import { ICXHeader, PERIODIC_PLUGIN_SWEEP, } from "../common/types.ts";
import { createSavingEntryFromLoadedEntry, createTextBlob, delay, fireAndForget, getDocDataAsArray, isDocContentSame, throttle } from "../lib/src/common/utils.ts";
import { Logger } from "../lib/src/common/logger.ts";
import { readString, decodeBinary, arrayBufferToBase64, digestHash } from "../lib/src/string_and_binary/strbin.ts";
import { serialized, shareRunningResult } from "../lib/src/concurrency/lock.ts";
import { LiveSyncCommands } from "./LiveSyncCommands.ts";
import { stripAllPrefixes } from "../lib/src/string_and_binary/path.ts";
import { PeriodicProcessor, disposeMemoObject, memoIfNotExist, memoObject, retrieveMemoObject, scheduleTask } from "../common/utils.ts";
import { PluginDialogModal } from "../common/dialogs.ts";
import { JsonResolveModal } from "../ui/JsonResolveModal.ts";
import { QueueProcessor } from '../lib/src/concurrency/processor.ts';
import { pluginScanningCount } from '../lib/src/mock_and_interop/stores.ts';
import type ObsidianLiveSyncPlugin from '../main.ts';
const d = "\u200b";
const d2 = "\n";
const delimiters = /(?<=[\n|\u200b])/g;
function serialize(data: PluginDataEx): string {
// For higher performance, create custom plug-in data strings.
@@ -30,7 +32,7 @@ function serialize(data: PluginDataEx): string {
ret += data.mtime + d2;
for (const file of data.files) {
ret += file.filename + d + (file.displayName ?? "") + d + (file.version ?? "") + d2;
const hash = digestHash((file.data ?? []).join());
const hash = digestHash((file.data ?? []));
ret += file.mtime + d + file.size + d + hash + d2;
for (const data of file.data ?? []) {
ret += data + d
@@ -39,41 +41,49 @@ function serialize(data: PluginDataEx): string {
}
return ret;
}
function fetchToken(source: string, from: number): [next: number, token: string] {
const limitIdx = source.indexOf(d2, from);
const limit = limitIdx == -1 ? source.length : limitIdx;
const delimiterIdx = source.indexOf(d, from);
const delimiter = delimiterIdx == -1 ? source.length : delimiterIdx;
const tokenEnd = Math.min(limit, delimiter);
let next = tokenEnd;
if (limit < delimiter) {
next = tokenEnd;
} else {
next = tokenEnd + 1
}
return [next, source.substring(from, tokenEnd)];
}
function getTokenizer(source: string) {
function getTokenizer(source: string[]) {
const sources = source.flatMap(e => e.split(delimiters))
sources[0] = sources[0].substring(1);
let pos = 0;
let lineRunOut = false;
const t = {
pos: 1,
next() {
const [next, token] = fetchToken(source, this.pos);
this.pos = next;
return token;
next(): string {
if (lineRunOut) {
return "";
}
if (pos >= sources.length) {
return "";
}
const item = sources[pos];
if (!item.endsWith(d2)) {
pos++;
} else {
lineRunOut = true;
}
if (item.endsWith(d) || item.endsWith(d2)) {
return item.substring(0, item.length - 1);
} else {
return item + this.next();
}
},
nextLine() {
const nextPos = source.indexOf(d2, this.pos);
if (nextPos == -1) {
this.pos = source.length;
if (lineRunOut) {
pos++;
} else {
this.pos = nextPos + 1;
while (!sources[pos].endsWith(d2)) {
pos++;
if (pos >= sources.length) break;
}
pos++;
}
lineRunOut = false;
}
}
return t;
}
function deserialize2(str: string): PluginDataEx {
function deserialize2(str: string[]): PluginDataEx {
const tokens = getTokenizer(str);
const ret = {} as PluginDataEx;
const category = tokens.next();
@@ -120,13 +130,16 @@ function deserialize2(str: string): PluginDataEx {
return result;
}
function deserialize<T>(str: string, def: T) {
function deserialize<T>(str: string[], def: T) {
try {
if (str[0] == ":") return deserialize2(str);
return JSON.parse(str) as T;
if (str[0][0] == ":") {
const o = deserialize2(str);
return o;
}
return JSON.parse(str.join("")) as T;
} catch (ex) {
try {
return parseYaml(str);
return parseYaml(str.join(""));
} catch (ex) {
return def;
}
@@ -277,14 +290,14 @@ export class ConfigSync extends LiveSyncCommands {
async loadPluginData(path: FilePathWithPrefix): Promise<PluginDataExDisplay | false> {
const wx = await this.localDatabase.getDBEntry(path, undefined, false, false);
if (wx) {
const data = deserialize(getDocData(wx.data), {}) as PluginDataEx;
const data = deserialize(getDocDataAsArray(wx.data), {}) as PluginDataEx;
const xFiles = [] as PluginDataExFile[];
let missingHash = false;
for (const file of data.files) {
const work = { ...file, data: [] as string[] };
if (!file.hash) {
// debugger;
const tempStr = getDocData(work.data);
const tempStr = getDocDataAsArray(work.data);
const hash = digestHash(tempStr);
file.hash = hash;
missingHash = true;
@@ -326,6 +339,7 @@ export class ConfigSync extends LiveSyncCommands {
if (saveRequired) {
this.plugin.saveSettingData();
}
}
pluginScanProcessor = new QueueProcessor(async (v: AnyEntry[]) => {
@@ -384,9 +398,9 @@ export class ConfigSync extends LiveSyncCommands {
const docB = await this.localDatabase.getDBEntry(dataB.documentPath);
if (docA && docB) {
const pluginDataA = deserialize(getDocData(docA.data), {}) as PluginDataEx;
const pluginDataA = deserialize(getDocDataAsArray(docA.data), {}) as PluginDataEx;
pluginDataA.documentPath = dataA.documentPath;
const pluginDataB = deserialize(getDocData(docB.data), {}) as PluginDataEx;
const pluginDataB = deserialize(getDocDataAsArray(docB.data), {}) as PluginDataEx;
pluginDataB.documentPath = dataB.documentPath;
// Use outer structure to wrap each data.
@@ -425,7 +439,7 @@ export class ConfigSync extends LiveSyncCommands {
if (dx == false) {
throw "Not found on database"
}
const loadedData = deserialize(getDocData(dx.data), {}) as PluginDataEx;
const loadedData = deserialize(getDocDataAsArray(dx.data), {}) as PluginDataEx;
for (const f of loadedData.files) {
Logger(`Applying ${f.filename} of ${data.displayName || data.name}..`);
try {
@@ -466,12 +480,7 @@ export class ConfigSync extends LiveSyncCommands {
Logger(`Plugin reloaded: ${pluginManifest.name}`, LOG_LEVEL_NOTICE, "plugin-reload-" + pluginManifest.id);
}
} else if (data.category == "CONFIG") {
scheduleTask("configReload", 250, async () => {
if (await askYesNo(this.app, "Do you want to restart and reload Obsidian now?") == "yes") {
// @ts-ignore
this.app.commands.executeCommandById("app:reload")
}
})
this.plugin.askReload();
}
return true;
} catch (ex) {
@@ -684,6 +693,7 @@ export class ConfigSync extends LiveSyncCommands {
children: [],
deleted: false,
type: "newnote",
eden: {}
};
} else {
if (old.mtime == mtime) {
@@ -692,7 +702,7 @@ export class ConfigSync extends LiveSyncCommands {
}
const oldC = await this.localDatabase.getDBEntryFromMeta(old, {}, false, false);
if (oldC) {
const d = await deserialize(getDocData(oldC.data), {}) as PluginDataEx;
const d = await deserialize(getDocDataAsArray(oldC.data), {}) as PluginDataEx;
const diffs = (d.files.map(previous => ({ prev: previous, curr: dt.files.find(e => e.filename == previous.filename) })).map(async e => {
try { return await isDocContentSame(e.curr?.data ?? [], e.prev.data) } catch (_) { return false }
}))
@@ -754,31 +764,33 @@ export class ConfigSync extends LiveSyncCommands {
async scanAllConfigFiles(showMessage: boolean) {
const logLevel = showMessage ? LOG_LEVEL_NOTICE : LOG_LEVEL_INFO;
Logger("Scanning customizing files.", logLevel, "scan-all-config");
const term = this.plugin.deviceAndVaultName;
if (term == "") {
Logger("We have to configure the device name", LOG_LEVEL_NOTICE);
return;
}
const filesAll = await this.scanInternalFiles();
const files = filesAll.filter(e => this.isTargetPath(e)).map(e => ({ key: this.filenameToUnifiedKey(e), file: e }));
const virtualPathsOfLocalFiles = [...new Set(files.map(e => e.key))];
const filesOnDB = ((await this.localDatabase.allDocsRaw({ startkey: ICXHeader + "", endkey: `${ICXHeader}\u{10ffff}`, include_docs: true })).rows.map(e => e.doc) as InternalFileEntry[]).filter(e => !e.deleted);
let deleteCandidate = filesOnDB.map(e => this.getPath(e)).filter(e => e.startsWith(`${ICXHeader}${term}/`));
for (const vp of virtualPathsOfLocalFiles) {
const p = files.find(e => e.key == vp)?.file;
if (!p) {
Logger(`scanAllConfigFiles - File not found: ${vp}`, LOG_LEVEL_VERBOSE);
continue;
await shareRunningResult("scanAllConfigFiles", async () => {
const logLevel = showMessage ? LOG_LEVEL_NOTICE : LOG_LEVEL_INFO;
Logger("Scanning customizing files.", logLevel, "scan-all-config");
const term = this.plugin.deviceAndVaultName;
if (term == "") {
Logger("We have to configure the device name", LOG_LEVEL_NOTICE);
return;
}
await this.storeCustomizationFiles(p);
deleteCandidate = deleteCandidate.filter(e => e != vp);
}
for (const vp of deleteCandidate) {
await this.deleteConfigOnDatabase(vp);
}
this.updatePluginList(false).then(/* fire and forget */);
const filesAll = await this.scanInternalFiles();
const files = filesAll.filter(e => this.isTargetPath(e)).map(e => ({ key: this.filenameToUnifiedKey(e), file: e }));
const virtualPathsOfLocalFiles = [...new Set(files.map(e => e.key))];
const filesOnDB = ((await this.localDatabase.allDocsRaw({ startkey: ICXHeader + "", endkey: `${ICXHeader}\u{10ffff}`, include_docs: true })).rows.map(e => e.doc) as InternalFileEntry[]).filter(e => !e.deleted);
let deleteCandidate = filesOnDB.map(e => this.getPath(e)).filter(e => e.startsWith(`${ICXHeader}${term}/`));
for (const vp of virtualPathsOfLocalFiles) {
const p = files.find(e => e.key == vp)?.file;
if (!p) {
Logger(`scanAllConfigFiles - File not found: ${vp}`, LOG_LEVEL_VERBOSE);
continue;
}
await this.storeCustomizationFiles(p);
deleteCandidate = deleteCandidate.filter(e => e != vp);
}
for (const vp of deleteCandidate) {
await this.deleteConfigOnDatabase(vp);
}
this.updatePluginList(false).then(/* fire and forget */);
});
}
async deleteConfigOnDatabase(prefixedFileName: FilePathWithPrefix, forceWrite = false) {

45
src/CmdHiddenFileSync.ts → src/features/CmdHiddenFileSync.ts
View File

@@ -1,16 +1,16 @@
import { normalizePath, type PluginManifest, type ListedFiles } from "./deps";
import { type EntryDoc, type LoadedEntry, type InternalFileEntry, type FilePathWithPrefix, type FilePath, LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE, MODE_SELECTIVE, MODE_PAUSED, type SavingEntry, type DocumentID } from "./lib/src/types";
import { type InternalFileInfo, ICHeader, ICHeaderEnd } from "./types";
import { readAsBlob, isDocContentSame, sendSignal, readContent, createBlob } from "./lib/src/utils";
import { Logger } from "./lib/src/logger";
import { PouchDB } from "./lib/src/pouchdb-browser.js";
import { isInternalMetadata, PeriodicProcessor } from "./utils";
import { serialized } from "./lib/src/lock";
import { JsonResolveModal } from "./JsonResolveModal";
import { LiveSyncCommands } from "./LiveSyncCommands";
import { addPrefix, stripAllPrefixes } from "./lib/src/path";
import { QueueProcessor } from "./lib/src/processor";
import { hiddenFilesEventCount, hiddenFilesProcessingCount } from "./lib/src/stores";
import { normalizePath, type PluginManifest, type ListedFiles } from "../deps.ts";
import { type EntryDoc, type LoadedEntry, type InternalFileEntry, type FilePathWithPrefix, type FilePath, LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE, MODE_SELECTIVE, MODE_PAUSED, type SavingEntry, type DocumentID } from "../lib/src/common/types.ts";
import { type InternalFileInfo, ICHeader, ICHeaderEnd } from "../common/types.ts";
import { readAsBlob, isDocContentSame, sendSignal, readContent, createBlob } from "../lib/src/common/utils.ts";
import { Logger } from "../lib/src/common/logger.ts";
import { PouchDB } from "../lib/src/pouchdb/pouchdb-browser.js";
import { isInternalMetadata, PeriodicProcessor } from "../common/utils.ts";
import { serialized } from "../lib/src/concurrency/lock.ts";
import { JsonResolveModal } from "../ui/JsonResolveModal.ts";
import { LiveSyncCommands } from "./LiveSyncCommands.ts";
import { addPrefix, stripAllPrefixes } from "../lib/src/string_and_binary/path.ts";
import { QueueProcessor } from "../lib/src/concurrency/processor.ts";
import { hiddenFilesEventCount, hiddenFilesProcessingCount } from "../lib/src/mock_and_interop/stores.ts";
export class HiddenFileSync extends LiveSyncCommands {
periodicInternalFileScanProcessor: PeriodicProcessor = new PeriodicProcessor(this.plugin, async () => this.settings.syncInternalFiles && this.localDatabase.isReady && await this.syncInternalFilesAndDatabase("push", false));
@@ -144,7 +144,7 @@ export class HiddenFileSync extends LiveSyncCommands {
Logger("something went wrong on resolving all conflicted internal files");
Logger(ex, LOG_LEVEL_VERBOSE);
}
await this.conflictResolutionProcessor.startPipeline().waitForPipeline();
await this.conflictResolutionProcessor.startPipeline().waitForAllProcessed();
}
async resolveByNewerEntry(id: DocumentID, path: FilePathWithPrefix, currentDoc: EntryDoc, currentRev: string, conflictedRev: string) {
@@ -388,7 +388,7 @@ export class HiddenFileSync extends LiveSyncCommands {
}, { suspended: true, batchSize: 1, concurrentLimit: 5, delay: 0 }))
.root
.enqueueAll(allFileNames)
.startPipeline().waitForPipeline();
.startPipeline().waitForAllDoneAndTerminate();
await this.kvDB.set("diff-caches-internal", caches);
@@ -432,13 +432,14 @@ export class HiddenFileSync extends LiveSyncCommands {
// If something changes left, notify for reloading Obsidian.
if (updatedCount != 0) {
this.plugin.askInPopup(`updated-any-hidden`, `Hidden files have been synchronized, Press {HERE} to reload Obsidian, or press elsewhere to dismiss this message.`, (anchor) => {
anchor.text = "HERE";
anchor.addEventListener("click", () => {
// @ts-ignore
this.app.commands.executeCommandById("app:reload");
if (!this.plugin.isReloadingScheduled) {
this.plugin.askInPopup(`updated-any-hidden`, `Hidden files have been synchronised, Press {HERE} to schedule a reload of Obsidian, or press elsewhere to dismiss this message.`, (anchor) => {
anchor.text = "HERE";
anchor.addEventListener("click", () => {
this.plugin.scheduleAppReload();
});
});
});
}
}
}
}
@@ -471,6 +472,7 @@ export class HiddenFileSync extends LiveSyncCommands {
children: [],
deleted: false,
type: "newnote",
eden: {},
};
} else {
if (await isDocContentSame(readAsBlob(old), content) && !forceWrite) {
@@ -521,6 +523,7 @@ export class HiddenFileSync extends LiveSyncCommands {
children: [],
deleted: true,
type: "newnote",
eden: {}
};
} else {
// Remove all conflicted before deleting.

24
src/CmdSetupLiveSync.ts → src/features/CmdSetupLiveSync.ts
View File

@@ -1,15 +1,15 @@
import { type EntryDoc, type ObsidianLiveSyncSettings, DEFAULT_SETTINGS, LOG_LEVEL_NOTICE, REMOTE_COUCHDB, REMOTE_MINIO } from "./lib/src/types";
import { configURIBase } from "./types";
import { Logger } from "./lib/src/logger";
import { PouchDB } from "./lib/src/pouchdb-browser.js";
import { askSelectString, askYesNo, askString } from "./utils";
import { decrypt, encrypt } from "./lib/src/e2ee_v2";
import { LiveSyncCommands } from "./LiveSyncCommands";
import { delay, fireAndForget } from "./lib/src/utils";
import { confirmWithMessage } from "./dialogs";
import { Platform } from "./deps";
import { fetchAllUsedChunks } from "./lib/src/utils_couchdb";
import type { LiveSyncCouchDBReplicator } from "./lib/src/LiveSyncReplicator.js";
import { type EntryDoc, type ObsidianLiveSyncSettings, DEFAULT_SETTINGS, LOG_LEVEL_NOTICE, REMOTE_COUCHDB, REMOTE_MINIO } from "../lib/src/common/types.ts";
import { configURIBase } from "../common/types.ts";
import { Logger } from "../lib/src/common/logger.ts";
import { PouchDB } from "../lib/src/pouchdb/pouchdb-browser.js";
import { askSelectString, askYesNo, askString } from "../common/utils.ts";
import { decrypt, encrypt } from "../lib/src/encryption/e2ee_v2.ts";
import { LiveSyncCommands } from "./LiveSyncCommands.ts";
import { delay, fireAndForget } from "../lib/src/common/utils.ts";
import { confirmWithMessage } from "../common/dialogs.ts";
import { Platform } from "../deps.ts";
import { fetchAllUsedChunks } from "../lib/src/pouchdb/utils_couchdb.ts";
import type { LiveSyncCouchDBReplicator } from "../lib/src/replication/couchdb/LiveSyncReplicator.js";
export class SetupLiveSync extends LiveSyncCommands {
onunload() { }

6
src/LiveSyncCommands.ts → src/features/LiveSyncCommands.ts
View File

@@ -1,6 +1,6 @@
import { type AnyEntry, type DocumentID, type EntryDoc, type EntryHasPath, type FilePath, type FilePathWithPrefix } from "./lib/src/types";
import { PouchDB } from "./lib/src/pouchdb-browser.js";
import type ObsidianLiveSyncPlugin from "./main";
import { type AnyEntry, type DocumentID, type EntryDoc, type EntryHasPath, type FilePath, type FilePathWithPrefix } from "../lib/src/common/types.ts";
import { PouchDB } from "../lib/src/pouchdb/pouchdb-browser.js";
import type ObsidianLiveSyncPlugin from "../main.ts";
export abstract class LiveSyncCommands {

2
src/lib

Submodule src/lib updated: 60d012f92d...302a2e7c0b

485
src/main.ts
View File

@@ -2,43 +2,46 @@ const isDebug = false;
import { type Diff, DIFF_DELETE, DIFF_EQUAL, DIFF_INSERT, diff_match_patch, stringifyYaml, parseYaml } from "./deps";
import { Notice, Plugin, TFile, addIcon, TFolder, normalizePath, TAbstractFile, Editor, MarkdownView, type RequestUrlParam, type RequestUrlResponse, requestUrl, type MarkdownFileInfo } from "./deps";
import { type EntryDoc, type LoadedEntry, type ObsidianLiveSyncSettings, type diff_check_result, type diff_result_leaf, type EntryBody, LOG_LEVEL, VER, DEFAULT_SETTINGS, type diff_result, FLAGMD_REDFLAG, SYNCINFO_ID, SALT_OF_PASSPHRASE, type ConfigPassphraseStore, type CouchDBConnection, FLAGMD_REDFLAG2, FLAGMD_REDFLAG3, PREFIXMD_LOGFILE, type DatabaseConnectingStatus, type EntryHasPath, type DocumentID, type FilePathWithPrefix, type FilePath, type AnyEntry, LOG_LEVEL_DEBUG, LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_URGENT, LOG_LEVEL_VERBOSE, type SavingEntry, MISSING_OR_ERROR, NOT_CONFLICTED, AUTO_MERGED, CANCELLED, LEAVE_TO_SUBSEQUENT, FLAGMD_REDFLAG2_HR, FLAGMD_REDFLAG3_HR, REMOTE_MINIO, REMOTE_COUCHDB, type BucketSyncSetting, } from "./lib/src/types";
import { type InternalFileInfo, type CacheData, type FileEventItem, FileWatchEventQueueMax } from "./types";
import { arrayToChunkedArray, createBlob, delay, determineTypeFromBlob, fireAndForget, getDocData, isAnyNote, isDocContentSame, isObjectDifferent, readContent, sendValue, throttle, type SimpleStore } from "./lib/src/utils";
import { Logger, setGlobalLogFunction } from "./lib/src/logger";
import { PouchDB } from "./lib/src/pouchdb-browser.js";
import { ConflictResolveModal } from "./ConflictResolveModal";
import { ObsidianLiveSyncSettingTab } from "./ObsidianLiveSyncSettingTab";
import { DocumentHistoryModal } from "./DocumentHistoryModal";
import { applyPatch, cancelAllPeriodicTask, cancelAllTasks, cancelTask, generatePatchObj, id2path, isObjectMargeApplicable, isSensibleMargeApplicable, flattenObject, path2id, scheduleTask, tryParseJSON, isValidPath, isInternalMetadata, isPluginMetadata, stripInternalMetadataPrefix, isChunk, askSelectString, askYesNo, askString, PeriodicProcessor, getPath, getPathWithoutPrefix, getPathFromTFile, performRebuildDB, memoIfNotExist, memoObject, retrieveMemoObject, disposeMemoObject, isCustomisationSyncMetadata, compareFileFreshness, BASE_IS_NEW, TARGET_IS_NEW, EVEN, compareMTime, markChangesAreSame } from "./utils";
import { encrypt, tryDecrypt } from "./lib/src/e2ee_v2";
import { balanceChunkPurgedDBs, enableCompression, enableEncryption, isCloudantURI, isErrorOfMissingDoc, isValidRemoteCouchDBURI, purgeUnreferencedChunks } from "./lib/src/utils_couchdb";
import { logStore, type LogEntry, collectingChunks, pluginScanningCount, hiddenFilesProcessingCount, hiddenFilesEventCount, logMessages } from "./lib/src/stores";
import { setNoticeClass } from "./lib/src/wrapper";
import { versionNumberString2Number, writeString, decodeBinary, readString } from "./lib/src/strbin";
import { addPrefix, isAcceptedAll, isPlainText, shouldBeIgnored, stripAllPrefixes } from "./lib/src/path";
import { isLockAcquired, serialized, shareRunningResult, skipIfDuplicated } from "./lib/src/lock";
import { StorageEventManager, StorageEventManagerObsidian } from "./StorageEventManager";
import { LiveSyncLocalDB, type LiveSyncLocalDBEnv } from "./lib/src/LiveSyncLocalDB";
import { LiveSyncAbstractReplicator, type LiveSyncReplicatorEnv } from "./lib/src/LiveSyncAbstractReplicator.js";
import { type KeyValueDatabase, OpenKeyValueDatabase } from "./KeyValueDB";
import { LiveSyncCommands } from "./LiveSyncCommands";
import { HiddenFileSync } from "./CmdHiddenFileSync";
import { SetupLiveSync } from "./CmdSetupLiveSync";
import { ConfigSync } from "./CmdConfigSync";
import { confirmWithMessage } from "./dialogs";
import { GlobalHistoryView, VIEW_TYPE_GLOBAL_HISTORY } from "./GlobalHistoryView";
import { LogPaneView, VIEW_TYPE_LOG } from "./LogPaneView";
import { LRUCache } from "./lib/src/LRUCache";
import { SerializedFileAccess } from "./SerializedFileAccess.js";
import { QueueProcessor } from "./lib/src/processor.js";
import { reactive, reactiveSource } from "./lib/src/reactive.js";
import { initializeStores } from "./stores.js";
import { JournalSyncMinio } from "./lib/src/JournalSyncMinio.js";
import { LiveSyncJournalReplicator, type LiveSyncJournalReplicatorEnv } from "./lib/src/LiveSyncJournalReplicator.js";
import { LiveSyncCouchDBReplicator, type LiveSyncCouchDBReplicatorEnv } from "./lib/src/LiveSyncReplicator.js";
import type { CheckPointInfo } from "./lib/src/JournalSyncTypes.js";
import { ObsHttpHandler } from "./ObsHttpHandler.js";
import { type EntryDoc, type LoadedEntry, type ObsidianLiveSyncSettings, type diff_check_result, type diff_result_leaf, type EntryBody, LOG_LEVEL, VER, DEFAULT_SETTINGS, type diff_result, FLAGMD_REDFLAG, SYNCINFO_ID, SALT_OF_PASSPHRASE, type ConfigPassphraseStore, type CouchDBConnection, FLAGMD_REDFLAG2, FLAGMD_REDFLAG3, PREFIXMD_LOGFILE, type DatabaseConnectingStatus, type EntryHasPath, type DocumentID, type FilePathWithPrefix, type FilePath, type AnyEntry, LOG_LEVEL_DEBUG, LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_URGENT, LOG_LEVEL_VERBOSE, type SavingEntry, MISSING_OR_ERROR, NOT_CONFLICTED, AUTO_MERGED, CANCELLED, LEAVE_TO_SUBSEQUENT, FLAGMD_REDFLAG2_HR, FLAGMD_REDFLAG3_HR, REMOTE_MINIO, REMOTE_COUCHDB, type BucketSyncSetting, TweakValuesShouldMatchedTemplate, confName, type TweakValues, } from "./lib/src/common/types.ts";
import { type InternalFileInfo, type CacheData, type FileEventItem, FileWatchEventQueueMax } from "./common/types.ts";
import { arrayToChunkedArray, createBlob, delay, determineTypeFromBlob, escapeMarkdownValue, extractObject, fireAndForget, getDocData, isAnyNote, isDocContentSame, isObjectDifferent, readContent, sendValue, throttle, type SimpleStore } from "./lib/src/common/utils.ts";
import { Logger, setGlobalLogFunction } from "./lib/src/common/logger.ts";
import { PouchDB } from "./lib/src/pouchdb/pouchdb-browser.js";
import { ConflictResolveModal } from "./ui/ConflictResolveModal.ts";
import { ObsidianLiveSyncSettingTab } from "./ui/ObsidianLiveSyncSettingTab.ts";
import { DocumentHistoryModal } from "./ui/DocumentHistoryModal.ts";
import { applyPatch, cancelAllPeriodicTask, cancelAllTasks, cancelTask, generatePatchObj, id2path, isObjectMargeApplicable, isSensibleMargeApplicable, flattenObject, path2id, scheduleTask, tryParseJSON, isValidPath, isInternalMetadata, isPluginMetadata, stripInternalMetadataPrefix, isChunk, askSelectString, askYesNo, askString, PeriodicProcessor, getPath, getPathWithoutPrefix, getPathFromTFile, performRebuildDB, memoIfNotExist, memoObject, retrieveMemoObject, disposeMemoObject, isCustomisationSyncMetadata, compareFileFreshness, BASE_IS_NEW, TARGET_IS_NEW, EVEN, compareMTime, markChangesAreSame } from "./common/utils.ts";
import { encrypt, tryDecrypt } from "./lib/src/encryption/e2ee_v2.ts";
import { balanceChunkPurgedDBs, enableCompression, enableEncryption, isCloudantURI, isErrorOfMissingDoc, isValidRemoteCouchDBURI, purgeUnreferencedChunks } from "./lib/src/pouchdb/utils_couchdb.ts";
import { logStore, type LogEntry, collectingChunks, pluginScanningCount, hiddenFilesProcessingCount, hiddenFilesEventCount, logMessages } from "./lib/src/mock_and_interop/stores.ts";
import { setNoticeClass } from "./lib/src/mock_and_interop/wrapper.ts";
import { versionNumberString2Number, writeString, decodeBinary, readString } from "./lib/src/string_and_binary/strbin.ts";
import { addPrefix, isAcceptedAll, isPlainText, shouldBeIgnored, stripAllPrefixes } from "./lib/src/string_and_binary/path.ts";
import { isLockAcquired, serialized, shareRunningResult, skipIfDuplicated } from "./lib/src/concurrency/lock.ts";
import { StorageEventManager, StorageEventManagerObsidian } from "./storages/StorageEventManager.ts";
import { LiveSyncLocalDB, type LiveSyncLocalDBEnv } from "./lib/src/pouchdb/LiveSyncLocalDB.ts";
import { LiveSyncAbstractReplicator, type LiveSyncReplicatorEnv } from "./lib/src/replication/LiveSyncAbstractReplicator.js";
import { type KeyValueDatabase, OpenKeyValueDatabase } from "./common/KeyValueDB.ts";
import { LiveSyncCommands } from "./features/LiveSyncCommands.ts";
import { HiddenFileSync } from "./features/CmdHiddenFileSync.ts";
import { SetupLiveSync } from "./features/CmdSetupLiveSync.ts";
import { ConfigSync } from "./features/CmdConfigSync.ts";
import { confirmWithMessage } from "./common/dialogs.ts";
import { GlobalHistoryView, VIEW_TYPE_GLOBAL_HISTORY } from "./ui/GlobalHistoryView.ts";
import { LogPaneView, VIEW_TYPE_LOG } from "./ui/LogPaneView.ts";
import { LRUCache } from "./lib/src/memory/LRUCache.ts";
import { SerializedFileAccess } from "./storages/SerializedFileAccess.js";
import { QueueProcessor, stopAllRunningProcessors } from "./lib/src/concurrency/processor.js";
import { reactive, reactiveSource, type ReactiveValue } from "./lib/src/dataobject/reactive.js";
import { initializeStores } from "./common/stores.js";
import { JournalSyncMinio } from "./lib/src/replication/journal/objectstore/JournalSyncMinio.js";
import { LiveSyncJournalReplicator, type LiveSyncJournalReplicatorEnv } from "./lib/src/replication/journal/LiveSyncJournalReplicator.js";
import { LiveSyncCouchDBReplicator, type LiveSyncCouchDBReplicatorEnv } from "./lib/src/replication/couchdb/LiveSyncReplicator.js";
import type { CheckPointInfo } from "./lib/src/replication/journal/JournalSyncTypes.js";
import { ObsHttpHandler } from "./common/ObsHttpHandler.js";
import { TestPaneView, VIEW_TYPE_TEST } from "./tests/TestPaneView.js"
import { $f, __onMissingTranslation, setLang } from "./lib/src/common/i18n.ts";
setNoticeClass(Notice);
@@ -84,6 +87,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin
settings!: ObsidianLiveSyncSettings;
localDatabase!: LiveSyncLocalDB;
replicator!: LiveSyncAbstractReplicator;
settingTab!: ObsidianLiveSyncSettingTab;
statusBar?: HTMLElement;
_suspended = false;
@@ -222,12 +226,17 @@ export default class ObsidianLiveSyncPlugin extends Plugin
}
Logger(`HTTP:${method}${size} to:${localURL} -> ${response.status}`, LOG_LEVEL_DEBUG);
if (Math.floor(response.status / 100) !== 2) {
const r = response.clone();
Logger(`The request may have failed. The reason sent by the server: ${r.status}: ${r.statusText}`);
try {
Logger(await (await r.blob()).text(), LOG_LEVEL_VERBOSE);
} catch (_) {
Logger("Cloud not parse response", LOG_LEVEL_VERBOSE);
if (method != "GET" && localURL.indexOf("/_local/") === -1 && !localURL.endsWith("/")) {
const r = response.clone();
Logger(`The request may have failed. The reason sent by the server: ${r.status}: ${r.statusText}`);
try {
Logger(await (await r.blob()).text(), LOG_LEVEL_VERBOSE);
} catch (_) {
Logger("Cloud not parse response", LOG_LEVEL_VERBOSE);
}
} else {
Logger(`Just checkpoint or some server information has been missing. The 404 error shown above is not an error.`, LOG_LEVEL_VERBOSE)
}
}
return response;
@@ -327,6 +336,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin
}
async onInitializeDatabase(db: LiveSyncLocalDB): Promise<void> {
this.kvDB = await OpenKeyValueDatabase(db.dbname + "-livesync-kv");
// this.trench = new Trench(this.simpleStore);
this.replicator = this.getNewReplicator();
}
async onResetDatabase(db: LiveSyncLocalDB): Promise<void> {
@@ -335,6 +345,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin
// localStorage.removeItem(lsKey);
await this.kvDB.destroy();
this.kvDB = await OpenKeyValueDatabase(db.dbname + "-livesync-kv");
// this.trench = new Trench(this.simpleStore);
this.replicator = this.getNewReplicator()
}
getReplicator() {
@@ -420,7 +431,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin
if (target) {
const targetItem = notes.find(e => e.dispPath == target)!;
this.resolveConflicted(targetItem.path);
await this.conflictCheckQueue.waitForPipeline();
await this.conflictCheckQueue.waitForAllProcessed();
return true;
}
return false;
@@ -480,6 +491,8 @@ export default class ObsidianLiveSyncPlugin extends Plugin
return (await ret).map(e => e.toString()).filter(e => e.startsWith("os-")).map(e => e.substring(3));
}
}
// trench!: Trench;
getMinioJournalSyncClient() {
const id = this.settings.accessKey
const key = this.settings.secretKey
@@ -840,12 +853,56 @@ Note: We can always able to read V1 format. It will be progressively converted.
VIEW_TYPE_LOG,
(leaf) => new LogPaneView(leaf, this)
);
// eslint-disable-next-line no-unused-labels
TEST: {
this.registerView(
VIEW_TYPE_TEST,
(leaf) => new TestPaneView(leaf, this)
);
(async () => {
if (await this.vaultAccess.adapterExists("_SHOWDIALOGAUTO.md"))
this.showView(VIEW_TYPE_TEST);
})()
this.addCommand({
id: "view-test",
name: "Open Test dialogue",
callback: () => {
this.showView(VIEW_TYPE_TEST);
}
});
}
}
async onload() {
logStore.pipeTo(new QueueProcessor(logs => logs.forEach(e => this.addLog(e.message, e.level, e.key)), { suspended: false, batchSize: 20, concurrentLimit: 1, delay: 0 })).startPipeline();
Logger("loading plugin");
this.addSettingTab(new ObsidianLiveSyncSettingTab(this.app, this));
// eslint-disable-next-line no-unused-labels
DEV: {
__onMissingTranslation((key) => {
const now = new Date();
const filename = `missing-translation-`
const time = now.toISOString().split("T")[0];
const outFile = `${filename}${time}.jsonl`;
const piece = JSON.stringify(
{
[key]: {}
}
)
const writePiece = piece.substring(1, piece.length - 1) + ",";
fireAndForget(async () => {
try {
await this.vaultAccess.adapterAppend(this.app.vault.configDir + "/ls-debug/" + outFile, writePiece + "\n")
} catch (ex) {
Logger(`Could not write ${outFile}`, LOG_LEVEL_VERBOSE);
Logger(`Missing translation: ${writePiece}`, LOG_LEVEL_VERBOSE);
}
});
})
}
this.settingTab = new ObsidianLiveSyncSettingTab(this.app, this);
this.addSettingTab(this.settingTab);
this.addUIs();
//@ts-ignore
const manifestVersion: string = MANIFEST_VERSION || "0.0.0";
@@ -855,7 +912,7 @@ Note: We can always able to read V1 format. It will be progressively converted.
this.manifestVersion = manifestVersion;
this.packageVersion = packageVersion;
Logger(`Self-hosted LiveSync v${manifestVersion} ${packageVersion} `);
Logger($f`Self-hosted LiveSync${" v"}${manifestVersion} ${packageVersion}`);
await this.loadSettings();
const lsKey = "obsidian-live-sync-ver" + this.getVaultName();
const last_version = localStorage.getItem(lsKey);
@@ -866,7 +923,7 @@ Note: We can always able to read V1 format. It will be progressively converted.
}
const lastVersion = ~~(versionNumberString2Number(manifestVersion) / 1000);
if (lastVersion > this.settings.lastReadUpdates && this.settings.isConfigured) {
Logger("Self-hosted LiveSync has undergone a major upgrade. Please open the setting dialog, and check the information pane.", LOG_LEVEL_NOTICE);
Logger($f`Self-hosted LiveSync has undergone a major upgrade. Please open the setting dialog, and check the information pane.`, LOG_LEVEL_NOTICE);
}
//@ts-ignore
@@ -881,7 +938,7 @@ Note: We can always able to read V1 format. It will be progressively converted.
this.settings.syncOnFileOpen = false;
this.settings.syncAfterMerge = false;
this.settings.periodicReplication = false;
this.settings.versionUpFlash = "Self-hosted LiveSync has been upgraded and some behaviors have changed incompatibly. All automatic synchronization is now disabled temporary. Ensure that other devices are also upgraded, and enable synchronization again.";
this.settings.versionUpFlash = $f`Self-hosted LiveSync has been upgraded and some behaviors have changed incompatibly. All automatic synchronization is now disabled temporary. Ensure that other devices are also upgraded, and enable synchronization again.`;
this.saveSettings();
}
localStorage.setItem(lsKey, `${VER}`);
@@ -926,6 +983,7 @@ Note: We can always able to read V1 format. It will be progressively converted.
onunload() {
cancelAllPeriodicTask();
cancelAllTasks();
stopAllRunningProcessors();
this._unloaded = true;
for (const addOn of this.addOns) {
addOn.onunload();
@@ -938,7 +996,7 @@ Note: We can always able to read V1 format. It will be progressively converted.
this.replicator.closeReplication();
this.localDatabase.close();
}
Logger("unloading plugin");
Logger($f`unloading plugin`);
}
async openDatabase() {
@@ -946,7 +1004,7 @@ Note: We can always able to read V1 format. It will be progressively converted.
await this.localDatabase.close();
}
const vaultName = this.getVaultName();
Logger("Waiting for ready...");
Logger($f`Waiting for ready...`);
this.localDatabase = new LiveSyncLocalDB(vaultName, this);
initializeStores(vaultName);
return await this.localDatabase.initializeDatabase();
@@ -1048,6 +1106,7 @@ Note: We can always able to read V1 format. It will be progressively converted.
}
this.settings = settings;
setLang(this.settings.displayLanguage);
if ("workingEncrypt" in this.settings) delete this.settings.workingEncrypt;
if ("workingPassphrase" in this.settings) delete this.settings.workingPassphrase;
@@ -1075,6 +1134,7 @@ Note: We can always able to read V1 format. It will be progressively converted.
this.deviceAndVaultName = localStorage.getItem(lsKey) || "";
this.ignoreFiles = this.settings.ignoreFiles.split(",").map(e => e.trim());
this.fileEventQueue.delay = (!this.settings.liveSync && this.settings.batchSave) ? 5000 : 100;
this.settingTab.requestReload()
}
async saveSettingData() {
@@ -1118,6 +1178,8 @@ Note: We can always able to read V1 format. It will be progressively converted.
}
await this.saveData(settings);
this.localDatabase.settings = this.settings;
setLang(this.settings.displayLanguage);
this.settingTab.requestReload();
this.fileEventQueue.delay = (!this.settings.liveSync && this.settings.batchSave) ? 5000 : 100;
this.ignoreFiles = this.settings.ignoreFiles.split(",").map(e => e.trim());
if (this.settings.settingSyncFile != "") {
@@ -1299,6 +1361,7 @@ We can perform a command in this file.
if (this._unloaded) {
Logger("Unload and remove the handler.", LOG_LEVEL_VERBOSE);
saveCommandDefinition.callback = this._initialCallback;
this._initialCallback = undefined;
} else {
Logger("Sync on Editor Save.", LOG_LEVEL_VERBOSE);
if (this.settings.syncOnEditorSave) {
@@ -1369,13 +1432,14 @@ We can perform a command in this file.
} else {
// suspend all temporary.
if (this.suspended) return;
if (!this.hasFocus) return;
await Promise.all(this.addOns.map(e => e.onResume()));
if (this.settings.remoteType == REMOTE_COUCHDB) {
if (this.settings.liveSync) {
this.replicator.openReplication(this.settings, true, false, false);
}
}
if (this.settings.syncOnStart) {
if (!this.settings.liveSync && this.settings.syncOnStart) {
this.replicator.openReplication(this.settings, false, false, false);
}
this.periodicSyncProcessor.enable(this.settings.periodicReplication ? this.settings.periodicReplicationInterval * 1000 : 0);
@@ -1416,55 +1480,64 @@ We can perform a command in this file.
}
async handleFileEvent(queue: FileEventItem): Promise<any> {
const file = queue.args.file;
const key = `file-last-proc-${queue.type}-${file.path}`;
const last = Number(await this.kvDB.get(key) || 0);
let mtime = file.mtime;
if (queue.type == "DELETE") {
await this.deleteFromDBbyPath(file.path);
mtime = file.mtime - 1;
const keyD1 = `file-last-proc-CREATE-${file.path}`;
const keyD2 = `file-last-proc-CHANGED-${file.path}`;
await this.kvDB.set(keyD1, mtime);
await this.kvDB.set(keyD2, mtime);
} else if (queue.type == "INTERNAL") {
await this.addOnHiddenFileSync.watchVaultRawEventsAsync(file.path);
await this.addOnConfigSync.watchVaultRawEventsAsync(file.path);
} else {
const targetFile = this.vaultAccess.getAbstractFileByPath(file.path);
if (!(targetFile instanceof TFile)) {
Logger(`Target file was not found: ${file.path}`, LOG_LEVEL_INFO);
return;
}
if (file.mtime == last) {
Logger(`File has been already scanned on ${queue.type}, skip: ${file.path}`, LOG_LEVEL_VERBOSE);
return;
}
// const cache = queue.args.cache;
if (queue.type == "CREATE" || queue.type == "CHANGED") {
fireAndForget(() => this.checkAndApplySettingFromMarkdown(queue.args.file.path, true));
const keyD1 = `file-last-proc-DELETED-${file.path}`;
const lockKey = `handleFile:${file.path}`;
return await serialized(lockKey, async () => {
// TODO CHECK
// console.warn(lockKey);
const key = `file-last-proc-${queue.type}-${file.path}`;
const last = Number(await this.kvDB.get(key) || 0);
let mtime = file.mtime;
if (queue.type == "DELETE") {
await this.deleteFromDBbyPath(file.path);
mtime = file.mtime - 1;
const keyD1 = `file-last-proc-CREATE-${file.path}`;
const keyD2 = `file-last-proc-CHANGED-${file.path}`;
await this.kvDB.set(keyD1, mtime);
if (!await this.updateIntoDB(targetFile, undefined)) {
Logger(`STORAGE -> DB: failed, cancel the relative operations: ${targetFile.path}`, LOG_LEVEL_INFO);
// cancel running queues and remove one of atomic operation
this.cancelRelativeEvent(queue);
await this.kvDB.set(keyD2, mtime);
} else if (queue.type == "INTERNAL") {
await this.addOnHiddenFileSync.watchVaultRawEventsAsync(file.path);
await this.addOnConfigSync.watchVaultRawEventsAsync(file.path);
} else {
const targetFile = this.vaultAccess.getAbstractFileByPath(file.path);
if (!(targetFile instanceof TFile)) {
Logger(`Target file was not found: ${file.path}`, LOG_LEVEL_INFO);
return;
}
if (file.mtime == last) {
Logger(`File has been already scanned on ${queue.type}, skip: ${file.path}`, LOG_LEVEL_VERBOSE);
return;
}
// const cache = queue.args.cache;
if (queue.type == "CREATE" || queue.type == "CHANGED") {
fireAndForget(() => this.checkAndApplySettingFromMarkdown(queue.args.file.path, true));
const keyD1 = `file-last-proc-DELETED-${file.path}`;
await this.kvDB.set(keyD1, mtime);
if (!await this.updateIntoDB(targetFile, undefined)) {
Logger(`STORAGE -> DB: failed, cancel the relative operations: ${targetFile.path}`, LOG_LEVEL_INFO);
// cancel running queues and remove one of atomic operation
this.cancelRelativeEvent(queue);
return;
}
}
if (queue.type == "RENAME") {
// Obsolete
await this.watchVaultRenameAsync(targetFile, queue.args.oldPath);
}
}
if (queue.type == "RENAME") {
// Obsolete
await this.watchVaultRenameAsync(targetFile, queue.args.oldPath);
}
}
await this.kvDB.set(key, mtime);
await this.kvDB.set(key, mtime);
});
}
pendingFileEventCount = reactiveSource(0);
processingFileEventCount = reactiveSource(0);
fileEventQueue =
new QueueProcessor(
(items: FileEventItem[]) => this.handleFileEvent(items[0]),
async (items: FileEventItem[]) => {
await this.handleFileEvent(items[0]);
return []
}
,
{ suspended: true, batchSize: 1, concurrentLimit: 5, delay: 100, yieldThreshold: FileWatchEventQueueMax, totalRemainingReactiveSource: this.pendingFileEventCount, processingEntitiesReactiveSource: this.processingFileEventCount }
).replaceEnqueueProcessor((items, newItem) => this.queueNextFileEvent(items, newItem));
@@ -1748,7 +1821,7 @@ We can perform a command in this file.
Logger(JSON.stringify(errors), LOG_LEVEL_VERBOSE);
}
this.replicationResultProcessor.enqueueAll(docs);
await this.replicationResultProcessor.waitForPipeline();
await this.replicationResultProcessor.waitForAllProcessed();
}
}
@@ -1888,7 +1961,7 @@ We can perform a command in this file.
observeForLogs() {
const padSpaces = `\u{2007}`.repeat(10);
// const emptyMark = `\u{2003}`;
const rerenderTimer = new Map<string, [ReturnType<typeof setTimeout>, number]>;
const rerenderTimer = new Map<string, [ReturnType<typeof setTimeout>, number]>();
const tick = reactiveSource(0);
function padLeftSp(num: number, mark: string) {
const numLen = `${num}`.length + 1;
@@ -1999,9 +2072,9 @@ We can perform a command in this file.
};
})
const statusBarLabels = reactive(() => {
const scheduleMessage = this.isReloadingScheduled ? `WARNING! RESTARTING OBSIDIAN IS SCHEDULED!\n` : "";
const { message } = statusLineLabel.value;
const status = this.statusLog.value;
const status = scheduleMessage + this.statusLog.value;
return {
message, status
}
@@ -2031,7 +2104,63 @@ We can perform a command in this file.
scheduleTask("log-hide", 3000, () => { this.statusLog.value = "" });
}
async askResolvingMismatchedTweaks(): Promise<"OK" | "CHECKAGAIN" | "IGNORE"> {
if (!this.replicator.tweakSettingsMismatched) {
return "OK";
}
const remoteSettings = this.replicator.mismatchedTweakValues;
const mustSettings = remoteSettings.map(e => extractObject(TweakValuesShouldMatchedTemplate, e));
const items = Object.entries(TweakValuesShouldMatchedTemplate);
// Making tables:
let table = `| Value name | Ours | ${mustSettings.map((_, i) => `Remote ${i + 1} |`).join("")}\n` +
`|: --- |: --- :${`|: --- :`.repeat(mustSettings.length)}|\n`
for (const v of items) {
const key = v[0] as keyof typeof TweakValuesShouldMatchedTemplate;
const value = mustSettings.map(e => e[key]);
table += `| ${confName(key)} | ${escapeMarkdownValue(this.settings[key])} | ${value.map((v) => `${escapeMarkdownValue(v)} |`).join("")}\n`;
}
const message = `
Configuration mismatching between the clients has been detected.
This can be harmful or extra capacity consumption. We have to make these value unified.
Configured values:
${table}
Please select a unification method.
However, even if we answer that you will \`Use mine\`, we will be prompted to accept it again on the other device and have to decide accept or not.`;
//TODO: apply this settings.
const CHOICE_USE_REMOTE = "Use Remote ";
const CHOICE_USR_MINE = "Use ours";
const CHOICE_DISMISS = "Dismiss";
// const ourConfig = extractObject(TweakValuesShouldMatchedTemplate, this.settings);
const CHOICE_AND_VALUES = [
...mustSettings.map((e, i) => [`${CHOICE_USE_REMOTE} ${i + 1}`, e]),
[CHOICE_USR_MINE, true],
[CHOICE_DISMISS, false]
]
const CHOICES = Object.fromEntries(CHOICE_AND_VALUES) as Record<string, TweakValues | boolean>;
const retKey = await confirmWithMessage(this, "Locked", message, Object.keys(CHOICES), CHOICE_DISMISS, 60);
if (!retKey) return "IGNORE";
const conf = CHOICES[retKey];
if (conf === true) {
await this.replicator.resetRemoteTweakSettings(this.settings);
Logger(`Tweak values on the remote server have been cleared, and will be overwritten in next synchronisation.`, LOG_LEVEL_NOTICE);
return "OK";
}
if (conf) {
this.settings = { ...this.settings, ...conf };
await this.saveSettingData();
Logger(`Tweak Values have been overwritten by the chosen one.`, LOG_LEVEL_NOTICE);
return "CHECKAGAIN";
}
return "IGNORE";
}
async replicate(showMessage: boolean = false) {
if (!this.isReady) return;
if (isLockAcquired("cleanup")) {
@@ -2047,58 +2176,63 @@ We can perform a command in this file.
await this.loadQueuedFiles();
const ret = await this.replicator.openReplication(this.settings, false, showMessage, false);
if (!ret) {
if (this.replicator.remoteLockedAndDeviceNotAccepted) {
if (this.replicator.remoteCleaned && this.settings.useIndexedDBAdapter) {
Logger(`The remote database has been cleaned.`, showMessage ? LOG_LEVEL_NOTICE : LOG_LEVEL_INFO);
await skipIfDuplicated("cleanup", async () => {
const count = await purgeUnreferencedChunks(this.localDatabase.localDatabase, true);
const message = `The remote database has been cleaned up.
if (this.replicator.tweakSettingsMismatched) {
await this.askResolvingMismatchedTweaks();
} else {
if (this.replicator?.remoteLockedAndDeviceNotAccepted) {
if (this.replicator.remoteCleaned && this.settings.useIndexedDBAdapter) {
Logger(`The remote database has been cleaned.`, showMessage ? LOG_LEVEL_NOTICE : LOG_LEVEL_INFO);
await skipIfDuplicated("cleanup", async () => {
const count = await purgeUnreferencedChunks(this.localDatabase.localDatabase, true);
const message = `The remote database has been cleaned up.
To synchronize, this device must be also cleaned up. ${count} chunk(s) will be erased from this device.
However, If there are many chunks to be deleted, maybe fetching again is faster.
We will lose the history of this device if we fetch the remote database again.
Even if you choose to clean up, you will see this option again if you exit Obsidian and then synchronise again.`
const CHOICE_FETCH = "Fetch again";
const CHOICE_CLEAN = "Cleanup";
const CHOICE_DISMISS = "Dismiss";
const ret = await confirmWithMessage(this, "Cleaned", message, [CHOICE_FETCH, CHOICE_CLEAN, CHOICE_DISMISS], CHOICE_DISMISS, 30);
if (ret == CHOICE_FETCH) {
await performRebuildDB(this, "localOnly");
}
if (ret == CHOICE_CLEAN) {
const replicator = this.getReplicator();
if (!(replicator instanceof LiveSyncCouchDBReplicator)) return;
const remoteDB = await replicator.connectRemoteCouchDBWithSetting(this.settings, this.getIsMobile(), true);
if (typeof remoteDB == "string") {
Logger(remoteDB, LOG_LEVEL_NOTICE);
return false;
const CHOICE_FETCH = "Fetch again";
const CHOICE_CLEAN = "Cleanup";
const CHOICE_DISMISS = "Dismiss";
const ret = await confirmWithMessage(this, "Cleaned", message, [CHOICE_FETCH, CHOICE_CLEAN, CHOICE_DISMISS], CHOICE_DISMISS, 30);
if (ret == CHOICE_FETCH) {
await performRebuildDB(this, "localOnly");
}
if (ret == CHOICE_CLEAN) {
const replicator = this.getReplicator();
if (!(replicator instanceof LiveSyncCouchDBReplicator)) return;
const remoteDB = await replicator.connectRemoteCouchDBWithSetting(this.settings, this.getIsMobile(), true);
if (typeof remoteDB == "string") {
Logger(remoteDB, LOG_LEVEL_NOTICE);
return false;
}
await purgeUnreferencedChunks(this.localDatabase.localDatabase, false);
this.localDatabase.hashCaches.clear();
// Perform the synchronisation once.
if (await this.replicator.openReplication(this.settings, false, showMessage, true)) {
await balanceChunkPurgedDBs(this.localDatabase.localDatabase, remoteDB.db);
await purgeUnreferencedChunks(this.localDatabase.localDatabase, false);
this.localDatabase.hashCaches.clear();
await this.getReplicator().markRemoteResolved(this.settings);
Logger("The local database has been cleaned up.", showMessage ? LOG_LEVEL_NOTICE : LOG_LEVEL_INFO)
} else {
Logger("Replication has been cancelled. Please try it again.", showMessage ? LOG_LEVEL_NOTICE : LOG_LEVEL_INFO)
}
// Perform the synchronisation once.
if (await this.replicator.openReplication(this.settings, false, showMessage, true)) {
await balanceChunkPurgedDBs(this.localDatabase.localDatabase, remoteDB.db);
await purgeUnreferencedChunks(this.localDatabase.localDatabase, false);
this.localDatabase.hashCaches.clear();
await this.getReplicator().markRemoteResolved(this.settings);
Logger("The local database has been cleaned up.", showMessage ? LOG_LEVEL_NOTICE : LOG_LEVEL_INFO)
} else {
Logger("Replication has been cancelled. Please try it again.", showMessage ? LOG_LEVEL_NOTICE : LOG_LEVEL_INFO)
}
}
});
} else {
const message = `
}
});
} else {
const message = `
The remote database has been rebuilt.
To synchronize, this device must fetch everything again once.
Or if you are sure know what had been happened, we can unlock the database from the setting dialog.
`
const CHOICE_FETCH = "Fetch again";
const CHOICE_DISMISS = "Dismiss";
const ret = await confirmWithMessage(this, "Locked", message, [CHOICE_FETCH, CHOICE_DISMISS], CHOICE_DISMISS, 10);
if (ret == CHOICE_FETCH) {
await performRebuildDB(this, "localOnly");
const CHOICE_FETCH = "Fetch again";
const CHOICE_DISMISS = "Dismiss";
const ret = await confirmWithMessage(this, "Locked", message, [CHOICE_FETCH, CHOICE_DISMISS], CHOICE_DISMISS, 10);
if (ret == CHOICE_FETCH) {
await performRebuildDB(this, "localOnly");
}
}
}
}
@@ -2125,14 +2259,30 @@ Or if you are sure know what had been happened, we can unlock the database from
}
}
async replicateAllToServer(showingNotice: boolean = false) {
async replicateAllToServer(showingNotice: boolean = false): Promise<boolean> {
if (!this.isReady) return false;
await Promise.all(this.addOns.map(e => e.beforeReplicate(showingNotice)));
return await this.replicator.replicateAllToServer(this.settings, showingNotice);
const ret = await this.replicator.replicateAllToServer(this.settings, showingNotice);
if (ret) return true;
if (this.replicator.tweakSettingsMismatched) {
const ret = await this.askResolvingMismatchedTweaks();
if (ret == "OK") return true;
if (ret == "CHECKAGAIN") return await this.replicateAllToServer(showingNotice);
if (ret == "IGNORE") return false;
}
return ret;
}
async replicateAllFromServer(showingNotice: boolean = false) {
async replicateAllFromServer(showingNotice: boolean = false): Promise<boolean> {
if (!this.isReady) return false;
return await this.replicator.replicateAllFromServer(this.settings, showingNotice);
const ret = await this.replicator.replicateAllFromServer(this.settings, showingNotice);
if (ret) return true;
if (this.replicator.tweakSettingsMismatched) {
const ret = await this.askResolvingMismatchedTweaks();
if (ret == "OK") return true;
if (ret == "CHECKAGAIN") return await this.replicateAllFromServer(showingNotice);
if (ret == "IGNORE") return false;
}
return ret;
}
async markRemoteLocked(lockByClean: boolean = false) {
@@ -2239,7 +2389,7 @@ Or if you are sure know what had been happened, we can unlock the database from
}
return;
}, { batchSize: 1, concurrentLimit: 10, delay: 0, suspended: true }, objects)
await processor.waitForPipeline();
await processor.waitForAllDoneAndTerminate();
const msg = `${procedureName} All done: DONE:${success}, FAILED:${failed}`;
updateLog(procedureName, msg)
}
@@ -2307,7 +2457,7 @@ Or if you are sure know what had been happened, we can unlock the database from
}
}
processPrepareSyncFile.startPipeline().onUpdateProgress(() => remainLog(processPrepareSyncFile.totalRemaining + processPrepareSyncFile.nowProcessing))
initProcess.push(processPrepareSyncFile.waitForPipeline());
initProcess.push(processPrepareSyncFile.waitForAllDoneAndTerminate());
await Promise.all(initProcess);
// this.setStatusBarText(`NOW TRACKING!`);
@@ -2873,6 +3023,7 @@ Or if you are sure know what had been happened, we can unlock the database from
children: [],
datatype: datatype,
type: datatype,
eden: {},
};
//upsert should locked
const msg = `STORAGE -> DB (${datatype}) `;
@@ -3111,5 +3262,61 @@ Or if you are sure know what had been happened, we can unlock the database from
// @ts-ignore
this.app.commands.executeCommandById(id)
}
_totalProcessingCount?: ReactiveValue<number>;
get isReloadingScheduled() {
return this._totalProcessingCount !== undefined;
}
askReload(message?: string) {
if (this.isReloadingScheduled) {
Logger(`Reloading is already scheduled`, LOG_LEVEL_VERBOSE);
return;
}
scheduleTask("configReload", 250, async () => {
const RESTART_NOW = "Yes, restart immediately";
const RESTART_AFTER_STABLE = "Yes, schedule a restart after stabilisation";
const RETRY_LATER = "No, Leave it to me";
const ret = await askSelectString(this.app, message || "Do you want to restart and reload Obsidian now?", [RESTART_AFTER_STABLE, RESTART_NOW, RETRY_LATER]);
if (ret == RESTART_NOW) {
this.performAppReload();
} else if (ret == RESTART_AFTER_STABLE) {
this.scheduleAppReload();
}
})
}
scheduleAppReload() {
if (!this._totalProcessingCount) {
const __tick = reactiveSource(0);
this._totalProcessingCount = reactive(() => {
const dbCount = this.databaseQueueCount.value;
const replicationCount = this.replicationResultCount.value;
const storageApplyingCount = this.storageApplyingCount.value;
const chunkCount = collectingChunks.value;
const pluginScanCount = pluginScanningCount.value;
const hiddenFilesCount = hiddenFilesEventCount.value + hiddenFilesProcessingCount.value;
const conflictProcessCount = this.conflictProcessQueueCount.value;
const e = this.pendingFileEventCount.value;
const proc = this.processingFileEventCount.value;
// eslint-disable-next-line @typescript-eslint/no-unused-vars
const __ = __tick.value;
return dbCount + replicationCount + storageApplyingCount + chunkCount + pluginScanCount + hiddenFilesCount + conflictProcessCount + e + proc;
})
this.registerInterval(setInterval(() => {
__tick.value++;
}, 1000) as unknown as number);
let stableCheck = 3;
this._totalProcessingCount.onChanged(e => {
if (e.value == 0) {
if (stableCheck-- <= 0) {
this.performAppReload();
}
Logger(`Obsidian will be restarted soon! (Within ${stableCheck} seconds)`, LOG_LEVEL_NOTICE, "restart-notice");
} else {
stableCheck = 3;
}
})
}
}
}

16
src/SerializedFileAccess.ts → src/storages/SerializedFileAccess.ts
View File

@@ -1,11 +1,11 @@
import { type App, TFile, type DataWriteOptions, TFolder, TAbstractFile } from "./deps";
import { serialized } from "./lib/src/lock";
import { Logger } from "./lib/src/logger";
import { isPlainText } from "./lib/src/path";
import type { FilePath } from "./lib/src/types";
import { createBinaryBlob, isDocContentSame } from "./lib/src/utils";
import type { InternalFileInfo } from "./types";
import { markChangesAreSame } from "./utils";
import { type App, TFile, type DataWriteOptions, TFolder, TAbstractFile } from "../deps.ts";
import { serialized } from "../lib/src/concurrency/lock.ts";
import { Logger } from "../lib/src/common/logger.ts";
import { isPlainText } from "../lib/src/string_and_binary/path.ts";
import type { FilePath } from "../lib/src/common/types.ts";
import { createBinaryBlob, isDocContentSame } from "../lib/src/common/utils.ts";
import type { InternalFileInfo } from "../common/types.ts";
import { markChangesAreSame } from "../common/utils.ts";
function getFileLockKey(file: TFile | TFolder | string) {
return `fl:${typeof (file) == "string" ? file : file.path}`;

16
src/StorageEventManager.ts → src/storages/StorageEventManager.ts
View File

@@ -1,11 +1,11 @@
import type { SerializedFileAccess } from "./SerializedFileAccess";
import { Plugin, TAbstractFile, TFile, TFolder } from "./deps";
import { Logger } from "./lib/src/logger";
import { shouldBeIgnored } from "./lib/src/path";
import type { QueueProcessor } from "./lib/src/processor";
import { LOG_LEVEL_NOTICE, type FilePath, type ObsidianLiveSyncSettings } from "./lib/src/types";
import { delay } from "./lib/src/utils";
import { type FileEventItem, type FileEventType, type FileInfo, type InternalFileInfo } from "./types";
import type { SerializedFileAccess } from "./SerializedFileAccess.ts";
import { Plugin, TAbstractFile, TFile, TFolder } from "../deps.ts";
import { Logger } from "../lib/src/common/logger.ts";
import { shouldBeIgnored } from "../lib/src/string_and_binary/path.ts";
import type { QueueProcessor } from "../lib/src/concurrency/processor.ts";
import { LOG_LEVEL_NOTICE, type FilePath, type ObsidianLiveSyncSettings } from "../lib/src/common/types.ts";
import { delay } from "../lib/src/common/utils.ts";
import { type FileEventItem, type FileEventType, type FileInfo, type InternalFileInfo } from "../common/types.ts";
export abstract class StorageEventManager {

50
src/tests/TestPane.svelte Normal file
View File

@@ -0,0 +1,50 @@
<script lang="ts">
import { onDestroy, onMount } from "svelte";
import type ObsidianLiveSyncPlugin from "../main";
import { perf_trench } from "./tests";
import { MarkdownRenderer } from "../deps";
export let plugin: ObsidianLiveSyncPlugin;
let performanceTestResult = "";
let functionCheckResult = "";
let testRunning = false;
let prefTestResultEl: HTMLDivElement;
let isReady = false;
$: {
if (performanceTestResult != "" && isReady) {
MarkdownRenderer.render(plugin.app, performanceTestResult, prefTestResultEl, "/", plugin);
}
}
async function performTest() {
try {
testRunning = true;
performanceTestResult = await perf_trench(plugin);
} finally {
testRunning = false;
}
}
function clearPerfTestResult() {
prefTestResultEl.empty();
}
onMount(() => {
isReady = true;
// performTest();
});
</script>
<h2>TESTBENCH: Self-hosted LiveSync</h2>
<h3>Function check</h3>
<pre>{functionCheckResult}</pre>
<h3>Performance test</h3>
<button on:click={() => performTest()} disabled={testRunning}>Test!</button>
<button on:click={() => clearPerfTestResult()}>Clear</button>
<div bind:this={prefTestResultEl}></div>
<style>
* {
box-sizing: border-box;
}
</style>

49
src/tests/TestPaneView.ts Normal file
View File

@@ -0,0 +1,49 @@
import {
ItemView,
WorkspaceLeaf
} from "obsidian";
import TestPaneComponent from "./TestPane.svelte"
import type ObsidianLiveSyncPlugin from "../main"
export const VIEW_TYPE_TEST = "ols-pane-test";
//Log view
export class TestPaneView extends ItemView {
component?: TestPaneComponent;
plugin: ObsidianLiveSyncPlugin;
icon = "view-log";
title: string = "Self-hosted LiveSync Test and Results"
navigation = true;
getIcon(): string {
return "view-log";
}
constructor(leaf: WorkspaceLeaf, plugin: ObsidianLiveSyncPlugin) {
super(leaf);
this.plugin = plugin;
}
getViewType() {
return VIEW_TYPE_TEST;
}
getDisplayText() {
return "Self-hosted LiveSync Test and Results";
}
// eslint-disable-next-line require-await
async onOpen() {
this.component = new TestPaneComponent({
target: this.contentEl,
props: {
plugin: this.plugin
},
});
}
// eslint-disable-next-line require-await
async onClose() {
this.component?.$destroy();
}
}

70
src/tests/tests.ts Normal file
View File

@@ -0,0 +1,70 @@
import { Trench } from "../lib/src/memory/memutil.ts";
import type ObsidianLiveSyncPlugin from "../main.ts";
type MeasureResult = [times: number, spent: number];
type NamedMeasureResult = [name: string, result: MeasureResult];
const measures = new Map<string, MeasureResult>();
function clearResult(name: string) {
measures.set(name, [0, 0]);
}
async function measureEach(name: string, proc: () => (void | Promise<void>)) {
const [times, spent] = measures.get(name) ?? [0, 0];
const start = performance.now();
const result = proc();
if (result instanceof Promise) await result;
const end = performance.now();
measures.set(name, [times + 1, spent + (end - start)]);
}
function formatNumber(num: number) {
return num.toLocaleString('en-US', { maximumFractionDigits: 2 });
}
async function measure(name: string, proc: () => (void | Promise<void>), times: number = 10000, duration: number = 1000): Promise<NamedMeasureResult> {
const from = Date.now();
let last = times;
clearResult(name);
do {
await measureEach(name, proc);
} while (last-- > 0 && (Date.now() - from) < duration)
return [name, measures.get(name) as MeasureResult];
}
// eslint-disable-next-line require-await
async function formatPerfResults(items: NamedMeasureResult[]) {
return `| Name | Runs | Each | Total |\n| --- | --- | --- | --- | \n` + items.map(e => `| ${e[0]} | ${e[1][0]} | ${e[1][0] != 0 ? formatNumber(e[1][1] / e[1][0]) : "-"} | ${formatNumber(e[1][0])} |`).join("\n");
}
export async function perf_trench(plugin: ObsidianLiveSyncPlugin) {
clearResult("trench");
const trench = new Trench(plugin.simpleStore);
const result = [] as NamedMeasureResult[];
result.push(await measure("trench-short-string", async () => {
const p = trench.evacuate("string");
await p();
}));
{
const testBinary = await plugin.vaultAccess.adapterReadBinary("testdata/10kb.png");
const uint8Array = new Uint8Array(testBinary);
result.push(await measure("trench-binary-10kb", async () => {
const p = trench.evacuate(uint8Array);
await p();
}));
}
{
const testBinary = await plugin.vaultAccess.adapterReadBinary("testdata/100kb.jpeg");
const uint8Array = new Uint8Array(testBinary);
result.push(await measure("trench-binary-100kb", async () => {
const p = trench.evacuate(uint8Array);
await p();
}));
}
{
const testBinary = await plugin.vaultAccess.adapterReadBinary("testdata/1mb.png");
const uint8Array = new Uint8Array(testBinary);
result.push(await measure("trench-binary-1mb", async () => {
const p = trench.evacuate(uint8Array);
await p();
}));
}
return formatPerfResults(result);
}

8
src/ConflictResolveModal.ts → src/ui/ConflictResolveModal.ts
View File

@@ -1,8 +1,8 @@
import { App, Modal } from "./deps";
import { App, Modal } from "../deps.ts";
import { DIFF_DELETE, DIFF_EQUAL, DIFF_INSERT } from "diff-match-patch";
import { CANCELLED, LEAVE_TO_SUBSEQUENT, RESULT_TIMED_OUT, type diff_result } from "./lib/src/types";
import { escapeStringToHTML } from "./lib/src/strbin";
import { delay, sendValue, waitForValue } from "./lib/src/utils";
import { CANCELLED, LEAVE_TO_SUBSEQUENT, RESULT_TIMED_OUT, type diff_result } from "../lib/src/common/types.ts";
import { escapeStringToHTML } from "../lib/src/string_and_binary/strbin.ts";
import { delay, sendValue, waitForValue } from "../lib/src/common/utils.ts";
export type MergeDialogResult = typeof LEAVE_TO_SUBSEQUENT | typeof CANCELLED | string;
export class ConflictResolveModal extends Modal {

18
src/DocumentHistoryModal.ts → src/ui/DocumentHistoryModal.ts
View File

@@ -1,12 +1,12 @@
import { TFile, Modal, App, DIFF_DELETE, DIFF_EQUAL, DIFF_INSERT, diff_match_patch } from "./deps";
import { getPathFromTFile, isValidPath } from "./utils";
import { decodeBinary, escapeStringToHTML, readString } from "./lib/src/strbin";
import ObsidianLiveSyncPlugin from "./main";
import { type DocumentID, type FilePathWithPrefix, type LoadedEntry, LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE } from "./lib/src/types";
import { Logger } from "./lib/src/logger";
import { isErrorOfMissingDoc } from "./lib/src/utils_couchdb";
import { getDocData, readContent } from "./lib/src/utils";
import { isPlainText, stripPrefix } from "./lib/src/path";
import { TFile, Modal, App, DIFF_DELETE, DIFF_EQUAL, DIFF_INSERT, diff_match_patch } from "../deps.ts";
import { getPathFromTFile, isValidPath } from "../common/utils.ts";
import { decodeBinary, escapeStringToHTML, readString } from "../lib/src/string_and_binary/strbin.ts";
import ObsidianLiveSyncPlugin from "../main.ts";
import { type DocumentID, type FilePathWithPrefix, type LoadedEntry, LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE } from "../lib/src/common/types.ts";
import { Logger } from "../lib/src/common/logger.ts";
import { isErrorOfMissingDoc } from "../lib/src/pouchdb/utils_couchdb.ts";
import { getDocData, readContent } from "../lib/src/common/utils.ts";
import { isPlainText, stripPrefix } from "../lib/src/string_and_binary/path.ts";
function isImage(path: string) {
const ext = path.split(".").splice(-1)[0].toLowerCase();

18
src/GlobalHistory.svelte → src/ui/GlobalHistory.svelte
View File

@@ -1,12 +1,12 @@
<script lang="ts">
import ObsidianLiveSyncPlugin from "./main";
import ObsidianLiveSyncPlugin from "../main";
import { onDestroy, onMount } from "svelte";
import type { AnyEntry, FilePathWithPrefix } from "./lib/src/types";
import { getDocData, isAnyNote, isDocContentSame, readAsBlob } from "./lib/src/utils";
import { diff_match_patch } from "./deps";
import type { AnyEntry, FilePathWithPrefix } from "../lib/src/common/types";
import { getDocData, isAnyNote, isDocContentSame, readAsBlob } from "../lib/src/common/utils";
import { diff_match_patch } from "../deps";
import { DocumentHistoryModal } from "./DocumentHistoryModal";
import { isPlainText, stripAllPrefixes } from "./lib/src/path";
import { TFile } from "./deps";
import { isPlainText, stripAllPrefixes } from "../lib/src/string_and_binary/path";
import { TFile } from "../deps";
export let plugin: ObsidianLiveSyncPlugin;
let showDiffInfo = false;
@@ -226,12 +226,18 @@
<td class="path">
<div class="filenames">
<span class="path">/{entry.dirname.split("/").join(`​/`)}</span>
<!-- svelte-ignore a11y-click-events-have-key-events -->
<!-- svelte-ignore a11y-no-static-element-interactions -->
<!-- svelte-ignore a11y-missing-attribute -->
<span class="filename"><a on:click={() => openFile(entry.path)}>{entry.filename}</a></span>
</div>
</td>
<td>
<span class="rev">
{#if entry.isPlain}
<!-- svelte-ignore a11y-click-events-have-key-events -->
<!-- svelte-ignore a11y-no-static-element-interactions -->
<!-- svelte-ignore a11y-missing-attribute -->
<a on:click={() => showHistory(entry.path, entry?.rev || "")}>{entry.rev}</a>
{:else}
{entry.rev}

14
src/GlobalHistoryView.ts → src/ui/GlobalHistoryView.ts
View File

@@ -1,18 +1,18 @@
import {
ItemView,
WorkspaceLeaf
} from "./deps";
} from "../deps.ts";
import GlobalHistoryComponent from "./GlobalHistory.svelte";
import type ObsidianLiveSyncPlugin from "./main";
import type ObsidianLiveSyncPlugin from "../main.ts";
export const VIEW_TYPE_GLOBAL_HISTORY = "global-history";
export class GlobalHistoryView extends ItemView {
component: GlobalHistoryComponent;
component?: GlobalHistoryComponent;
plugin: ObsidianLiveSyncPlugin;
icon: "clock";
title: string;
navigation: true;
icon = "clock";
title: string = "";
navigation = true;
getIcon(): string {
return "clock";
@@ -44,6 +44,6 @@ export class GlobalHistoryView extends ItemView {
// eslint-disable-next-line require-await
async onClose() {
this.component.$destroy();
this.component?.$destroy();
}
}

36
src/JsonResolveModal.ts → src/ui/JsonResolveModal.ts
View File

@@ -1,32 +1,32 @@
import { App, Modal } from "./deps";
import { type FilePath, type LoadedEntry } from "./lib/src/types";
import { App, Modal } from "../deps.ts";
import { type FilePath, type LoadedEntry } from "../lib/src/common/types.ts";
import JsonResolvePane from "./JsonResolvePane.svelte";
import { waitForSignal } from "./lib/src/utils";
import { waitForSignal } from "../lib/src/common/utils.ts";
export class JsonResolveModal extends Modal {
// result: Array<[number, string]>;
filename: FilePath;
callback: (keepRev: string, mergedStr?: string) => Promise<void>;
callback?: (keepRev?: string, mergedStr?: string) => Promise<void>;
docs: LoadedEntry[];
component: JsonResolvePane;
component?: JsonResolvePane;
nameA: string;
nameB: string;
defaultSelect: string;
constructor(app: App, filename: FilePath, docs: LoadedEntry[], callback: (keepRev: string, mergedStr?: string) => Promise<void>, nameA?: string, nameB?: string, defaultSelect?: string) {
constructor(app: App, filename: FilePath, docs: LoadedEntry[], callback: (keepRev?: string, mergedStr?: string) => Promise<void>, nameA?: string, nameB?: string, defaultSelect?: string) {
super(app);
this.callback = callback;
this.filename = filename;
this.docs = docs;
this.nameA = nameA;
this.nameB = nameB;
this.defaultSelect = defaultSelect;
this.nameA = nameA || "";
this.nameB = nameB || "";
this.defaultSelect = defaultSelect || "";
waitForSignal(`cancel-internal-conflict:${filename}`).then(() => this.close());
}
async UICallback(keepRev: string, mergedStr?: string) {
async UICallback(keepRev?: string, mergedStr?: string) {
this.close();
await this.callback(keepRev, mergedStr);
this.callback = null;
await this.callback?.(keepRev, mergedStr);
this.callback = undefined;
}
onOpen() {
@@ -34,7 +34,7 @@ export class JsonResolveModal extends Modal {
this.titleEl.setText("Conflicted Setting");
contentEl.empty();
if (this.component == null) {
if (this.component == undefined) {
this.component = new JsonResolvePane({
target: contentEl,
props: {
@@ -43,7 +43,7 @@ export class JsonResolveModal extends Modal {
nameA: this.nameA,
nameB: this.nameB,
defaultSelect: this.defaultSelect,
callback: (keepRev: string, mergedStr: string) => this.UICallback(keepRev, mergedStr),
callback: (keepRev, mergedStr) => this.UICallback(keepRev, mergedStr),
},
});
}
@@ -55,12 +55,12 @@ export class JsonResolveModal extends Modal {
const { contentEl } = this;
contentEl.empty();
// contentEl.empty();
if (this.callback != null) {
this.callback(null);
if (this.callback != undefined) {
this.callback(undefined);
}
if (this.component != null) {
if (this.component != undefined) {
this.component.$destroy();
this.component = null;
this.component = undefined;
}
}
}

10
src/JsonResolvePane.svelte → src/ui/JsonResolvePane.svelte
View File

@@ -1,9 +1,9 @@
<script lang="ts">
import { type Diff, DIFF_DELETE, DIFF_INSERT, diff_match_patch } from "./deps";
import type { FilePath, LoadedEntry } from "./lib/src/types";
import { decodeBinary, readString } from "./lib/src/strbin";
import { getDocData } from "./lib/src/utils";
import { mergeObject } from "./utils";
import { type Diff, DIFF_DELETE, DIFF_INSERT, diff_match_patch } from "../deps";
import type { FilePath, LoadedEntry } from "../lib/src/common/types";
import { decodeBinary, readString } from "../lib/src/string_and_binary/strbin";
import { getDocData } from "../lib/src/common/utils";
import { mergeObject } from "../common/utils";
export let docs: LoadedEntry[] = [];
export let callback: (keepRev?: string, mergedStr?: string) => Promise<void> = async (_, __) => {

6
src/LogPane.svelte → src/ui/LogPane.svelte
View File

@@ -1,8 +1,8 @@
<script lang="ts">
import { onDestroy, onMount } from "svelte";
import { logMessages } from "./lib/src/stores";
import type { ReactiveInstance } from "./lib/src/reactive";
import { Logger } from "./lib/src/logger";
import { logMessages } from "../lib/src/mock_and_interop/stores";
import type { ReactiveInstance } from "../lib/src/dataobject/reactive";
import { Logger } from "../lib/src/common/logger";
let unsubscribe: () => void;
let messages = [] as string[];

12
src/LogPaneView.ts → src/ui/LogPaneView.ts
View File

@@ -3,16 +3,16 @@ import {
WorkspaceLeaf
} from "obsidian";
import LogPaneComponent from "./LogPane.svelte";
import type ObsidianLiveSyncPlugin from "./main";
import type ObsidianLiveSyncPlugin from "../main.ts";
export const VIEW_TYPE_LOG = "log-log";
//Log view
export class LogPaneView extends ItemView {
component: LogPaneComponent;
component?: LogPaneComponent;
plugin: ObsidianLiveSyncPlugin;
icon: "view-log";
title: string;
navigation: true;
icon = "view-log";
title: string = "";
navigation = true;
getIcon(): string {
return "view-log";
@@ -43,6 +43,6 @@ export class LogPaneView extends ItemView {
// eslint-disable-next-line require-await
async onClose() {
this.component.$destroy();
this.component?.$destroy();
}
}

2491
src/ui/ObsidianLiveSyncSettingTab.ts Normal file
View File

File diff suppressed because it is too large Load Diff

16
src/PluginPane.svelte → src/ui/PluginPane.svelte
View File

@@ -1,12 +1,12 @@
<script lang="ts">
import { onMount } from "svelte";
import ObsidianLiveSyncPlugin from "./main";
import { type PluginDataExDisplay, pluginIsEnumerating, pluginList } from "./CmdConfigSync";
import PluginCombo from "./PluginCombo.svelte";
import ObsidianLiveSyncPlugin from "../main";
import { type PluginDataExDisplay, pluginIsEnumerating, pluginList } from "../features/CmdConfigSync";
import PluginCombo from "./components/PluginCombo.svelte";
import { Menu } from "obsidian";
import { unique } from "./lib/src/utils";
import { MODE_SELECTIVE, MODE_AUTOMATIC, MODE_PAUSED, type SYNC_MODE, type PluginSyncSettingEntry } from "./lib/src/types";
import { normalizePath } from "./deps";
import { unique } from "../lib/src/common/utils";
import { MODE_SELECTIVE, MODE_AUTOMATIC, MODE_PAUSED, type SYNC_MODE, type PluginSyncSettingEntry } from "../lib/src/common/types";
import { normalizePath } from "../deps";
export let plugin: ObsidianLiveSyncPlugin;
$: hideNotApplicable = false;
@@ -366,10 +366,10 @@
</div>
<style>
span.spacer {
/* span.spacer {
min-width: 1px;
flex-grow: 1;
}
} */
h3 {
position: sticky;
top: 0;

6
src/MultipleRegExpControl.svelte → src/ui/components/MultipleRegExpControl.svelte
View File

@@ -31,6 +31,7 @@
<ul>
{#each patterns as pattern, idx}
<!-- svelte-ignore a11y-label-has-associated-control -->
<li><label>{modified[idx]}{status[idx]}</label><input type="text" bind:value={pattern} class={modified[idx]} /><button class="iconbutton" on:click={() => remove(idx)}>🗑</button></li>
{/each}
<li>
@@ -72,12 +73,7 @@
li input {
min-width: 10em;
}
li.buttons {
}
button.iconbutton {
max-width: 4em;
}
span.spacer {
flex-grow: 1;
}
</style>

14
src/PluginCombo.svelte → src/ui/components/PluginCombo.svelte
View File

@@ -1,11 +1,11 @@
<script lang="ts">
import type { PluginDataExDisplay } from "./CmdConfigSync";
import { Logger } from "./lib/src/logger";
import { versionNumberString2Number } from "./lib/src/strbin";
import { type FilePath, LOG_LEVEL_NOTICE } from "./lib/src/types";
import { getDocData } from "./lib/src/utils";
import type ObsidianLiveSyncPlugin from "./main";
import { askString, scheduleTask } from "./utils";
import type { PluginDataExDisplay } from "../../features/CmdConfigSync";
import { Logger } from "../../lib/src/common/logger";
import { versionNumberString2Number } from "../../lib/src/string_and_binary/strbin";
import { type FilePath, LOG_LEVEL_NOTICE } from "../../lib/src/common/types";
import { getDocData } from "../../lib/src/common/utils";
import type ObsidianLiveSyncPlugin from "../../main";
import { askString, scheduleTask } from "../../common/utils";
export let list: PluginDataExDisplay[] = [];
export let thisTerm = "";

331
src/ui/settingConstants.ts Normal file
View File

@@ -0,0 +1,331 @@
import { $t } from "src/lib/src/common/i18n";
import { DEFAULT_SETTINGS, configurationNames, type ConfigurationItem, type FilterBooleanKeys, type FilterNumberKeys, type FilterStringKeys, type ObsidianLiveSyncSettings } from "src/lib/src/common/types";
export type OnDialogSettings = {
configPassphrase: string,
preset: "" | "PERIODIC" | "LIVESYNC" | "DISABLE",
syncMode: "ONEVENTS" | "PERIODIC" | "LIVESYNC"
dummy: number,
deviceAndVaultName: string,
}
export const OnDialogSettingsDefault: OnDialogSettings = {
configPassphrase: "",
preset: "",
syncMode: "ONEVENTS",
dummy: 0,
deviceAndVaultName: "",
}
export const AllSettingDefault =
{ ...DEFAULT_SETTINGS, ...OnDialogSettingsDefault }
export type AllSettings = ObsidianLiveSyncSettings & OnDialogSettings;
export type AllStringItemKey = FilterStringKeys<AllSettings>;
export type AllNumericItemKey = FilterNumberKeys<AllSettings>;
export type AllBooleanItemKey = FilterBooleanKeys<AllSettings>;
export type AllSettingItemKey = AllStringItemKey | AllNumericItemKey | AllBooleanItemKey;
export type ValueOf<T extends AllSettingItemKey> =
T extends AllStringItemKey ? string :
T extends AllNumericItemKey ? number :
T extends AllBooleanItemKey ? boolean :
AllSettings[T];
export const SettingInformation: Partial<Record<keyof AllSettings, ConfigurationItem>> = {
"liveSync": {
"name": "Sync Mode"
},
"couchDB_URI": {
"name": "URI",
"placeHolder": "https://........"
},
"couchDB_USER": {
"name": "Username",
"desc": "username"
},
"couchDB_PASSWORD": {
"name": "Password",
"desc": "password"
},
"couchDB_DBNAME": {
"name": "Database name"
},
"passphrase": {
"name": "Passphrase",
"desc": "Encrypting passphrase. If you change the passphrase of an existing database, overwriting the remote database is strongly recommended."
},
"showStatusOnEditor": {
"name": "Show status inside the editor",
"desc": "Reflected after reboot"
},
"showOnlyIconsOnEditor": {
"name": "Show status as icons only"
},
"showStatusOnStatusbar": {
"name": "Show status on the status bar",
"desc": "Reflected after reboot."
},
"lessInformationInLog": {
"name": "Show only notifications",
"desc": "Prevent logging and show only notification"
},
"showVerboseLog": {
"name": "Verbose Log",
"desc": "Show verbose log"
},
"hashCacheMaxCount": {
"name": "Memory cache size (by total items)"
},
"hashCacheMaxAmount": {
"name": "Memory cache size (by total characters)",
"desc": "(Mega chars)"
},
"writeCredentialsForSettingSync": {
"name": "Write credentials in the file",
"desc": "(Not recommended) If set, credentials will be stored in the file."
},
"notifyAllSettingSyncFile": {
"name": "Notify all setting files"
},
"configPassphrase": {
"name": "Passphrase of sensitive configuration items",
"desc": "This passphrase will not be copied to another device. It will be set to `Default` until you configure it again."
},
"configPassphraseStore": {
"name": "Encrypting sensitive configuration items"
},
"syncOnSave": {
"name": "Sync on Save",
"desc": "When you save a file, sync automatically"
},
"syncOnEditorSave": {
"name": "Sync on Editor Save",
"desc": "When you save a file in the editor, sync automatically"
},
"syncOnFileOpen": {
"name": "Sync on File Open",
"desc": "When you open a file, sync automatically"
},
"syncOnStart": {
"name": "Sync on Start",
"desc": "Start synchronization after launching Obsidian."
},
"syncAfterMerge": {
"name": "Sync after merging file",
"desc": "Sync automatically after merging files"
},
"trashInsteadDelete": {
"name": "Use the trash bin",
"desc": "Do not delete files that are deleted in remote, just move to trash."
},
"doNotDeleteFolder": {
"name": "Keep empty folder",
"desc": "Normally, a folder is deleted when it becomes empty after a synchronization. Enabling this will prevent it from getting deleted"
},
"resolveConflictsByNewerFile": {
"name": "Always overwrite with a newer file (beta)",
"desc": "(Def off) Resolve conflicts by newer files automatically."
},
"checkConflictOnlyOnOpen": {
"name": "Postpone resolution of inactive files"
},
"showMergeDialogOnlyOnActive": {
"name": "Postpone manual resolution of inactive files"
},
"disableMarkdownAutoMerge": {
"name": "Always resolve conflicts manually",
"desc": "If this switch is turned on, a merge dialog will be displayed, even if the sensible-merge is possible automatically. (Turn on to previous behavior)"
},
"writeDocumentsIfConflicted": {
"name": "Always reflect synchronized changes even if the note has a conflict",
"desc": "Turn on to previous behavior"
},
"syncInternalFilesInterval": {
"name": "Scan hidden files periodically",
"desc": "Seconds, 0 to disable"
},
"batchSave": {
"name": "Batch database update",
"desc": "Reducing the frequency with which on-disk changes are reflected into the DB"
},
"readChunksOnline": {
"name": "Fetch chunks on demand",
"desc": "(ex. Read chunks online) If this option is enabled, LiveSync reads chunks online directly instead of replicating them locally. Increasing Custom chunk size is recommended."
},
"syncMaxSizeInMB": {
"name": "Maximum file size",
"desc": "(MB) If this is set, changes to local and remote files that are larger than this will be skipped. If the file becomes smaller again, a newer one will be used."
},
"useIgnoreFiles": {
"name": "(Beta) Use ignore files",
"desc": "If this is set, changes to local files which are matched by the ignore files will be skipped. Remote changes are determined using local ignore files."
},
"ignoreFiles": {
"name": "Ignore files",
"desc": "We can use multiple ignore files, e.g.) `.gitignore, .dockerignore`"
},
"batch_size": {
"name": "Batch size",
"desc": "Number of change feed items to process at a time. Defaults to 50. Minimum is 2."
},
"batches_limit": {
"name": "Batch limit",
"desc": "Number of batches to process at a time. Defaults to 40. Minimum is 2. This along with batch size controls how many docs are kept in memory at a time."
},
"useTimeouts": {
"name": "Use timeouts instead of heartbeats",
"desc": "If this option is enabled, PouchDB will hold the connection open for 60 seconds, and if no change arrives in that time, close and reopen the socket, instead of holding it open indefinitely. Useful when a proxy limits request duration but can increase resource usage."
},
"concurrencyOfReadChunksOnline": {
"name": "Batch size of on-demand fetching"
},
"minimumIntervalOfReadChunksOnline": {
"name": "The delay for consecutive on-demand fetches"
},
"suspendFileWatching": {
"name": "Suspend file watching",
"desc": "Stop watching for file change."
},
"suspendParseReplicationResult": {
"name": "Suspend database reflecting",
"desc": "Stop reflecting database changes to storage files."
},
"writeLogToTheFile": {
"name": "Write logs into the file",
"desc": "Warning! This will have a serious impact on performance. And the logs will not be synchronised under the default name. Please be careful with logs; they often contain your confidential information."
},
"deleteMetadataOfDeletedFiles": {
"name": "Do not keep metadata of deleted files."
},
"useIndexedDBAdapter": {
"name": "Use an old adapter for compatibility",
"desc": "Before v0.17.16, we used an old adapter for the local database. Now the new adapter is preferred. However, it needs local database rebuilding. Please disable this toggle when you have enough time. If leave it enabled, also while fetching from the remote database, you will be asked to disable this."
},
"watchInternalFileChanges": {
"name": "Scan changes on customization sync",
"desc": "Do not use internal API"
},
"doNotSuspendOnFetching": {
"name": "Fetch database with previous behaviour"
},
"disableCheckingConfigMismatch": {
"name": "Do not check configuration mismatch before replication"
},
"usePluginSync": {
"name": "Enable customization sync"
},
"autoSweepPlugins": {
"name": "Scan customization automatically",
"desc": "Scan customization before replicating."
},
"autoSweepPluginsPeriodic": {
"name": "Scan customization periodically",
"desc": "Scan customization every 1 minute."
},
"notifyPluginOrSettingUpdated": {
"name": "Notify customized",
"desc": "Notify when other device has newly customized."
},
"remoteType": {
"name": "Remote Type",
"desc": "Remote server type"
},
"endpoint": {
"name": "Endpoint URL",
"placeHolder": "https://........"
},
"accessKey": {
"name": "Access Key"
},
"secretKey": {
"name": "Secret Key"
},
"region": {
"name": "Region",
"placeHolder": "auto"
},
"bucket": {
"name": "Bucket Name"
},
"useCustomRequestHandler": {
"name": "Use Custom HTTP Handler",
"desc": "If your Object Storage could not configured accepting CORS, enable this."
},
"maxChunksInEden": {
"name": "Maximum Incubating Chunks",
"desc": "The maximum number of chunks that can be incubated within the document. Chunks exceeding this number will immediately graduate to independent chunks."
},
"maxTotalLengthInEden": {
"name": "Maximum Incubating Chunk Size",
"desc": "The maximum total size of chunks that can be incubated within the document. Chunks exceeding this size will immediately graduate to independent chunks."
},
"maxAgeInEden": {
"name": "Maximum Incubation Period",
"desc": "The maximum duration for which chunks can be incubated within the document. Chunks exceeding this period will graduate to independent chunks."
},
"settingSyncFile": {
"name": "Filename",
"desc": "If you set this, all settings are saved in a markdown file. You will be notified when new settings arrive. You can set different files by the platform."
},
"preset": {
"name": "Presets",
"desc": "Apply preset configuration"
},
"syncMode": {
name: "Sync Mode",
},
"periodicReplicationInterval": {
"name": "Periodic Sync interval",
"desc": "Interval (sec)"
},
"syncInternalFilesBeforeReplication": {
"name": "Scan for hidden files before replication"
},
"automaticallyDeleteMetadataOfDeletedFiles": {
"name": "Delete old metadata of deleted files on start-up",
"desc": "(Days passed, 0 to disable automatic-deletion)"
},
"additionalSuffixOfDatabaseName": {
"name": "Database suffix",
"desc": "LiveSync could not handle multiple vaults which have same name without different prefix, This should be automatically configured."
},
"hashAlg": {
"name": configurationNames["hashAlg"]?.name || "",
"desc": "xxhash64 is the current default."
},
"deviceAndVaultName": {
"name": "Device name",
"desc": "Unique name between all synchronized devices. To edit this setting, please disable customization sync once."
},
"displayLanguage": {
"name": "Display Language",
"desc": "Not all messages have been translated. And, please revert to \"Default\" when reporting errors."
}
}
function translateInfo(infoSrc: ConfigurationItem | undefined | false) {
if (!infoSrc) return false;
const info = { ...infoSrc };
info.name = $t(info.name);
if (info.desc) {
info.desc = $t(info.desc);
}
return info;
}
function _getConfig(key: AllSettingItemKey) {
if (key in configurationNames) {
return configurationNames[key as keyof ObsidianLiveSyncSettings];
}
if (key in SettingInformation) {
return SettingInformation[key as keyof ObsidianLiveSyncSettings];
}
return false;
}
export function getConfig(key: AllSettingItemKey) {
return translateInfo(_getConfig(key));
}
export function getConfName(key: AllSettingItemKey) {
const conf = getConfig(key);
if (!conf) return `${key} (No info)`;
return conf.name;
}

17
styles.css
View File

@@ -133,6 +133,7 @@
top: var(--view-header-height);
right: 1em;
}
.canvas-wrapper::before {
right: 48px;
}
@@ -270,6 +271,22 @@ div.sls-setting-menu-btn {
content: "✏";
}
.sls-item-dirty-help::after {
content: " ❓";
}
.sls-item-invalid-value {
background-color: rgba(var(--background-modifier-error-rgb), 0.3) !important;
}
.sls-setting-disabled input[type=text],
.sls-setting-disabled input[type=number],
.sls-setting-disabled input[type=password] {
filter: brightness(80%);
color: var(--text-muted);
}
.sls-setting-hidden {
display: none;
}

52
updates.md
View File

@@ -18,26 +18,36 @@ I have a lot of respect for that plugin, even though it is sometimes treated as
Hooray for open source, and generous licences, and the sharing of knowledge by experts.
#### Version history
- 0.23.3
- Fixed: No longer unwanted `\f` in journal sync.
- 0.23.2
- Sorry for all the fixes to experimental features. (These things were also critical for dogfooding). The next release would be the main fixes! Thank you for your patience and understanding!
- 0.23.9
- Fixed:
- Journal Sync will not hang up during big replication, especially the initial one.
- All changes which have been replicated while rebuilding will not be postponed (Previous behaviour).
- Improved:
- Now Journal Sync works efficiently in download and parse, or pack and upload.
- Less server storage and faster packing/unpacking usage by the new chunk format.
- 0.23.1
- Fixed:
- Now journal synchronisation considers untransferred each from sent and received.
- Journal sync now handles retrying.
- Journal synchronisation no longer considers the synchronisation of chunks as revision updates (Simply ignored).
- Journal sync now splits the journal pack to prevent mobile device rebooting.
- Maintenance menus which had been on the command palette are now back in the maintain pane on the setting dialogue.
- Improved:
- Now all changes which have been replicated while rebuilding will be postponed.
- 0.23.0
- No longer unexpected parallel replication is performed.
- Now we can set the device name and enable customised synchronisation again.
- 0.23.8
- New feature:
- Now we can use Object Storage.
- Now we are ready for i18n.
- Patch or PR of `rosetta.ts` are welcome!
- The setting dialogue has been refined. Very controllable, clearly displayed disabled items, and ready to i18n.
- Fixed:
- Many memory leaks have been rescued.
- Chunk caches now work well.
- Many trivial but potential bugs are fixed.
- No longer error messages will be shown on retrieving checkpoint or server information.
- Now we can check and correct tweak mismatch during the setup
- Improved:
- Customisation synchronisation has got more smoother.
- Tidied
- Practically unused functions have been removed or are being prepared for removal.
- Many of the type-errors and lint errors have been corrected.
- Unused files have been removed.
- Note:
- From this version, some test files have been included. However, they are not enabled and released in the release build.
- To try them, please run Self-hosted LiveSync in the dev build.
- 0.23.7
- Fixed:
- No longer missing tasks which have queued as the same key (e.g., for the same operation to the same file).
- This occurs, for example, with hidden files that have been changed multiple times in a very short period of time, such as `appearance.json`. Thanks for the report!
- Some trivial issues have been fixed.
- New feature:
- Reloading Obsidian can be scheduled until that file and database operations are stable.
Older notes is in [updates_old.md](https://github.com/vrtmrz/obsidian-livesync/blob/main/updates_old.md).

69
updates_old.md
View File

@@ -1,3 +1,72 @@
### 0.23.0
Incredibly new features!
Now, we can use object storage (MinIO, S3, R2 or anything you like) for synchronising! Moreover, despite that, we can use all the features as if we were using CouchDB.
Note: As this is a pretty experimental feature, hence we have some limitations.
- This is built on the append-only architecture. It will not shrink used storage if we do not perform a rebuild.
- A bit fragile. However, our version x.yy.0 is always so.
- When the first synchronisation, the entire history to date is transferred. For this reason, it is preferable to do this under the WiFi network.
- Do not worry, from the second synchronisation, we always transfer only differences.
I hope this feature empowers users to maintain independence and self-host their data, offering an alternative for those who prefer to manage their own storage solutions and avoid being stuck on the right side of a sudden change in business model.
Of course, I use Self-hosted MinIO for testing and recommend this. It is for the same reason as using CouchDB. -- open, controllable, auditable and indeed already audited by numerous eyes.
Let me write one more acknowledgement.
I have a lot of respect for that plugin, even though it is sometimes treated as if it is a competitor, remotely-save. I think it is a great architecture that embodies a different approach to my approach of recreating history. This time, with all due respect, I have used some of its code as a reference.
Hooray for open source, and generous licences, and the sharing of knowledge by experts.
#### Version history
- 0.23.6:
- Fixed:
- Now the remote chunks could be decrypted even if we are using `Incubate chunks in Document`. (The note of 0.23.6 has been fixed).
- Chunk retrieving with `Incubate chunks in document` got more efficiently.
- No longer task processor misses the completed tasks.
- Replication is no longer started automatically during changes in window visibility (e.g., task switching on the desktop) when off-focused.
- 0.23.5:
- New feature:
- Now we can check configuration mismatching between clients before synchronisation.
- Default: enabled / Preferred: enabled / We can disable this by the `Do not check configuration mismatch before replication` toggle in the `Hatch` pane.
- It detects configuration mismatches and prevents synchronisation failures and wasted storage.
- Now we can perform remote database compaction from the `Maintenance` pane.
- Fixed:
- We can detect the bucket could not be reachable.
- Note:
- Known inexplicable behaviour: Recently, (Maybe while enabling `Incubate chunks in Document` and `Fetch chunks on demand` or some more toggles), our customisation sync data is sometimes corrupted. It will be addressed by the next release.
- 0.23.4
- Fixed:
- No longer experimental configuration is shown on the Minimal Setup.
- New feature:
- We can now use `Incubate Chunks in Document` to reduce non-well-formed chunks.
- Default: disabled / Preferred: enabled in all devices.
- When we enabled this toggle, newly created chunks are temporarily kept within the document, and graduated to become independent chunks once stabilised.
- The [design document](https://github.com/vrtmrz/obsidian-livesync/blob/3925052f9290b3579e45a4b716b3679c833d8ca0/docs/design_docs_of_keep_newborn_chunks.md) has been also available..
- 0.23.3
- Fixed: No longer unwanted `\f` in journal sync.
- 0.23.2
- Sorry for all the fixes to experimental features. (These things were also critical for dogfooding). The next release would be the main fixes! Thank you for your patience and understanding!
- Fixed:
- Journal Sync will not hang up during big replication, especially the initial one.
- All changes which have been replicated while rebuilding will not be postponed (Previous behaviour).
- Improved:
- Now Journal Sync works efficiently in download and parse, or pack and upload.
- Less server storage and faster packing/unpacking usage by the new chunk format.
- 0.23.1
- Fixed:
- Now journal synchronisation considers untransferred each from sent and received.
- Journal sync now handles retrying.
- Journal synchronisation no longer considers the synchronisation of chunks as revision updates (Simply ignored).
- Journal sync now splits the journal pack to prevent mobile device rebooting.
- Maintenance menus which had been on the command palette are now back in the maintain pane on the setting dialogue.
- Improved:
- Now all changes which have been replicated while rebuilding will be postponed.
- 0.23.0
- New feature:
- Now we can use Object Storage.
### 0.22.0
A few years passed since Self-hosted LiveSync was born, and our codebase had been very complicated. This could be patient now, but it should be a tremendous hurt.
Therefore at v0.22.0, for future maintainability, I refined task scheduling logic totally.