public/obsidian-livesync
1
0
Fork 0
mirror of https://github.com/vrtmrz/obsidian-livesync.git synced 2026-02-22 20:18:48 +00:00
Code Issues Packages Projects Releases Wiki Activity

Compare commits

base: public:0.19.4
Branches Tags
public:main public:beta public:snyk-upgrade-7fd31e6f474568014502e7205847f257 public:snyk-fix-90f3b71450f7543a1c61724d8afa0e71 public:refactor_services_2 public:refactor_services public:dev public:snyk-upgrade-fe9625385aa4be849c7157033fd2abb0 public:snyk-upgrade-99a31e70c822894595544b0c9813b11a public:svelteui public:disenchant public:snyk-fix-61f76ef0bce2a276df9552c2fc80a14f public:snyk-upgrade-2baa7938ba318464c3bad5c3fd5cbda5 public:snyk-upgrade-7af62a575eab20e8fb4e9565bf1adcfc public:snyk-upgrade-b32f281b31a81483f126b8fb6c54a957 public:snyk-upgrade-bd61b1103442d3faecc9fc726c34e9a4 public:snyk-upgrade-5b2d6584430c061b5187611a5d7fb64a public:snyk-fix-0e9fbbcb374fcb8d5e8cb027d8414fb7 public:snyk-upgrade-5242daf91e706d1ae500c9ea6545636e public:snyk-upgrade-70434a6822fbad83424135dfad9df909 public:snyk-upgrade-8fa01ccad0352125f1866d41e57d35e1 public:snyk-upgrade-39eb97b4655e70091f718a88fccd57c1 public:snyk-upgrade-2b99d54d19e207887e36c3071b796925 public:forprivacy public:snyk-upgrade-dad9088d0a333677e0157e3deef7fc38 public:snyk-upgrade-b9a76d74b3e5723759af5b3319aabb80 public:snyk-upgrade-7da6459d8ff5d293d80de9d80061d912 public:snyk-upgrade-6a3786fc4cbe394542e9d1b6058c46e7 public:snyk-upgrade-8fc9e62771dc41b0f23dc0efefb5c949
public:0.25.43-patched-9 public:0.25.43-patched-8 public:0.25.43-patched-7 public:0.25.43-patched-6 public:0.25.43-patched-5 public:0.25.43-patched-4 public:0.25.43-patched-3 public:0.25.43-patched-2 public:0.25.43-patched-1 public:0.25.43 public:0.25.42 public:0.25.41-patched-3 public:0.25.41-patched-2 public:0.25.41-patched-1 public:0.25.41 public:0.25.40 public:0.25.39 public:0.25.38 public:0.25.37 public:0.25.36 public:0.25.35 public:0.25.34 public:0.25.33 public:0.25.32 public:0.25.31 public:0.25.30 public:0.25.30-dev public:0.25.29 public:0.25.28 public:0.25.27 public:0.25.26 public:0.25.25 public:0.25.24 public:0.25.24.beta3 public:0.25.24.beta2 public:0.25.24.beta1 public:0.25.23 public:0.25.23.beta1 public:0.25.22 public:0.25.21 public:0.25.21.beta2 public:0.25.21.beta1 public:0.25.20 public:0.25.19 public:0.25.18 public:0.25.17 public:0.25.16 public:0.25.15 public:0.25.14 public:0.25.13 public:0.25.12 public:0.25.11 public:0.25.10 public:0.25.9 public:0.25.8 public:0.25.7 public:0.25.6 public:0.25.5 public:0.25.4 public:0.25.3 public:0.25.2 public:0.25.1 public:0.25.0 public:0.25.0-beta1 public:0.24.31 public:0.24.30 public:0.24.29 public:0.24.28 public:0.24.27 public:0.24.26 public:0.24.25 public:0.24.24 public:0.24.23 public:0.24.22 public:0.24.21 public:0.24.20 public:0.24.19 public:0.24.18 public:0.24.17 public:0.24.16 public:0.24.15 public:0.24.14 public:0.24.13 public:0.24.12 public:0.24.11 public:0.24.10 public:0.24.9 public:0.24.8 public:0.24.7 public:0.24.6 public:0.24.5 public:0.24.4 public:0.24.3 public:0.24.2 public:0.24.1 public:0.24.0 public:0.24.0.dev-rc8 public:0.24.0.dev-rc7 public:0.24.0.dev-rc6 public:0.24.0.dev-rc5 public:0.24.0.dev-rc4 public:0.24.0.dev-rc3 public:0.24.0.dev-rc2 public:0.23.23 public:0.23.22 public:0.23.21 public:0.23.20 public:0.23.19 public:0.23.18 public:0.23.17 public:0.23.16 public:0.23.15 public:0.23.14 public:0.23.13 public:0.23.12 public:0.23.11 public:0.23.10 public:0.23.9 public:0.23.8 public:0.23.7 public:0.23.6 public:0.23.5 public:0.23.4 public:0.23.3 public:0.23.2 public:0.23.1 public:0.23.0 public:0.22.19 public:0.22.18 public:0.22.17 public:0.22.16 public:0.22.15 public:0.22.14 public:0.22.13 public:0.22.12 public:0.22.11 public:0.22.10 public:0.22.9 public:0.22.8 public:0.22.7 public:0.22.6 public:0.22.5 public:0.22.4 public:0.22.3 public:0.22.2 public:0.22.1 public:0.22.0 public:0.21.5 public:0.21.4 public:0.21.3 public:0.21.2 public:0.21.1 public:0.21.0 public:0.20.7 public:0.20.6 public:0.20.5 public:0.20.4 public:0.20.3 public:0.20.2 public:0.20.1 public:0.20.0 public:0.19.23 public:0.19.22 public:0.19.21 public:0.19.20 public:0.19.19 public:0.19.18 public:0.19.17 public:0.19.16 public:0.19.15 public:0.19.14 public:0.19.13 public:0.19.12 public:0.19.11 public:0.19.10 public:0.19.9 public:0.19.8 public:0.19.7 public:0.19.6 public:0.19.5 public:0.19.4 public:0.19.3 public:0.19.2 public:0.19.1 public:0.19.0 public:0.18.6 public:0.18.5 public:0.18.4 public:0.18.3 public:0.18.2 public:0.18.1 public:0.18.0 public:0.17.34 public:0.17.33 public:0.17.32 public:0.17.31 public:0.17.30 public:0.17.29 public:0.17.28 public:0.17.27 public:0.17.26 public:0.17.25 public:0.17.24 public:0.17.23 public:0.17.22 public:0.17.21 public:0.17.20 public:0.17.19 public:0.17.18 public:0.17.17 public:0.17.16 public:0.17.15 public:0.17.14 public:0.17.13 public:0.17.12 public:0.17.11 public:0.17.10 public:0.17.9 public:0.17.8 public:0.17.7 public:0.17.6 public:0.17.5 public:0.17.4 public:0.17.3 public:0.17.2 public:0.17.1 public:0.17.0 public:0.16.8 public:0.16.7 public:0.16.6 public:0.16.5 public:0.16.4 public:0.16.3 public:0.16.2 public:0.16.1 public:0.16.0 public:0.15.10 public:0.15.9 public:0.15.8 public:0.15.7 public:0.15.6 public:0.15.5 public:0.15.4 public:0.15.3 public:0.15.2 public:0.15.1 public:0.15.0 public:0.14.8 public:0.14.7 public:0.14.6 public:0.14.5 public:0.14.4 public:0.14.3 public:0.14.2 public:0.14.1 public:0.14.0 public:0.13.4 public:0.13.3 public:0.13.2 public:0.13.1 public:0.13.0 public:0.12.5 public:0.12.4 public:0.12.3 public:0.12.2 public:0.12.1 public:0.12.0 public:0.11.10 public:0.11.9 public:0.11.8 public:0.11.7 public:0.11.6 public:0.11.5 public:0.11.4 public:0.11.3 public:0.11.2 public:0.11.1 public:0.11.0 public:0.10.1 public:0.10.0 public:0.9.0 public:0.8.8 public:0.8.7 public:0.8.6 public:0.8.5 public:0.8.3 public:0.8.2 public:0.8.1 public:0.8.0 public:0.7.2 public:0.7.1 public:0.7.0 public:0.6.1 public:0.6.0 public:0.5.0 public:0.4.1 public:0.4.0 public:0.3.1 public:0.3.0 public:0.2.1 public:0.2.0 public:0.1.26 public:0.1.25 public:0.1.24 public:0.1.23 public:0.1.22 public:0.1.21 public:0.1.20 public:0.1.19 public:0.1.18 public:0.1.17 public:0.1.16 public:0.1.15 public:0.1.14 public:0.1.13 public:0.1.12 public:0.1.11 public:0.1.10 public:0.1.9 public:0.1.8 public:0.1.7 public:0.1.6 public:0.1.5 public:0.1.4 public:0.1.3 public:0.1.2 public:0.1.1 public:0.1.0 public:0.0.9 public:0.0.8 public:0.0.7 public:0.0.6
...
compare: public:0.21.2
Branches Tags
public:beta public:main public:snyk-upgrade-7fd31e6f474568014502e7205847f257 public:snyk-fix-90f3b71450f7543a1c61724d8afa0e71 public:refactor_services_2 public:refactor_services public:dev public:snyk-upgrade-fe9625385aa4be849c7157033fd2abb0 public:snyk-upgrade-99a31e70c822894595544b0c9813b11a public:svelteui public:disenchant public:snyk-fix-61f76ef0bce2a276df9552c2fc80a14f public:snyk-upgrade-2baa7938ba318464c3bad5c3fd5cbda5 public:snyk-upgrade-7af62a575eab20e8fb4e9565bf1adcfc public:snyk-upgrade-b32f281b31a81483f126b8fb6c54a957 public:snyk-upgrade-bd61b1103442d3faecc9fc726c34e9a4 public:snyk-upgrade-5b2d6584430c061b5187611a5d7fb64a public:snyk-fix-0e9fbbcb374fcb8d5e8cb027d8414fb7 public:snyk-upgrade-5242daf91e706d1ae500c9ea6545636e public:snyk-upgrade-70434a6822fbad83424135dfad9df909 public:snyk-upgrade-8fa01ccad0352125f1866d41e57d35e1 public:snyk-upgrade-39eb97b4655e70091f718a88fccd57c1 public:snyk-upgrade-2b99d54d19e207887e36c3071b796925 public:forprivacy public:snyk-upgrade-dad9088d0a333677e0157e3deef7fc38 public:snyk-upgrade-b9a76d74b3e5723759af5b3319aabb80 public:snyk-upgrade-7da6459d8ff5d293d80de9d80061d912 public:snyk-upgrade-6a3786fc4cbe394542e9d1b6058c46e7 public:snyk-upgrade-8fc9e62771dc41b0f23dc0efefb5c949
public:0.25.43-patched-9 public:0.25.43-patched-8 public:0.25.43-patched-7 public:0.25.43-patched-6 public:0.25.43-patched-5 public:0.25.43-patched-4 public:0.25.43-patched-3 public:0.25.43-patched-2 public:0.25.43-patched-1 public:0.25.43 public:0.25.42 public:0.25.41-patched-3 public:0.25.41-patched-2 public:0.25.41-patched-1 public:0.25.41 public:0.25.40 public:0.25.39 public:0.25.38 public:0.25.37 public:0.25.36 public:0.25.35 public:0.25.34 public:0.25.33 public:0.25.32 public:0.25.31 public:0.25.30 public:0.25.30-dev public:0.25.29 public:0.25.28 public:0.25.27 public:0.25.26 public:0.25.25 public:0.25.24 public:0.25.24.beta3 public:0.25.24.beta2 public:0.25.24.beta1 public:0.25.23 public:0.25.23.beta1 public:0.25.22 public:0.25.21 public:0.25.21.beta2 public:0.25.21.beta1 public:0.25.20 public:0.25.19 public:0.25.18 public:0.25.17 public:0.25.16 public:0.25.15 public:0.25.14 public:0.25.13 public:0.25.12 public:0.25.11 public:0.25.10 public:0.25.9 public:0.25.8 public:0.25.7 public:0.25.6 public:0.25.5 public:0.25.4 public:0.25.3 public:0.25.2 public:0.25.1 public:0.25.0 public:0.25.0-beta1 public:0.24.31 public:0.24.30 public:0.24.29 public:0.24.28 public:0.24.27 public:0.24.26 public:0.24.25 public:0.24.24 public:0.24.23 public:0.24.22 public:0.24.21 public:0.24.20 public:0.24.19 public:0.24.18 public:0.24.17 public:0.24.16 public:0.24.15 public:0.24.14 public:0.24.13 public:0.24.12 public:0.24.11 public:0.24.10 public:0.24.9 public:0.24.8 public:0.24.7 public:0.24.6 public:0.24.5 public:0.24.4 public:0.24.3 public:0.24.2 public:0.24.1 public:0.24.0 public:0.24.0.dev-rc8 public:0.24.0.dev-rc7 public:0.24.0.dev-rc6 public:0.24.0.dev-rc5 public:0.24.0.dev-rc4 public:0.24.0.dev-rc3 public:0.24.0.dev-rc2 public:0.23.23 public:0.23.22 public:0.23.21 public:0.23.20 public:0.23.19 public:0.23.18 public:0.23.17 public:0.23.16 public:0.23.15 public:0.23.14 public:0.23.13 public:0.23.12 public:0.23.11 public:0.23.10 public:0.23.9 public:0.23.8 public:0.23.7 public:0.23.6 public:0.23.5 public:0.23.4 public:0.23.3 public:0.23.2 public:0.23.1 public:0.23.0 public:0.22.19 public:0.22.18 public:0.22.17 public:0.22.16 public:0.22.15 public:0.22.14 public:0.22.13 public:0.22.12 public:0.22.11 public:0.22.10 public:0.22.9 public:0.22.8 public:0.22.7 public:0.22.6 public:0.22.5 public:0.22.4 public:0.22.3 public:0.22.2 public:0.22.1 public:0.22.0 public:0.21.5 public:0.21.4 public:0.21.3 public:0.21.2 public:0.21.1 public:0.21.0 public:0.20.7 public:0.20.6 public:0.20.5 public:0.20.4 public:0.20.3 public:0.20.2 public:0.20.1 public:0.20.0 public:0.19.23 public:0.19.22 public:0.19.21 public:0.19.20 public:0.19.19 public:0.19.18 public:0.19.17 public:0.19.16 public:0.19.15 public:0.19.14 public:0.19.13 public:0.19.12 public:0.19.11 public:0.19.10 public:0.19.9 public:0.19.8 public:0.19.7 public:0.19.6 public:0.19.5 public:0.19.4 public:0.19.3 public:0.19.2 public:0.19.1 public:0.19.0 public:0.18.6 public:0.18.5 public:0.18.4 public:0.18.3 public:0.18.2 public:0.18.1 public:0.18.0 public:0.17.34 public:0.17.33 public:0.17.32 public:0.17.31 public:0.17.30 public:0.17.29 public:0.17.28 public:0.17.27 public:0.17.26 public:0.17.25 public:0.17.24 public:0.17.23 public:0.17.22 public:0.17.21 public:0.17.20 public:0.17.19 public:0.17.18 public:0.17.17 public:0.17.16 public:0.17.15 public:0.17.14 public:0.17.13 public:0.17.12 public:0.17.11 public:0.17.10 public:0.17.9 public:0.17.8 public:0.17.7 public:0.17.6 public:0.17.5 public:0.17.4 public:0.17.3 public:0.17.2 public:0.17.1 public:0.17.0 public:0.16.8 public:0.16.7 public:0.16.6 public:0.16.5 public:0.16.4 public:0.16.3 public:0.16.2 public:0.16.1 public:0.16.0 public:0.15.10 public:0.15.9 public:0.15.8 public:0.15.7 public:0.15.6 public:0.15.5 public:0.15.4 public:0.15.3 public:0.15.2 public:0.15.1 public:0.15.0 public:0.14.8 public:0.14.7 public:0.14.6 public:0.14.5 public:0.14.4 public:0.14.3 public:0.14.2 public:0.14.1 public:0.14.0 public:0.13.4 public:0.13.3 public:0.13.2 public:0.13.1 public:0.13.0 public:0.12.5 public:0.12.4 public:0.12.3 public:0.12.2 public:0.12.1 public:0.12.0 public:0.11.10 public:0.11.9 public:0.11.8 public:0.11.7 public:0.11.6 public:0.11.5 public:0.11.4 public:0.11.3 public:0.11.2 public:0.11.1 public:0.11.0 public:0.10.1 public:0.10.0 public:0.9.0 public:0.8.8 public:0.8.7 public:0.8.6 public:0.8.5 public:0.8.3 public:0.8.2 public:0.8.1 public:0.8.0 public:0.7.2 public:0.7.1 public:0.7.0 public:0.6.1 public:0.6.0 public:0.5.0 public:0.4.1 public:0.4.0 public:0.3.1 public:0.3.0 public:0.2.1 public:0.2.0 public:0.1.26 public:0.1.25 public:0.1.24 public:0.1.23 public:0.1.22 public:0.1.21 public:0.1.20 public:0.1.19 public:0.1.18 public:0.1.17 public:0.1.16 public:0.1.15 public:0.1.14 public:0.1.13 public:0.1.12 public:0.1.11 public:0.1.10 public:0.1.9 public:0.1.8 public:0.1.7 public:0.1.6 public:0.1.5 public:0.1.4 public:0.1.3 public:0.1.2 public:0.1.1 public:0.1.0 public:0.0.9 public:0.0.8 public:0.0.7 public:0.0.6

95 Commits
0.19.4 ... 0.21.2

Author SHA1 Message Date
vorotamoroz
b337a05b5a bump 2023-11-27 07:13:15 +00:00
vorotamoroz
9ea6bee9d1 - Fixed: 
- No longer files are broken while rebuilding.
    - Now, Large binary files can be written correctly on a mobile platform.
    - Any decoding errors now make zero-byte files.
  - Modified:
    - All files are processed sequentially for each.
 2023-11-27 06:55:55 +00:00
vorotamoroz
9747c26d50 bump 2023-11-25 02:22:26 +09:00
vorotamoroz
bb4b764586 - Fixed: 
- No more infinity loops on larger files.
    - Show message on decode error.
  - Refactored:
    - Fixed to avoid obsolete global variables.
 2023-11-25 02:21:44 +09:00
vorotamoroz
279b4b41e5 bump 2023-11-24 10:32:46 +00:00
vorotamoroz
b644fb791d - Changes and performance improvements: 
- Now the saving files are processed by Blob.
    - The V2-Format has been reverted.
    - New encoding format has been enabled in default.
    - WARNING: Since this version, the compatibilities with older Filesystem LiveSync have been lost.
 2023-11-24 10:31:58 +00:00
vorotamoroz
ac9428e96b Fixed 
- To better replication, path obfuscation is now deterministic even if with E2EE.
 2023-11-15 08:44:03 +00:00
vorotamoroz
280d9e1dd9 Fixed: Fixed the issue of TOML editing. 2023-11-07 01:07:58 +00:00
vorotamoroz
f7209e566c bump 2023-10-24 10:07:29 +01:00
vorotamoroz
4a9ab2d1de Fixed: 
- No longer enumerating file names is broken.
 2023-10-24 10:07:17 +01:00
vorotamoroz
cb74b5ee93 - Fixed 
- Now empty file could be decoded.
    - Local files are no longer pre-saved before fetching from a remote database.
    - No longer deadlock while applying customisation sync.
    - Configuration with multiple files is now able to be applied correctly.
    - Deleting folder propagation now works without enabling the use of a trash bin.
 2023-10-24 09:54:56 +01:00
vorotamoroz
60eecd7001 bump 2023-10-17 12:00:59 +09:00
vorotamoroz
4bd7b54bcd Fixed: 
- Now the files which having digit or character prefixes in the path will not be ignored.
 2023-10-17 12:00:19 +09:00
vorotamoroz
8923c73d1b bump 2023-10-14 23:08:34 +09:00
vorotamoroz
11e64b13e2 The text-input-dialogue is no longer broken. 2023-10-14 23:07:51 +09:00
vorotamoroz
983d9248ed bump 2023-10-13 04:23:59 +01:00
vorotamoroz
7240e84328 - New feature: 
- We can launch Customization sync from the Ribbon if we enable it.
- Fixed:
  - Setup URI is now back to the previous spec; be encrypted by V1.
  - The Settings dialogue is now registered at the beginning of the start-up process.
- Improved:
  - Enumerating documents has been faster.
 2023-10-13 04:22:24 +01:00
vorotamoroz
0d55ae2532 Merge pull request #298 from LiamSwayne/patch-1 
Grammar fix
 2023-10-04 13:07:42 +09:00
Liam Swayne
dbd284f5dd grammar fix 2023-10-03 22:26:30 -04:00
vorotamoroz
c000a02f4a bump 2023-10-02 10:39:54 +01:00
vorotamoroz
79754f48d6 New feature: 
- We can delete all data of customization sync from the `Delete all customization sync data` on the `Hatch` pane.
Fixed:
- Prevent keep restarting on iOS by yielding microtasks.
 2023-10-02 10:38:54 +01:00
vorotamoroz
dd7a40630b bump 2023-10-02 09:54:56 +01:00
vorotamoroz
14406f8213 - Fixed: 
- No more UI freezing and keep restarting on iOS.
  - Diff of Non-markdown documents are now shown correctly.
- Improved:
  - Performance has been a bit improved.
  - Customization sync has gotten faster.
    - However, We lost forward compatibility again (only for this feature). Please update all devices.
- Misc
  - Terser configuration has been more aggressive.
 2023-10-02 09:53:41 +01:00
vorotamoroz
3bbd9c048d bump 2023-09-29 18:57:54 +09:00
vorotamoroz
d91c4f50b4 - Improved: 
- A New binary file handling implemented
  - A new encrypted format has been implemented
  - Now the chunk sizes will be adjusted for efficient sync
- Fixed:
  - levels of exception in some logs have been fixed
- Tidied:
  - Some Lint warnings have been suppressed.
 2023-09-29 18:55:46 +09:00
vorotamoroz
395b7fbc42 bump 2023-09-22 18:03:09 +09:00
vorotamoroz
3773e57429 Improved: 
- We can open the log pane also from the command palette now.
- Now, the hidden file scanning interval could be configured to 0.
- `Check database configuration` now points out that we do not have administrator permission.
 2023-09-22 17:59:32 +09:00
vorotamoroz
4835fce62a bump 2023-09-21 09:44:50 +01:00
vorotamoroz
ff814be4a0 Fixed: 
- Now the synchronisation will begin without our interaction.
- No longer puts the configuration of the remote database into the log while checking configuration.
- Some outdated description notes have been removed.
- Options that are meaningless depending on other settings configured are now hidden.
 2023-09-21 09:44:07 +01:00
vorotamoroz
b271b63efa bump 2023-09-20 07:05:18 +01:00
vorotamoroz
23419e476a - Fixed: 
- Hidden files are no longer handled in the initial replication.
  - Report from `Making report` fixed
 2023-09-20 07:04:31 +01:00
vorotamoroz
b9bd1f17b8 bump 2023-09-19 09:58:04 +01:00
vorotamoroz
bcce277c36 New feature: 
- `Sync on Editor save` has been implemented
- Now we can use the `Hidden file sync` and the `Customization sync` cooperatively.
- We can ignore specific plugins in Customization sync.
- Now the message of leftover conflicted files accepts our click.

Refactored:
- Parallelism functions made more explicit.
- Type errors have been reduced.

Fixed:
- Now documents would not be overwritten if they are conflicted.
- Some error messages have been fixed.
- Missing dialogue titles have been shown now.
 2023-09-19 09:53:48 +01:00
vorotamoroz
5acbbe479e Merge pull request #276 from cgcel/main 
Add Quick setup Chinese translation.
 2023-09-13 17:13:55 +09:00
vorotamoroz
c9f9d511e0 bump 2023-09-05 09:16:49 +01:00
vorotamoroz
b8cb94c498 Fixed: 
- Resolving conflicted revision has become more robust.
- LiveSync now try to keep local changes when fetching from the rebuilt remote database.
- Now, all files will be restored after performing `fetch` immediately.
 2023-09-05 09:16:11 +01:00
cgcel
52c736f6b9 update: format markdown titles. 2023-09-01 11:01:06 +08:00
cgcel
ebd1cb7777 add: Quick setup Chinese translation. 2023-09-01 10:58:07 +08:00
vorotamoroz
10decb7909 Merge pull request #259 from jt-wang/main 
Upload colab notebook into the repo, and update setup_flyio.md to reference it
 2023-08-24 13:04:02 +09:00
vorotamoroz
e0aab8d69d bump 2023-08-24 04:03:03 +01:00
vorotamoroz
618600c753 Fixed: 
- Now the empty (or deleted) file could be conflict-resolved.
 2023-08-24 04:00:56 +01:00
Jingtao Wang
d1aba87e37 Updated with latest changes 2023-08-23 19:22:07 -07:00
Jingtao Wang
db889f635e Merge pull request #1 from jt-wang/move-colab-script-into-repo 
Upload deploy_couchdb_to_flyio_v2_with_swap.ipynb into the repo, and reference it from fly.io setup doc.
 2023-08-15 14:55:37 -07:00
Jingtao Wang
dd80e634f5 Update setup_flyio.md to reference colab notebook within the repo 2023-08-15 14:54:43 -07:00
Jingtao Wang
bec6fc1a74 Upload deploy_couchdb_to_flyio_v2_with_swap.ipynb 
Upload https://gist.github.com/vrtmrz/37c3efd7842e49947aaaa7f665e5020a into the repo for unified management.
 2023-08-15 14:49:24 -07:00
vorotamoroz
5c96c7f99b bump 2023-08-08 10:45:50 +01:00
vorotamoroz
7b9724f713 Fixed: 
- Now nested ignore files could be parsed correctly.
- The unexpected deletion of hidden files in some cases has been corrected.
- Hidden file change is no longer reflected on the device which has made the change itself.
 2023-08-08 10:42:07 +01:00
vorotamoroz
4cd12c85ed update the changelog. 2023-08-04 10:03:51 +01:00
vorotamoroz
90651540f9 Update release.yml 2023-08-04 17:52:13 +09:00
vorotamoroz
9e504d5002 bump 2023-08-04 09:45:14 +01:00
vorotamoroz
faaa94423c New feature: 
- (Beta) ignore files handling

Fixed:
- Buttons on lock-detected-dialogue now can be shown in narrow-width devices.

Improved:
- Some constant has been flattened to be evaluated.
- The usage of the deprecated API of obsidian has been reduced.
- Now the indexedDB adapter will be enabled while the importing configuration.

Misc:
- Compiler, framework, and dependencies have been upgraded.
- Due to standing for these impacts (especially in esbuild and svelte,) terser has been introduced.
  Feel free to notify your opinion to me! I do not like to obfuscate the code too.
 2023-08-04 09:45:04 +01:00
vorotamoroz
a7c179fc86 Merge pull request #247 from cgcel/patch-1 
Update setup_own_server_cn.md
 2023-07-31 12:49:50 +09:00
vorotamoroz
ed1a670b9b bump 2023-07-26 17:35:06 +09:00
vorotamoroz
6c3c265bd6 Dependency updates 2023-07-26 17:32:07 +09:00
vorotamoroz
9d68025f2a Fixed: 
- Now storing files after cleaning up is correct works.

Improved:
- Cleaning the local database up got incredibly fastened.
 2023-07-26 17:31:55 +09:00
GC Chen
e70972c8f9 Update setup_own_server_cn.md 2023-07-25 22:31:17 +08:00
vorotamoroz
7607be7729 bump 2023-07-25 19:20:18 +09:00
vorotamoroz
db9d428ab4 Fixed: 
- Internal documents are now ignored.
- Merge dialogue now respond immediately to button pressing.
- Periodic processing now works fine
- The checking interval of detecting conflicted has got shorter
- Replication is now cancelled while cleaning up
- The database locking by the cleaning up is now carefully unlocked
- Missing chunks message is correctly reported

New feature:
- Suspend database reflecting has been implemented
- Now fetch suspends the reflecting database and storage changes temporarily to improve the performance.
- We can choose the action when the remote database has been cleaned
- Merge dialogue now show `↲` before the new line.

Improved:
- Now progress is reported while the cleaning up and fetch process
- Cancelled replication is now detected
 2023-07-25 19:16:39 +09:00
vorotamoroz
0a2caea3c7 bump 2023-07-25 00:30:17 +09:00
vorotamoroz
b1d1ba0e6b - Implemented: 
- Database clean-up is now in beta 2!
    We can shrink the remote database by deleting unused chunks, with keeping history.
    Note: Local database is not cleaned up totally. We have to `Fetch` again to let it done.
    **Note2**: Still in beta. Please back your vault up anything before.
- Fixed:
  - The log updates are not thinned out now.
 2023-07-25 00:29:47 +09:00
vorotamoroz
5e844372cb Merge pull request #236 from bioluks/documentation-rproxy 
CouchDB config corrections, reverse proxy documentation, other additions
 2023-07-20 10:33:37 +09:00
Meriç Aşkın
99c6911e96 Merge branch 'vrtmrz:main' into documentation-rproxy 2023-07-18 04:25:53 +02:00
vorotamoroz
dc880d7d4e Merge pull request #244 from samyarkd/patch-1 
fix: separate languages
 2023-07-18 09:24:04 +09:00
Samyar
c157fef76c fix: separate languages 
when i first saw it i thought it's one connected sentence
so, i thought it would be nice to separate them using a dash.
 2023-07-14 22:14:51 +03:30
bioluks
2b2011dc49 Added docker-compose, table of contents, a new reverse proxies section populated with traefik for now 2023-07-04 01:52:48 +02:00
bioluks
ae451e005e Moved enable_cors to the right section. Added explanation for difference of versions. Added bind_address for making sure the container uses all interfaces given. Added spaces between 'origins' and removed spaces between the 'methods' elements because it's like this in the official Documentation. Added a write permission warning since many newbies had this mistake with couchdb. 2023-07-04 00:15:00 +02:00
vorotamoroz
8a75d41cbb bump 2023-07-03 18:45:05 +09:00
vorotamoroz
5252cc0372 Improved: 
- Boot-up performance has been improved.
- Customisation sync performance has been improved.
- Synchronising performance has been improved.
 2023-07-03 18:44:02 +09:00
vorotamoroz
5022155317 Fix documentation 2023-06-19 16:31:52 +09:00
vorotamoroz
d36f925c65 bump 2023-06-15 18:13:43 +09:00
vorotamoroz
3ae33e0500 Refactored: External dependency merged. 2023-06-15 18:07:11 +09:00
vorotamoroz
13e442a0c7 Improvements: 
- Hashing ChunkID has been improved.
- Logging keeps 400 lines now.

Refactored:
- Import statement has been fixed about types.
 2023-06-15 17:55:58 +09:00
vorotamoroz
6288716966 - Fixed 
- Fixed the issue about fixing the database.
 2023-06-09 19:28:29 +09:00
vorotamoroz
47d2cf9733 bump 2023-06-09 18:48:59 +09:00
vorotamoroz
ae6a9ecee4 - New feature (For fixing a problem): 
- We can fix the database obfuscated and plain paths that have been mixed up.
- Improvements
  - Customisation Sync performance has been improved.
 2023-06-09 18:48:10 +09:00
vorotamoroz
2289bea8d9 bump 2023-06-07 17:32:28 +09:00
vorotamoroz
cda90259c5 - New feature: 
- Vault history: A tab has been implemented to give a birds-eye view of the changes that have occurred in the vault.
- Improved:
  - Log dialogue is now shown as one of tabs.
- Fixed:
  - Some minor issues has been fixed.
 2023-06-07 17:29:53 +09:00
vorotamoroz
432a211f80 Merge pull request #224 from antoKeinanen/main 
[Feature] Add password protection to askString function
 2023-06-07 17:04:31 +09:00
antoKeinanen
eaf8c4998e feat: add password protection for required inputs 2023-06-05 13:27:06 +03:00
antoKeinanen
55601f7910 feat: add option for password protection in askString function 2023-06-05 13:24:50 +03:00
vorotamoroz
13e70475d9 Add new documentation 
Thanks for your discussion!!
 2023-06-02 14:36:00 +09:00
vorotamoroz
2572177879 Merge pull request #222 from Hugo-Persson/add-troubleshooting-guide 
Added troubleshooting guide
 2023-06-02 09:25:18 +09:00
Hugo Persson
e82a2560e4 Added troubleshooting guide 2023-06-01 19:54:42 +02:00
vorotamoroz
09146591eb bump 2023-06-01 17:06:23 +09:00
vorotamoroz
69c6e57df3 Fix: 
- Fixed Setup wizard
- Set initial pane to General settings.
 2023-06-01 17:01:42 +09:00
vorotamoroz
5e181a8ec4 Update docs 2023-06-01 16:19:14 +09:00
vorotamoroz
4354cc3054 bump 2023-06-01 13:05:29 +09:00
vorotamoroz
0664427c63 Refined: 
- Configuration dialogue refined.
 2023-06-01 13:02:56 +09:00
vorotamoroz
49c4736d69 Improved: 
- Confirmation for new adapters while rebuilding.
- Batched file is now shown in digits.

Fixed:
- Some framework have been upgraded.
 2023-06-01 12:47:41 +09:00
vorotamoroz
f0ce8f0e05 Fixed: 
- Import declarations
- Logging has been tweaked
Improved:
 2023-06-01 12:36:10 +09:00
vorotamoroz
0a70afc5a3 Update issue templates 2023-05-24 11:51:53 +09:00
vorotamoroz
431239a736 Merge pull request #218 from garlic-hub/garlic-hub-patch-1 
Update setup_own_server.md
 2023-05-23 17:46:35 +09:00
vorotamoroz
1ceb671683 bump 2023-05-23 17:40:47 +09:00
vorotamoroz
ea40e5918c Fixed: 
- Now hidden file synchronisation would not be hanged, even if so many files exist.

Improved:
- Customisation sync works more smoothly.
 2023-05-23 17:39:02 +09:00
garlic-hub
64681729ff Update setup_own_server.md 2023-05-23 04:49:50 +00:00
53 changed files with 6941 additions and 3336 deletions
Expand all files Collapse all files

3
.eslintignore
View File

@@ -1,4 +1,5 @@
node_modules
build
.eslintrc.js.bak
src/lib/src/patches/pouchdb-utils
src/lib/src/patches/pouchdb-utils
esbuild.config.mjs

78
.github/ISSUE_TEMPLATE/issue-report.md vendored Normal file
View File

@@ -0,0 +1,78 @@
---
name: Issue report
about: Create a report to help us improve
title: ''
labels: ''
assignees: ''
---
Thank you for taking the time to report this issue!
To improve the process, I would like to ask you to let me know the information in advance.
All instructions and examples, and empty entries can be deleted.
Just for your information, a [filled example](https://docs.vrtmrz.net/LiveSync/hintandtrivia/Issue+example) is also written.
## Abstract
The synchronisation hung up immediately after connecting.
## Expected behaviour
- Synchronisation ends with the message `Replication completed`
- Everything synchronised
## Actually happened
- Synchronisation has been cancelled with the message `TypeError ... ` (captured in the attached log, around LL.10-LL.12)
- No files synchronised
## Reproducing procedure
1. Configure LiveSync as in the attached material.
2. Click the replication button on the ribbon.
3. Synchronising has begun.
4. About two or three seconds later, we got the error `TypeError ... `.
5. Replication has been stopped. No files synchronised.
Note: If you do not catch the reproducing procedure, please let me know the frequency and signs.
## Report materials
If the information is not available, do not hesitate to report it as it is. You can also of course omit it if you think this is indeed unnecessary. If it is necessary, I will ask you.
### Report from the LiveSync
For more information, please refer to [Making the report](https://docs.vrtmrz.net/LiveSync/hintandtrivia/Making+the+report).
<details>
<summary>Report from hatch</summary>
```
<!-- paste here -->
```
</details>
### Obsidian debug info
<details>
<summary>Debug info</summary>
```
<!-- paste here -->
```
</details>
### Plug-in log
We can see the log by tapping the Document box icon. If you noticed something suspicious, please let me know.
Note: **Please enable `Verbose Log`**. For detail, refer to [Logging](https://docs.vrtmrz.net/LiveSync/hintandtrivia/Logging), please.
<details>
<summary>Plug-in log</summary>
```
<!-- paste here -->
```
</details>
### Network log
Network logs displayed in DevTools will possibly help with connection-related issues. To capture that, please refer to [DevTools](https://docs.vrtmrz.net/LiveSync/hintandtrivia/DevTools).
### Screenshots
If applicable, please add screenshots to help explain your problem.
### Other information, insights and intuition.
Please provide any additional context or information about the problem.

2
.github/workflows/release.yml vendored
View File

@@ -17,7 +17,7 @@ jobs:
- name: Use Node.js
uses: actions/setup-node@v1
with:
node-version: '14.x' # You might need to adjust this value to your own version
node-version: '18.x' # You might need to adjust this value to your own version
# Get the version number and put it in a variable
- name: Get Version
id: version

1
.gitignore vendored
View File

@@ -8,6 +8,7 @@ package-lock.json
# build
main.js
main_org.js
*.js.map
# obsidian

22
README.md
View File

@@ -1,6 +1,6 @@
# Self-hosted LiveSync
[Japanese docs](./README_ja.md) [Chinese docs](./README_cn.md).
[Japanese docs](./README_ja.md) - [Chinese docs](./README_cn.md).
Self-hosted LiveSync is a community-implemented synchronization plugin.
A self-hosted or purchased CouchDB acts as the intermediate server. Available on every obsidian-compatible platform.
@@ -34,11 +34,10 @@ Useful for researchers, engineers and developers with a need to keep their notes
### Get your database ready.
First, get your database ready. IBM Cloudant is preferred for testing. Or you can use your own server with CouchDB. For more information, refer below:
1. [Setup IBM Cloudant](docs/setup_cloudant.md)
2. [Setup your CouchDB](docs/setup_own_server.md)
Note: More information about alternative hosting methods is needed! Currently, [using fly.io](https://github.com/vrtmrz/obsidian-livesync/discussions/85) is being discussed.
First, get your database ready. fly.io is preferred for testing. Or you can use your own server with CouchDB. For more information, refer below:
1. [Setup fly.io](docs/setup_flyio.md)
2. [Setup IBM Cloudant](docs/setup_cloudant.md)
3. [Setup your CouchDB](docs/setup_own_server.md)
### Configure the plugin
@@ -54,12 +53,6 @@ Please open the configuration link again and Answer below:
If you answered `No` to both, your databases will be rebuilt by the content on your device. And the remote database will lock out other devices. You have to synchronize all your devices again. (When this time, almost all your files should be synchronized with a timestamp. So you can use an existing vault).
## Test Server
~~Setting up an instance of Cloudant or local CouchDB is a little complicated, so I set up a [Tasting server for self-hosted-livesync](https://olstaste.vrtmrz.net/). Try it out for free!~~
Now (30 May 2023) suspending while the server transfer.
Note: Please read "Limitations" carefully. Do not send your private vault.
## Information in StatusBar
Synchronization status is shown in statusbar.
@@ -73,7 +66,7 @@ Synchronization status is shown in statusbar.
- ↓ Downloaded chunks and metadata
- ⏳ Number of pending processes
- 🧩 Number of files waiting for their chunks.
If you have deleted or renamed files, please wait until ⏳ icon disappeared.
If you have deleted or renamed files, please wait until ⏳ icon disappears.
## Hints
@@ -94,6 +87,9 @@ If you have deleted or renamed files, please wait until ⏳ icon disappeared.
Repo is here: [obsidian-livesync-webclip](https://github.com/vrtmrz/obsidian-livesync-webclip). (Docs are a work in progress.)
## Troubleshooting
If you are having problems getting the plugin working see: [Troubleshooting](docs/troubleshooting.md)
## License
The source code is licensed under the MIT License.

318
deploy_couchdb_to_flyio_v2_with_swap.ipynb Normal file
View File

@@ -0,0 +1,318 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "view-in-github"
},
"source": [
"<a href=\"https://colab.research.google.com/gist/vrtmrz/37c3efd7842e49947aaaa7f665e5020a/deploy_couchdb_to_flyio_v2_with_swap.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "HiRV7G8Gk1Rs"
},
"source": [
"History:\n",
"- 18, May, 2023: Initial.\n",
"- 19, Jun., 2023: Patched for enabling swap.\n",
"- 22, Aug., 2023: Generating Setup-URI implemented.\n",
"- 7, Nov., 2023: Fixed the issue of TOML editing."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "2Vh0mEQEZuAK"
},
"outputs": [],
"source": [
"# Configurations\n",
"import os\n",
"os.environ['region']=\"nrt\"\n",
"os.environ['couchUser']=\"alkcsa93\"\n",
"os.environ['couchPwd']=\"c349usdfnv48fsasd\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "SPmbB0jZauQ1"
},
"outputs": [],
"source": [
"# Delete once (Do not care about `cannot remove './fly.toml': No such file or directory`)\n",
"!rm ./fly.toml"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Nze7QoxLZ7Yx"
},
"outputs": [],
"source": [
"# Installation\n",
"# You have to set up your account in here.\n",
"!curl -L https://fly.io/install.sh | sh\n",
"!/root/.fly/bin/flyctl auth signup"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "MVJwsIYrbgtx"
},
"outputs": [],
"source": [
"# Generate server\n",
"!/root/.fly/bin/flyctl launch --auto-confirm --generate-name --detach --no-deploy --region ${region}\n",
"!/root/.fly/bin/fly volumes create --region ${region} couchdata --size 2 --yes"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "2RSoO9o-i2TT"
},
"outputs": [],
"source": [
"# Check the toml once.\n",
"!cat fly.toml"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "zUtPZLVnbvdQ"
},
"outputs": [],
"source": [
"# Modify the TOML and generate Dockerfile\n",
"!pip install mergedeep\n",
"from mergedeep import merge\n",
"import toml\n",
"fly = toml.load('fly.toml')\n",
"override = {\n",
" \"http_service\":{\n",
" \"internal_port\":5984\n",
" },\n",
" \"build\":{\n",
" \"dockerfile\":\"./Dockerfile\"\n",
" },\n",
" \"mounts\":{\n",
" \"source\":\"couchdata\",\n",
" \"destination\":\"/opt/couchdb/data\"\n",
" },\n",
" \"env\":{\n",
" \"COUCHDB_USER\":os.environ['couchUser'],\n",
" \"ERL_FLAGS\":\"-couch_ini /opt/couchdb/etc/default.ini /opt/couchdb/etc/default.d/ /opt/couchdb/etc/local.d /opt/couchdb/etc/local.ini /opt/couchdb/data/persistence.ini\",\n",
" }\n",
"}\n",
"out = merge(fly,override)\n",
"with open('fly.toml', 'wt') as fp:\n",
" toml.dump(out, fp)\n",
" fp.close()\n",
"\n",
"# Make the Dockerfile to modify the permission of the ini file. If you want to use a specific version, you should change `latest` here.\n",
"dockerfile = '''FROM couchdb:latest\n",
"RUN sed -i '2itouch /opt/couchdb/data/persistence.ini && chmod +w /opt/couchdb/data/persistence.ini && fallocate -l 512M /swapfile && chmod 0600 /swapfile && mkswap /swapfile && echo 10 > /proc/sys/vm/swappiness && swapon /swapfile && echo 1 > /proc/sys/vm/overcommit_memory' /docker-entrypoint.sh\n",
"'''\n",
"with open(\"./Dockerfile\",\"wt\") as fp:\n",
" fp.write(dockerfile)\n",
" fp.close()\n",
"\n",
"!echo ------\n",
"!cat fly.toml\n",
"!echo ------\n",
"!cat Dockerfile"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "xWdsTCI6bzk2"
},
"outputs": [],
"source": [
"# Configure password\n",
"!/root/.fly/bin/flyctl secrets set COUCHDB_PASSWORD=${couchPwd}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "k0WIQlShcXGa"
},
"outputs": [],
"source": [
"# Deploy server\n",
"# Be sure to shutdown after the test.\n",
"!/root/.fly/bin/flyctl deploy --detach --remote-only\n",
"!/root/.fly/bin/flyctl status"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "0ySggkdlfq7M"
},
"outputs": [],
"source": [
"import subprocess, json\n",
"result = subprocess.run([\"/root/.fly/bin/flyctl\",\"status\",\"-j\"], capture_output=True, text=True)\n",
"if result.returncode==0:\n",
" hostname = json.loads(result.stdout)[\"Hostname\"]\n",
" os.environ['couchHost']=\"https://%s\" % (hostname)\n",
" print(\"Your couchDB server is https://%s/\" % (hostname))\n",
"else:\n",
" print(\"Something occured.\")\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "cGlSzVqlQG_z"
},
"outputs": [],
"source": [
"# Finish setting up the CouchDB\n",
"# Please repeat until the request is completed without error messages\n",
"# i.e., You have to redo this block while \"curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to xxxx\" is showing.\n",
"#\n",
"# Note: A few minutes might be required to be booted.\n",
"!curl -X POST \"${couchHost}/_cluster_setup\" -H \"Content-Type: application/json\" -d \"{\\\"action\\\":\\\"enable_single_node\\\",\\\"username\\\":\\\"${couchUser}\\\",\\\"password\\\":\\\"${couchPwd}\\\",\\\"bind_address\\\":\\\"0.0.0.0\\\",\\\"port\\\":5984,\\\"singlenode\\\":true}\" --user \"${couchUser}:${couchPwd}\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "JePzrsHypY18"
},
"outputs": [],
"source": [
"# Please repeat until all lines are completed without error messages\n",
"!curl -X PUT \"${couchHost}/_node/nonode@nohost/_config/chttpd/require_valid_user\" -H \"Content-Type: application/json\" -d '\"true\"' --user \"${couchUser}:${couchPwd}\"\n",
"!curl -X PUT \"${couchHost}/_node/nonode@nohost/_config/chttpd_auth/require_valid_user\" -H \"Content-Type: application/json\" -d '\"true\"' --user \"${couchUser}:${couchPwd}\"\n",
"!curl -X PUT \"${couchHost}/_node/nonode@nohost/_config/httpd/WWW-Authenticate\" -H \"Content-Type: application/json\" -d '\"Basic realm=\\\"couchdb\\\"\"' --user \"${couchUser}:${couchPwd}\"\n",
"!curl -X PUT \"${couchHost}/_node/nonode@nohost/_config/httpd/enable_cors\" -H \"Content-Type: application/json\" -d '\"true\"' --user \"${couchUser}:${couchPwd}\"\n",
"!curl -X PUT \"${couchHost}/_node/nonode@nohost/_config/chttpd/enable_cors\" -H \"Content-Type: application/json\" -d '\"true\"' --user \"${couchUser}:${couchPwd}\"\n",
"!curl -X PUT \"${couchHost}/_node/nonode@nohost/_config/chttpd/max_http_request_size\" -H \"Content-Type: application/json\" -d '\"4294967296\"' --user \"${couchUser}:${couchPwd}\"\n",
"!curl -X PUT \"${couchHost}/_node/nonode@nohost/_config/couchdb/max_document_size\" -H \"Content-Type: application/json\" -d '\"50000000\"' --user \"${couchUser}:${couchPwd}\"\n",
"!curl -X PUT \"${couchHost}/_node/nonode@nohost/_config/cors/credentials\" -H \"Content-Type: application/json\" -d '\"true\"' --user \"${couchUser}:${couchPwd}\"\n",
"!curl -X PUT \"${couchHost}/_node/nonode@nohost/_config/cors/origins\" -H \"Content-Type: application/json\" -d '\"app://obsidian.md,capacitor://localhost,http://localhost\"' --user \"${couchUser}:${couchPwd}\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "YfSOomsoXbGS"
},
"source": [
"Now, our CouchDB has been surely installed and configured. Cheers!\n",
"\n",
"In the steps that follow, create a setup-URI.\n",
"\n",
"This URI could be imported directly into Self-hosted LiveSync, to configure the use of the CouchDB which we configured now."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "416YncOqXdNn"
},
"outputs": [],
"source": [
"# Database config\n",
"import random, string\n",
"\n",
"def randomname(n):\n",
" return ''.join(random.choices(string.ascii_letters + string.digits, k=n))\n",
"\n",
"# The database name\n",
"os.environ['database']=\"obsidiannote\"\n",
"# The passphrase to E2EE\n",
"os.environ['passphrase']=randomname(20)\n",
"\n",
"print(\"Your database:\"+os.environ['database'])\n",
"print(\"Your passphrase:\"+os.environ['passphrase'])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "C4d7C0HAXgsr"
},
"outputs": [],
"source": [
"# Install deno for make setup uri\n",
"!curl -fsSL https://deno.land/x/install/install.sh | sh"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "hQL_Dx-PXise"
},
"outputs": [],
"source": [
"# Fetch module for encrypting a Setup URI\n",
"!curl -o encrypt.ts https://gist.githubusercontent.com/vrtmrz/f9d1d95ee2ca3afa1a924a2c6759b854/raw/d7a070d864a6f61403d8dc74208238d5741aeb5a/encrypt.ts"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "o0gX_thFXlIZ"
},
"outputs": [],
"source": [
"# Make buttons!\n",
"from IPython.display import HTML\n",
"result = subprocess.run([\"/root/.deno/bin/deno\",\"run\",\"-A\",\"encrypt.ts\"], capture_output=True, text=True)\n",
"text=\"\"\n",
"if result.returncode==0:\n",
" text = result.stdout.strip()\n",
" result = HTML(f\"<button onclick=navigator.clipboard.writeText('{text}')>Copy setup uri</button><br>Importing passphrase is `welcome`. <br>If you want to synchronise in live mode, please apply a preset after setup.)\")\n",
"else:\n",
" result = \"Failed to encrypt the setup URI\"\n",
"result"
]
}
],
"metadata": {
"colab": {
"include_colab_link": true,
"private_outputs": true,
"provenance": []
},
"gpuClass": "standard",
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 0
}

46
docker-compose.traefik.yml Normal file
View File

@@ -0,0 +1,46 @@
# For details and other explanations about this file refer to:
# https://github.com/vrtmrz/obsidian-livesync/blob/main/docs/setup_own_server.md#traefik
version: "2.1"
services:
couchdb:
image: couchdb:latest
container_name: obsidian-livesync
user: 1000:1000
environment:
- COUCHDB_USER=username
- COUCHDB_PASSWORD=password
volumes:
- ./data:/opt/couchdb/data
- ./local.ini:/opt/couchdb/etc/local.ini
# Ports not needed when already passed to Traefik
#ports:
# - 5984:5984
restart: unless-stopped
networks:
- proxy
labels:
- "traefik.enable=true"
# The Traefik Network
- "traefik.docker.network=proxy"
# Don't forget to replace 'obsidian-livesync.example.org' with your own domain
- "traefik.http.routers.obsidian-livesync.rule=Host(`obsidian-livesync.example.org`)"
# The 'websecure' entryPoint is basically your HTTPS entrypoint. Check the next code snippet if you are encountering problems only; you probably have a working traefik configuration if this is not your first container you are reverse proxying.
- "traefik.http.routers.obsidian-livesync.entrypoints=websecure"
- "traefik.http.routers.obsidian-livesync.service=obsidian-livesync"
- "traefik.http.services.obsidian-livesync.loadbalancer.server.port=5984"
- "traefik.http.routers.obsidian-livesync.tls=true"
# Replace the string 'letsencrypt' with your own certificate resolver
- "traefik.http.routers.obsidian-livesync.tls.certresolver=letsencrypt"
- "traefik.http.routers.obsidian-livesync.middlewares=obsidiancors"
# The part needed for CORS to work on Traefik 2.x starts here
- "traefik.http.middlewares.obsidiancors.headers.accesscontrolallowmethods=GET,PUT,POST,HEAD,DELETE"
- "traefik.http.middlewares.obsidiancors.headers.accesscontrolallowheaders=accept,authorization,content-type,origin,referer"
- "traefik.http.middlewares.obsidiancors.headers.accesscontrolalloworiginlist=app://obsidian.md,capacitor://localhost,http://localhost"
- "traefik.http.middlewares.obsidiancors.headers.accesscontrolmaxage=3600"
- "traefik.http.middlewares.obsidiancors.headers.addvaryheader=true"
- "traefik.http.middlewares.obsidiancors.headers.accessControlAllowCredentials=true"
networks:
proxy:
external: true

62
docs/quick_setup.md
View File

@@ -1,17 +1,20 @@
# Quick setup
The Setup wizard has been implemented since v0.15.0 simplifying the initial setup.
Note: Subsequent devices should be set up using the `Copy setup URI` and `Open setup URI` functionality.
[Japanese docs](./quick_setup_ja.md) - [Chinese docs](./quick_setup_cn.md).
## How to open and use the wizard
Open the `🪄 Setup wizard` in the settings dialogue. If the plugin has not been configured before, it should already be open.
The plugin has so many configuration options to deal with different circumstances. However, there are not so many settings that are actually used. Therefore, `The Setup wizard` has been implemented to simplify the initial setup.
Note: Subsequent devices are recommended to be set up using the `Copy setup URI` and `Open setup URI`.
## The Setup wizard
Open the `🧙‍♂️ Setup wizard` in the settings dialogue. If the plugin has not been configured before, it should already be open.
![](../images/quick_setup_1.png)
### Discard the existing configuration and set up
- Discard the existing configuration and set up
If you have changed any settings, this button allows you to discard all changes before setting up.
### Do not discard the existing configuration and set up
- Do not discard the existing configuration and set up
Simply reconfigure. Be careful. In wizard mode, you cannot see all configuration items, even if they have been configured.
Pressing `Next` on one of the above options will put the configuration dialog into wizard mode.
@@ -30,17 +33,10 @@ Enter the information for the database we have set up.
![](../images/quick_setup_3.png)
### End to End Encryption
![](../images/quick_setup_4.png)
#### Test database connection and Check database configuration
When End to End encryption is enabled, a third party will be less likely to be able to read your Remote database in the event of a data breach/leak (assuming they do not know the Passphrase). We strongly recommend enabling it.
Encryption is based on 256-bit AES-GCM.
This setting can be disabled if you are inside a closed network and it is clear that you will not be accessed by third parties.
### Test database connection and Check database configuration
Here we can check the status of the connection to the database and the database settings.
We can check the connectivity to the database, and the database settings.
![](../images/quick_setup_5.png)
@@ -49,38 +45,36 @@ Check whether we can connect to the database. If it fails, there are several pos
#### Check database configuration
Check the database settings and fix any deficiencies on the spot.
Check the database settings and fix any problems on the spot.
![](../images/quick_setup_6.png)
This item may vary depending on the connection. In the above case, press all three Fix buttons.
If the Fix buttons disappear and all become check marks, we are done.
### Confidentiality configuration
![](../images/quick_setup_4.png)
Encrypt your database in case of unintended database exposure; enable End to End encryption and the contents of your notes will be encrypted at the moment it leaves the device. We strongly recommend enabling it. And `Path Obfuscation` also obfuscates filenames. Now stable and recommended.
Encryption is based on 256-bit AES-GCM.
These setting can be disabled if you are inside a closed network and it is clear that you will not be accessed by third parties.
![](../images/quick_setup_7.png)
### Next
Go to the Local Database configuration.
#### Next
Go to the Sync Settings.
### Discard existing database and proceed
Discard the contents of the Remote database and go to the Local Database configuration.
#### Discard existing database and proceed
Discard the contents of the Remote database and go to the Sync Settings.
## Local Database configuration
![](../images/quick_setup_8.png)
Configure the local database. If we already have a Vault with Self-hosted LiveSync installed which has the same directory name as the one we are currently setting up, please specify a different suffix than the Vault you have already set up here.
## Miscellaneous
Finally, finish the miscellaneous configurations and select a preset for synchronisation.
### Sync Settings
Finally, finish the wizard by selecting a preset for synchronisation.
![](../images/quick_setup_9_1.png)
The `Show status inside editor` can be enabled to your liking. If enabled, the status is displayed in the top right-hand corner of the editor. Learn how to read the status bar [here](/README.md#information-in-statusbar).
![](../images/quick_setup_9_2.png)
From Presets, select the synchronisation method we want to use and `Apply` to initialise and build the local and remote databases as required.
If `All done!' is displayed, we are done. Automatically, `Copy setup URI` will open and we will be asked for a passphrase to encrypt the `Setup URI`.
Select any synchronisation methods we want to use and `Apply` to initialise and build the local and remote databases as required. If `All done!` is displayed, we are done. Automatically, `Copy setup URI` will open and we will be asked for a passphrase to encrypt the `Setup URI`.
![](../images/quick_setup_10.png)

93
docs/quick_setup_cn.md Normal file
View File

@@ -0,0 +1,93 @@
# 快速配置 (Quick setup)
该插件有较多配置项, 可以应对不同的情况. 不过, 实际使用的设置并不多. 因此, 我们采用了 "设置向导 (The Setup wizard)" 来简化初始设置.
Note: 建议使用 `Copy setup URI` and `Open setup URI` 来设置后续设备.
## 设置向导 (The Setup wizard)
在设置对话框中打开 `🧙‍♂️ Setup wizard`. 如果之前未配置插件, 则会自动打开该页面.
![quick_setup_1](../images/quick_setup_1.png)
- 放弃现有配置并进行设置
如果您先前有过任何设置, 此按钮允许您在设置前放弃所有更改.
- 保留现有配置和设置
快速重新配置. 请注意, 在向导模式下, 您无法看到所有已经配置过的配置项.
在上述选项中按下 `Next`, 配置对话框将进入向导模式 (wizard mode).
### 向导模式 (Wizard mode)
![quick_setup_2](../images/quick_setup_2.png)
接下来将介绍如何逐步使用向导模式.
## 配置远程数据库
### 开始配置远程数据库
输入已部署好的数据库的信息.
![quick_setup_3](../images/quick_setup_3.png)
#### 测试数据库连接并检查数据库配置
我们可以检查数据库的连接性和数据库设置.
![quick_setup_5](../images/quick_setup_5.png)
#### 测试数据库连接
检查是否能成功连接数据库. 如果连接失败, 可能是多种原因导致的, 但请先点击 `Check database configuration` 来检查数据库配置是否有问题.
#### 检查数据库配置
检查数据库设置并修复问题.
![quick_setup_6](../images/quick_setup_6.png)
Config check 的显示内容可能因不同连接而异. 在上图情况下, 按下所有三个修复按钮.
如果修复按钮消失, 全部变为复选标记, 则表示修复完成.
### 加密配置
![quick_setup_4](../images/quick_setup_4.png)
为您的数据库加密, 以防数据库意外曝光; 启用端到端加密后, 笔记内容在离开设备时就会被加密. 我们强烈建议启用该功能. `路径混淆 (Path Obfuscation)` 还能混淆文件名. 现已稳定并推荐使用.
加密基于 256 位 AES-GCM.
如果你在一个封闭的网络中, 而且很明显第三方不会访问你的文件, 则可以禁用这些设置.
![quick_setup_7](../images/quick_setup_7.png)
#### Next
转到同步设置.
#### 放弃现有数据库并继续
清除远程数据库的内容, 然后转到同步设置.
### 同步设置
最后, 选择一个同步预设完成向导.
![quick_setup_9_1](../images/quick_setup_9_1.png)
选择我们要使用的任何同步方法, 然后 `Apply` 初始化并按要求建立本地和远程数据库. 如果显示 `All done!`, 我们就完成了. `Copy setup URI` 将自动打开，并要求我们输入密码以加密 `Setup URI`.
![quick_setup_10](../images/quick_setup_10.png)
根据需要设置密码。.
设置 URI (Setup URI) 将被复制到剪贴板, 然后您可以通过某种方式将其传输到第二个及后续设备.
## 如何设置第二单元和后续单元 (the second and subsequent units)
在第一台设备上安装 Self-hosted LiveSync 后, 从命令面板上选择 `Open setup URI`, 然后输入您传输的设置 URI (Setup URI). 然后输入密码，安装向导就会打开.
在弹窗中选择以下内容.
- `Importing LiveSync's conf, OK?` 选择 `Yes`
- `How would you like to set it up?`. 选择 `Set it up as secondary or subsequent device`
然后, 配置将生效并开始复制. 您的文件很快就会同步! 您可能需要关闭设置对话框并重新打开, 才能看到设置字段正确填充, 但它们都将设置好.

38
docs/quick_setup_ja.md
View File

@@ -1,10 +1,10 @@
# Quick setup
v0.15.0からSetup wizardが実装されました。これで、初回セットアップがシンプルになります。
このプラグインには、いろいろな状況に対応するための非常に多くの設定オプションがあります。しかし、実際に使用する設定項目はそれほど多くはありません。そこで、初期設定を簡略化するために、「セットアップウィザード」を実装しています。
※なお、次のデバイスからは、`Copy setup URI`と`Open setup URI`を使ってセットアップしてください。
## Wizardの使い方
`🪄 Setup wizard` から開きます。もしセットアップされていなかったり、同期設定が何も有効になっていない場合はデフォルトで開いています。
`🧙‍♂️ Setup wizard` から開きます。もしセットアップされていなかったり、同期設定が何も有効になっていない場合はデフォルトで開いています。
![](../images/quick_setup_1.png)
@@ -32,20 +32,12 @@ v0.15.0からSetup wizardが実装されました。これで、初回セット
これらはデータベースをセットアップした際に決めた情報です。
### End to End暗号化の設定
![](../images/quick_setup_4.png)
End to End暗号化を有効にした場合、万が一Remote databaseの内容が流出してもPassphraseを知らない第三者にそれを読まれる可能性が低くなります。そのため、有効化を強く推奨します。
暗号化は256bitのAES-GCMを採用しています。
この設定は、あなたが閉じたネットワークの内側にいて、かつ第三者からアクセスされない事が明確な場合には無効にできます。
### Test database connectionとCheck database configuraion
### Test database connectionとCheck database configuration
ここで、データベースへの接続状況と、データベース設定を確認します。
![](../images/quick_setup_5.png)
#### Test Database Connection
データベースに接続出来るか自体を確認します。失敗する場合はいくつか理由がありますが、一度Check database configurationを行ってそちらでも失敗するか確認してください。
データベースに接続できるか自体を確認します。失敗する場合はいくつか理由がありますが、一度Check database configurationを行ってそちらでも失敗するか確認してください。
#### Check database configuration
データベースの設定を確認し、不備がある場合はその場で修正します。
@@ -55,6 +47,15 @@ End to End暗号化を有効にした場合、万が一Remote databaseの内容
この項目は接続先によって異なる場合があります。上記の場合、みっつのFixボタンを順にすべて押してください。
Fixボタンがなくなり、すべてチェックマークになれば完了です。
### 機密性設定
![](../images/quick_setup_4.png)
意図しないデータベースの暴露に備えて、End to End Encryptionを有効にします。この項目を有効にした場合、デバイスを出る瞬間にノートの内容が暗号化されます。`Path Obfuscation`を有効にすると、ファイル名も難読化されます。現在は安定しているため、こちらも推奨されます。
暗号化には256bitのAES-GCMを採用しています。
これらの設定は、あなたが閉じたネットワークの内側にいて、かつ第三者からアクセスされない事が明確な場合には無効にできます。
![](../images/quick_setup_7.png)
### Next
@@ -63,20 +64,13 @@ Fixボタンがなくなり、すべてチェックマークになれば完了
### Discard exist database and proceed
すでにRemote databaseがある場合、Remote databaseの内容を破棄してから次へ進みます
## Local Database confiuration
![](../images/quick_setup_8.png)
ローカルのデータベースを設定します。もし、すでにSelf-hosted LiveSyncをインストールしたVaultがあり、そのVaultと同じデータベース名を使用している場合は、ここですでに設定したVaultとは異なるsuffixを指定してください。
## Miscellaneous
最後にその他の設定を行います。
## Sync Settings
最後に同期方法の設定を行います。
![](../images/quick_setup_9_1.png)
`Show status inside editor`はお好みで有効化してください。有効にするとエディターの右上にステータスが表示されます。
![](../images/quick_setup_9_2.png)
Presetsから、使用する同期方法を選び`Apply`を行うと、必要に応じてローカル・リモートのデータベースを初期化・構築します。
Presetsから、いずれかの同期方法を選び`Apply`を行うと、必要に応じてローカル・リモートのデータベースを初期化・構築します。
All done! と表示されれば完了です。自動的に、`Copy setup URI`が開き、`Setup URI`を暗号化するパスフレーズを聞かれます。
![](../images/quick_setup_10.png)

287
docs/setup_flyio.md Normal file
View File

@@ -0,0 +1,287 @@
# Setup CouchDB on fly.io
In some cases, the use of IBM Cloudant was found to be hard. We looked for alternatives, but there were no services available. Therefore, we have to build our own servers, which is quite a challenge. In this situation, with fly.io, most of the processes are simplified and only CouchDB needs to be configured.
This is how to configure fly.io and CouchDB on it for Self-hosted LiveSync.
It generally falls within the `Free Allowances` range in most cases.
**[Automatic setup using Colaboratory](#automatic-setup-using-colaboratory) is recommended, after reading this document through. It is reproducible and hard to fail.**
## Required materials
- A valid credit or debit card.
## Warning
- It will be `your` instance. Check the log regularly.
## Prerequisites
For simplicity, the following description assumes that the settings are as shown in the table below. Please read it in accordance with the actual settings you wish to make.
| Used in the text | Meaning and where to use | Memo |
| ---------------- | --------------------------- | ------------------------------------------------------------------------ |
| campanella | Username | It is less likely to fail if it consists only of letters and numbers. |
| dfusiuada9suy | Password | |
| nrt | Region to make the instance | We can use any [region](https://fly.io/docs/reference/regions/) near us. |
## Steps with your computer
If you want to avoid installing anything, please skip to [Automatic setup using Colaboratory](#automatic-setup-using-colaboratory).
### 1. Install flyctl
- Mac or Linux
```sh
$ curl -L https://fly.io/install.sh | sh
```
- Windows
```powershell
$ iwr https://fly.io/install.ps1 -useb | iex
```
### 2. Sign up or Sign in to fly.io
- Sign up
```bash
$ fly auth signup
```
- Sign in
```bash
$ fly auth login
```
For more information, please refer [Sign up](https://fly.io/docs/hands-on/sign-up/) and [Sign in](https://fly.io/docs/hands-on/sign-in/).
### 3. Make configuration files
Please be careful, `nrt` is the region where near to Japan. Please use your preferred region.
1. Make fly.toml
```
$ flyctl launch --generate-name --detach --no-deploy --region nrt
Creating app in /home/vrtmrz/dev/fly/demo
Scanning source code
Could not find a Dockerfile, nor detect a runtime or framework from source code. Continuing with a blank app.
automatically selected personal organization: vorotamoroz
App will use 'nrt' region as primary
Created app 'billowing-dawn-6619' in organization 'personal'
Admin URL: https://fly.io/apps/billowing-dawn-6619
Hostname: billowing-dawn-6619.fly.dev
Wrote config file fly.toml
```
`billowing-dawn-6619` is an automatically generated name. It is used as the hostname. Please note it in something.
Note: we can specify this without `--generate-name`, but does not recommend in the trial phases.
1. Make volume
```
$ flyctl volumes create --region nrt couchdata --size 2 --yes
ID: vol_g67340kxgmmvydxw
Name: couchdata
App: billowing-dawn-6619
Region: nrt
Zone: 35b7
Size GB: 2
Encrypted: true
Created at: 02 Jun 23 01:19 UTC
```
3. Edit fly.toml
Changes:
- Change exposing port from `8080` to `5984`
- Mounting the volume `couchdata` created in step 2 under `/opt/couchdb/data`
- Set `campanella` for the administrator of CouchDB
- Customise CouchDB to use persistent ini-file; which is located under the data directory.
- To use Dockerfile
```diff
# fly.toml app configuration file generated for billowing-dawn-6619 on 2023-06-02T10:18:59+09:00
#
# See https://fly.io/docs/reference/configuration/ for information about how to use this file.
#
app = "billowing-dawn-6619"
primary_region = "nrt"
[http_service]
- internal_port = 8080
+ internal_port = 5984
force_https = true
auto_stop_machines = true
auto_start_machines = true
min_machines_running = 0
+[mounts]
+ source="couchdata"
+ destination="/opt/couchdb/data"
+
+[env]
+ COUCHDB_USER = "campanella"
+ ERL_FLAGS="-couch_ini /opt/couchdb/etc/default.ini /opt/couchdb/etc/default.d/ /opt/couchdb/etc/local.d /opt/couchdb/etc/local.ini /opt/couchdb/data/persistence.ini"
+
+[build]
+ dockerfile = "./Dockerfile"
```
4. Make `Dockerfile`
Create a Dockerfile that patches the start-up script to fix ini file permissions.
```dockerfile
FROM couchdb:latest
RUN sed -i '2itouch /opt/couchdb/data/persistence.ini && chmod +w /opt/couchdb/data/persistence.ini' /docker-entrypoint.sh
```
5. Set credential
```
flyctl secrets set COUCHDB_PASSWORD=dfusiuada9suy
```
### 4. Deploy
```
$ flyctl deploy --detach --remote-only
==> Verifying app config
Validating /home/vrtmrz/dev/fly/demo/fly.toml
Platform: machines
✓ Configuration is valid
--> Verified app config
==> Building image
Remote builder fly-builder-bold-sky-4515 ready
==> Creating build context
--> Creating build context done
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
-------------:SNIPPED:-------------
Watch your app at https://fly.io/apps/billowing-dawn-6619/monitoring
Provisioning ips for billowing-dawn-6619
Dedicated ipv6: 2a09:8280:1::2d:240f
Shared ipv4: 66.241.125.213
Add a dedicated ipv4 with: fly ips allocate-v4
This deployment will:
* create 1 "app" machine
No machines in group app, launching a new machine
Machine e7845d1f297183 [app] has state: started
Finished launching new machines
NOTE: The machines for [app] have services with 'auto_stop_machines = true' that will be stopped when idling
```
Now your CouchDB has been launched. (Do not forget to delete it if no longer need).
If failed, please check by `flyctl doctor`. Failure of remote build may be resolved by `flyctl` wireguard reset` or something.
```
$ flyctl status
App
Name = billowing-dawn-6619
Owner = personal
Hostname = billowing-dawn-6619.fly.dev
Image = billowing-dawn-6619:deployment-01H1WWB3CK5Z9ZX71KHBSDGHF1
Platform = machines
Machines
PROCESS ID VERSION REGION STATE CHECKS LAST UPDATED
app e7845d1f297183 1 nrt started 2023-06-02T01:43:34Z
```
### 5. Apply CouchDB configuration
After the initial setup, CouchDB needs some more customisations to be used from Self-hosted LiveSync. It can be configured in browsers or by HTTP-REST APIs.
This section is set up using the REST API.
1. Prepare environment variables.
- Mac or Linux:
```bash
export couchHost=https://billowing-dawn-6619.fly.dev
export couchUser=campanella
export couchPwd=dfusiuada9suy
```
- Windows
```powershell
set couchHost https://billowing-dawn-6619.fly.dev
set couchUser campanella
set couchPwd dfusiuada9suy
$creds = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes("${couchUser}:${couchPwd}"))
```
2. Perform cluster setup
- Mac or Linux
```bash
curl -X POST "${couchHost}/_cluster_setup" -H "Content-Type: application/json" -d "{\"action\":\"enable_single_node\",\"username\":\"${couchUser}\",\"password\":\"${couchPwd}\",\"bind_address\":\"0.0.0.0\",\"port\":5984,\"singlenode\":true}" --user "${couchUser}:${couchPwd}"
```
- Windows
```powershell
iwr -UseBasicParsing -Method 'POST' -ContentType 'application/json; charset=utf-8' -Headers @{ 'Authorization' = 'Basic ' + $creds } "${couchHost}/_cluster_setup" -Body "{""action"":""enable_single_node"",""username"":""${couchUser}"",""password"":""${couchPwd}"",""bind_address"":""0.0.0.0"",""port"":5984,""singlenode"":true}"
```
Note: if the response code is not 200. We have to retry the request once again.
If you run the request several times and it does not result in 200, something is wrong. Please report it.
3. Configure parameters
- Mac or Linux
```bash
curl -X PUT "${couchHost}/_node/nonode@nohost/_config/chttpd/require_valid_user" -H "Content-Type: application/json" -d '"true"' --user "${couchUser}:${couchPwd}"
curl -X PUT "${couchHost}/_node/nonode@nohost/_config/chttpd_auth/require_valid_user" -H "Content-Type: application/json" -d '"true"' --user "${couchUser}:${couchPwd}"
curl -X PUT "${couchHost}/_node/nonode@nohost/_config/httpd/WWW-Authenticate" -H "Content-Type: application/json" -d '"Basic realm=\"couchdb\""' --user "${couchUser}:${couchPwd}"
curl -X PUT "${couchHost}/_node/nonode@nohost/_config/httpd/enable_cors" -H "Content-Type: application/json" -d '"true"' --user "${couchUser}:${couchPwd}"
curl -X PUT "${couchHost}/_node/nonode@nohost/_config/chttpd/enable_cors" -H "Content-Type: application/json" -d '"true"' --user "${couchUser}:${couchPwd}"
curl -X PUT "${couchHost}/_node/nonode@nohost/_config/chttpd/max_http_request_size" -H "Content-Type: application/json" -d '"4294967296"' --user "${couchUser}:${couchPwd}"
curl -X PUT "${couchHost}/_node/nonode@nohost/_config/couchdb/max_document_size" -H "Content-Type: application/json" -d '"50000000"' --user "${couchUser}:${couchPwd}"
curl -X PUT "${couchHost}/_node/nonode@nohost/_config/cors/credentials" -H "Content-Type: application/json" -d '"true"' --user "${couchUser}:${couchPwd}"
curl -X PUT "${couchHost}/_node/nonode@nohost/_config/cors/origins" -H "Content-Type: application/json" -d '"app://obsidian.md,capacitor://localhost,http://localhost"' --user "${couchUser}:${couchPwd}"
```
- Windows
```powershell
iwr -UseBasicParsing -Method 'PUT' -ContentType 'application/json; charset=utf-8' -Headers @{ 'Authorization' = 'Basic ' + $creds } "${couchHost}/_node/nonode@nohost/_config/chttpd/require_valid_user" -Body '"true"'
iwr -UseBasicParsing -Method 'PUT' -ContentType 'application/json; charset=utf-8' -Headers @{ 'Authorization' = 'Basic ' + $creds } "${couchHost}/_node/nonode@nohost/_config/chttpd_auth/require_valid_user" -Body '"true"'
iwr -UseBasicParsing -Method 'PUT' -ContentType 'application/json; charset=utf-8' -Headers @{ 'Authorization' = 'Basic ' + $creds } "${couchHost}/_node/nonode@nohost/_config/httpd/WWW-Authenticate" -Body '"Basic realm=\"couchdb\""'
iwr -UseBasicParsing -Method 'PUT' -ContentType 'application/json; charset=utf-8' -Headers @{ 'Authorization' = 'Basic ' + $creds } "${couchHost}/_node/nonode@nohost/_config/httpd/enable_cors" -Body '"true"'
iwr -UseBasicParsing -Method 'PUT' -ContentType 'application/json; charset=utf-8' -Headers @{ 'Authorization' = 'Basic ' + $creds } "${couchHost}/_node/nonode@nohost/_config/chttpd/enable_cors" -Body '"true"'
iwr -UseBasicParsing -Method 'PUT' -ContentType 'application/json; charset=utf-8' -Headers @{ 'Authorization' = 'Basic ' + $creds } "${couchHost}/_node/nonode@nohost/_config/chttpd/max_http_request_size" -Body '"4294967296"'
iwr -UseBasicParsing -Method 'PUT' -ContentType 'application/json; charset=utf-8' -Headers @{ 'Authorization' = 'Basic ' + $creds } "${couchHost}/_node/nonode@nohost/_config/couchdb/max_document_size" -Body '"50000000"'
iwr -UseBasicParsing -Method 'PUT' -ContentType 'application/json; charset=utf-8' -Headers @{ 'Authorization' = 'Basic ' + $creds } "${couchHost}/_node/nonode@nohost/_config/cors/credentials" -Body '"true"'
iwr -UseBasicParsing -Method 'PUT' -ContentType 'application/json; charset=utf-8' -Headers @{ 'Authorization' = 'Basic ' + $creds } "${couchHost}/_node/nonode@nohost/_config/cors/origins" -Body '"app://obsidian.md,capacitor://localhost,http://localhost"'
```
Note: Each of these should also be repeated until finished in 200.
### 6. Use it from Self-hosted LiveSync
Now the CouchDB is ready to use from Self-hosted LiveSync. We can use `https://billowing-dawn-6619.fly.dev` in URI, `campanella` in `Username` and `dfusiuada9suy` in `Password` on Self-hosted LiveSync. `Database name` could be anything you want.
`Enhance chunk size` could be up to around `100`.
## Automatic setup using Colaboratory
We can perform all these steps by using [this Colaboratory notebook](/deploy_couchdb_to_flyio_v2_with_swap.ipynb) without installing anything.
## After testing / before creating a new instance
**Be sure to delete the instance. We can check instances on the [Dashboard](https://fly.io/dashboard/personal)**

149
docs/setup_own_server.md
View File

@@ -1,13 +1,24 @@
# Setup CouchDB to your server
# Setup a CouchDB server
## Table of Contents
- [Configure](#configure)
- [Run](#run)
- [Docker CLI](#docker-cli)
- [Docker Compose](#docker-compose)
- [Access from a mobile device](#access-from-a-mobile-device)
- [Testing from a mobile](#testing-from-a-mobile)
- [Setting up your domain](#setting-up-your-domain)
- [Reverse Proxies](#reverse-proxies)
- [Traefik](#traefik)
---
## Install CouchDB and access from a PC or Mac
## Configure
The easiest way to set up the CouchDB is using the [docker image]((https://hub.docker.com/_/couchdb)).
The easiest way to set up a CouchDB instance is using the official [docker image](https://hub.docker.com/_/couchdb).
But some additional configurations are required in `local.ini` to use from Self-hosted LiveSync, like below:
Some initial configuration is required. Create a `local.ini` to use Self-hosted LiveSync as follows ([CouchDB has to be version 3.2 or higher](https://docs.couchdb.org/en/latest/config/http.html#chttpd/enable_cors), if lower `enable_cors = true` has to be under section `[httpd]` ):
```
```ini
[couchdb]
single_node=true
max_document_size = 50000000
@@ -15,6 +26,7 @@ max_document_size = 50000000
[chttpd]
require_valid_user = true
max_http_request_size = 4294967296
enable_cors = true
[chttpd_auth]
require_valid_user = true
@@ -22,38 +34,72 @@ authentication_redirect = /_utils/session.html
[httpd]
WWW-Authenticate = Basic realm="couchdb"
enable_cors = true
bind_address = 0.0.0.0
[cors]
origins = app://obsidian.md,capacitor://localhost,http://localhost
origins = app://obsidian.md, capacitor://localhost, http://localhost
credentials = true
headers = accept, authorization, content-type, origin, referer
methods = GET, PUT, POST, HEAD, DELETE
methods = GET,PUT,POST,HEAD,DELETE
max_age = 3600
```
Make `local.ini` and run with docker run like this, you can launch the CouchDB.
## Run
### Docker CLI
You can launch CouchDB using your `local.ini` like this:
```
$ docker run --rm -it -e COUCHDB_USER=admin -e COUCHDB_PASSWORD=password -v /path/to/local.ini:/opt/couchdb/etc/local.ini -p 5984:5984 couchdb
```
*Remember to replace the path with the path to your local.ini*
Note: At this time, the file owner of local.ini became 5984:5984. It's the limitation docker image. please change the owner before editing local.ini again.
If you could confirm that Self-hosted LiveSync can sync with the server, launch the docker image as a background as you like.
Example to run docker in detached mode:
Run in detached mode:
```
$ docker run -d --restart always -e COUCHDB_USER=admin -e COUCHDB_PASSWORD=password -v /path/to/local.ini:/opt/couchdb/etc/local.ini -p 5984:5984 couchdb
```
*Remember to replace the path with the path to your local.ini*
### Docker Compose
Create a directory, place your `local.ini` within it, and create a `docker-compose.yml` alongside it. Make sure to have write permissions for `local.ini` and the about to be created `data` folder after the container start. The directory structure should look similar to this:
```
obsidian-livesync
├── docker-compose.yml
└── local.ini
```
A good place to start for `docker-compose.yml`:
```yaml
version: "2.1"
services:
couchdb:
image: couchdb
container_name: obsidian-livesync
user: 1000:1000
environment:
- COUCHDB_USER=admin
- COUCHDB_PASSWORD=password
volumes:
- ./data:/opt/couchdb/data
- ./local.ini:/opt/couchdb/etc/local.ini
ports:
- 5984:5984
restart: unless-stopped
```
And finally launch the container
```
# -d will launch detached so the container runs in background
docker compose up -d
```
## Access from a mobile device
If you want to access Self-hosted LiveSync from mobile devices, you need a valid SSL certificate.
### Testing from a mobile
In the testing phase, [localhost.run](http://localhost.run/) or something like services is very useful.
In the testing phase, [localhost.run](https://localhost.run/) or something like services is very useful.
example on using localhost.run)
example using localhost.run:
```
$ ssh -R 80:localhost:5984 nokey@localhost.run
Warning: Permanently added the RSA host key for IP address '35.171.254.69' to the list of known hosts.
@@ -94,6 +140,77 @@ Set the A record of your domain to point to your server, and host reverse proxy
Note: Mounting CouchDB on the top directory is not recommended.
Using Caddy is a handy way to serve the server with SSL automatically.
I have published [docker-compose.yml and ini files](https://github.com/vrtmrz/self-hosted-livesync-server) that launch Caddy and CouchDB at once. Please try it out.
I have published [docker-compose.yml and ini files](https://github.com/vrtmrz/self-hosted-livesync-server) that launch Caddy and CouchDB at once. If you are using Traefik you can check the [Reverse Proxies](#reverse-proxies) section below.
And, be sure to check the server log and be careful of malicious access.
## Reverse Proxies
### Traefik
If you are using Traefik, this [docker-compose.yml](https://github.com/vrtmrz/obsidian-livesync/blob/main/docker-compose.traefik.yml) file (also pasted below) has all the right CORS parameters set. It assumes you have an external network called `proxy`.
```yaml
version: "2.1"
services:
couchdb:
image: couchdb:latest
container_name: obsidian-livesync
user: 1000:1000
environment:
- COUCHDB_USER=username
- COUCHDB_PASSWORD=password
volumes:
- ./data:/opt/couchdb/data
- ./local.ini:/opt/couchdb/etc/local.ini
# Ports not needed when already passed to Traefik
#ports:
# - 5984:5984
restart: unless-stopped
networks:
- proxy
labels:
- "traefik.enable=true"
# The Traefik Network
- "traefik.docker.network=proxy"
# Don't forget to replace 'obsidian-livesync.example.org' with your own domain
- "traefik.http.routers.obsidian-livesync.rule=Host(`obsidian-livesync.example.org`)"
# The 'websecure' entryPoint is basically your HTTPS entrypoint. Check the next code snippet if you are encountering problems only; you probably have a working traefik configuration if this is not your first container you are reverse proxying.
- "traefik.http.routers.obsidian-livesync.entrypoints=websecure"
- "traefik.http.routers.obsidian-livesync.service=obsidian-livesync"
- "traefik.http.services.obsidian-livesync.loadbalancer.server.port=5984"
- "traefik.http.routers.obsidian-livesync.tls=true"
# Replace the string 'letsencrypt' with your own certificate resolver
- "traefik.http.routers.obsidian-livesync.tls.certresolver=letsencrypt"
- "traefik.http.routers.obsidian-livesync.middlewares=obsidiancors"
# The part needed for CORS to work on Traefik 2.x starts here
- "traefik.http.middlewares.obsidiancors.headers.accesscontrolallowmethods=GET,PUT,POST,HEAD,DELETE"
- "traefik.http.middlewares.obsidiancors.headers.accesscontrolallowheaders=accept,authorization,content-type,origin,referer"
- "traefik.http.middlewares.obsidiancors.headers.accesscontrolalloworiginlist=app://obsidian.md,capacitor://localhost,http://localhost"
- "traefik.http.middlewares.obsidiancors.headers.accesscontrolmaxage=3600"
- "traefik.http.middlewares.obsidiancors.headers.addvaryheader=true"
- "traefik.http.middlewares.obsidiancors.headers.accessControlAllowCredentials=true"
networks:
proxy:
external: true
```
Partial `traefik.yml` config file mentioned in above:
```yml
...
entryPoints:
web:
address: ":80"
http:
redirections:
entryPoint:
to: "websecure"
scheme: "https"
websecure:
address: ":443"
...
```

72
docs/setup_own_server_cn.md
View File

@@ -1,8 +1,19 @@
# 在你自己的服务器上设置 CouchDB
## 目录
- [配置 CouchDB](#配置-CouchDB)
- [运行 CouchDB](#运行-CouchDB)
- [Docker CLI](#docker-cli)
- [Docker Compose](#docker-compose)
- [创建数据库](#创建数据库)
- [从移动设备访问](#从移动设备访问)
- [移动设备测试](#移动设备测试)
- [设置你的域名](#设置你的域名)
---
> 注：提供了 [docker-compose.yml 和 ini 文件](https://github.com/vrtmrz/self-hosted-livesync-server) 可以同时启动 Caddy 和 CouchDB。推荐直接使用该 docker-compose 配置进行搭建。（若使用，请查阅链接中的文档，而不是这个文档）
## 安装 CouchDB 并从 PC 或 Mac 上访问
## 配置 CouchDB
设置 CouchDB 的最简单方法是使用 [CouchDB docker image]((https://hub.docker.com/_/couchdb)).
@@ -33,17 +44,62 @@ methods = GET, PUT, POST, HEAD, DELETE
max_age = 3600
```
创建 `local.ini` 并用如下指令启动 CouchDB：
```
$ docker run --rm -it -e COUCHDB_USER=admin -e COUCHDB_PASSWORD=password -v .local.ini:/opt/couchdb/etc/local.ini -p 5984:5984 couchdb
```
Note: 此时 local.ini 的文件所有者会变成 5984:5984。这是 docker 镜像的限制，请修改文件所有者后再编辑 local.ini。
## 运行 CouchDB
在确定 Self-hosted LiveSync 可以和服务器同步后，可以后台启动 docker 镜像：
### Docker CLI
你可以通过指定 `local.ini` 配置运行 CouchDB:
```
$ docker run -d --restart always -e COUCHDB_USER=admin -e COUCHDB_PASSWORD=password -v .local.ini:/opt/couchdb/etc/local.ini -p 5984:5984 couchdb
$ docker run --rm -it -e COUCHDB_USER=admin -e COUCHDB_PASSWORD=password -v /path/to/local.ini:/opt/couchdb/etc/local.ini -p 5984:5984 couchdb
```
*记得将上述命令中的 local.ini 挂载路径替换成实际的存放路径*
后台运行:
```
$ docker run -d --restart always -e COUCHDB_USER=admin -e COUCHDB_PASSWORD=password -v /path/to/local.ini:/opt/couchdb/etc/local.ini -p 5984:5984 couchdb
```
*记得将上述命令中的 local.ini 挂载路径替换成实际的存放路径*
### Docker Compose
创建一个文件夹, 将你的 `local.ini` 放在文件夹内, 然后在文件夹内创建 `docker-compose.yml`. 请确保对 `local.ini` 有读写权限并且确保在容器运行后能创建 `data` 文件夹. 文件夹结构大概如下:
```
obsidian-livesync
├── docker-compose.yml
└── local.ini
```
可以参照以下内容编辑 `docker-compose.yml`:
```yaml
version: "2.1"
services:
couchdb:
image: couchdb
container_name: obsidian-livesync
user: 1000:1000
environment:
- COUCHDB_USER=admin
- COUCHDB_PASSWORD=password
volumes:
- ./data:/opt/couchdb/data
- ./local.ini:/opt/couchdb/etc/local.ini
ports:
- 5984:5984
restart: unless-stopped
```
最后, 创建并启动容器:
```
# -d will launch detached so the container runs in background
docker compose up -d
```
## 创建数据库
CouchDB 部署成功后, 需要手动创建一个数据库, 方便插件连接并同步.
1. 访问 `http://localhost:5984/_utils`, 输入帐号密码后进入管理页面
2. 点击 Create Database, 然后根据个人喜好创建数据库
## 从移动设备访问
如果你想要从移动设备访问 Self-hosted LiveSync，你需要一个合法的 SSL 证书。

2
docs/setup_own_server_ja.md
View File

@@ -12,10 +12,10 @@ max_document_size = 50000000
[chttpd]
require_valid_user = true
max_http_request_size = 4294967296
[chttpd_auth]
require_valid_user = true
max_http_request_size = 4294967296
authentication_redirect = /_utils/session.html
[httpd]

12
docs/troubleshooting.md Normal file
View File

@@ -0,0 +1,12 @@
# Info
In this document some tips for solving issues will be given.
# Issue with syncing with mobile device
- If you are getting problem like could not sync files when doing initial sync. Try lower batch size
- Open setting
- Open Live Sync settings
- Go to sync settings
- Go down to Advanced settings.
- Lower "Batch size" and "Batch limit" and try to sync again.
- If you keep getting error keep lowering until you find the sweet spot.

185
esbuild.config.mjs
View File

@@ -1,46 +1,165 @@
//@ts-check
import esbuild from "esbuild";
import process from "process";
import builtins from "builtin-modules";
import sveltePlugin from "esbuild-svelte";
import sveltePreprocess from "svelte-preprocess";
import fs from "node:fs";
// import terser from "terser";
import { minify } from "terser";
const banner = `/*
THIS IS A GENERATED/BUNDLED FILE BY ESBUILD
THIS IS A GENERATED/BUNDLED FILE BY ESBUILD AND TERSER
if you want to view the source, please visit the github repository of this plugin
*/
`;
const prod = process.argv[2] === "production";
const manifestJson = JSON.parse(fs.readFileSync("./manifest.json"));
const packageJson = JSON.parse(fs.readFileSync("./package.json"));
const terserOpt = {
sourceMap: (!prod ? {
url: "inline"
} : {}),
format: {
indent_level: 2,
beautify: true,
comments: "some",
ecma: 2018,
preamble: banner,
webkit: true
},
parse: {
// parse options
},
compress: {
// compress options
defaults: false,
evaluate: true,
inline: 3,
join_vars: true,
loops: true,
passes: prod ? 4 : 1,
reduce_vars: true,
reduce_funcs: true,
arrows: true,
collapse_vars: true,
comparisons: true,
lhs_constants: true,
hoist_props: true,
side_effects: true,
if_return: true,
ecma: 2018,
unused: true,
},
// mangle: {
// // mangle options
// keep_classnames: true,
// keep_fnames: true,
// properties: {
// // mangle property options
// }
// },
ecma: 2018, // specify one of: 5, 2015, 2016, etc.
enclose: false, // or specify true, or "args:values"
keep_classnames: true,
keep_fnames: true,
ie8: false,
module: false,
// nameCache: null, // or specify a name cache object
safari10: false,
toplevel: false
}
const manifestJson = JSON.parse(fs.readFileSync("./manifest.json") + "");
const packageJson = JSON.parse(fs.readFileSync("./package.json") + "");
const updateInfo = JSON.stringify(fs.readFileSync("./updates.md") + "");
esbuild
.build({
banner: {
js: banner,
},
entryPoints: ["src/main.ts"],
bundle: true,
define: {
"MANIFEST_VERSION": `"${manifestJson.version}"`,
"PACKAGE_VERSION": `"${packageJson.version}"`,
"UPDATE_INFO": `${updateInfo}`,
"global":"window",
},
external: ["obsidian", "electron", "crypto"],
format: "cjs",
watch: !prod,
target: "es2018",
logLevel: "info",
sourcemap: prod ? false : "inline",
treeShaking: true,
platform: "browser",
plugins: [
sveltePlugin({
preprocess: sveltePreprocess(),
compilerOptions: { css: true },
}),
],
outfile: "main.js",
})
.catch(() => process.exit(1));
/** @type esbuild.Plugin[] */
const plugins = [{
name: 'my-plugin',
setup(build) {
let count = 0;
build.onEnd(async result => {
if (count++ === 0) {
console.log('first build:', result);
} else {
console.log('subsequent build:');
}
if (prod) {
console.log("Performing terser");
const src = fs.readFileSync("./main_org.js").toString();
// @ts-ignore
const ret = await minify(src, terserOpt);
if (ret && ret.code) {
fs.writeFileSync("./main.js", ret.code);
}
console.log("Finished terser");
} else {
fs.copyFileSync("./main_org.js", "./main.js");
}
});
},
}];
const context = await esbuild.context({
banner: {
js: banner,
},
entryPoints: ["src/main.ts"],
bundle: true,
define: {
"MANIFEST_VERSION": `"${manifestJson.version}"`,
"PACKAGE_VERSION": `"${packageJson.version}"`,
"UPDATE_INFO": `${updateInfo}`,
"global": "window",
},
external: [
"obsidian",
"electron",
"crypto",
"@codemirror/autocomplete",
"@codemirror/collab",
"@codemirror/commands",
"@codemirror/language",
"@codemirror/lint",
"@codemirror/search",
"@codemirror/state",
"@codemirror/view",
"@lezer/common",
"@lezer/highlight",
"@lezer/lr"],
// minifyWhitespace: true,
format: "cjs",
target: "es2018",
logLevel: "info",
platform: "browser",
sourcemap: prod ? false : "inline",
treeShaking: true,
outfile: "main_org.js",
minifyWhitespace: false,
minifySyntax: false,
minifyIdentifiers: false,
minify: false,
// keepNames: true,
plugins: [
sveltePlugin({
preprocess: sveltePreprocess(),
compilerOptions: { css: true, preserveComments: true },
}),
...plugins
],
})
if (prod) {
await context.rebuild();
process.exit(0);
} else {
await context.watch();
}

BIN
images/quick_setup_1.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 39 KiB

After

Width:  |  Height:  |  Size: 25 KiB

BIN
images/quick_setup_2.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 6.2 KiB

BIN
images/quick_setup_4.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 31 KiB

After

Width:  |  Height:  |  Size: 20 KiB

BIN
images/quick_setup_9_1.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 35 KiB

After

Width:  |  Height:  |  Size: 6.4 KiB

2
manifest.json
View File

@@ -1,7 +1,7 @@
{
"id": "obsidian-livesync",
"name": "Self-hosted LiveSync",
"version": "0.19.4",
"version": "0.21.2",
"minAppVersion": "0.9.12",
"description": "Community implementation of self-hosted livesync. Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.",
"author": "vorotamoroz",

3400
package-lock.json generated
View File

File diff suppressed because it is too large Load Diff

45
package.json
View File

@@ -1,6 +1,6 @@
{
"name": "obsidian-livesync",
"version": "0.19.4",
"version": "0.21.2",
"description": "Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.",
"main": "main.js",
"type": "module",
@@ -13,40 +13,51 @@
"author": "vorotamoroz",
"license": "MIT",
"devDependencies": {
"@tsconfig/svelte": "^5.0.0",
"@types/diff-match-patch": "^1.0.32",
"@types/node": "^20.2.5",
"@types/pouchdb": "^6.4.0",
"@types/pouchdb-adapter-http": "^6.1.3",
"@types/pouchdb-adapter-idb": "^6.1.4",
"@types/pouchdb-browser": "^6.1.3",
"@typescript-eslint/eslint-plugin": "^5.54.0",
"@typescript-eslint/parser": "^5.54.0",
"@types/pouchdb-core": "^7.0.11",
"@types/pouchdb-mapreduce": "^6.1.7",
"@types/pouchdb-replication": "^6.4.4",
"@types/transform-pouch": "^1.0.2",
"@typescript-eslint/eslint-plugin": "^6.2.1",
"@typescript-eslint/parser": "^6.2.1",
"builtin-modules": "^3.3.0",
"esbuild": "0.15.15",
"esbuild-svelte": "^0.7.3",
"eslint": "^8.35.0",
"esbuild": "0.18.17",
"esbuild-svelte": "^0.7.4",
"eslint": "^8.46.0",
"eslint-config-airbnb-base": "^15.0.0",
"eslint-plugin-import": "^2.27.5",
"eslint-plugin-import": "^2.28.0",
"events": "^3.3.0",
"obsidian": "^1.1.1",
"postcss": "^8.4.21",
"obsidian": "^1.4.11",
"postcss": "^8.4.27",
"postcss-load-config": "^4.0.1",
"pouchdb-adapter-http": "^8.0.1",
"pouchdb-adapter-idb": "^8.0.1",
"pouchdb-adapter-indexeddb": "^8.0.1",
"pouchdb-core": "^8.0.1",
"pouchdb-errors": "^8.0.1",
"pouchdb-find": "^8.0.1",
"pouchdb-mapreduce": "^8.0.1",
"pouchdb-merge": "^8.0.1",
"pouchdb-replication": "^8.0.1",
"pouchdb-utils": "^8.0.1",
"svelte": "^3.55.1",
"svelte-preprocess": "^5.0.1",
"svelte": "^4.1.2",
"svelte-preprocess": "^5.0.4",
"terser": "^5.19.2",
"transform-pouch": "^2.0.0",
"tslib": "^2.5.0",
"typescript": "^4.9.5"
"tslib": "^2.6.1",
"typescript": "^5.1.6"
},
"dependencies": {
"diff-match-patch": "^1.0.5",
"esbuild": "0.15.15",
"esbuild-svelte": "^0.7.3",
"idb": "^7.1.1",
"xxhash-wasm": "^0.4.2"
"minimatch": "^9.0.3",
"xxhash-wasm": "0.4.2",
"xxhash-wasm-102": "npm:xxhash-wasm@^1.0.2"
}
}
}

361
src/CmdConfigSync.ts
View File

@@ -1,33 +1,138 @@
import { writable } from 'svelte/store';
import { Notice, PluginManifest, stringifyYaml, parseYaml } from "./deps";
import { Notice, type PluginManifest, parseYaml, normalizePath } from "./deps";
import { EntryDoc, LoadedEntry, LOG_LEVEL, InternalFileEntry, FilePathWithPrefix, FilePath, DocumentID } from "./lib/src/types";
import type { EntryDoc, LoadedEntry, InternalFileEntry, FilePathWithPrefix, FilePath, DocumentID, AnyEntry, SavingEntry } from "./lib/src/types";
import { LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE, MODE_SELECTIVE } from "./lib/src/types";
import { ICXHeader, PERIODIC_PLUGIN_SWEEP, } from "./types";
import { delay, getDocData } from "./lib/src/utils";
import { createTextBlob, delay, getDocData } from "./lib/src/utils";
import { Logger } from "./lib/src/logger";
import { PouchDB } from "./lib/src/pouchdb-browser.js";
import { WrappedNotice } from "./lib/src/wrapper";
import { base64ToArrayBuffer, arrayBufferToBase64, readString, writeString, uint8ArrayToHexString } from "./lib/src/strbin";
import { runWithLock } from "./lib/src/lock";
import { readString, crc32CKHash, decodeBinary, arrayBufferToBase64 } from "./lib/src/strbin";
import { serialized } from "./lib/src/lock";
import { LiveSyncCommands } from "./LiveSyncCommands";
import { stripAllPrefixes } from "./lib/src/path";
import { PeriodicProcessor, askYesNo, disposeMemoObject, memoIfNotExist, memoObject, retrieveMemoObject, scheduleTask } from "./utils";
import { Semaphore } from "./lib/src/semaphore";
import { PluginDialogModal } from "./dialogs";
import { JsonResolveModal } from "./JsonResolveModal";
import { pipeGeneratorToGenerator, processAllGeneratorTasksWithConcurrencyLimit } from './lib/src/task';
const d = "\u200b";
const d2 = "\n";
function serialize(data: PluginDataEx): string {
// For higher performance, create custom plug-in data strings.
// Self-hosted LiveSync uses `\n` to split chunks. Therefore, grouping together those with similar entropy would work nicely.
let ret = "";
ret += ":";
ret += data.category + d + data.name + d + data.term + d2;
ret += (data.version ?? "") + d2;
ret += data.mtime + d2;
for (const file of data.files) {
ret += file.filename + d + (file.displayName ?? "") + d + (file.version ?? "") + d2;
ret += file.mtime + d + file.size + d2;
for (const data of file.data ?? []) {
ret += data + d
}
ret += d2;
}
return ret;
}
function fetchToken(source: string, from: number): [next: number, token: string] {
const limitIdx = source.indexOf(d2, from);
const limit = limitIdx == -1 ? source.length : limitIdx;
const delimiterIdx = source.indexOf(d, from);
const delimiter = delimiterIdx == -1 ? source.length : delimiterIdx;
const tokenEnd = Math.min(limit, delimiter);
let next = tokenEnd;
if (limit < delimiter) {
next = tokenEnd;
} else {
next = tokenEnd + 1
}
return [next, source.substring(from, tokenEnd)];
}
function getTokenizer(source: string) {
const t = {
pos: 1,
next() {
const [next, token] = fetchToken(source, this.pos);
this.pos = next;
return token;
},
nextLine() {
const nextPos = source.indexOf(d2, this.pos);
if (nextPos == -1) {
this.pos = source.length;
} else {
this.pos = nextPos + 1;
}
}
}
return t;
}
function deserialize2(str: string): PluginDataEx {
const tokens = getTokenizer(str);
const ret = {} as PluginDataEx;
const category = tokens.next();
const name = tokens.next();
const term = tokens.next();
tokens.nextLine();
const version = tokens.next();
tokens.nextLine();
const mtime = Number(tokens.next());
tokens.nextLine();
const result: PluginDataEx = Object.assign(ret,
{ category, name, term, version, mtime, files: [] as PluginDataExFile[] })
let filename = "";
do {
filename = tokens.next();
if (!filename) break;
const displayName = tokens.next();
const version = tokens.next();
tokens.nextLine();
const mtime = Number(tokens.next());
const size = Number(tokens.next());
tokens.nextLine();
const data = [] as string[];
let piece = "";
do {
piece = tokens.next();
if (piece == "") break;
data.push(piece);
} while (piece != "");
result.files.push(
{
filename,
displayName,
version,
mtime,
size,
data
}
)
tokens.nextLine();
} while (filename);
return result;
}
function deserialize<T>(str: string, def: T) {
try {
if (str[0] == ":") return deserialize2(str);
return JSON.parse(str) as T;
} catch (ex) {
try {
return parseYaml(str);
} catch (ex) {
return def;
}
}
}
export const pluginList = writable([] as PluginDataExDisplay[]);
export const pluginIsEnumerating = writable(false);
const hashString = (async (key: string) => {
const buff = writeString(key);
const digest = await crypto.subtle.digest('SHA-256', buff);
return uint8ArrayToHexString(new Uint8Array(digest));
})
export type PluginDataExFile = {
filename: string,
data?: string[],
@@ -99,6 +204,7 @@ export class ConfigSync extends LiveSyncCommands {
},
});
}
getFileCategory(filePath: string): "CONFIG" | "THEME" | "SNIPPET" | "PLUGIN_MAIN" | "PLUGIN_ETC" | "PLUGIN_DATA" | "" {
if (filePath.split("/").length == 2 && filePath.endsWith(".json")) return "CONFIG";
if (filePath.split("/").length == 4 && filePath.startsWith(`${this.app.vault.configDir}/themes/`)) return "THEME";
@@ -131,7 +237,7 @@ export class ConfigSync extends LiveSyncCommands {
Logger("Scanning customizations : done");
} catch (ex) {
Logger("Scanning customizations : failed");
Logger(ex, LOG_LEVEL.VERBOSE);
Logger(ex, LOG_LEVEL_VERBOSE);
}
}
@@ -156,83 +262,104 @@ export class ConfigSync extends LiveSyncCommands {
pluginList.set(this.pluginList)
await this.updatePluginList(showMessage);
}
async loadPluginData(path: FilePathWithPrefix): Promise<PluginDataExDisplay | false> {
const wx = await this.localDatabase.getDBEntry(path, null, false, false);
if (wx) {
const data = deserialize(getDocData(wx.data), {}) as PluginDataEx;
const xFiles = [] as PluginDataExFile[];
for (const file of data.files) {
const work = { ...file };
const tempStr = getDocData(work.data);
work.data = [crc32CKHash(tempStr)];
xFiles.push(work);
}
return ({
...data,
documentPath: this.getPath(wx),
files: xFiles
}) as PluginDataExDisplay;
}
return false;
}
createMissingConfigurationEntry() {
let saveRequired = false;
for (const v of this.pluginList) {
const key = `${v.category}/${v.name}`;
if (!(key in this.plugin.settings.pluginSyncExtendedSetting)) {
this.plugin.settings.pluginSyncExtendedSetting[key] = {
key,
mode: MODE_SELECTIVE,
files: []
}
}
if (this.plugin.settings.pluginSyncExtendedSetting[key].files.sort().join(",").toLowerCase() !=
v.files.map(e => e.filename).sort().join(",").toLowerCase()) {
this.plugin.settings.pluginSyncExtendedSetting[key].files = v.files.map(e => e.filename).sort();
saveRequired = true;
}
}
if (saveRequired) {
this.plugin.saveSettingData();
}
}
async updatePluginList(showMessage: boolean, updatedDocumentPath?: FilePathWithPrefix): Promise<void> {
const logLevel = showMessage ? LOG_LEVEL.NOTICE : LOG_LEVEL.INFO;
const logLevel = showMessage ? LOG_LEVEL_NOTICE : LOG_LEVEL_INFO;
// pluginList.set([]);
if (!this.settings.usePluginSync) {
this.pluginList = [];
pluginList.set(this.pluginList)
return;
}
await runWithLock("update-plugin-list", false, async () => {
// if (updatedDocumentPath != "") pluginList.update(e => e.filter(ee => ee.documentPath != updatedDocumentPath));
// const work: Record<string, Record<string, Record<string, Record<string, PluginDataEntryEx>>>> = {};
const entries = [] as PluginDataExDisplay[]
const plugins = this.localDatabase.findEntries(ICXHeader + "", `${ICXHeader}\u{10ffff}`, { include_docs: true });
const semaphore = Semaphore(4);
const processes = [] as Promise<void>[];
let count = 0;
pluginIsEnumerating.set(true);
let processed = false;
try {
for await (const plugin of plugins) {
const path = plugin.path || this.getPath(plugin);
if (updatedDocumentPath && updatedDocumentPath != path) {
continue;
}
processed = true;
const oldEntry = (this.pluginList.find(e => e.documentPath == path));
if (oldEntry && oldEntry.mtime == plugin.mtime) continue;
processes.push((async (v) => {
const release = await semaphore.acquire(1);
await Promise.resolve(); // Just to prevent warning.
scheduleTask("update-plugin-list-task", 200, async () => {
await serialized("update-plugin-list", async () => {
try {
const updatedDocumentId = updatedDocumentPath ? await this.path2id(updatedDocumentPath) : "";
const plugins = updatedDocumentPath ?
this.localDatabase.findEntries(updatedDocumentId, updatedDocumentId + "\u{10ffff}", { include_docs: true, key: updatedDocumentId, limit: 1 }) :
this.localDatabase.findEntries(ICXHeader + "", `${ICXHeader}\u{10ffff}`, { include_docs: true });
let count = 0;
pluginIsEnumerating.set(true);
for await (const v of processAllGeneratorTasksWithConcurrencyLimit(20, pipeGeneratorToGenerator(plugins, async plugin => {
const path = plugin.path || this.getPath(plugin);
if (updatedDocumentPath && updatedDocumentPath != path) {
return false;
}
const oldEntry = (this.pluginList.find(e => e.documentPath == path));
if (oldEntry && oldEntry.mtime == plugin.mtime) return false;
try {
Logger(`Enumerating files... ${count++}`, logLevel, "get-plugins");
Logger(`plugin-${path}`, LOG_LEVEL.VERBOSE);
const wx = await this.localDatabase.getDBEntry(path, null, false, false);
if (wx) {
const data = parseYaml(getDocData(wx.data)) as PluginDataEx;
const xFiles = [] as PluginDataExFile[];
for (const file of data.files) {
const work = { ...file };
const tempStr = getDocData(work.data);
work.data = [await hashString(tempStr)];
xFiles.push(work);
}
entries.push({
...data,
documentPath: this.getPath(wx),
files: xFiles
});
}
count++;
if (count % 10 == 0) Logger(`Enumerating files... ${count}`, logLevel, "get-plugins");
Logger(`plugin-${path}`, LOG_LEVEL_VERBOSE);
return this.loadPluginData(path);
// return entries;
} catch (ex) {
//TODO
Logger(`Something happened at enumerating customization :${v.path}`, LOG_LEVEL.NOTICE);
Logger(`Something happened at enumerating customization :${path}`, LOG_LEVEL_NOTICE);
console.warn(ex);
} finally {
release();
}
return false;
}))) {
if ("ok" in v) {
if (v.ok !== false) {
let newList = [...this.pluginList];
const item = v.ok;
newList = newList.filter(x => x.documentPath != item.documentPath);
newList.push(item)
if (updatedDocumentPath != "") newList = newList.filter(e => e.documentPath != updatedDocumentPath);
this.pluginList = newList;
pluginList.set(newList);
}
}
}
)(plugin));
Logger(`All files enumerated`, logLevel, "get-plugins");
pluginIsEnumerating.set(false);
this.createMissingConfigurationEntry();
} finally {
pluginIsEnumerating.set(false);
}
await Promise.all(processes);
let newList = [...this.pluginList];
for (const item of entries) {
newList = newList.filter(x => x.documentPath != item.documentPath);
newList.push(item)
}
if (updatedDocumentPath != "" && !processed) newList = newList.filter(e => e.documentPath != updatedDocumentPath);
this.pluginList = newList;
pluginList.set(newList);
Logger(`All files enumerated`, logLevel, "get-plugins");
} finally {
pluginIsEnumerating.set(false);
}
});
pluginIsEnumerating.set(false);
});
// return entries;
}
@@ -241,9 +368,9 @@ export class ConfigSync extends LiveSyncCommands {
const docB = await this.localDatabase.getDBEntry(dataB.documentPath);
if (docA && docB) {
const pluginDataA = parseYaml(getDocData(docA.data)) as PluginDataEx;
const pluginDataA = deserialize(getDocData(docA.data), {}) as PluginDataEx;
pluginDataA.documentPath = dataA.documentPath;
const pluginDataB = parseYaml(getDocData(docB.data)) as PluginDataEx;
const pluginDataB = deserialize(getDocData(docB.data), {}) as PluginDataEx;
pluginDataB.documentPath = dataB.documentPath;
// Use outer structure to wrap each data.
@@ -256,8 +383,8 @@ export class ConfigSync extends LiveSyncCommands {
const fileA = { ...pluginDataA.files[0], ctime: pluginDataA.files[0].mtime, _id: `${pluginDataA.documentPath}` as DocumentID };
const fileB = pluginDataB.files[0];
const docAx = { ...docA, ...fileA } as LoadedEntry, docBx = { ...docB, ...fileB } as LoadedEntry
return runWithLock("config:merge-data", false, () => new Promise((res) => {
Logger("Opening data-merging dialog", LOG_LEVEL.VERBOSE);
return serialized("config:merge-data", () => new Promise((res) => {
Logger("Opening data-merging dialog", LOG_LEVEL_VERBOSE);
// const docs = [docA, docB];
const path = stripAllPrefixes(docAx.path.split("/").slice(-1).join("/") as FilePath);
const modal = new JsonResolveModal(this.app, path, [docAx, docBx], async (keep, result) => {
@@ -266,7 +393,7 @@ export class ConfigSync extends LiveSyncCommands {
res(await this.applyData(pluginDataA, result));
} catch (ex) {
Logger("Could not apply merged file");
Logger(ex, LOG_LEVEL.VERBOSE);
Logger(ex, LOG_LEVEL_VERBOSE);
res(false);
}
}, "📡", "🛰️", "B");
@@ -282,7 +409,7 @@ export class ConfigSync extends LiveSyncCommands {
if (dx == false) {
throw "Not found on database"
}
const loadedData = parseYaml(getDocData(dx.data)) as PluginDataEx;
const loadedData = deserialize(getDocData(dx.data), {}) as PluginDataEx;
for (const f of loadedData.files) {
Logger(`Applying ${f.filename} of ${data.displayName || data.name}..`);
try {
@@ -290,16 +417,16 @@ export class ConfigSync extends LiveSyncCommands {
const path = `${baseDir}/${f.filename}`;
await this.ensureDirectoryEx(path);
if (!content) {
const dt = base64ToArrayBuffer(f.data);
await this.app.vault.adapter.writeBinary(path, dt);
const dt = decodeBinary(f.data);
await this.vaultAccess.adapterWrite(path, dt);
} else {
await this.app.vault.adapter.write(path, content);
await this.vaultAccess.adapterWrite(path, content);
}
Logger(`Applying ${f.filename} of ${data.displayName || data.name}.. Done`);
} catch (ex) {
Logger(`Applying ${f.filename} of ${data.displayName || data.name}.. Failed`);
Logger(ex, LOG_LEVEL.VERBOSE);
Logger(ex, LOG_LEVEL_VERBOSE);
}
}
@@ -307,7 +434,7 @@ export class ConfigSync extends LiveSyncCommands {
await this.storeCustomizationFiles(uPath);
await this.updatePluginList(true, uPath);
await delay(100);
Logger(`Config ${data.displayName || data.name} has been applied`, LOG_LEVEL.NOTICE);
Logger(`Config ${data.displayName || data.name} has been applied`, LOG_LEVEL_NOTICE);
if (data.category == "PLUGIN_DATA" || data.category == "PLUGIN_MAIN") {
//@ts-ignore
const manifests = Object.values(this.app.plugins.manifests) as any as PluginManifest[];
@@ -315,12 +442,12 @@ export class ConfigSync extends LiveSyncCommands {
const enabledPlugins = this.app.plugins.enabledPlugins as Set<string>;
const pluginManifest = manifests.find((manifest) => enabledPlugins.has(manifest.id) && manifest.dir == `${baseDir}/plugins/${data.name}`);
if (pluginManifest) {
Logger(`Unloading plugin: ${pluginManifest.name}`, LOG_LEVEL.NOTICE, "plugin-reload-" + pluginManifest.id);
Logger(`Unloading plugin: ${pluginManifest.name}`, LOG_LEVEL_NOTICE, "plugin-reload-" + pluginManifest.id);
// @ts-ignore
await this.app.plugins.unloadPlugin(pluginManifest.id);
// @ts-ignore
await this.app.plugins.loadPlugin(pluginManifest.id);
Logger(`Plugin reloaded: ${pluginManifest.name}`, LOG_LEVEL.NOTICE, "plugin-reload-" + pluginManifest.id);
Logger(`Plugin reloaded: ${pluginManifest.name}`, LOG_LEVEL_NOTICE, "plugin-reload-" + pluginManifest.id);
}
} else if (data.category == "CONFIG") {
scheduleTask("configReload", 250, async () => {
@@ -333,7 +460,7 @@ export class ConfigSync extends LiveSyncCommands {
return true;
} catch (ex) {
Logger(`Applying ${data.displayName || data.name}.. Failed`);
Logger(ex, LOG_LEVEL.VERBOSE);
Logger(ex, LOG_LEVEL_VERBOSE);
return false;
}
}
@@ -342,11 +469,11 @@ export class ConfigSync extends LiveSyncCommands {
if (data.documentPath) {
await this.deleteConfigOnDatabase(data.documentPath);
await this.updatePluginList(false, data.documentPath);
Logger(`Delete: ${data.documentPath}`, LOG_LEVEL.NOTICE);
Logger(`Delete: ${data.documentPath}`, LOG_LEVEL_NOTICE);
}
return true;
} catch (ex) {
Logger(`Failed to delete: ${data.documentPath}`, LOG_LEVEL.NOTICE);
Logger(`Failed to delete: ${data.documentPath}`, LOG_LEVEL_NOTICE);
return false;
}
@@ -354,7 +481,7 @@ export class ConfigSync extends LiveSyncCommands {
async parseReplicationResultItem(docs: PouchDB.Core.ExistingDocument<EntryDoc>) {
if (docs._id.startsWith(ICXHeader)) {
if (this.plugin.settings.usePluginSync) {
await this.updatePluginList(false, docs.path ? docs.path : this.getPath(docs));
await this.updatePluginList(false, (docs as AnyEntry).path ? (docs as AnyEntry).path : this.getPath((docs as AnyEntry)));
}
if (this.plugin.settings.usePluginSync && this.plugin.settings.notifyPluginOrSettingUpdated) {
if (!this.pluginDialog || (this.pluginDialog && !this.pluginDialog.isOpened())) {
@@ -410,15 +537,16 @@ export class ConfigSync extends LiveSyncCommands {
this.periodicPluginSweepProcessor.enable(this.settings.autoSweepPluginsPeriodic && !this.settings.watchInternalFileChanges ? (PERIODIC_PLUGIN_SWEEP * 1000) : 0);
return;
}
recentProcessedInternalFiles = [] as string[];
async makeEntryFromFile(path: FilePath): Promise<false | PluginDataExFile> {
const stat = await this.app.vault.adapter.stat(path);
const stat = await this.vaultAccess.adapterStat(path);
let version: string | undefined;
let displayName: string | undefined;
if (!stat) {
return false;
}
const contentBin = await this.app.vault.adapter.readBinary(path);
const contentBin = await this.vaultAccess.adapterReadBinary(path);
let content: string[];
try {
content = await arrayBufferToBase64(contentBin);
@@ -433,12 +561,12 @@ export class ConfigSync extends LiveSyncCommands {
displayName = `${json.name}`;
}
} catch (ex) {
Logger(`Configuration sync data: ${path} looks like manifest, but could not read the version`, LOG_LEVEL.INFO);
Logger(`Configuration sync data: ${path} looks like manifest, but could not read the version`, LOG_LEVEL_INFO);
}
}
} catch (ex) {
Logger(`The file ${path} could not be encoded`);
Logger(ex, LOG_LEVEL.VERBOSE);
Logger(ex, LOG_LEVEL_VERBOSE);
return false;
}
const mtime = stat.mtime;
@@ -464,8 +592,12 @@ export class ConfigSync extends LiveSyncCommands {
}
async storeCustomizationFiles(path: FilePath, termOverRide?: string) {
const term = termOverRide || this.plugin.deviceAndVaultName;
if (term == "") {
Logger("We have to configure the device name", LOG_LEVEL_NOTICE);
return;
}
const vf = this.filenameToUnifiedKey(path, term);
return await runWithLock(`plugin-${vf}`, false, async () => {
return await serialized(`plugin-${vf}`, async () => {
const category = this.getFileCategory(path);
let mtime = 0;
let fileTargets = [] as FilePath[];
@@ -497,7 +629,7 @@ export class ConfigSync extends LiveSyncCommands {
for (const target of fileTargets) {
const data = await this.makeEntryFromFile(target);
if (data == false) {
Logger(`Config: skipped: ${target} `, LOG_LEVEL.VERBOSE);
// Logger(`Config: skipped: ${target} `, LOG_LEVEL_VERBOSE);
continue;
}
if (data.version) {
@@ -520,10 +652,10 @@ export class ConfigSync extends LiveSyncCommands {
return
}
const content = stringifyYaml(dt);
const content = createTextBlob(serialize(dt));
try {
const old = await this.localDatabase.getDBEntryMeta(prefixedFileName, null, false);
let saveData: LoadedEntry;
let saveData: SavingEntry;
if (old === false) {
saveData = {
_id: id,
@@ -532,14 +664,14 @@ export class ConfigSync extends LiveSyncCommands {
mtime,
ctime: mtime,
datatype: "newnote",
size: content.length,
size: content.size,
children: [],
deleted: false,
type: "newnote",
};
} else {
if (old.mtime == mtime) {
// Logger(`STORAGE --> DB:${file.path}: (hidden) Not changed`, LOG_LEVEL.VERBOSE);
// Logger(`STORAGE --> DB:${file.path}: (hidden) Not changed`, LOG_LEVEL_VERBOSE);
return true;
}
saveData =
@@ -547,7 +679,7 @@ export class ConfigSync extends LiveSyncCommands {
...old,
data: content,
mtime,
size: content.length,
size: content.size,
datatype: "newnote",
children: [],
deleted: false,
@@ -560,7 +692,7 @@ export class ConfigSync extends LiveSyncCommands {
return ret;
} catch (ex) {
Logger(`STORAGE --> DB:${prefixedFileName}: (config) Failed`);
Logger(ex, LOG_LEVEL.VERBOSE);
Logger(ex, LOG_LEVEL_VERBOSE);
return false;
}
})
@@ -569,10 +701,17 @@ export class ConfigSync extends LiveSyncCommands {
async watchVaultRawEventsAsync(path: FilePath) {
if (!this.settings.usePluginSync) return false;
if (!this.isTargetPath(path)) return false;
const stat = await this.app.vault.adapter.stat(path);
const stat = await this.vaultAccess.adapterStat(path);
// Make sure that target is a file.
if (stat && stat.type != "file")
return false;
const configDir = normalizePath(this.app.vault.configDir);
const synchronisedInConfigSync = Object.values(this.settings.pluginSyncExtendedSetting).filter(e => e.mode != MODE_SELECTIVE).map(e => e.files).flat().map(e => `${configDir}/${e}`.toLowerCase());
if (synchronisedInConfigSync.some(e => e.startsWith(path.toLowerCase()))) {
Logger(`Customization file skipped: ${path}`, LOG_LEVEL_VERBOSE);
return;
}
const storageMTime = ~~((stat && stat.mtime || 0) / 1000);
const key = `${path}-${storageMTime}`;
if (this.recentProcessedInternalFiles.contains(key)) {
@@ -587,11 +726,11 @@ export class ConfigSync extends LiveSyncCommands {
async scanAllConfigFiles(showMessage: boolean) {
const logLevel = showMessage ? LOG_LEVEL.NOTICE : LOG_LEVEL.INFO;
const logLevel = showMessage ? LOG_LEVEL_NOTICE : LOG_LEVEL_INFO;
Logger("Scanning customizing files.", logLevel, "scan-all-config");
const term = this.plugin.deviceAndVaultName;
if (term == "") {
Logger("We have to configure the device name", LOG_LEVEL.NOTICE);
Logger("We have to configure the device name", LOG_LEVEL_NOTICE);
return;
}
const filesAll = await this.scanInternalFiles();
@@ -613,7 +752,7 @@ export class ConfigSync extends LiveSyncCommands {
// const id = await this.path2id(prefixedFileName);
const mtime = new Date().getTime();
await runWithLock("file-x-" + prefixedFileName, false, async () => {
await serialized("file-x-" + prefixedFileName, async () => {
try {
const old = await this.localDatabase.getDBEntryMeta(prefixedFileName, null, false) as InternalFileEntry | false;
let saveData: InternalFileEntry;
@@ -639,7 +778,7 @@ export class ConfigSync extends LiveSyncCommands {
Logger(`STORAGE -x> DB:${prefixedFileName}: (config) Done`);
} catch (ex) {
Logger(`STORAGE -x> DB:${prefixedFileName}: (config) Failed`);
Logger(ex, LOG_LEVEL.VERBOSE);
Logger(ex, LOG_LEVEL_VERBOSE);
return false;
}
});

296
src/CmdHiddenFileSync.ts
View File

@@ -1,14 +1,13 @@
import { Notice, normalizePath, PluginManifest } from "./deps";
import { EntryDoc, LoadedEntry, LOG_LEVEL, InternalFileEntry, FilePathWithPrefix, FilePath } from "./lib/src/types";
import { InternalFileInfo, ICHeader, ICHeaderEnd } from "./types";
import { delay, isDocContentSame } from "./lib/src/utils";
import { normalizePath, type PluginManifest } from "./deps";
import { type EntryDoc, type LoadedEntry, type InternalFileEntry, type FilePathWithPrefix, type FilePath, LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE, MODE_SELECTIVE, MODE_PAUSED, type SavingEntry } from "./lib/src/types";
import { type InternalFileInfo, ICHeader, ICHeaderEnd } from "./types";
import { Parallels, createBinaryBlob, delay, isDocContentSame } from "./lib/src/utils";
import { Logger } from "./lib/src/logger";
import { PouchDB } from "./lib/src/pouchdb-browser.js";
import { disposeMemoObject, memoIfNotExist, memoObject, retrieveMemoObject, scheduleTask, isInternalMetadata, PeriodicProcessor } from "./utils";
import { scheduleTask, isInternalMetadata, PeriodicProcessor } from "./utils";
import { WrappedNotice } from "./lib/src/wrapper";
import { base64ToArrayBuffer, arrayBufferToBase64 } from "./lib/src/strbin";
import { runWithLock } from "./lib/src/lock";
import { Semaphore } from "./lib/src/semaphore";
import { decodeBinary, encodeBinary } from "./lib/src/strbin";
import { serialized } from "./lib/src/lock";
import { JsonResolveModal } from "./JsonResolveModal";
import { LiveSyncCommands } from "./LiveSyncCommands";
import { addPrefix, stripAllPrefixes } from "./lib/src/path";
@@ -45,7 +44,7 @@ export class HiddenFileSync extends LiveSyncCommands {
Logger("Synchronizing hidden files done");
} catch (ex) {
Logger("Synchronizing hidden files failed");
Logger(ex, LOG_LEVEL.VERBOSE);
Logger(ex, LOG_LEVEL_VERBOSE);
}
}
}
@@ -78,7 +77,7 @@ export class HiddenFileSync extends LiveSyncCommands {
procInternalFiles: string[] = [];
async execInternalFile() {
await runWithLock("execInternal", false, async () => {
await serialized("execInternal", async () => {
const w = [...this.procInternalFiles];
this.procInternalFiles = [];
Logger(`Applying hidden ${w.length} files change...`);
@@ -96,7 +95,15 @@ export class HiddenFileSync extends LiveSyncCommands {
recentProcessedInternalFiles = [] as string[];
async watchVaultRawEventsAsync(path: FilePath) {
if (!this.settings.syncInternalFiles) return;
const stat = await this.app.vault.adapter.stat(path);
// Exclude files handled by customization sync
const configDir = normalizePath(this.app.vault.configDir);
const synchronisedInConfigSync = !this.settings.usePluginSync ? [] : Object.values(this.settings.pluginSyncExtendedSetting).filter(e => e.mode == MODE_SELECTIVE || e.mode == MODE_PAUSED).map(e => e.files).flat().map(e => `${configDir}/${e}`.toLowerCase());
if (synchronisedInConfigSync.some(e => e.startsWith(path.toLowerCase()))) {
Logger(`Hidden file skipped: ${path} is synchronized in customization sync.`, LOG_LEVEL_VERBOSE);
return;
}
const stat = await this.vaultAccess.adapterStat(path);
// sometimes folder is coming.
if (stat && stat.type != "file")
return;
@@ -162,20 +169,20 @@ export class HiddenFileSync extends LiveSyncCommands {
const commonBase = revFrom._revs_info.filter(e => e.status == "available" && Number(e.rev.split("-")[0]) < conflictedRevNo).first()?.rev ?? "";
const result = await this.plugin.mergeObject(path, commonBase, doc._rev, conflictedRev);
if (result) {
Logger(`Object merge:${path}`, LOG_LEVEL.INFO);
Logger(`Object merge:${path}`, LOG_LEVEL_INFO);
const filename = stripAllPrefixes(path);
const isExists = await this.app.vault.adapter.exists(filename);
const isExists = await this.plugin.vaultAccess.adapterExists(filename);
if (!isExists) {
await this.ensureDirectoryEx(filename);
}
await this.app.vault.adapter.write(filename, result);
const stat = await this.app.vault.adapter.stat(filename);
await this.plugin.vaultAccess.adapterWrite(filename, result);
const stat = await this.vaultAccess.adapterStat(filename);
await this.storeInternalFileToDatabase({ path: filename, ...stat });
await this.extractInternalFileFromDatabase(filename);
await this.localDatabase.removeRaw(id, revB);
return this.resolveConflictOnInternalFile(path);
} else {
Logger(`Object merge is not applicable.`, LOG_LEVEL.VERBOSE);
Logger(`Object merge is not applicable.`, LOG_LEVEL_VERBOSE);
}
const docAMerge = await this.localDatabase.getDBEntry(path, { rev: revA });
@@ -204,24 +211,30 @@ export class HiddenFileSync extends LiveSyncCommands {
return this.resolveConflictOnInternalFile(path);
} catch (ex) {
Logger(`Failed to resolve conflict (Hidden): ${path}`);
Logger(ex, LOG_LEVEL.VERBOSE);
Logger(ex, LOG_LEVEL_VERBOSE);
return false;
}
}
//TODO: Tidy up. Even though it is experimental feature, So dirty...
async syncInternalFilesAndDatabase(direction: "push" | "pull" | "safe" | "pullForce" | "pushForce", showMessage: boolean, files: InternalFileInfo[] | false = false, targetFiles: string[] | false = false) {
async syncInternalFilesAndDatabase(direction: "push" | "pull" | "safe" | "pullForce" | "pushForce", showMessage: boolean, filesAll: InternalFileInfo[] | false = false, targetFiles: string[] | false = false) {
await this.resolveConflictOnInternalFiles();
const logLevel = showMessage ? LOG_LEVEL.NOTICE : LOG_LEVEL.INFO;
const logLevel = showMessage ? LOG_LEVEL_NOTICE : LOG_LEVEL_INFO;
Logger("Scanning hidden files.", logLevel, "sync_internal");
const ignorePatterns = this.settings.syncInternalFilesIgnorePatterns
.replace(/\n| /g, "")
.split(",").filter(e => e).map(e => new RegExp(e, "i"));
if (!files)
files = await this.scanInternalFiles();
const configDir = normalizePath(this.app.vault.configDir);
let files: InternalFileInfo[] =
filesAll ? filesAll : (await this.scanInternalFiles())
const synchronisedInConfigSync = !this.settings.usePluginSync ? [] : Object.values(this.settings.pluginSyncExtendedSetting).filter(e => e.mode == MODE_SELECTIVE || e.mode == MODE_PAUSED).map(e => e.files).flat().map(e => `${configDir}/${e}`.toLowerCase());
files = files.filter(file => synchronisedInConfigSync.every(filterFile => !file.path.toLowerCase().startsWith(filterFile)))
const filesOnDB = ((await this.localDatabase.allDocsRaw({ startkey: ICHeader, endkey: ICHeaderEnd, include_docs: true })).rows.map(e => e.doc) as InternalFileEntry[]).filter(e => !e.deleted);
const allFileNamesSrc = [...new Set([...files.map(e => normalizePath(e.path)), ...filesOnDB.map(e => stripAllPrefixes(this.getPath(e)))])];
const allFileNames = allFileNamesSrc.filter(filename => !targetFiles || (targetFiles && targetFiles.indexOf(filename) !== -1));
const allFileNames = allFileNamesSrc.filter(filename => !targetFiles || (targetFiles && targetFiles.indexOf(filename) !== -1)).filter(path => synchronisedInConfigSync.every(filterFile => !path.toLowerCase().startsWith(filterFile)))
function compareMTime(a: number, b: number) {
const wa = ~~(a / 1000);
const wb = ~~(b / 1000);
@@ -254,37 +267,38 @@ export class HiddenFileSync extends LiveSyncCommands {
c = pieces.shift();
}
};
const p = [] as Promise<void>[];
const semaphore = Semaphore(10);
// Cache update time information for files which have already been processed (mainly for files that were skipped due to the same content)
let caches: { [key: string]: { storageMtime: number; docMtime: number; }; } = {};
caches = await this.kvDB.get<{ [key: string]: { storageMtime: number; docMtime: number; }; }>("diff-caches-internal") || {};
const filesMap = files.reduce((acc, cur) => {
acc[cur.path] = cur;
return acc;
}, {} as { [key: string]: InternalFileInfo; });
const filesOnDBMap = filesOnDB.reduce((acc, cur) => {
acc[stripAllPrefixes(this.getPath(cur))] = cur;
return acc;
}, {} as { [key: string]: InternalFileEntry; });
const para = Parallels();
for (const filename of allFileNames) {
if (!filename) continue;
processed++;
if (processed % 100 == 0)
if (processed % 100 == 0) {
Logger(`Hidden file: ${processed}/${fileCount}`, logLevel, "sync_internal");
}
if (!filename) continue;
if (ignorePatterns.some(e => filename.match(e)))
continue;
if (await this.plugin.isIgnoredByIgnoreFiles(filename)) {
continue;
}
const fileOnStorage = filename in filesMap ? filesMap[filename] : undefined;
const fileOnDatabase = filename in filesOnDBMap ? filesOnDBMap[filename] : undefined;
const fileOnStorage = files.find(e => e.path == filename);
const fileOnDatabase = filesOnDB.find(e => stripAllPrefixes(this.getPath(e)) == filename);
const addProc = async (p: () => Promise<void>): Promise<void> => {
const releaser = await semaphore.acquire(1);
try {
return p();
} catch (ex) {
Logger("Some process failed", logLevel);
Logger(ex);
} finally {
releaser();
}
};
const cache = filename in caches ? caches[filename] : { storageMtime: 0, docMtime: 0 };
p.push(addProc(async () => {
const xFileOnStorage = fileOnStorage;
const xFileOnDatabase = fileOnDatabase;
await para.wait(5);
const proc = (async (xFileOnStorage: InternalFileInfo, xFileOnDatabase: InternalFileEntry) => {
if (xFileOnStorage && xFileOnDatabase) {
// Both => Synchronize
if ((direction != "pullForce" && direction != "pushForce") && xFileOnDatabase.mtime == cache.docMtime && xFileOnStorage.mtime == cache.storageMtime) {
@@ -326,14 +340,15 @@ export class HiddenFileSync extends LiveSyncCommands {
throw new Error("Invalid state on hidden file sync");
// Something corrupted?
}
}));
});
para.add(proc(fileOnStorage, fileOnDatabase))
}
await Promise.all(p);
await para.all();
await this.kvDB.set("diff-caches-internal", caches);
// When files has been retrieved from the database. they must be reloaded.
if ((direction == "pull" || direction == "pullForce") && filesChanged != 0) {
const configDir = normalizePath(this.app.vault.configDir);
// Show notification to restart obsidian when something has been changed in configDir.
if (configDir in updatedFolders) {
// Numbers of updated files that is below of configDir.
@@ -350,78 +365,33 @@ export class HiddenFileSync extends LiveSyncCommands {
updatedCount -= updatedFolders[manifest.dir];
const updatePluginId = manifest.id;
const updatePluginName = manifest.name;
const fragment = createFragment((doc) => {
doc.createEl("span", null, (a) => {
a.appendText(`Files in ${updatePluginName} has been updated, Press `);
a.appendChild(a.createEl("a", null, (anchor) => {
anchor.text = "HERE";
anchor.addEventListener("click", async () => {
Logger(`Unloading plugin: ${updatePluginName}`, LOG_LEVEL.NOTICE, "plugin-reload-" + updatePluginId);
// @ts-ignore
await this.app.plugins.unloadPlugin(updatePluginId);
// @ts-ignore
await this.app.plugins.loadPlugin(updatePluginId);
Logger(`Plugin reloaded: ${updatePluginName}`, LOG_LEVEL.NOTICE, "plugin-reload-" + updatePluginId);
});
}));
a.appendText(` to reload ${updatePluginName}, or press elsewhere to dismiss this message.`);
this.plugin.askInPopup(`updated-${updatePluginId}`, `Files in ${updatePluginName} has been updated, Press {HERE} to reload ${updatePluginName}, or press elsewhere to dismiss this message.`, (anchor) => {
anchor.text = "HERE";
anchor.addEventListener("click", async () => {
Logger(`Unloading plugin: ${updatePluginName}`, LOG_LEVEL_NOTICE, "plugin-reload-" + updatePluginId);
// @ts-ignore
await this.app.plugins.unloadPlugin(updatePluginId);
// @ts-ignore
await this.app.plugins.loadPlugin(updatePluginId);
Logger(`Plugin reloaded: ${updatePluginName}`, LOG_LEVEL_NOTICE, "plugin-reload-" + updatePluginId);
});
});
const updatedPluginKey = "popupUpdated-" + updatePluginId;
scheduleTask(updatedPluginKey, 1000, async () => {
const popup = await memoIfNotExist(updatedPluginKey, () => new Notice(fragment, 0));
//@ts-ignore
const isShown = popup?.noticeEl?.isShown();
if (!isShown) {
memoObject(updatedPluginKey, new Notice(fragment, 0));
}
scheduleTask(updatedPluginKey + "-close", 20000, () => {
const popup = retrieveMemoObject<Notice>(updatedPluginKey);
if (!popup)
return;
//@ts-ignore
if (popup?.noticeEl?.isShown()) {
popup.hide();
}
disposeMemoObject(updatedPluginKey);
});
});
}
);
}
}
} catch (ex) {
Logger("Error on checking plugin status.");
Logger(ex, LOG_LEVEL.VERBOSE);
Logger(ex, LOG_LEVEL_VERBOSE);
}
// If something changes left, notify for reloading Obsidian.
if (updatedCount != 0) {
const fragment = createFragment((doc) => {
doc.createEl("span", null, (a) => {
a.appendText(`Hidden files have been synchronized, Press `);
a.appendChild(a.createEl("a", null, (anchor) => {
anchor.text = "HERE";
anchor.addEventListener("click", () => {
// @ts-ignore
this.app.commands.executeCommandById("app:reload");
});
}));
a.appendText(` to reload obsidian, or press elsewhere to dismiss this message.`);
});
});
scheduleTask("popupUpdated-" + configDir, 1000, () => {
//@ts-ignore
const isShown = this.confirmPopup?.noticeEl?.isShown();
if (!isShown) {
this.confirmPopup = new Notice(fragment, 0);
}
scheduleTask("popupClose" + configDir, 20000, () => {
this.confirmPopup?.hide();
this.confirmPopup = null;
this.plugin.askInPopup(`updated-any-hidden`, `Hidden files have been synchronized, Press {HERE} to reload Obsidian, or press elsewhere to dismiss this message.`, (anchor) => {
anchor.text = "HERE";
anchor.addEventListener("click", () => {
// @ts-ignore
this.app.commands.executeCommandById("app:reload");
});
});
}
@@ -432,22 +402,26 @@ export class HiddenFileSync extends LiveSyncCommands {
}
async storeInternalFileToDatabase(file: InternalFileInfo, forceWrite = false) {
if (await this.plugin.isIgnoredByIgnoreFiles(file.path)) {
return
}
const id = await this.path2id(file.path, ICHeader);
const prefixedFileName = addPrefix(file.path, ICHeader);
const contentBin = await this.app.vault.adapter.readBinary(file.path);
let content: string[];
const contentBin = await this.plugin.vaultAccess.adapterReadBinary(file.path);
let content: Blob;
try {
content = await arrayBufferToBase64(contentBin);
content = createBinaryBlob(contentBin);
} catch (ex) {
Logger(`The file ${file.path} could not be encoded`);
Logger(ex, LOG_LEVEL.VERBOSE);
Logger(ex, LOG_LEVEL_VERBOSE);
return false;
}
const mtime = file.mtime;
return await runWithLock("file-" + prefixedFileName, false, async () => {
return await serialized("file-" + prefixedFileName, async () => {
try {
const old = await this.localDatabase.getDBEntry(prefixedFileName, null, false, false);
let saveData: LoadedEntry;
let saveData: SavingEntry;
if (old === false) {
saveData = {
_id: id,
@@ -463,7 +437,7 @@ export class HiddenFileSync extends LiveSyncCommands {
};
} else {
if (isDocContentSame(old.data, content) && !forceWrite) {
// Logger(`STORAGE --> DB:${file.path}: (hidden) Not changed`, LOG_LEVEL.VERBOSE);
// Logger(`STORAGE --> DB:${file.path}: (hidden) Not changed`, LOG_LEVEL_VERBOSE);
return;
}
saveData =
@@ -483,7 +457,7 @@ export class HiddenFileSync extends LiveSyncCommands {
return ret;
} catch (ex) {
Logger(`STORAGE --> DB:${file.path}: (hidden) Failed`);
Logger(ex, LOG_LEVEL.VERBOSE);
Logger(ex, LOG_LEVEL_VERBOSE);
return false;
}
});
@@ -493,7 +467,10 @@ export class HiddenFileSync extends LiveSyncCommands {
const id = await this.path2id(filename, ICHeader);
const prefixedFileName = addPrefix(filename, ICHeader);
const mtime = new Date().getTime();
await runWithLock("file-" + prefixedFileName, false, async () => {
if (await this.plugin.isIgnoredByIgnoreFiles(filename)) {
return
}
await serialized("file-" + prefixedFileName, async () => {
try {
const old = await this.localDatabase.getDBEntryMeta(prefixedFileName, null, true) as InternalFileEntry | false;
let saveData: InternalFileEntry;
@@ -527,17 +504,19 @@ export class HiddenFileSync extends LiveSyncCommands {
Logger(`STORAGE -x> DB:${filename}: (hidden) Done`);
} catch (ex) {
Logger(`STORAGE -x> DB:${filename}: (hidden) Failed`);
Logger(ex, LOG_LEVEL.VERBOSE);
Logger(ex, LOG_LEVEL_VERBOSE);
return false;
}
});
}
async extractInternalFileFromDatabase(filename: FilePath, force = false) {
const isExists = await this.app.vault.adapter.exists(filename);
const isExists = await this.plugin.vaultAccess.adapterExists(filename);
const prefixedFileName = addPrefix(filename, ICHeader);
return await runWithLock("file-" + prefixedFileName, false, async () => {
if (await this.plugin.isIgnoredByIgnoreFiles(filename)) {
return;
}
return await serialized("file-" + prefixedFileName, async () => {
try {
// Check conflicted status
//TODO option
@@ -546,7 +525,7 @@ export class HiddenFileSync extends LiveSyncCommands {
throw new Error(`File not found on database.:${filename}`);
// Prevent overwrite for Prevent overwriting while some conflicted revision exists.
if (fileOnDB?._conflicts?.length) {
Logger(`Hidden file ${filename} has conflicted revisions, to keep in safe, writing to storage has been prevented`, LOG_LEVEL.INFO);
Logger(`Hidden file ${filename} has conflicted revisions, to keep in safe, writing to storage has been prevented`, LOG_LEVEL_INFO);
return;
}
const deleted = "deleted" in fileOnDB ? fileOnDB.deleted : false;
@@ -555,43 +534,43 @@ export class HiddenFileSync extends LiveSyncCommands {
Logger(`STORAGE <x- DB:${filename}: deleted (hidden) Deleted on DB, but the file is already not found on storage.`);
} else {
Logger(`STORAGE <x- DB:${filename}: deleted (hidden).`);
await this.app.vault.adapter.remove(filename);
await this.plugin.vaultAccess.adapterRemove(filename);
try {
//@ts-ignore internalAPI
await app.vault.adapter.reconcileInternalFile(filename);
await this.app.vault.adapter.reconcileInternalFile(filename);
} catch (ex) {
Logger("Failed to call internal API(reconcileInternalFile)", LOG_LEVEL.VERBOSE);
Logger(ex, LOG_LEVEL.VERBOSE);
Logger("Failed to call internal API(reconcileInternalFile)", LOG_LEVEL_VERBOSE);
Logger(ex, LOG_LEVEL_VERBOSE);
}
}
return true;
}
if (!isExists) {
await this.ensureDirectoryEx(filename);
await this.app.vault.adapter.writeBinary(filename, base64ToArrayBuffer(fileOnDB.data), { mtime: fileOnDB.mtime, ctime: fileOnDB.ctime });
await this.plugin.vaultAccess.adapterWrite(filename, decodeBinary(fileOnDB.data), { mtime: fileOnDB.mtime, ctime: fileOnDB.ctime });
try {
//@ts-ignore internalAPI
await app.vault.adapter.reconcileInternalFile(filename);
await this.app.vault.adapter.reconcileInternalFile(filename);
} catch (ex) {
Logger("Failed to call internal API(reconcileInternalFile)", LOG_LEVEL.VERBOSE);
Logger(ex, LOG_LEVEL.VERBOSE);
Logger("Failed to call internal API(reconcileInternalFile)", LOG_LEVEL_VERBOSE);
Logger(ex, LOG_LEVEL_VERBOSE);
}
Logger(`STORAGE <-- DB:${filename}: written (hidden,new${force ? ", force" : ""})`);
return true;
} else {
const contentBin = await this.app.vault.adapter.readBinary(filename);
const content = await arrayBufferToBase64(contentBin);
if (content == fileOnDB.data && !force) {
// Logger(`STORAGE <-- DB:${filename}: skipped (hidden) Not changed`, LOG_LEVEL.VERBOSE);
const contentBin = await this.plugin.vaultAccess.adapterReadBinary(filename);
const content = await encodeBinary(contentBin);
if (isDocContentSame(content, fileOnDB.data) && !force) {
// Logger(`STORAGE <-- DB:${filename}: skipped (hidden) Not changed`, LOG_LEVEL_VERBOSE);
return true;
}
await this.app.vault.adapter.writeBinary(filename, base64ToArrayBuffer(fileOnDB.data), { mtime: fileOnDB.mtime, ctime: fileOnDB.ctime });
await this.plugin.vaultAccess.adapterWrite(filename, decodeBinary(fileOnDB.data), { mtime: fileOnDB.mtime, ctime: fileOnDB.ctime });
try {
//@ts-ignore internalAPI
await app.vault.adapter.reconcileInternalFile(filename);
await this.app.vault.adapter.reconcileInternalFile(filename);
} catch (ex) {
Logger("Failed to call internal API(reconcileInternalFile)", LOG_LEVEL.VERBOSE);
Logger(ex, LOG_LEVEL.VERBOSE);
Logger("Failed to call internal API(reconcileInternalFile)", LOG_LEVEL_VERBOSE);
Logger(ex, LOG_LEVEL_VERBOSE);
}
Logger(`STORAGE <-- DB:${filename}: written (hidden, overwrite${force ? ", force" : ""})`);
return true;
@@ -599,7 +578,7 @@ export class HiddenFileSync extends LiveSyncCommands {
}
} catch (ex) {
Logger(`STORAGE <-- DB:${filename}: written (hidden, overwrite${force ? ", force" : ""}) Failed`);
Logger(ex, LOG_LEVEL.VERBOSE);
Logger(ex, LOG_LEVEL_VERBOSE);
return false;
}
});
@@ -608,8 +587,8 @@ export class HiddenFileSync extends LiveSyncCommands {
showJSONMergeDialogAndMerge(docA: LoadedEntry, docB: LoadedEntry): Promise<boolean> {
return runWithLock("conflict:merge-data", false, () => new Promise((res) => {
Logger("Opening data-merging dialog", LOG_LEVEL.VERBOSE);
return serialized("conflict:merge-data", () => new Promise((res) => {
Logger("Opening data-merging dialog", LOG_LEVEL_VERBOSE);
const docs = [docA, docB];
const path = stripAllPrefixes(docA.path);
const modal = new JsonResolveModal(this.app, path, [docA, docB], async (keep, result) => {
@@ -634,19 +613,19 @@ export class HiddenFileSync extends LiveSyncCommands {
}
}
if (!keep && result) {
const isExists = await this.app.vault.adapter.exists(filename);
const isExists = await this.plugin.vaultAccess.adapterExists(filename);
if (!isExists) {
await this.ensureDirectoryEx(filename);
}
await this.app.vault.adapter.write(filename, result);
const stat = await this.app.vault.adapter.stat(filename);
await this.plugin.vaultAccess.adapterWrite(filename, result);
const stat = await this.plugin.vaultAccess.adapterStat(filename);
await this.storeInternalFileToDatabase({ path: filename, ...stat }, true);
try {
//@ts-ignore internalAPI
await app.vault.adapter.reconcileInternalFile(filename);
await this.app.vault.adapter.reconcileInternalFile(filename);
} catch (ex) {
Logger("Failed to call internal API(reconcileInternalFile)", LOG_LEVEL.VERBOSE);
Logger(ex, LOG_LEVEL.VERBOSE);
Logger("Failed to call internal API(reconcileInternalFile)", LOG_LEVEL_VERBOSE);
Logger(ex, LOG_LEVEL_VERBOSE);
}
Logger(`STORAGE <-- DB:${filename}: written (hidden,merged)`);
}
@@ -657,7 +636,7 @@ export class HiddenFileSync extends LiveSyncCommands {
res(true);
} catch (ex) {
Logger("Could not merge conflicted json");
Logger(ex, LOG_LEVEL.VERBOSE);
Logger(ex, LOG_LEVEL_VERBOSE);
res(false);
}
});
@@ -666,21 +645,27 @@ export class HiddenFileSync extends LiveSyncCommands {
}
async scanInternalFiles(): Promise<InternalFileInfo[]> {
const configDir = normalizePath(this.app.vault.configDir);
const ignoreFilter = this.settings.syncInternalFilesIgnorePatterns
.replace(/\n| /g, "")
.split(",").filter(e => e).map(e => new RegExp(e, "i"));
const synchronisedInConfigSync = !this.settings.usePluginSync ? [] : Object.values(this.settings.pluginSyncExtendedSetting).filter(e => e.mode == MODE_SELECTIVE || e.mode == MODE_PAUSED).map(e => e.files).flat().map(e => `${configDir}/${e}`.toLowerCase());
const root = this.app.vault.getRoot();
const findRoot = root.path;
const filenames = (await this.getFiles(findRoot, [], null, ignoreFilter)).filter(e => e.startsWith(".")).filter(e => !e.startsWith(".trash"));
const files = filenames.map(async (e) => {
const files = filenames.filter(path => synchronisedInConfigSync.every(filterFile => !path.toLowerCase().startsWith(filterFile))).map(async (e) => {
return {
path: e as FilePath,
stat: await this.app.vault.adapter.stat(e)
stat: await this.plugin.vaultAccess.adapterStat(e)
};
});
const result: InternalFileInfo[] = [];
for (const f of files) {
const w = await f;
if (await this.plugin.isIgnoredByIgnoreFiles(w.path)) {
continue
}
result.push({
...w,
...w.stat
@@ -699,12 +684,18 @@ export class HiddenFileSync extends LiveSyncCommands {
) {
const w = await this.app.vault.adapter.list(path);
let files = [
const filesSrc = [
...w.files
.filter((e) => !ignoreList.some((ee) => e.endsWith(ee)))
.filter((e) => !filter || filter.some((ee) => e.match(ee)))
.filter((e) => !ignoreFilter || ignoreFilter.every((ee) => !e.match(ee))),
.filter((e) => !ignoreFilter || ignoreFilter.every((ee) => !e.match(ee)))
];
let files = [] as string[];
for (const file of filesSrc) {
if (!await this.plugin.isIgnoredByIgnoreFiles(file)) {
files.push(file);
}
}
L1: for (const v of w.folders) {
for (const ignore of ignoreList) {
@@ -715,6 +706,9 @@ export class HiddenFileSync extends LiveSyncCommands {
if (ignoreFilter && ignoreFilter.some(e => v.match(e))) {
continue L1;
}
if (await this.plugin.isIgnoredByIgnoreFiles(v)) {
continue L1;
}
files = files.concat(await this.getFiles(v, ignoreList, filter, ignoreFilter));
}
return files;

77
src/CmdPluginAndTheirSettings.ts
View File

@@ -1,14 +1,15 @@
import { normalizePath, PluginManifest } from "./deps";
import { DocumentID, EntryDoc, FilePathWithPrefix, LoadedEntry, LOG_LEVEL } from "./lib/src/types";
import { PluginDataEntry, PERIODIC_PLUGIN_SWEEP, PluginList, DevicePluginList, PSCHeader, PSCHeaderEnd } from "./types";
import { getDocData, isDocContentSame } from "./lib/src/utils";
import { normalizePath, type PluginManifest } from "./deps";
import type { DocumentID, EntryDoc, FilePathWithPrefix, LoadedEntry, SavingEntry } from "./lib/src/types";
import { LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE } from "./lib/src/types";
import { type PluginDataEntry, PERIODIC_PLUGIN_SWEEP, type PluginList, type DevicePluginList, PSCHeader, PSCHeaderEnd } from "./types";
import { createTextBlob, getDocData, isDocContentSame } from "./lib/src/utils";
import { Logger } from "./lib/src/logger";
import { PouchDB } from "./lib/src/pouchdb-browser.js";
import { isPluginMetadata, PeriodicProcessor } from "./utils";
import { PluginDialogModal } from "./dialogs";
import { NewNotice } from "./lib/src/wrapper";
import { versionNumberString2Number } from "./lib/src/strbin";
import { runWithLock } from "./lib/src/lock";
import { serialized, skipIfDuplicated } from "./lib/src/lock";
import { LiveSyncCommands } from "./LiveSyncCommands";
export class PluginAndTheirSettings extends LiveSyncCommands {
@@ -78,7 +79,7 @@ export class PluginAndTheirSettings extends LiveSyncCommands {
Logger("Scanning plugins done");
} catch (ex) {
Logger("Scanning plugins failed");
Logger(ex, LOG_LEVEL.VERBOSE);
Logger(ex, LOG_LEVEL_VERBOSE);
}
}
@@ -147,7 +148,7 @@ export class PluginAndTheirSettings extends LiveSyncCommands {
});
NewNotice(fragment, 10000);
} else {
Logger("Everything is up to date.", LOG_LEVEL.NOTICE);
Logger("Everything is up to date.", LOG_LEVEL_NOTICE);
}
}
@@ -163,10 +164,10 @@ export class PluginAndTheirSettings extends LiveSyncCommands {
if (specificPluginPath != "") {
specificPlugin = manifests.find(e => e.dir.endsWith("/" + specificPluginPath))?.id ?? "";
}
await runWithLock("sweepplugin", true, async () => {
const logLevel = showMessage ? LOG_LEVEL.NOTICE : LOG_LEVEL.INFO;
await skipIfDuplicated("sweepplugin", async () => {
const logLevel = showMessage ? LOG_LEVEL_NOTICE : LOG_LEVEL_INFO;
if (!this.deviceAndVaultName) {
Logger("You have to set your device name.", LOG_LEVEL.NOTICE);
Logger("You have to set your device name.", LOG_LEVEL_NOTICE);
return;
}
Logger("Scanning plugins", logLevel);
@@ -175,7 +176,7 @@ export class PluginAndTheirSettings extends LiveSyncCommands {
endkey: `ps:${this.deviceAndVaultName}-${specificPlugin}\u{10ffff}`,
include_docs: true,
});
// Logger("OLD DOCS.", LOG_LEVEL.VERBOSE);
// Logger("OLD DOCS.", LOG_LEVEL_VERBOSE);
// sweep current plugin.
const procs = manifests.map(async (m) => {
const pluginDataEntryID = `ps:${this.deviceAndVaultName}-${m.id}` as DocumentID;
@@ -183,20 +184,19 @@ export class PluginAndTheirSettings extends LiveSyncCommands {
if (specificPlugin && m.id != specificPlugin) {
return;
}
Logger(`Reading plugin:${m.name}(${m.id})`, LOG_LEVEL.VERBOSE);
Logger(`Reading plugin:${m.name}(${m.id})`, LOG_LEVEL_VERBOSE);
const path = normalizePath(m.dir) + "/";
const adapter = this.app.vault.adapter;
const files = ["manifest.json", "main.js", "styles.css", "data.json"];
const pluginData: { [key: string]: string; } = {};
for (const file of files) {
const thePath = path + file;
if (await adapter.exists(thePath)) {
pluginData[file] = await adapter.read(thePath);
if (await this.plugin.vaultAccess.adapterExists(thePath)) {
pluginData[file] = await this.plugin.vaultAccess.adapterRead(thePath);
}
}
let mtime = 0;
if (await adapter.exists(path + "/data.json")) {
mtime = (await adapter.stat(path + "/data.json")).mtime;
if (await this.plugin.vaultAccess.adapterExists(path + "/data.json")) {
mtime = (await this.plugin.vaultAccess.adapterStat(path + "/data.json")).mtime;
}
const p: PluginDataEntry = {
@@ -210,19 +210,20 @@ export class PluginAndTheirSettings extends LiveSyncCommands {
mtime: mtime,
type: "plugin",
};
const d: LoadedEntry = {
const blob = createTextBlob(JSON.stringify(p));
const d: SavingEntry = {
_id: p._id,
path: p._id as string as FilePathWithPrefix,
data: JSON.stringify(p),
data: blob,
ctime: mtime,
mtime: mtime,
size: 0,
size: blob.size,
children: [],
datatype: "plain",
type: "plain"
};
Logger(`check diff:${m.name}(${m.id})`, LOG_LEVEL.VERBOSE);
await runWithLock("plugin-" + m.id, false, async () => {
Logger(`check diff:${m.name}(${m.id})`, LOG_LEVEL_VERBOSE);
await serialized("plugin-" + m.id, async () => {
const old = await this.localDatabase.getDBEntry(p._id as string as FilePathWithPrefix /* This also should be explained */, null, false, false);
if (old !== false) {
const oldData = { data: old.data, deleted: old._deleted };
@@ -236,7 +237,7 @@ export class PluginAndTheirSettings extends LiveSyncCommands {
Logger(`Plugin saved:${m.name}`, logLevel);
});
} catch (ex) {
Logger(`Plugin save failed:${m.name}`, LOG_LEVEL.NOTICE);
Logger(`Plugin save failed:${m.name}`, LOG_LEVEL_NOTICE);
} finally {
oldDocs.rows = oldDocs.rows.filter((e) => e.id != pluginDataEntryID);
}
@@ -258,57 +259,55 @@ export class PluginAndTheirSettings extends LiveSyncCommands {
}
return e.doc;
});
Logger(`Deleting old plugin:(${delDocs.length})`, LOG_LEVEL.VERBOSE);
Logger(`Deleting old plugin:(${delDocs.length})`, LOG_LEVEL_VERBOSE);
await this.localDatabase.bulkDocsRaw(delDocs);
Logger(`Scan plugin done.`, logLevel);
});
}
async applyPluginData(plugin: PluginDataEntry) {
await runWithLock("plugin-" + plugin.manifest.id, false, async () => {
await serialized("plugin-" + plugin.manifest.id, async () => {
const pluginTargetFolderPath = normalizePath(plugin.manifest.dir) + "/";
const adapter = this.app.vault.adapter;
// @ts-ignore
const stat = this.app.plugins.enabledPlugins.has(plugin.manifest.id) == true;
if (stat) {
// @ts-ignore
await this.app.plugins.unloadPlugin(plugin.manifest.id);
Logger(`Unload plugin:${plugin.manifest.id}`, LOG_LEVEL.NOTICE);
Logger(`Unload plugin:${plugin.manifest.id}`, LOG_LEVEL_NOTICE);
}
if (plugin.dataJson)
await adapter.write(pluginTargetFolderPath + "data.json", plugin.dataJson);
Logger("wrote:" + pluginTargetFolderPath + "data.json", LOG_LEVEL.NOTICE);
await this.plugin.vaultAccess.adapterWrite(pluginTargetFolderPath + "data.json", plugin.dataJson);
Logger("wrote:" + pluginTargetFolderPath + "data.json", LOG_LEVEL_NOTICE);
if (stat) {
// @ts-ignore
await this.app.plugins.loadPlugin(plugin.manifest.id);
Logger(`Load plugin:${plugin.manifest.id}`, LOG_LEVEL.NOTICE);
Logger(`Load plugin:${plugin.manifest.id}`, LOG_LEVEL_NOTICE);
}
});
}
async applyPlugin(plugin: PluginDataEntry) {
await runWithLock("plugin-" + plugin.manifest.id, false, async () => {
await serialized("plugin-" + plugin.manifest.id, async () => {
// @ts-ignore
const stat = this.app.plugins.enabledPlugins.has(plugin.manifest.id) == true;
if (stat) {
// @ts-ignore
await this.app.plugins.unloadPlugin(plugin.manifest.id);
Logger(`Unload plugin:${plugin.manifest.id}`, LOG_LEVEL.NOTICE);
Logger(`Unload plugin:${plugin.manifest.id}`, LOG_LEVEL_NOTICE);
}
const pluginTargetFolderPath = normalizePath(plugin.manifest.dir) + "/";
const adapter = this.app.vault.adapter;
if ((await adapter.exists(pluginTargetFolderPath)) === false) {
await adapter.mkdir(pluginTargetFolderPath);
if ((await this.plugin.vaultAccess.adapterExists(pluginTargetFolderPath)) === false) {
await this.app.vault.adapter.mkdir(pluginTargetFolderPath);
}
await adapter.write(pluginTargetFolderPath + "main.js", plugin.mainJs);
await adapter.write(pluginTargetFolderPath + "manifest.json", plugin.manifestJson);
await this.plugin.vaultAccess.adapterWrite(pluginTargetFolderPath + "main.js", plugin.mainJs);
await this.plugin.vaultAccess.adapterWrite(pluginTargetFolderPath + "manifest.json", plugin.manifestJson);
if (plugin.styleCss)
await adapter.write(pluginTargetFolderPath + "styles.css", plugin.styleCss);
await this.plugin.vaultAccess.adapterWrite(pluginTargetFolderPath + "styles.css", plugin.styleCss);
if (stat) {
// @ts-ignore
await this.app.plugins.loadPlugin(plugin.manifest.id);
Logger(`Load plugin:${plugin.manifest.id}`, LOG_LEVEL.NOTICE);
Logger(`Load plugin:${plugin.manifest.id}`, LOG_LEVEL_NOTICE);
}
});
}

124
src/CmdSetupLiveSync.ts
View File

@@ -1,4 +1,4 @@
import { EntryDoc, ObsidianLiveSyncSettings, LOG_LEVEL, DEFAULT_SETTINGS } from "./lib/src/types";
import { type EntryDoc, type ObsidianLiveSyncSettings, DEFAULT_SETTINGS, LOG_LEVEL_NOTICE } from "./lib/src/types";
import { configURIBase } from "./types";
import { Logger } from "./lib/src/logger";
import { PouchDB } from "./lib/src/pouchdb-browser.js";
@@ -8,6 +8,7 @@ import { LiveSyncCommands } from "./LiveSyncCommands";
import { delay } from "./lib/src/utils";
import { confirmWithMessage } from "./dialogs";
import { Platform } from "./deps";
import { fetchAllUsedChunks } from "./lib/src/utils_couchdb";
export class SetupLiveSync extends LiveSyncCommands {
onunload() { }
@@ -19,6 +20,11 @@ export class SetupLiveSync extends LiveSyncCommands {
name: "Copy the setup URI",
callback: this.command_copySetupURI.bind(this),
});
this.plugin.addCommand({
id: "livesync-copysetupuri-short",
name: "Copy the setup URI (With customization sync)",
callback: this.command_copySetupURIWithSync.bind(this),
});
this.plugin.addCommand({
id: "livesync-copysetupurifull",
@@ -40,11 +46,14 @@ export class SetupLiveSync extends LiveSyncCommands {
}
async realizeSettingSyncMode() { }
async command_copySetupURI() {
const encryptingPassphrase = await askString(this.app, "Encrypt your settings", "The passphrase to encrypt the setup URI", "");
async command_copySetupURI(stripExtra = true) {
const encryptingPassphrase = await askString(this.app, "Encrypt your settings", "The passphrase to encrypt the setup URI", "", true);
if (encryptingPassphrase === false)
return;
const setting = { ...this.settings, configPassphraseStore: "", encryptedCouchDBConnection: "", encryptedPassphrase: "" };
if (stripExtra) {
delete setting.pluginSyncExtendedSetting;
}
const keys = Object.keys(setting) as (keyof ObsidianLiveSyncSettings)[];
for (const k of keys) {
if (JSON.stringify(k in setting ? setting[k] : "") == JSON.stringify(k in DEFAULT_SETTINGS ? DEFAULT_SETTINGS[k] : "*")) {
@@ -54,24 +63,27 @@ export class SetupLiveSync extends LiveSyncCommands {
const encryptedSetting = encodeURIComponent(await encrypt(JSON.stringify(setting), encryptingPassphrase, false));
const uri = `${configURIBase}${encryptedSetting}`;
await navigator.clipboard.writeText(uri);
Logger("Setup URI copied to clipboard", LOG_LEVEL.NOTICE);
Logger("Setup URI copied to clipboard", LOG_LEVEL_NOTICE);
}
async command_copySetupURIFull() {
const encryptingPassphrase = await askString(this.app, "Encrypt your settings", "The passphrase to encrypt the setup URI", "");
const encryptingPassphrase = await askString(this.app, "Encrypt your settings", "The passphrase to encrypt the setup URI", "", true);
if (encryptingPassphrase === false)
return;
const setting = { ...this.settings, configPassphraseStore: "", encryptedCouchDBConnection: "", encryptedPassphrase: "" };
const encryptedSetting = encodeURIComponent(await encrypt(JSON.stringify(setting), encryptingPassphrase, false));
const uri = `${configURIBase}${encryptedSetting}`;
await navigator.clipboard.writeText(uri);
Logger("Setup URI copied to clipboard", LOG_LEVEL.NOTICE);
Logger("Setup URI copied to clipboard", LOG_LEVEL_NOTICE);
}
async command_copySetupURIWithSync() {
this.command_copySetupURI(false);
}
async command_openSetupURI() {
const setupURI = await askString(this.app, "Easy setup", "Set up URI", `${configURIBase}aaaaa`);
if (setupURI === false)
return;
if (!setupURI.startsWith(`${configURIBase}`)) {
Logger("Set up URI looks wrong.", LOG_LEVEL.NOTICE);
Logger("Set up URI looks wrong.", LOG_LEVEL_NOTICE);
return;
}
const config = decodeURIComponent(setupURI.substring(configURIBase.length));
@@ -81,7 +93,7 @@ export class SetupLiveSync extends LiveSyncCommands {
async setupWizard(confString: string) {
try {
const oldConf = JSON.parse(JSON.stringify(this.settings));
const encryptingPassphrase = await askString(this.app, "Passphrase", "The passphrase to decrypt your setup URI", "");
const encryptingPassphrase = await askString(this.app, "Passphrase", "The passphrase to decrypt your setup URI", "", true);
if (encryptingPassphrase === false)
return;
const newConf = await JSON.parse(await decrypt(confString, encryptingPassphrase, false));
@@ -98,11 +110,17 @@ export class SetupLiveSync extends LiveSyncCommands {
newSettingW.encryptedCouchDBConnection = "";
const setupJustImport = "Just import setting";
const setupAsNew = "Set it up as secondary or subsequent device";
const setupAsMerge = "Secondary device but try keeping local changes";
const setupAgain = "Reconfigure and reconstitute the data";
const setupManually = "Leave everything to me";
newSettingW.syncInternalFiles = false;
newSettingW.usePluginSync = false;
const setupType = await askSelectString(this.app, "How would you like to set it up?", [setupAsNew, setupAgain, setupJustImport, setupManually]);
// Migrate completely obsoleted configuration.
if (!newSettingW.useIndexedDBAdapter) {
newSettingW.useIndexedDBAdapter = true;
}
const setupType = await askSelectString(this.app, "How would you like to set it up?", [setupAsNew, setupAgain, setupAsMerge, setupJustImport, setupManually]);
if (setupType == setupJustImport) {
this.plugin.settings = newSettingW;
this.plugin.usedPassphrase = "";
@@ -111,6 +129,10 @@ export class SetupLiveSync extends LiveSyncCommands {
this.plugin.settings = newSettingW;
this.plugin.usedPassphrase = "";
await this.fetchLocal();
} else if (setupType == setupAsMerge) {
this.plugin.settings = newSettingW;
this.plugin.usedPassphrase = "";
await this.fetchLocalWithKeepLocal();
} else if (setupType == setupAgain) {
const confirm = "I know this operation will rebuild all my databases with files on this device, and files that are on the remote database and I didn't synchronize to any other devices will be lost and want to proceed indeed.";
if (await askSelectString(this.app, "Do you really want to do this?", ["Cancel", confirm]) != confirm) {
@@ -134,13 +156,13 @@ export class SetupLiveSync extends LiveSyncCommands {
await this.plugin.replicate(true);
await this.plugin.markRemoteUnlocked();
}
Logger("Configuration loaded.", LOG_LEVEL.NOTICE);
Logger("Configuration loaded.", LOG_LEVEL_NOTICE);
return;
}
if (keepLocalDB == "no" && keepRemoteDB == "no") {
const reset = await askYesNo(this.app, "Drop everything?");
if (reset != "yes") {
Logger("Cancelled", LOG_LEVEL.NOTICE);
Logger("Cancelled", LOG_LEVEL_NOTICE);
this.plugin.settings = oldConf;
return;
}
@@ -175,17 +197,17 @@ export class SetupLiveSync extends LiveSyncCommands {
}
}
Logger("Configuration loaded.", LOG_LEVEL.NOTICE);
Logger("Configuration loaded.", LOG_LEVEL_NOTICE);
} else {
Logger("Cancelled.", LOG_LEVEL.NOTICE);
Logger("Cancelled.", LOG_LEVEL_NOTICE);
}
} catch (ex) {
Logger("Couldn't parse or decrypt configuration uri.", LOG_LEVEL.NOTICE);
Logger("Couldn't parse or decrypt configuration uri.", LOG_LEVEL_NOTICE);
}
}
suspendExtraSync() {
Logger("Hidden files and plugin synchronization have been temporarily disabled. Please enable them after the fetching, if you need them.", LOG_LEVEL.NOTICE)
Logger("Hidden files and plugin synchronization have been temporarily disabled. Please enable them after the fetching, if you need them.", LOG_LEVEL_NOTICE)
this.plugin.settings.syncInternalFiles = false;
this.plugin.settings.usePluginSync = false;
this.plugin.settings.autoSweepPlugins = false;
@@ -230,7 +252,7 @@ Of course, we are able to disable these features.`
return;
}
if (mode != "CUSTOMIZE") {
Logger("Gathering files for enabling Hidden File Sync", LOG_LEVEL.NOTICE);
Logger("Gathering files for enabling Hidden File Sync", LOG_LEVEL_NOTICE);
if (mode == "FETCH") {
await this.plugin.addOnHiddenFileSync.syncInternalFilesAndDatabase("pullForce", true);
} else if (mode == "OVERWRITE") {
@@ -240,7 +262,7 @@ Of course, we are able to disable these features.`
}
this.plugin.settings.syncInternalFiles = true;
await this.plugin.saveSettings();
Logger(`Done! Restarting the app is strongly recommended!`, LOG_LEVEL.NOTICE);
Logger(`Done! Restarting the app is strongly recommended!`, LOG_LEVEL_NOTICE);
} else if (mode == "CUSTOMIZE") {
if (!this.plugin.deviceAndVaultName) {
let name = await askString(this.app, "Device name", "Please set this device name", `desktop`);
@@ -279,13 +301,59 @@ Of course, we are able to disable these features.`
this.plugin.settings.liveSync = false;
this.plugin.settings.periodicReplication = false;
this.plugin.settings.syncOnSave = false;
this.plugin.settings.syncOnEditorSave = false;
this.plugin.settings.syncOnStart = false;
this.plugin.settings.syncOnFileOpen = false;
this.plugin.settings.syncAfterMerge = false;
//this.suspendExtraSync();
}
async suspendReflectingDatabase() {
if (this.plugin.settings.doNotSuspendOnFetching) return;
Logger(`Suspending reflection: Database and storage changes will not be reflected in each other until completely finished the fetching.`, LOG_LEVEL_NOTICE);
this.plugin.settings.suspendParseReplicationResult = true;
this.plugin.settings.suspendFileWatching = true;
await this.plugin.saveSettings();
}
async resumeReflectingDatabase() {
if (this.plugin.settings.doNotSuspendOnFetching) return;
Logger(`Database and storage reflection has been resumed!`, LOG_LEVEL_NOTICE);
this.plugin.settings.suspendParseReplicationResult = false;
this.plugin.settings.suspendFileWatching = false;
await this.plugin.syncAllFiles(true);
await this.plugin.loadQueuedFiles();
this.plugin.procQueuedFiles();
await this.plugin.saveSettings();
}
async askUseNewAdapter() {
if (!this.plugin.settings.useIndexedDBAdapter) {
const message = `Now this plugin has been configured to use the old database adapter for keeping compatibility. Do you want to deactivate it?`;
const CHOICE_YES = "Yes, disable and use latest";
const CHOICE_NO = "No, keep compatibility";
const choices = [CHOICE_YES, CHOICE_NO];
const ret = await confirmWithMessage(this.plugin, "Database adapter", message, choices, CHOICE_YES, 10);
if (ret == CHOICE_YES) {
this.plugin.settings.useIndexedDBAdapter = false;
}
}
}
async fetchRemoteChunks() {
if (!this.plugin.settings.doNotSuspendOnFetching && this.plugin.settings.readChunksOnline) {
Logger(`Fetching chunks`, LOG_LEVEL_NOTICE);
const remoteDB = await this.plugin.getReplicator().connectRemoteCouchDBWithSetting(this.settings, this.plugin.getIsMobile(), true);
if (typeof remoteDB == "string") {
Logger(remoteDB, LOG_LEVEL_NOTICE);
} else {
await fetchAllUsedChunks(this.localDatabase.localDatabase, remoteDB.db);
}
Logger(`Fetching chunks done`, LOG_LEVEL_NOTICE);
}
}
async fetchLocal() {
this.suspendExtraSync();
this.askUseNewAdapter();
await this.suspendReflectingDatabase();
await this.plugin.realizeSettingSyncMode();
await this.plugin.resetLocalDatabase();
await delay(1000);
@@ -296,6 +364,27 @@ Of course, we are able to disable these features.`
await this.plugin.replicateAllFromServer(true);
await delay(1000);
await this.plugin.replicateAllFromServer(true);
await this.fetchRemoteChunks();
await this.resumeReflectingDatabase();
await this.askHiddenFileConfiguration({ enableFetch: true });
}
async fetchLocalWithKeepLocal() {
this.suspendExtraSync();
this.askUseNewAdapter();
await this.suspendReflectingDatabase();
await this.plugin.realizeSettingSyncMode();
await this.plugin.resetLocalDatabase();
await delay(1000);
await this.plugin.initializeDatabase(true);
await this.plugin.markRemoteResolved();
await this.plugin.openDatabase();
this.plugin.isReady = true;
await delay(500);
await this.plugin.replicateAllFromServer(true);
await delay(1000);
await this.plugin.replicateAllFromServer(true);
await this.fetchRemoteChunks();
await this.resumeReflectingDatabase();
await this.askHiddenFileConfiguration({ enableFetch: true });
}
async rebuildRemote() {
@@ -313,6 +402,7 @@ Of course, we are able to disable these features.`
}
async rebuildEverything() {
this.suspendExtraSync();
this.askUseNewAdapter();
await this.plugin.realizeSettingSyncMode();
await this.plugin.resetLocalDatabase();
await delay(1000);

21
src/ConflictResolveModal.ts
View File

@@ -1,6 +1,6 @@
import { App, Modal } from "./deps";
import { DIFF_DELETE, DIFF_EQUAL, DIFF_INSERT } from "diff-match-patch";
import { diff_result } from "./lib/src/types";
import { type diff_result } from "./lib/src/types";
import { escapeStringToHTML } from "./lib/src/strbin";
export class ConflictResolveModal extends Modal {
@@ -18,10 +18,8 @@ export class ConflictResolveModal extends Modal {
onOpen() {
const { contentEl } = this;
this.titleEl.setText("Conflicting changes");
contentEl.empty();
contentEl.createEl("h2", { text: "This document has conflicted changes." });
contentEl.createEl("span", { text: this.filename });
const div = contentEl.createDiv("");
div.addClass("op-scrollable");
@@ -30,11 +28,11 @@ export class ConflictResolveModal extends Modal {
const x1 = v[0];
const x2 = v[1];
if (x1 == DIFF_DELETE) {
diff += "<span class='deleted'>" + escapeStringToHTML(x2) + "</span>";
diff += "<span class='deleted'>" + escapeStringToHTML(x2).replace(/\n/g, "<span class='ls-mark-cr'></span>\n") + "</span>";
} else if (x1 == DIFF_EQUAL) {
diff += "<span class='normal'>" + escapeStringToHTML(x2) + "</span>";
diff += "<span class='normal'>" + escapeStringToHTML(x2).replace(/\n/g, "<span class='ls-mark-cr'></span>\n") + "</span>";
} else if (x1 == DIFF_INSERT) {
diff += "<span class='added'>" + escapeStringToHTML(x2) + "</span>";
diff += "<span class='added'>" + escapeStringToHTML(x2).replace(/\n/g, "<span class='ls-mark-cr'></span>\n") + "</span>";
}
}
@@ -48,23 +46,26 @@ export class ConflictResolveModal extends Modal {
`;
contentEl.createEl("button", { text: "Keep A" }, (e) => {
e.addEventListener("click", async () => {
await this.callback(this.result.right.rev);
const callback = this.callback;
this.callback = null;
this.close();
await callback(this.result.right.rev);
});
});
contentEl.createEl("button", { text: "Keep B" }, (e) => {
e.addEventListener("click", async () => {
await this.callback(this.result.left.rev);
const callback = this.callback;
this.callback = null;
this.close();
await callback(this.result.left.rev);
});
});
contentEl.createEl("button", { text: "Concat both" }, (e) => {
e.addEventListener("click", async () => {
await this.callback("");
const callback = this.callback;
this.callback = null;
this.close();
await callback("");
});
});
contentEl.createEl("button", { text: "Not now" }, (e) => {

77
src/DocumentHistoryModal.ts
View File

@@ -1,9 +1,8 @@
import { TFile, Modal, App } from "./deps";
import { TFile, Modal, App, DIFF_DELETE, DIFF_EQUAL, DIFF_INSERT, diff_match_patch } from "./deps";
import { getPathFromTFile, isValidPath } from "./utils";
import { base64ToArrayBuffer, base64ToString, escapeStringToHTML } from "./lib/src/strbin";
import { decodeBinary, escapeStringToHTML, readString } from "./lib/src/strbin";
import ObsidianLiveSyncPlugin from "./main";
import { DIFF_DELETE, DIFF_EQUAL, DIFF_INSERT, diff_match_patch } from "diff-match-patch";
import { DocumentID, FilePathWithPrefix, LoadedEntry, LOG_LEVEL } from "./lib/src/types";
import { type DocumentID, type FilePathWithPrefix, type LoadedEntry, LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE } from "./lib/src/types";
import { Logger } from "./lib/src/logger";
import { isErrorOfMissingDoc } from "./lib/src/utils_couchdb";
import { getDocData } from "./lib/src/utils";
@@ -11,45 +10,47 @@ import { stripPrefix } from "./lib/src/path";
export class DocumentHistoryModal extends Modal {
plugin: ObsidianLiveSyncPlugin;
range: HTMLInputElement;
contentView: HTMLDivElement;
info: HTMLDivElement;
fileInfo: HTMLDivElement;
range!: HTMLInputElement;
contentView!: HTMLDivElement;
info!: HTMLDivElement;
fileInfo!: HTMLDivElement;
showDiff = false;
id: DocumentID;
id?: DocumentID;
file: FilePathWithPrefix;
revs_info: PouchDB.Core.RevisionInfo[] = [];
currentDoc: LoadedEntry;
currentDoc?: LoadedEntry;
currentText = "";
currentDeleted = false;
initialRev?: string;
constructor(app: App, plugin: ObsidianLiveSyncPlugin, file: TFile | FilePathWithPrefix, id: DocumentID) {
constructor(app: App, plugin: ObsidianLiveSyncPlugin, file: TFile | FilePathWithPrefix, id?: DocumentID, revision?: string) {
super(app);
this.plugin = plugin;
this.file = (file instanceof TFile) ? getPathFromTFile(file) : file;
this.id = id;
if (!file) {
this.file = this.plugin.id2path(id, null);
this.initialRev = revision;
if (!file && id) {
this.file = this.plugin.id2path(id);
}
if (localStorage.getItem("ols-history-highlightdiff") == "1") {
this.showDiff = true;
}
}
async loadFile() {
async loadFile(initialRev: string) {
if (!this.id) {
this.id = await this.plugin.path2id(this.file);
}
const db = this.plugin.localDatabase;
try {
const w = await db.localDatabase.get(this.id, { revs_info: true });
this.revs_info = w._revs_info.filter((e) => e?.status == "available");
this.range.max = `${this.revs_info.length - 1}`;
this.revs_info = w._revs_info?.filter((e) => e?.status == "available") ?? [];
this.range.max = `${Math.max(this.revs_info.length - 1, 0)}`;
this.range.value = this.range.max;
this.fileInfo.setText(`${this.file} / ${this.revs_info.length} revisions`);
await this.loadRevs();
await this.loadRevs(initialRev);
} catch (ex) {
if (isErrorOfMissingDoc(ex)) {
this.range.max = "0";
@@ -59,28 +60,37 @@ export class DocumentHistoryModal extends Modal {
this.contentView.setText(`History of this file was not recorded.`);
} else {
this.contentView.setText(`Error occurred.`);
Logger(ex, LOG_LEVEL.VERBOSE);
Logger(ex, LOG_LEVEL_VERBOSE);
}
}
}
async loadRevs() {
async loadRevs(initialRev?: string) {
if (this.revs_info.length == 0) return;
const db = this.plugin.localDatabase;
if (initialRev) {
const rIndex = this.revs_info.findIndex(e => e.rev == initialRev);
if (rIndex >= 0) {
this.range.value = `${this.revs_info.length - 1 - rIndex}`;
}
}
const index = this.revs_info.length - 1 - (this.range.value as any) / 1;
const rev = this.revs_info[index];
const w = await db.getDBEntry(this.file, { rev: rev.rev }, false, false, true);
await this.showExactRev(rev.rev);
}
async showExactRev(rev: string) {
const db = this.plugin.localDatabase;
const w = await db.getDBEntry(this.file, { rev: rev }, false, false, true);
this.currentText = "";
this.currentDeleted = false;
if (w === false) {
this.currentDeleted = true;
this.info.innerHTML = "";
this.contentView.innerHTML = `Could not read this revision<br>(${rev.rev})`;
this.contentView.innerHTML = `Could not read this revision<br>(${rev})`;
} else {
this.currentDoc = w;
this.info.innerHTML = `Modified:${new Date(w.mtime).toLocaleString()}`;
let result = "";
const w1data = w.datatype == "plain" ? getDocData(w.data) : base64ToString(w.data);
this.currentDeleted = w.deleted;
const w1data = w.datatype == "plain" ? getDocData(w.data) : readString(new Uint8Array(decodeBinary(w.data)));
this.currentDeleted = !!w.deleted;
this.currentText = w1data;
if (this.showDiff) {
const prevRevIdx = this.revs_info.length - 1 - ((this.range.value as any) / 1 - 1);
@@ -89,7 +99,7 @@ export class DocumentHistoryModal extends Modal {
const w2 = await db.getDBEntry(this.file, { rev: oldRev }, false, false, true);
if (w2 != false) {
const dmp = new diff_match_patch();
const w2data = w2.datatype == "plain" ? getDocData(w2.data) : base64ToString(w2.data);
const w2data = w2.datatype == "plain" ? getDocData(w2.data) : readString(new Uint8Array(decodeBinary(w2.data)));
const diff = dmp.diff_main(w2data, w1data);
dmp.diff_cleanupSemantic(diff);
for (const v of diff) {
@@ -120,9 +130,8 @@ export class DocumentHistoryModal extends Modal {
onOpen() {
const { contentEl } = this;
this.titleEl.setText("Document History");
contentEl.empty();
contentEl.createEl("h2", { text: "Document History" });
this.fileInfo = contentEl.createDiv("");
this.fileInfo.addClass("op-info");
const divView = contentEl.createDiv("");
@@ -158,7 +167,7 @@ export class DocumentHistoryModal extends Modal {
.addClass("op-info");
this.info = contentEl.createDiv("");
this.info.addClass("op-info");
this.loadFile();
this.loadFile(this.initialRev);
const div = contentEl.createDiv({ text: "Loading old revisions..." });
this.contentView = div;
div.addClass("op-scrollable");
@@ -168,7 +177,7 @@ export class DocumentHistoryModal extends Modal {
e.addClass("mod-cta");
e.addEventListener("click", async () => {
await navigator.clipboard.writeText(this.currentText);
Logger(`Old content copied to clipboard`, LOG_LEVEL.NOTICE);
Logger(`Old content copied to clipboard`, LOG_LEVEL_NOTICE);
});
});
async function focusFile(path: string) {
@@ -179,7 +188,7 @@ export class DocumentHistoryModal extends Modal {
const leaf = app.workspace.getLeaf(false);
await leaf.openFile(targetFile);
} else {
Logger("The file could not view on the editor", LOG_LEVEL.NOTICE)
Logger("The file could not view on the editor", LOG_LEVEL_NOTICE)
}
}
buttons.createEl("button", { text: "Back to this revision" }, (e) => {
@@ -188,19 +197,19 @@ export class DocumentHistoryModal extends Modal {
// const pathToWrite = this.plugin.id2path(this.id, true);
const pathToWrite = stripPrefix(this.file);
if (!isValidPath(pathToWrite)) {
Logger("Path is not valid to write content.", LOG_LEVEL.INFO);
Logger("Path is not valid to write content.", LOG_LEVEL_INFO);
}
if (this.currentDoc?.datatype == "plain") {
await this.app.vault.adapter.write(pathToWrite, getDocData(this.currentDoc.data));
await this.plugin.vaultAccess.adapterWrite(pathToWrite, getDocData(this.currentDoc.data));
await focusFile(pathToWrite);
this.close();
} else if (this.currentDoc?.datatype == "newnote") {
await this.app.vault.adapter.writeBinary(pathToWrite, base64ToArrayBuffer(this.currentDoc.data));
await this.plugin.vaultAccess.adapterWrite(pathToWrite, decodeBinary(this.currentDoc.data));
await focusFile(pathToWrite);
this.close();
} else {
Logger(`Could not parse entry`, LOG_LEVEL.NOTICE);
Logger(`Could not parse entry`, LOG_LEVEL_NOTICE);
}
});
});

327
src/GlobalHistory.svelte Normal file
View File

@@ -0,0 +1,327 @@
<script lang="ts">
import ObsidianLiveSyncPlugin from "./main";
import { onDestroy, onMount } from "svelte";
import type { AnyEntry, FilePathWithPrefix } from "./lib/src/types";
import { createBinaryBlob, getDocData, isDocContentSame } from "./lib/src/utils";
import { diff_match_patch } from "./deps";
import { DocumentHistoryModal } from "./DocumentHistoryModal";
import { isPlainText, stripAllPrefixes } from "./lib/src/path";
import { TFile } from "./deps";
export let plugin: ObsidianLiveSyncPlugin;
let showDiffInfo = false;
let showChunkCorrected = false;
let checkStorageDiff = false;
let range_from_epoch = Date.now() - 3600000 * 24 * 7;
let range_to_epoch = Date.now() + 3600000 * 24 * 2;
const timezoneOffset = new Date().getTimezoneOffset();
let dispDateFrom = new Date(range_from_epoch - timezoneOffset).toISOString().split("T")[0];
let dispDateTo = new Date(range_to_epoch - timezoneOffset).toISOString().split("T")[0];
$: {
range_from_epoch = new Date(dispDateFrom).getTime() + timezoneOffset;
range_to_epoch = new Date(dispDateTo).getTime() + timezoneOffset;
getHistory(showDiffInfo, showChunkCorrected, checkStorageDiff);
}
function mtimeToDate(mtime: number) {
return new Date(mtime).toLocaleString();
}
type HistoryData = {
id: string;
rev: string;
path: string;
dirname: string;
filename: string;
mtime: number;
mtimeDisp: string;
isDeleted: boolean;
size: number;
changes: string;
chunks: string;
isPlain: boolean;
};
let history = [] as HistoryData[];
let loading = false;
async function fetchChanges(): Promise<HistoryData[]> {
try {
const db = plugin.localDatabase;
let result = [] as typeof history;
for await (const docA of db.findAllNormalDocs()) {
if (docA.mtime < range_from_epoch) {
continue;
}
if (docA.type != "newnote" && docA.type != "plain") continue;
const path = plugin.getPath(docA as AnyEntry);
const isPlain = isPlainText(docA.path);
const revs = await db.getRaw(docA._id, { revs_info: true });
let p: string = undefined;
const reversedRevs = revs._revs_info.reverse();
const DIFF_DELETE = -1;
const DIFF_EQUAL = 0;
const DIFF_INSERT = 1;
for (const revInfo of reversedRevs) {
if (revInfo.status == "available") {
const doc =
(!isPlain && showDiffInfo) || (checkStorageDiff && revInfo.rev == docA._rev)
? await db.getDBEntry(path, { rev: revInfo.rev }, false, false, true)
: await db.getDBEntryMeta(path, { rev: revInfo.rev }, true);
if (doc === false) continue;
const rev = revInfo.rev;
const mtime = "mtime" in doc ? doc.mtime : 0;
if (range_from_epoch > mtime) {
continue;
}
if (range_to_epoch < mtime) {
continue;
}
let diffDetail = "";
if (showDiffInfo && !isPlain) {
const data = getDocData(doc.data);
if (p === undefined) {
p = data;
}
if (p != data) {
const dmp = new diff_match_patch();
const diff = dmp.diff_main(p, data);
dmp.diff_cleanupSemantic(diff);
p = data;
const pxinit = {
[DIFF_DELETE]: 0,
[DIFF_EQUAL]: 0,
[DIFF_INSERT]: 0,
} as { [key: number]: number };
const px = diff.reduce((p, c) => ({ ...p, [c[0]]: (p[c[0]] ?? 0) + c[1].length }), pxinit);
diffDetail = `-${px[DIFF_DELETE]}, +${px[DIFF_INSERT]}`;
}
}
const isDeleted = doc._deleted || (doc as any)?.deleted || false;
if (isDeleted) {
diffDetail += " 🗑️";
}
if (rev == docA._rev) {
if (checkStorageDiff) {
const abs = plugin.vaultAccess.getAbstractFileByPath(stripAllPrefixes(plugin.getPath(docA)));
if (abs instanceof TFile) {
let result = false;
if (isPlainText(docA.path)) {
const data = await plugin.vaultAccess.adapterRead(abs);
result = isDocContentSame(data, doc.data);
} else {
const data = await plugin.vaultAccess.adapterReadBinary(abs);
const dataEEncoded = createBinaryBlob(data);
result = isDocContentSame(dataEEncoded, doc.data);
}
if (result) {
diffDetail += " ⚖️";
} else {
diffDetail += " ⚠️";
}
}
}
}
const docPath = plugin.getPath(doc as AnyEntry);
const [filename, ...pathItems] = docPath.split("/").reverse();
let chunksStatus = "";
if (showChunkCorrected) {
const chunks = (doc as any)?.children ?? [];
const loadedChunks = await db.allDocsRaw({ keys: [...chunks] });
const totalCount = loadedChunks.rows.length;
const errorCount = loadedChunks.rows.filter((e) => "error" in e).length;
if (errorCount == 0) {
chunksStatus = `✅ ${totalCount}`;
} else {
chunksStatus = `🔎 ${errorCount} ✅ ${totalCount}`;
}
}
result.push({
id: doc._id,
rev: doc._rev,
path: docPath,
dirname: pathItems.reverse().join("/"),
filename: filename,
mtime: mtime,
mtimeDisp: mtimeToDate(mtime),
size: (doc as any)?.size ?? 0,
isDeleted: isDeleted,
changes: diffDetail,
chunks: chunksStatus,
isPlain: isPlain,
});
}
}
}
return [...result].sort((a, b) => b.mtime - a.mtime);
} finally {
loading = false;
}
}
async function getHistory(showDiffInfo: boolean, showChunkCorrected: boolean, checkStorageDiff: boolean) {
loading = true;
const newDisplay = [];
const page = await fetchChanges();
newDisplay.push(...page);
history = [...newDisplay];
}
function nextWeek() {
dispDateTo = new Date(range_to_epoch - timezoneOffset + 3600 * 1000 * 24 * 7).toISOString().split("T")[0];
}
function prevWeek() {
dispDateFrom = new Date(range_from_epoch - timezoneOffset - 3600 * 1000 * 24 * 7).toISOString().split("T")[0];
}
onMount(async () => {
await getHistory(showDiffInfo, showChunkCorrected, checkStorageDiff);
});
onDestroy(() => {});
function showHistory(file: string, rev: string) {
new DocumentHistoryModal(plugin.app, plugin, file as unknown as FilePathWithPrefix, null, rev).open();
}
function openFile(file: string) {
plugin.app.workspace.openLinkText(file, file);
}
</script>
<div class="globalhistory">
<h1>Vault history</h1>
<div class="control">
<div class="row"><label for="">From:</label><input type="date" bind:value={dispDateFrom} disabled={loading} /></div>
<div class="row"><label for="">To:</label><input type="date" bind:value={dispDateTo} disabled={loading} /></div>
<div class="row">
<label for="">Info:</label>
<label><input type="checkbox" bind:checked={showDiffInfo} disabled={loading} /><span>Diff</span></label>
<label><input type="checkbox" bind:checked={showChunkCorrected} disabled={loading} /><span>Chunks</span></label>
<label><input type="checkbox" bind:checked={checkStorageDiff} disabled={loading} /><span>File integrity</span></label>
</div>
</div>
{#if loading}
<div class="">Gathering information...</div>
{/if}
<table>
<tr>
<th> Date </th>
<th> Path </th>
<th> Rev </th>
<th> Stat </th>
{#if showChunkCorrected}
<th> Chunks </th>
{/if}
</tr>
<tr>
<td colspan="5" class="more">
{#if loading}
<div class="" />
{:else}
<div><button on:click={() => nextWeek()}>+1 week</button></div>
{/if}
</td>
</tr>
{#each history as entry}
<tr>
<td class="mtime">
{entry.mtimeDisp}
</td>
<td class="path">
<div class="filenames">
<span class="path">/{entry.dirname.split("/").join(`​/`)}</span>
<span class="filename"><a on:click={() => openFile(entry.path)}>{entry.filename}</a></span>
</div>
</td>
<td>
<span class="rev">
{#if entry.isPlain}
<a on:click={() => showHistory(entry.path, entry.rev)}>{entry.rev}</a>
{:else}
{entry.rev}
{/if}
</span>
</td>
<td>
{entry.changes}
</td>
{#if showChunkCorrected}
<td>
{entry.chunks}
</td>
{/if}
</tr>
{/each}
<tr>
<td colspan="5" class="more">
{#if loading}
<div class="" />
{:else}
<div><button on:click={() => prevWeek()}>+1 week</button></div>
{/if}
</td>
</tr>
</table>
</div>
<style>
* {
box-sizing: border-box;
}
.globalhistory {
margin-bottom: 2em;
}
table {
width: 100%;
}
.more > div {
display: flex;
}
.more > div > button {
flex-grow: 1;
}
th {
position: sticky;
top: 0;
backdrop-filter: blur(10px);
}
td.mtime {
white-space: break-spaces;
}
td.path {
word-break: break-word;
}
.row {
display: flex;
flex-direction: row;
flex-wrap: wrap;
}
.row > label {
display: flex;
align-items: center;
min-width: 5em;
}
.row > input {
flex-grow: 1;
}
.filenames {
display: flex;
flex-direction: column;
}
.filenames > .path {
font-size: 70%;
}
.rev {
text-overflow: ellipsis;
max-width: 3em;
display: inline-block;
overflow: hidden;
white-space: nowrap;
}
</style>

49
src/GlobalHistoryView.ts Normal file
View File

@@ -0,0 +1,49 @@
import {
ItemView,
WorkspaceLeaf
} from "./deps";
import GlobalHistoryComponent from "./GlobalHistory.svelte";
import type ObsidianLiveSyncPlugin from "./main";
export const VIEW_TYPE_GLOBAL_HISTORY = "global-history";
export class GlobalHistoryView extends ItemView {
component: GlobalHistoryComponent;
plugin: ObsidianLiveSyncPlugin;
icon: "clock";
title: string;
navigation: true;
getIcon(): string {
return "clock";
}
constructor(leaf: WorkspaceLeaf, plugin: ObsidianLiveSyncPlugin) {
super(leaf);
this.plugin = plugin;
}
getViewType() {
return VIEW_TYPE_GLOBAL_HISTORY;
}
getDisplayText() {
return "Vault history";
}
// eslint-disable-next-line require-await
async onOpen() {
this.component = new GlobalHistoryComponent({
target: this.contentEl,
props: {
plugin: this.plugin,
},
});
}
// eslint-disable-next-line require-await
async onClose() {
this.component.$destroy();
}
}

6
src/JsonResolveModal.ts
View File

@@ -1,5 +1,5 @@
import { App, Modal } from "./deps";
import { FilePath, LoadedEntry } from "./lib/src/types";
import { type FilePath, type LoadedEntry } from "./lib/src/types";
import JsonResolvePane from "./JsonResolvePane.svelte";
export class JsonResolveModal extends Modal {
@@ -29,7 +29,7 @@ export class JsonResolveModal extends Modal {
onOpen() {
const { contentEl } = this;
this.titleEl.setText("Conflicted Setting");
contentEl.empty();
if (this.component == null) {
@@ -41,7 +41,7 @@ export class JsonResolveModal extends Modal {
nameA: this.nameA,
nameB: this.nameB,
defaultSelect: this.defaultSelect,
callback: (keepRev, mergedStr) => this.UICallback(keepRev, mergedStr),
callback: (keepRev: string, mergedStr: string) => this.UICallback(keepRev, mergedStr),
},
});
}

7
src/JsonResolvePane.svelte
View File

@@ -1,7 +1,7 @@
<script lang="ts">
import { Diff, DIFF_DELETE, DIFF_INSERT, diff_match_patch } from "diff-match-patch";
import { type Diff, DIFF_DELETE, DIFF_INSERT, diff_match_patch } from "./deps";
import type { FilePath, LoadedEntry } from "./lib/src/types";
import { base64ToString } from "./lib/src/strbin";
import { decodeBinary, readString } from "./lib/src/strbin";
import { getDocData } from "./lib/src/utils";
import { mergeObject } from "./utils";
@@ -26,7 +26,7 @@
let mode: SelectModes = defaultSelect as SelectModes;
function docToString(doc: LoadedEntry) {
return doc.datatype == "plain" ? getDocData(doc.data) : base64ToString(doc.data);
return doc.datatype == "plain" ? getDocData(doc.data) : readString(new Uint8Array(decodeBinary(doc.data)));
}
function revStringToRevNumber(rev: string) {
return rev.split("-")[0];
@@ -104,7 +104,6 @@
] as ["" | "A" | "B" | "AB" | "BA", string][];
</script>
<h1>Conflicted settings</h1>
<h2>{filename}</h2>
{#if !docA || !docB}
<div class="message">Just for a minute, please!</div>

2
src/KeyValueDB.ts
View File

@@ -1,4 +1,4 @@
import { deleteDB, IDBPDatabase, openDB } from "idb";
import { deleteDB, type IDBPDatabase, openDB } from "idb";
export interface KeyValueDatabase {
get<T>(key: string): Promise<T>;
set<T>(key: string, value: T): Promise<IDBValidKey>;

5
src/LiveSyncCommands.ts
View File

@@ -1,4 +1,4 @@
import { AnyEntry, DocumentID, EntryDoc, EntryHasPath, FilePath, FilePathWithPrefix } from "./lib/src/types";
import { type AnyEntry, type DocumentID, type EntryDoc, type EntryHasPath, type FilePath, type FilePathWithPrefix } from "./lib/src/types";
import { PouchDB } from "./lib/src/pouchdb-browser.js";
import type ObsidianLiveSyncPlugin from "./main";
@@ -14,6 +14,9 @@ export abstract class LiveSyncCommands {
get localDatabase() {
return this.plugin.localDatabase;
}
get vaultAccess() {
return this.plugin.vaultAccess;
}
id2path(id: DocumentID, entry?: EntryHasPath, stripPrefix?: boolean): FilePathWithPrefix {
return this.plugin.id2path(id, entry, stripPrefix);
}

2
src/LogDisplayModal.ts
View File

@@ -14,9 +14,9 @@ export class LogDisplayModal extends Modal {
onOpen() {
const { contentEl } = this;
this.titleEl.setText("Sync status");
contentEl.empty();
contentEl.createEl("h2", { text: "Sync Status" });
const div = contentEl.createDiv("");
div.addClass("op-scrollable");
div.addClass("op-pre");

81
src/LogPane.svelte Normal file
View File

@@ -0,0 +1,81 @@
<script lang="ts">
import { onDestroy, onMount } from "svelte";
import { logMessageStore } from "./lib/src/stores";
let unsubscribe: () => void;
let messages = [] as string[];
let wrapRight = false;
let autoScroll = true;
let suspended = false;
onMount(async () => {
unsubscribe = logMessageStore.observe((e) => {
if (!suspended) {
messages = [...e];
if (autoScroll) {
if (scroll) scroll.scrollTop = scroll.scrollHeight;
}
}
});
logMessageStore.invalidate();
setTimeout(() => {
if (scroll) scroll.scrollTop = scroll.scrollHeight;
}, 100);
});
onDestroy(() => {
if (unsubscribe) unsubscribe();
});
let scroll: HTMLDivElement;
</script>
<div class="logpane">
<!-- <h1>Self-hosted LiveSync Log</h1> -->
<div class="control">
<div class="row">
<label><input type="checkbox" bind:checked={wrapRight} /><span>Wrap</span></label>
<label><input type="checkbox" bind:checked={autoScroll} /><span>Auto scroll</span></label>
<label><input type="checkbox" bind:checked={suspended} /><span>Pause</span></label>
</div>
</div>
<div class="log" bind:this={scroll}>
{#each messages as line}
<pre class:wrap-right={wrapRight}>{line}</pre>
{/each}
</div>
</div>
<style>
* {
box-sizing: border-box;
}
.logpane {
display: flex;
height: 100%;
flex-direction: column;
}
.log {
overflow-y: scroll;
user-select: text;
padding-bottom: 2em;
}
.log > pre {
margin: 0;
}
.log > pre.wrap-right {
word-break: break-all;
max-width: 100%;
width: 100%;
white-space: normal;
}
.row {
display: flex;
flex-direction: row;
justify-content: flex-end;
}
.row > label {
display: flex;
align-items: center;
min-width: 5em;
margin-right: 1em;
}
</style>

48
src/LogPaneView.ts Normal file
View File

@@ -0,0 +1,48 @@
import {
ItemView,
WorkspaceLeaf
} from "obsidian";
import LogPaneComponent from "./LogPane.svelte";
import type ObsidianLiveSyncPlugin from "./main";
export const VIEW_TYPE_LOG = "log-log";
//Log view
export class LogPaneView extends ItemView {
component: LogPaneComponent;
plugin: ObsidianLiveSyncPlugin;
icon: "view-log";
title: string;
navigation: true;
getIcon(): string {
return "view-log";
}
constructor(leaf: WorkspaceLeaf, plugin: ObsidianLiveSyncPlugin) {
super(leaf);
this.plugin = plugin;
}
getViewType() {
return VIEW_TYPE_LOG;
}
getDisplayText() {
return "Self-hosted LiveSync Log";
}
// eslint-disable-next-line require-await
async onOpen() {
this.component = new LogPaneComponent({
target: this.contentEl,
props: {
},
});
}
// eslint-disable-next-line require-await
async onClose() {
this.component.$destroy();
}
}

1686
src/ObsidianLiveSyncSettingTab.ts
View File

File diff suppressed because it is too large Load Diff

14
src/PluginCombo.svelte
View File

@@ -2,7 +2,7 @@
import type { PluginDataExDisplay } from "./CmdConfigSync";
import { Logger } from "./lib/src/logger";
import { versionNumberString2Number } from "./lib/src/strbin";
import { FilePath, LOG_LEVEL } from "./lib/src/types";
import { type FilePath, LOG_LEVEL_NOTICE } from "./lib/src/types";
import { getDocData } from "./lib/src/utils";
import type ObsidianLiveSyncPlugin from "./main";
import { askString, scheduleTask } from "./utils";
@@ -108,7 +108,7 @@
}
}
})
.reduce((p, c) => p | c, 0);
.reduce((p, c) => p | (c as number), 0 as number);
if (matchingStatus == 0b0000100) {
equivalency = "⚖️ Same";
canApply = false;
@@ -207,21 +207,21 @@
const local = list.find((e) => e.term == thisTerm);
const selectedItem = list.find((e) => e.term == selected);
if (selectedItem && (await applyData(selectedItem))) {
scheduleTask("update-plugin-list", 250, () => addOn.updatePluginList(true, local.documentPath));
addOn.updatePluginList(true, local?.documentPath);
}
}
async function compareSelected() {
const local = list.find((e) => e.term == thisTerm);
const selectedItem = list.find((e) => e.term == selected);
if (local && selectedItem && (await compareData(local, selectedItem))) {
scheduleTask("update-plugin-list", 250, () => addOn.updatePluginList(true, local.documentPath));
addOn.updatePluginList(true, local.documentPath);
}
}
async function deleteSelected() {
const selectedItem = list.find((e) => e.term == selected);
// const deletedPath = selectedItem.documentPath;
if (selectedItem && (await deleteData(selectedItem))) {
scheduleTask("update-plugin-list", 250, () => addOn.reloadPluginList(true));
addOn.reloadPluginList(true);
}
}
async function duplicateItem() {
@@ -229,7 +229,7 @@
const duplicateTermName = await askString(plugin.app, "Duplicate", "device name", "");
if (duplicateTermName) {
if (duplicateTermName.contains("/")) {
Logger(`We can not use "/" to the device name`, LOG_LEVEL.NOTICE);
Logger(`We can not use "/" to the device name`, LOG_LEVEL_NOTICE);
return;
}
const key = `${plugin.app.vault.configDir}/${local.files[0].filename}`;
@@ -274,7 +274,7 @@
{/if}
{:else}
<span class="spacer" />
<span class="message even">All devices are even</span>
<span class="message even">All the same or non-existent</span>
<button disabled />
<button disabled />
{/if}

232
src/PluginPane.svelte
View File

@@ -1,11 +1,15 @@
<script lang="ts">
import { onMount } from "svelte";
import ObsidianLiveSyncPlugin from "./main";
import { PluginDataExDisplay, pluginIsEnumerating, pluginList } from "./CmdConfigSync";
import { type PluginDataExDisplay, pluginIsEnumerating, pluginList } from "./CmdConfigSync";
import PluginCombo from "./PluginCombo.svelte";
import { Menu } from "obsidian";
import { unique } from "./lib/src/utils";
import { MODE_SELECTIVE, MODE_AUTOMATIC, MODE_PAUSED, type SYNC_MODE, type PluginSyncSettingEntry } from "./lib/src/types";
import { normalizePath } from "./deps";
export let plugin: ObsidianLiveSyncPlugin;
$: hideNotApplicable = true;
$: hideNotApplicable = false;
$: thisTerm = plugin.deviceAndVaultName;
const addOn = plugin.addOnConfigSync;
@@ -13,7 +17,7 @@
let list: PluginDataExDisplay[] = [];
let selectNewestPulse = 0;
let hideEven = true;
let hideEven = false;
let loading = false;
let applyAllPluse = 0;
let isMaintenanceMode = false;
@@ -80,6 +84,54 @@
async function deleteData(data: PluginDataExDisplay): Promise<boolean> {
return await addOn.deleteData(data);
}
function askMode(evt: MouseEvent, title: string, key: string) {
const menu = new Menu();
menu.addItem((item) => item.setTitle(title).setIsLabel(true));
menu.addSeparator();
const prevMode = automaticList.get(key) ?? MODE_SELECTIVE;
for (const mode of [MODE_SELECTIVE, MODE_AUTOMATIC, MODE_PAUSED]) {
menu.addItem((item) => {
item.setTitle(`${getIcon(mode as SYNC_MODE)}:${TITLES[mode]}`)
.onClick((e) => {
if (mode === MODE_AUTOMATIC) {
askOverwriteModeForAutomatic(evt, key);
} else {
setMode(key, mode as SYNC_MODE);
}
})
.setChecked(prevMode == mode)
.setDisabled(prevMode == mode);
});
}
menu.showAtMouseEvent(evt);
}
function applyAutomaticSync(key: string, direction: "pushForce" | "pullForce" | "safe") {
setMode(key, MODE_AUTOMATIC);
const configDir = normalizePath(plugin.app.vault.configDir);
const files = (plugin.settings.pluginSyncExtendedSetting[key]?.files ?? []).map((e) => `${configDir}/${e}`);
plugin.addOnHiddenFileSync.syncInternalFilesAndDatabase(direction, true, false, files);
}
function askOverwriteModeForAutomatic(evt: MouseEvent, key: string) {
const menu = new Menu();
menu.addItem((item) => item.setTitle("Initial Action").setIsLabel(true));
menu.addSeparator();
menu.addItem((item) => {
item.setTitle(`↑: Overwrite Remote`).onClick((e) => {
applyAutomaticSync(key, "pushForce");
});
})
.addItem((item) => {
item.setTitle(`↓: Overwrite Local`).onClick((e) => {
applyAutomaticSync(key, "pullForce");
});
})
.addItem((item) => {
item.setTitle(`⇅: Use newer`).onClick((e) => {
applyAutomaticSync(key, "safe");
});
});
menu.showAtMouseEvent(evt);
}
$: options = {
thisTerm,
@@ -92,11 +144,84 @@
plugin,
isMaintenanceMode,
};
const ICON_EMOJI_PAUSED = `⛔`;
const ICON_EMOJI_AUTOMATIC = `✨`;
const ICON_EMOJI_SELECTIVE = `🔀`;
const ICONS: { [key: number]: string } = {
[MODE_SELECTIVE]: ICON_EMOJI_SELECTIVE,
[MODE_PAUSED]: ICON_EMOJI_PAUSED,
[MODE_AUTOMATIC]: ICON_EMOJI_AUTOMATIC,
};
const TITLES: { [key: number]: string } = {
[MODE_SELECTIVE]: "Selective",
[MODE_PAUSED]: "Ignore",
[MODE_AUTOMATIC]: "Automatic",
};
const PREFIX_PLUGIN_ALL = "PLUGIN_ALL";
const PREFIX_PLUGIN_DATA = "PLUGIN_DATA";
const PREFIX_PLUGIN_MAIN = "PLUGIN_MAIN";
function setMode(key: string, mode: SYNC_MODE) {
if (key.startsWith(PREFIX_PLUGIN_ALL + "/")) {
setMode(PREFIX_PLUGIN_DATA + key.substring(PREFIX_PLUGIN_ALL.length), mode);
setMode(PREFIX_PLUGIN_MAIN + key.substring(PREFIX_PLUGIN_ALL.length), mode);
}
const files = unique(
list
.filter((e) => `${e.category}/${e.name}` == key)
.map((e) => e.files)
.flat()
.map((e) => e.filename)
);
automaticList.set(key, mode);
automaticListDisp = automaticList;
if (!(key in plugin.settings.pluginSyncExtendedSetting)) {
plugin.settings.pluginSyncExtendedSetting[key] = {
key,
mode,
files: [],
};
}
plugin.settings.pluginSyncExtendedSetting[key].files = files;
plugin.settings.pluginSyncExtendedSetting[key].mode = mode;
plugin.saveSettingData();
}
function getIcon(mode: SYNC_MODE) {
if (mode in ICONS) {
return ICONS[mode];
} else {
("");
}
}
let automaticList = new Map<string, SYNC_MODE>();
let automaticListDisp = new Map<string, SYNC_MODE>();
// apply current configuration to the dialogue
for (const { key, mode } of Object.values(plugin.settings.pluginSyncExtendedSetting)) {
automaticList.set(key, mode);
}
automaticListDisp = automaticList;
let displayKeys: Record<string, string[]> = {};
$: {
const extraKeys = Object.keys(plugin.settings.pluginSyncExtendedSetting);
displayKeys = [
...list,
...extraKeys
.map((e) => `${e}///`.split("/"))
.filter((e) => e[0] && e[1])
.map((e) => ({ category: e[0], name: e[1], displayName: e[1] })),
]
.sort((a, b) => (a.displayName ?? a.name).localeCompare(b.displayName ?? b.name))
.reduce((p, c) => ({ ...p, [c.category]: unique(c.category in p ? [...p[c.category], c.displayName ?? c.name] : [c.displayName ?? c.name]) }), {} as Record<string, string[]>);
}
</script>
<div>
<div>
<h1>Customization sync</h1>
<div class="buttons">
<button on:click={() => scanAgain()}>Scan changes</button>
<button on:click={() => replicate()}>Sync once</button>
@@ -119,15 +244,24 @@
{#if list.length == 0}
<div class="center">No Items.</div>
{:else}
{#each Object.entries(displays) as [key, label]}
{#each Object.entries(displays).filter(([key, _]) => key in displayKeys) as [key, label]}
<div>
<h3>{label}</h3>
{#each groupBy(filterList(list, [key]), "name") as [name, listX]}
{#each displayKeys[key] as name}
{@const bindKey = `${key}/${name}`}
{@const mode = automaticListDisp.get(bindKey) ?? MODE_SELECTIVE}
<div class="labelrow {hideEven ? 'hideeven' : ''}">
<div class="title">
{name}
<button class="status" on:click={(evt) => askMode(evt, `${key}/${name}`, bindKey)}>
{getIcon(mode)}
</button>
<span class="name">{name}</span>
</div>
<PluginCombo {...options} list={listX} hidden={false} />
{#if mode == MODE_SELECTIVE}
<PluginCombo {...options} list={list.filter((e) => e.category == key && e.name == name)} hidden={false} />
{:else}
<div class="statusnote">{TITLES[mode]}</div>
{/if}
</div>
{/each}
</div>
@@ -135,20 +269,55 @@
<div>
<h3>Plugins</h3>
{#each groupBy(filterList(list, ["PLUGIN_MAIN", "PLUGIN_DATA", "PLUGIN_ETC"]), "name") as [name, listX]}
{@const bindKeyAll = `${PREFIX_PLUGIN_ALL}/${name}`}
{@const modeAll = automaticListDisp.get(bindKeyAll) ?? MODE_SELECTIVE}
{@const bindKeyMain = `${PREFIX_PLUGIN_MAIN}/${name}`}
{@const modeMain = automaticListDisp.get(bindKeyMain) ?? MODE_SELECTIVE}
{@const bindKeyData = `${PREFIX_PLUGIN_DATA}/${name}`}
{@const modeData = automaticListDisp.get(bindKeyData) ?? MODE_SELECTIVE}
<div class="labelrow {hideEven ? 'hideeven' : ''}">
<div class="title">
{name}
<button class="status" on:click={(evt) => askMode(evt, `${PREFIX_PLUGIN_ALL}/${name}`, bindKeyAll)}>
{getIcon(modeAll)}
</button>
<span class="name">{name}</span>
</div>
<PluginCombo {...options} list={listX} hidden={true} />
</div>
<div class="filerow {hideEven ? 'hideeven' : ''}">
<div class="filetitle">Main</div>
<PluginCombo {...options} list={filterList(listX, ["PLUGIN_MAIN"])} hidden={false} />
</div>
<div class="filerow {hideEven ? 'hideeven' : ''}">
<div class="filetitle">Data</div>
<PluginCombo {...options} list={filterList(listX, ["PLUGIN_DATA"])} hidden={false} />
{#if modeAll == MODE_SELECTIVE}
<PluginCombo {...options} list={listX} hidden={true} />
{/if}
</div>
{#if modeAll == MODE_SELECTIVE}
<div class="filerow {hideEven ? 'hideeven' : ''}">
<div class="filetitle">
<button class="status" on:click={(evt) => askMode(evt, `${PREFIX_PLUGIN_MAIN}/${name}/MAIN`, bindKeyMain)}>
{getIcon(modeMain)}
</button>
<span class="name">MAIN</span>
</div>
{#if modeMain == MODE_SELECTIVE}
<PluginCombo {...options} list={filterList(listX, ["PLUGIN_MAIN"])} hidden={false} />
{:else}
<div class="statusnote">{TITLES[modeMain]}</div>
{/if}
</div>
<div class="filerow {hideEven ? 'hideeven' : ''}">
<div class="filetitle">
<button class="status" on:click={(evt) => askMode(evt, `${PREFIX_PLUGIN_DATA}/${name}`, bindKeyData)}>
{getIcon(modeData)}
</button>
<span class="name">DATA</span>
</div>
{#if modeData == MODE_SELECTIVE}
<PluginCombo {...options} list={filterList(listX, ["PLUGIN_DATA"])} hidden={false} />
{:else}
<div class="statusnote">{TITLES[modeData]}</div>
{/if}
</div>
{:else}
<div class="noterow">
<div class="statusnote">{TITLES[modeAll]}</div>
</div>
{/if}
{/each}
</div>
{/if}
@@ -162,6 +331,15 @@
</div>
<style>
span.spacer {
min-width: 1px;
flex-grow: 1;
}
h3 {
position: sticky;
top: 0;
background-color: var(--modal-background);
}
.labelrow {
margin-left: 0.4em;
display: flex;
@@ -183,6 +361,24 @@
.labelrow.hideeven:has(.even) {
display: none;
}
.noterow {
min-height: 2em;
display: flex;
}
button.status {
flex-grow: 0;
margin: 2px 4px;
min-width: 3em;
max-width: 4em;
}
.statusnote {
display: flex;
justify-content: flex-end;
padding-right: var(--size-4-12);
align-items: center;
min-width: 10em;
flex-grow: 1;
}
.title {
color: var(--text-normal);

118
src/SerializedFileAccess.ts Normal file
View File

@@ -0,0 +1,118 @@
import { type App, TFile, type DataWriteOptions, TFolder, TAbstractFile } from "./deps";
import { serialized } from "./lib/src/lock";
import type { FilePath } from "./lib/src/types";
function getFileLockKey(file: TFile | TFolder | string) {
return `fl:${typeof (file) == "string" ? file : file.path}`;
}
function toArrayBuffer(arr: Uint8Array | ArrayBuffer | DataView): ArrayBufferLike {
if (arr instanceof Uint8Array) {
return arr.buffer;
}
if (arr instanceof DataView) {
return arr.buffer;
}
return arr;
}
export class SerializedFileAccess {
app: App
constructor(app: App) {
this.app = app;
}
async adapterStat(file: TFile | string) {
const path = file instanceof TFile ? file.path : file;
return await serialized(getFileLockKey(path), () => this.app.vault.adapter.stat(path));
}
async adapterExists(file: TFile | string) {
const path = file instanceof TFile ? file.path : file;
return await serialized(getFileLockKey(path), () => this.app.vault.adapter.exists(path));
}
async adapterRemove(file: TFile | string) {
const path = file instanceof TFile ? file.path : file;
return await serialized(getFileLockKey(path), () => this.app.vault.adapter.remove(path));
}
async adapterRead(file: TFile | string) {
const path = file instanceof TFile ? file.path : file;
return await serialized(getFileLockKey(path), () => this.app.vault.adapter.read(path));
}
async adapterReadBinary(file: TFile | string) {
const path = file instanceof TFile ? file.path : file;
return await serialized(getFileLockKey(path), () => this.app.vault.adapter.readBinary(path));
}
async adapterWrite(file: TFile | string, data: string | ArrayBuffer | Uint8Array, options?: DataWriteOptions) {
const path = file instanceof TFile ? file.path : file;
if (typeof (data) === "string") {
return await serialized(getFileLockKey(path), () => this.app.vault.adapter.write(path, data, options));
} else {
return await serialized(getFileLockKey(path), () => this.app.vault.adapter.writeBinary(path, toArrayBuffer(data), options));
}
}
async vaultCacheRead(file: TFile) {
return await serialized(getFileLockKey(file), () => this.app.vault.cachedRead(file));
}
async vaultRead(file: TFile) {
return await serialized(getFileLockKey(file), () => this.app.vault.read(file));
}
async vaultReadBinary(file: TFile) {
return await serialized(getFileLockKey(file), () => this.app.vault.readBinary(file));
}
async vaultModify(file: TFile, data: string | ArrayBuffer | Uint8Array, options?: DataWriteOptions) {
if (typeof (data) === "string") {
return await serialized(getFileLockKey(file), () => this.app.vault.modify(file, data, options));
} else {
return await serialized(getFileLockKey(file), () => this.app.vault.modifyBinary(file, toArrayBuffer(data), options));
}
}
async vaultCreate(path: string, data: string | ArrayBuffer | Uint8Array, options?: DataWriteOptions): Promise<TFile> {
if (typeof (data) === "string") {
return await serialized(getFileLockKey(path), () => this.app.vault.create(path, data, options));
} else {
return await serialized(getFileLockKey(path), () => this.app.vault.createBinary(path, toArrayBuffer(data), options));
}
}
async delete(file: TFile | TFolder, force = false) {
return await serialized(getFileLockKey(file), () => this.app.vault.delete(file, force));
}
async trash(file: TFile | TFolder, force = false) {
return await serialized(getFileLockKey(file), () => this.app.vault.trash(file, force));
}
getAbstractFileByPath(path: FilePath | string): TAbstractFile | null {
// Disabled temporary.
return this.app.vault.getAbstractFileByPath(path);
// // Hidden API but so useful.
// // @ts-ignore
// if ("getAbstractFileByPathInsensitive" in app.vault && (app.vault.adapter?.insensitive ?? false)) {
// // @ts-ignore
// return app.vault.getAbstractFileByPathInsensitive(path);
// } else {
// return app.vault.getAbstractFileByPath(path);
// }
}
touchedFiles: string[] = [];
touch(file: TFile | FilePath) {
const f = file instanceof TFile ? file : this.getAbstractFileByPath(file) as TFile;
const key = `${f.path}-${f.stat.mtime}-${f.stat.size}`;
this.touchedFiles.unshift(key);
this.touchedFiles = this.touchedFiles.slice(0, 100);
}
recentlyTouched(file: TFile) {
const key = `${file.path}-${file.stat.mtime}-${file.stat.size}`;
if (this.touchedFiles.indexOf(key) == -1) return false;
return true;
}
clearTouched() {
this.touchedFiles = [];
}
}

51
src/StorageEventManager.ts
View File

@@ -1,9 +1,9 @@
import { Plugin_2, TAbstractFile, TFile, TFolder } from "./deps";
import type { SerializedFileAccess } from "./SerializedFileAccess";
import { Plugin, TAbstractFile, TFile, TFolder } from "./deps";
import { isPlainText, shouldBeIgnored } from "./lib/src/path";
import { getGlobalStore } from "./lib/src/store";
import { FilePath, ObsidianLiveSyncSettings } from "./lib/src/types";
import { FileEventItem, FileEventType, FileInfo, InternalFileInfo, queueItem } from "./types";
import { recentlyTouched } from "./utils";
import { type FilePath, type ObsidianLiveSyncSettings } from "./lib/src/types";
import { type FileEventItem, type FileEventType, type FileInfo, type InternalFileInfo, type queueItem } from "./types";
export abstract class StorageEventManager {
@@ -12,12 +12,14 @@ export abstract class StorageEventManager {
abstract getQueueLength(): number;
}
type LiveSyncForStorageEventManager = Plugin_2 &
type LiveSyncForStorageEventManager = Plugin &
{
settings: ObsidianLiveSyncSettings
ignoreFiles: string[],
vaultAccess: SerializedFileAccess
} & {
isTargetFile: (file: string | TAbstractFile) => boolean,
procFileEvent: (applyBatch?: boolean) => Promise<boolean>
isTargetFile: (file: string | TAbstractFile) => Promise<boolean>,
procFileEvent: (applyBatch?: boolean) => Promise<any>,
};
@@ -35,12 +37,12 @@ export class StorageEventManagerObsidian extends StorageEventManager {
this.watchVaultDelete = this.watchVaultDelete.bind(this);
this.watchVaultRename = this.watchVaultRename.bind(this);
this.watchVaultRawEvents = this.watchVaultRawEvents.bind(this);
plugin.registerEvent(app.vault.on("modify", this.watchVaultChange));
plugin.registerEvent(app.vault.on("delete", this.watchVaultDelete));
plugin.registerEvent(app.vault.on("rename", this.watchVaultRename));
plugin.registerEvent(app.vault.on("create", this.watchVaultCreate));
plugin.registerEvent(plugin.app.vault.on("modify", this.watchVaultChange));
plugin.registerEvent(plugin.app.vault.on("delete", this.watchVaultDelete));
plugin.registerEvent(plugin.app.vault.on("rename", this.watchVaultRename));
plugin.registerEvent(plugin.app.vault.on("create", this.watchVaultCreate));
//@ts-ignore : Internal API
plugin.registerEvent(app.vault.on("raw", this.watchVaultRawEvents));
plugin.registerEvent(plugin.app.vault.on("raw", this.watchVaultRawEvents));
}
watchVaultCreate(file: TAbstractFile, ctx?: any) {
@@ -64,9 +66,18 @@ export class StorageEventManagerObsidian extends StorageEventManager {
}
// Watch raw events (Internal API)
watchVaultRawEvents(path: FilePath) {
if (this.plugin.settings.useIgnoreFiles && this.plugin.ignoreFiles.some(e => path.endsWith(e.trim()))) {
// If it is one of ignore files, refresh the cached one.
this.plugin.isTargetFile(path).then(() => this._watchVaultRawEvents(path));
} else {
this._watchVaultRawEvents(path);
}
}
_watchVaultRawEvents(path: FilePath) {
if (!this.plugin.settings.syncInternalFiles && !this.plugin.settings.usePluginSync) return;
if (!this.plugin.settings.watchInternalFileChanges) return;
if (!path.startsWith(app.vault.configDir)) return;
if (!path.startsWith(this.plugin.app.vault.configDir)) return;
const ignorePatterns = this.plugin.settings.syncInternalFilesIgnorePatterns
.replace(/\n| /g, "")
.split(",").filter(e => e).map(e => new RegExp(e, "i"));
@@ -77,7 +88,6 @@ export class StorageEventManagerObsidian extends StorageEventManager {
file: { path, mtime: 0, ctime: 0, size: 0 }
}], null);
}
// Cache file and waiting to can be proceed.
async appendWatchEvent(params: { type: FileEventType, file: TAbstractFile | InternalFileInfo, oldPath?: string }[], ctx?: any) {
let forcePerform = false;
@@ -90,21 +100,20 @@ export class StorageEventManagerObsidian extends StorageEventManager {
const file = param.file;
const oldPath = param.oldPath;
if (file instanceof TFolder) continue;
if (!this.plugin.isTargetFile(file.path)) continue;
if (!await this.plugin.isTargetFile(file.path)) continue;
if (this.plugin.settings.suspendFileWatching) continue;
let cache: null | string | ArrayBuffer;
// new file or something changed, cache the changes.
if (file instanceof TFile && (type == "CREATE" || type == "CHANGED")) {
if (recentlyTouched(file)) {
if (this.plugin.vaultAccess.recentlyTouched(file)) {
continue;
}
if (!isPlainText(file.name)) {
cache = await app.vault.readBinary(file);
cache = await this.plugin.vaultAccess.vaultReadBinary(file);
} else {
// cache = await this.app.vault.read(file);
cache = await app.vault.cachedRead(file);
if (!cache) cache = await app.vault.read(file);
cache = await this.plugin.vaultAccess.vaultCacheRead(file);
if (!cache) cache = await this.plugin.vaultAccess.vaultRead(file);
}
}
if (type == "DELETE" || type == "RENAME") {
@@ -112,7 +121,7 @@ export class StorageEventManagerObsidian extends StorageEventManager {
}
if (this.plugin.settings.batchSave) {
if (this.plugin.settings.batchSave && !this.plugin.settings.liveSync) {
// if the latest event is the same type, omit that
// a.md MODIFY <- this should be cancelled when a.md MODIFIED
// b.md MODIFY <- this should be cancelled when b.md MODIFIED

9
src/deps.ts
View File

@@ -1,12 +1,13 @@
import { FilePath } from "./lib/src/types";
import { type FilePath } from "./lib/src/types";
export {
addIcon, App, DataWriteOptions, debounce, Editor, FuzzySuggestModal, MarkdownRenderer, MarkdownView, Modal, Notice, Platform, Plugin, PluginManifest,
PluginSettingTab, Plugin_2, requestUrl, RequestUrlParam, RequestUrlResponse, sanitizeHTMLToDom, Setting, stringifyYaml, TAbstractFile, TextAreaComponent, TFile, TFolder,
parseYaml
addIcon, App, debounce, Editor, FuzzySuggestModal, MarkdownRenderer, MarkdownView, Modal, Notice, Platform, Plugin, PluginSettingTab, requestUrl, sanitizeHTMLToDom, Setting, stringifyYaml, TAbstractFile, TextAreaComponent, TFile, TFolder,
parseYaml, ItemView, WorkspaceLeaf
} from "obsidian";
export type { DataWriteOptions, PluginManifest, RequestUrlParam, RequestUrlResponse, MarkdownFileInfo } from "obsidian";
import {
normalizePath as normalizePath_
} from "obsidian";
const normalizePath = normalizePath_ as <T extends string | FilePath>(from: T) => T;
export { normalizePath }
export { type Diff, DIFF_DELETE, DIFF_EQUAL, DIFF_INSERT, diff_match_patch } from "diff-match-patch";

22
src/dialogs.ts
View File

@@ -20,6 +20,7 @@ export class PluginDialogModal extends Modal {
onOpen() {
const { contentEl } = this;
this.titleEl.setText("Customization Sync (Beta2)")
if (this.component == null) {
this.component = new PluginPane({
target: contentEl,
@@ -38,27 +39,27 @@ export class PluginDialogModal extends Modal {
export class InputStringDialog extends Modal {
result: string | false = false;
onSubmit: (result: string | boolean) => void;
onSubmit: (result: string | false) => void;
title: string;
key: string;
placeholder: string;
isManuallyClosed = false;
isPassword = false;
constructor(app: App, title: string, key: string, placeholder: string, onSubmit: (result: string | false) => void) {
constructor(app: App, title: string, key: string, placeholder: string, isPassword: boolean, onSubmit: (result: string | false) => void) {
super(app);
this.onSubmit = onSubmit;
this.title = title;
this.placeholder = placeholder;
this.key = key;
this.isPassword = isPassword;
}
onOpen() {
const { contentEl } = this;
contentEl.createEl("h1", { text: this.title });
// For enter to submit
const formEl = contentEl.createEl("form");
new Setting(formEl).setName(this.key).addText((text) =>
this.titleEl.setText(this.title);
const formEl = contentEl.createDiv();
new Setting(formEl).setName(this.key).setClass(this.isPassword ? "password-input" : "normal-input").addText((text) =>
text.onChange((value) => {
this.result = value;
})
@@ -142,7 +143,7 @@ export class MessageBox extends Modal {
timer: ReturnType<typeof setInterval> = undefined;
defaultButtonComponent: ButtonComponent | undefined;
onSubmit: (result: string | boolean) => void;
onSubmit: (result: string | false) => void;
constructor(plugin: Plugin, title: string, contentMd: string, buttons: string[], defaultAction: (typeof buttons)[number], timeout: number, onSubmit: (result: (typeof buttons)[number] | false) => void) {
super(plugin.app);
@@ -173,16 +174,17 @@ export class MessageBox extends Modal {
onOpen() {
const { contentEl } = this;
this.titleEl.setText(this.title);
contentEl.addEventListener("click", () => {
if (this.timer) {
clearInterval(this.timer);
this.timer = undefined;
}
})
contentEl.createEl("h1", { text: this.title });
const div = contentEl.createDiv();
MarkdownRenderer.renderMarkdown(this.contentMd, div, "/", null);
MarkdownRenderer.render(this.plugin.app, this.contentMd, div, "/", this.plugin);
const buttonSetting = new Setting(contentEl);
buttonSetting.controlEl.style.flexWrap = "wrap";
for (const button of this.buttons) {
buttonSetting.addButton((btn) => {
btn

2
src/lib

Submodule src/lib updated: ec4ecacb43...7e79c27035

1166
src/main.ts
View File

File diff suppressed because it is too large Load Diff

4
src/types.ts
View File

@@ -1,5 +1,5 @@
import { PluginManifest, TFile } from "./deps";
import { DatabaseEntry, EntryBody, FilePath } from "./lib/src/types";
import { type PluginManifest, TFile } from "./deps";
import { type DatabaseEntry, type EntryBody, type FilePath } from "./lib/src/types";
export interface PluginDataEntry extends DatabaseEntry {
deviceVaultName: string;

324
src/utils.ts
View File

@@ -1,12 +1,12 @@
import { DataWriteOptions, normalizePath, TFile, Platform, TAbstractFile, App, Plugin_2, RequestUrlParam, requestUrl } from "./deps";
import { normalizePath, TFile, Platform, TAbstractFile, App, Plugin, type RequestUrlParam, requestUrl } from "./deps";
import { path2id_base, id2path_base, isValidFilenameInLinux, isValidFilenameInDarwin, isValidFilenameInWidows, isValidFilenameInAndroid, stripAllPrefixes } from "./lib/src/path";
import { Logger } from "./lib/src/logger";
import { AnyEntry, DocumentID, EntryDoc, EntryHasPath, FilePath, FilePathWithPrefix, LOG_LEVEL, NewEntry } from "./lib/src/types";
import { CHeader, ICHeader, ICHeaderLength, PSCHeader } from "./types";
import { LOG_LEVEL_VERBOSE, type AnyEntry, type DocumentID, type EntryHasPath, type FilePath, type FilePathWithPrefix } from "./lib/src/types";
import { CHeader, ICHeader, ICHeaderLength, ICXHeader, PSCHeader } from "./types";
import { InputStringDialog, PopoverSelectString } from "./dialogs";
import ObsidianLiveSyncPlugin from "./main";
import { runWithLock } from "./lib/src/lock";
import { writeString } from "./lib/src/strbin";
// For backward compatibility, using the path for determining id.
// Only CouchDB unacceptable ID (that starts with an underscore) has been prefixed with "/".
@@ -303,20 +303,6 @@ export function flattenObject(obj: Record<string | number | symbol, any>, path:
return ret;
}
export function modifyFile(file: TFile, data: string | ArrayBuffer, options?: DataWriteOptions) {
if (typeof (data) === "string") {
return app.vault.modify(file, data, options);
} else {
return app.vault.modifyBinary(file, data, options);
}
}
export function createFile(path: string, data: string | ArrayBuffer, options?: DataWriteOptions): Promise<TFile> {
if (typeof (data) === "string") {
return app.vault.create(path, data, options);
} else {
return app.vault.createBinary(path, data, options);
}
}
export function isValidPath(filename: string) {
if (Platform.isDesktop) {
@@ -328,42 +314,14 @@ export function isValidPath(filename: string) {
if (Platform.isAndroidApp) return isValidFilenameInAndroid(filename);
if (Platform.isIosApp) return isValidFilenameInDarwin(filename);
//Fallback
Logger("Could not determine platform for checking filename", LOG_LEVEL.VERBOSE);
Logger("Could not determine platform for checking filename", LOG_LEVEL_VERBOSE);
return isValidFilenameInWidows(filename);
}
let touchedFiles: string[] = [];
export function getAbstractFileByPath(path: FilePath): TAbstractFile | null {
// Disabled temporary.
return app.vault.getAbstractFileByPath(path);
// // Hidden API but so useful.
// // @ts-ignore
// if ("getAbstractFileByPathInsensitive" in app.vault && (app.vault.adapter?.insensitive ?? false)) {
// // @ts-ignore
// return app.vault.getAbstractFileByPathInsensitive(path);
// } else {
// return app.vault.getAbstractFileByPath(path);
// }
}
export function trimPrefix(target: string, prefix: string) {
return target.startsWith(prefix) ? target.substring(prefix.length) : target;
}
export function touch(file: TFile | FilePath) {
const f = file instanceof TFile ? file : getAbstractFileByPath(file) as TFile;
const key = `${f.path}-${f.stat.mtime}-${f.stat.size}`;
touchedFiles.unshift(key);
touchedFiles = touchedFiles.slice(0, 100);
}
export function recentlyTouched(file: TFile) {
const key = `${file.path}-${file.stat.mtime}-${file.stat.size}`;
if (touchedFiles.indexOf(key) == -1) return false;
return true;
}
export function clearTouched() {
touchedFiles = [];
}
/**
* returns is internal chunk of file
@@ -388,6 +346,9 @@ export function isChunk(str: string): boolean {
export function isPluginMetadata(str: string): boolean {
return str.startsWith(PSCHeader);
}
export function isCustomisationSyncMetadata(str: string): boolean {
return str.startsWith(ICXHeader);
}
export const askYesNo = (app: App, message: string): Promise<"yes" | "no"> => {
return new Promise((res) => {
@@ -405,9 +366,9 @@ export const askSelectString = (app: App, message: string, items: string[]): Pro
};
export const askString = (app: App, title: string, key: string, placeholder: string): Promise<string | false> => {
export const askString = (app: App, title: string, key: string, placeholder: string, isPassword?: boolean): Promise<string | false> => {
return new Promise((res) => {
const dialog = new InputStringDialog(app, title, key, placeholder, (result) => res(result));
const dialog = new InputStringDialog(app, title, key, placeholder, isPassword, (result) => res(result));
dialog.open();
});
};
@@ -416,8 +377,8 @@ export const askString = (app: App, title: string, key: string, placeholder: str
export class PeriodicProcessor {
_process: () => Promise<any>;
_timer?: number;
_plugin: Plugin_2;
constructor(plugin: Plugin_2, process: () => Promise<any>) {
_plugin: Plugin;
constructor(plugin: Plugin, process: () => Promise<any>) {
this._plugin = plugin;
this._process = process;
}
@@ -431,22 +392,17 @@ export class PeriodicProcessor {
enable(interval: number) {
this.disable();
if (interval == 0) return;
this._timer = window.setInterval(() => this._process().then(() => { }), interval);
this._timer = window.setInterval(() => this.process().then(() => { }), interval);
this._plugin.registerInterval(this._timer);
}
disable() {
if (this._timer) clearInterval(this._timer);
if (this._timer !== undefined) window.clearInterval(this._timer);
this._timer = undefined;
}
}
function sizeToHumanReadable(size: number | undefined) {
if (!size) return "-";
const i = Math.floor(Math.log(size) / Math.log(1024));
return Number.parseInt((size / Math.pow(1024, i)).toFixed(2)) + ' ' + ['B', 'kB', 'MB', 'GB', 'TB'][i];
}
export const _requestToCouchDBFetch = async (baseUri: string, username: string, password: string, path?: string, body?: string | any, method?: string) => {
const utf8str = String.fromCharCode.apply(null, new TextEncoder().encode(`${username}:${password}`));
const utf8str = String.fromCharCode.apply(null, [...writeString(`${username}:${password}`)]);
const encoded = window.btoa(utf8str);
const authHeader = "Basic " + encoded;
const transformedHeaders: Record<string, string> = { authorization: authHeader, "content-type": "application/json" };
@@ -462,7 +418,7 @@ export const _requestToCouchDBFetch = async (baseUri: string, username: string,
}
export const _requestToCouchDB = async (baseUri: string, username: string, password: string, origin: string, path?: string, body?: any, method?: string) => {
const utf8str = String.fromCharCode.apply(null, new TextEncoder().encode(`${username}:${password}`));
const utf8str = String.fromCharCode.apply(null, [...writeString(`${username}:${password}`)]);
const encoded = window.btoa(utf8str);
const authHeader = "Basic " + encoded;
const transformedHeaders: Record<string, string> = { authorization: authHeader, origin: origin };
@@ -492,249 +448,3 @@ export async function performRebuildDB(plugin: ObsidianLiveSyncPlugin, method: "
await plugin.addOnSetup.rebuildEverything();
}
}
export const gatherChunkUsage = async (db: PouchDB.Database<EntryDoc>) => {
const used = new Map();
const unreferenced = new Map();
const removed = new Map();
const missing = new Map();
const xx = await db.allDocs({ startkey: "h:", endkey: `h:\u{10ffff}` });
for (const xxd of xx.rows) {
const chunk = xxd.id
unreferenced.set(chunk, xxd.value.rev);
}
const x = await db.find({ limit: 999999999, selector: { children: { $exists: true, $type: "array" } }, fields: ["_id", "path", "mtime", "children"] });
for (const temp of x.docs) {
for (const chunk of (temp as NewEntry).children) {
used.set(chunk, (used.has(chunk) ? used.get(chunk) : 0) + 1);
if (unreferenced.has(chunk)) {
removed.set(chunk, unreferenced.get(chunk));
unreferenced.delete(chunk);
} else {
if (!removed.has(chunk)) {
if (!missing.has(temp._id)) {
missing.set(temp._id, []);
}
missing.get(temp._id).push(chunk);
}
}
}
}
return { used, unreferenced, missing };
}
export const localDatabaseCleanUp = async (plugin: ObsidianLiveSyncPlugin, force: boolean, dryRun: boolean) => {
await runWithLock("clean-up:local", true, async () => {
const db = plugin.localDatabase.localDatabase;
if ((db as any)?.adapter != "indexeddb") {
if (force && !dryRun) {
Logger("Fetch from the remote database", LOG_LEVEL.NOTICE, "clean-up-db");
await performRebuildDB(plugin, "localOnly");
return;
} else {
Logger("This feature requires enabling `Use new adapter`. Please enable it", LOG_LEVEL.NOTICE, "clean-up-db");
return;
}
}
Logger(`The remote database has been locked for garbage collection`, LOG_LEVEL.NOTICE, "clean-up-db");
Logger(`Gathering chunk usage information`, LOG_LEVEL.NOTICE, "clean-up-db");
const { unreferenced, missing } = await gatherChunkUsage(db);
if (missing.size != 0) {
Logger(`Some chunks are not found! We have to rescue`, LOG_LEVEL.NOTICE);
Logger(missing, LOG_LEVEL.VERBOSE);
} else {
Logger(`All chunks are OK`, LOG_LEVEL.NOTICE);
}
const payload = {} as Record<string, string[]>;
for (const [id, rev] of unreferenced) {
payload[id] = [rev];
}
const removeItems = Object.keys(payload).length;
if (removeItems == 0) {
Logger(`No unreferenced chunks found (Local)`, LOG_LEVEL.NOTICE);
await plugin.markRemoteResolved();
}
if (dryRun) {
Logger(`There are ${removeItems} unreferenced chunks (Local)`, LOG_LEVEL.NOTICE);
return;
}
Logger(`Deleting unreferenced chunks: ${removeItems}`, LOG_LEVEL.NOTICE, "clean-up-db");
for (const [id, rev] of unreferenced) {
//@ts-ignore
const ret = await db.purge(id, rev);
Logger(ret, LOG_LEVEL.VERBOSE);
}
plugin.localDatabase.refreshSettings();
Logger(`Compacting local database...`, LOG_LEVEL.NOTICE, "clean-up-db");
await db.compact();
await plugin.markRemoteResolved();
Logger("Done!", LOG_LEVEL.NOTICE, "clean-up-db");
})
}
export const balanceChunks = async (plugin: ObsidianLiveSyncPlugin, dryRun: boolean) => {
await runWithLock("clean-up:balance", true, async () => {
const localDB = plugin.localDatabase.localDatabase;
Logger(`Gathering chunk usage information`, LOG_LEVEL.NOTICE, "clean-up-db");
const ret = await plugin.replicator.connectRemoteCouchDBWithSetting(plugin.settings, plugin.isMobile);
if (typeof ret === "string") {
Logger(`Connect error: ${ret}`, LOG_LEVEL.NOTICE, "clean-up-db");
return;
}
const localChunks = new Map<string, string>();
const xx = await localDB.allDocs({ startkey: "h:", endkey: `h:\u{10ffff}` });
for (const xxd of xx.rows) {
const chunk = xxd.id
localChunks.set(chunk, xxd.value.rev);
}
// const info = ret.info;
const remoteDB = ret.db;
const remoteChunks = new Map<string, string>();
const xxr = await remoteDB.allDocs({ startkey: "h:", endkey: `h:\u{10ffff}` });
for (const xxd of xxr.rows) {
const chunk = xxd.id
remoteChunks.set(chunk, xxd.value.rev);
}
const localToRemote = new Map<string, string>([...localChunks]);
const remoteToLocal = new Map<string, string>([...remoteChunks]);
for (const id of new Set([...localChunks.keys(), ...remoteChunks.keys()])) {
if (remoteChunks.has(id)) {
localToRemote.delete(id);
}
if (localChunks.has(id)) {
remoteToLocal.delete(id);
}
}
function arrayToChunkedArray<T>(src: T[], size = 25) {
const ret = [] as T[][];
let i = 0;
while (i < src.length) {
ret.push(src.slice(i, i += size));
}
return ret;
}
if (localToRemote.size == 0) {
Logger(`No chunks need to be sent`, LOG_LEVEL.NOTICE);
} else {
Logger(`${localToRemote.size} chunks need to be sent`, LOG_LEVEL.NOTICE);
if (!dryRun) {
const w = arrayToChunkedArray([...localToRemote]);
for (const chunk of w) {
for (const [id,] of chunk) {
const queryRet = await localDB.allDocs({ keys: [id], include_docs: true });
const docs = queryRet.rows.filter(e => !("error" in e)).map(x => x.doc);
const ret = await remoteDB.bulkDocs(docs, { new_edits: false });
Logger(ret, LOG_LEVEL.VERBOSE);
}
}
Logger(`Done! ${remoteToLocal.size} chunks have been sent`, LOG_LEVEL.NOTICE);
}
}
if (remoteToLocal.size == 0) {
Logger(`No chunks need to be retrieved`, LOG_LEVEL.NOTICE);
} else {
Logger(`${remoteToLocal.size} chunks need to be retrieved`, LOG_LEVEL.NOTICE);
if (!dryRun) {
const w = arrayToChunkedArray([...remoteToLocal]);
for (const chunk of w) {
for (const [id,] of chunk) {
const queryRet = await remoteDB.allDocs({ keys: [id], include_docs: true });
const docs = queryRet.rows.filter(e => !("error" in e)).map(x => x.doc);
const ret = await localDB.bulkDocs(docs, { new_edits: false });
Logger(ret, LOG_LEVEL.VERBOSE);
}
}
Logger(`Done! ${remoteToLocal.size} chunks have been retrieved`, LOG_LEVEL.NOTICE);
}
}
})
}
export const remoteDatabaseCleanup = async (plugin: ObsidianLiveSyncPlugin, dryRun: boolean) => {
const getSize = function (info: PouchDB.Core.DatabaseInfo, key: "active" | "external" | "file") {
return Number.parseInt((info as any)?.sizes?.[key] ?? 0);
}
await runWithLock("clean-up:remote", true, async () => {
const CHUNK_SIZE = 100;
function makeChunkedArrayFromArray<T>(items: T[]): T[][] {
const chunked = [];
for (let i = 0; i < items.length; i += CHUNK_SIZE) {
chunked.push(items.slice(i, i + CHUNK_SIZE));
}
return chunked;
}
try {
const ret = await plugin.replicator.connectRemoteCouchDBWithSetting(plugin.settings, plugin.isMobile);
if (typeof ret === "string") {
Logger(`Connect error: ${ret}`, LOG_LEVEL.NOTICE, "clean-up-db");
return;
}
const info = ret.info;
Logger(JSON.stringify(info), LOG_LEVEL.VERBOSE, "clean-up-db");
Logger(`Database active-size: ${sizeToHumanReadable(getSize(info, "active"))}, external-size:${sizeToHumanReadable(getSize(info, "external"))}, file-size: ${sizeToHumanReadable(getSize(info, "file"))}`, LOG_LEVEL.NOTICE);
if (!dryRun) {
Logger(`The remote database has been locked for garbage collection`, LOG_LEVEL.NOTICE, "clean-up-db");
await plugin.markRemoteLocked(true);
}
Logger(`Gathering chunk usage information`, LOG_LEVEL.NOTICE, "clean-up-db");
const db = ret.db;
const { unreferenced, missing } = await gatherChunkUsage(db);
if (missing.size != 0) {
Logger(`Some chunks are not found! We have to rescue`, LOG_LEVEL.NOTICE);
Logger(missing, LOG_LEVEL.VERBOSE);
} else {
Logger(`All chunks are OK`, LOG_LEVEL.NOTICE);
}
const payload = {} as Record<string, string[]>;
for (const [id, rev] of unreferenced) {
payload[id] = [rev];
}
const removeItems = Object.keys(payload).length;
if (removeItems == 0) {
Logger(`No unreferenced chunk found (Remote)`, LOG_LEVEL.NOTICE);
return;
}
if (dryRun) {
Logger(`There are ${removeItems} unreferenced chunks (Remote)`, LOG_LEVEL.NOTICE);
return;
}
Logger(`Deleting unreferenced chunks: ${removeItems}`, LOG_LEVEL.NOTICE, "clean-up-db");
const buffer = makeChunkedArrayFromArray(Object.entries(payload));
for (const chunkedPayload of buffer) {
const rets = await _requestToCouchDBFetch(
`${plugin.settings.couchDB_URI}/${plugin.settings.couchDB_DBNAME}`,
plugin.settings.couchDB_USER,
plugin.settings.couchDB_PASSWORD,
"_purge",
chunkedPayload.reduce((p, c) => ({ ...p, [c[0]]: c[1] }), {}), "POST");
// const result = await rets();
Logger(JSON.stringify(await rets.json()), LOG_LEVEL.VERBOSE);
}
Logger(`Compacting database...`, LOG_LEVEL.NOTICE, "clean-up-db");
await db.compact();
const endInfo = await db.info();
Logger(`Processed database active-size: ${sizeToHumanReadable(getSize(endInfo, "active"))}, external-size:${sizeToHumanReadable(getSize(endInfo, "external"))}, file-size: ${sizeToHumanReadable(getSize(endInfo, "file"))}`, LOG_LEVEL.NOTICE);
Logger(`Reduced sizes: active-size: ${sizeToHumanReadable(getSize(info, "active") - getSize(endInfo, "active"))}, external-size:${sizeToHumanReadable(getSize(info, "external") - getSize(endInfo, "external"))}, file-size: ${sizeToHumanReadable(getSize(info, "file") - getSize(endInfo, "file"))}`, LOG_LEVEL.NOTICE);
Logger(JSON.stringify(endInfo), LOG_LEVEL.VERBOSE, "clean-up-db");
Logger(`Local database cleaning up...`);
await localDatabaseCleanUp(plugin, true, false);
} catch (ex) {
Logger("Failed to clean up db.")
Logger(ex, LOG_LEVEL.VERBOSE);
}
});
}

19
styles.css
View File

@@ -251,4 +251,23 @@ div.sls-setting-menu-btn {
.sls-item-dirty::before {
content: "✏";
}
.sls-setting-hidden {
display: none;
}
.password-input > .setting-item-control >input {
-webkit-text-security: disc;
}
span.ls-mark-cr::after {
user-select: none;
content: "↲";
color: var(--text-muted);
font-size: 0.8em;
}
.deleted span.ls-mark-cr::after {
color: var(--text-on-accent);
}

10
tsconfig.json
View File

@@ -1,4 +1,6 @@
{
"extends": "@tsconfig/svelte/tsconfig.json",
"inlineSourceMap": true,
"compilerOptions": {
"baseUrl": ".",
"module": "ESNext",
@@ -6,16 +8,22 @@
"allowJs": true,
"noImplicitAny": true,
"moduleResolution": "node",
"types": [
"svelte",
"node"
],
// "importsNotUsedAsValues": "error",
"importHelpers": false,
"alwaysStrict": true,
"allowImportingTsExtensions": true,
"lib": [
"es2018",
"DOM",
"ES5",
"ES6",
"ES7",
"es2019.array"
"es2019.array",
"ES2020.BigInt",
]
},
"include": [

60
updates.md
View File

@@ -1,39 +1,31 @@
### 0.19.0
#### Customization sync
Since `Plugin and their settings` have been broken, so I tried to fix it, not just fix it, but fix it the way it should be.
Now, we have `Customization sync`.
It is a real shame that the compatibility between these features has been broken. However, this new feature is surely useful and I believe that worth getting over the pain.
We can use the new feature with the same configuration. Only the menu on the command palette has been changed. The dialog can be opened by `Show customization sync dialog`.
I hope you will give it a try.
### 0.21.0
The E2EE encryption V2 format has been reverted. That was probably the cause of the glitch.
Instead, to maintain efficiency, files are treated with Blob until just before saving. Along with this, the old-fashioned encryption format has also been discontinued.
There are both forward and backwards compatibilities, with recent versions. However, unfortunately, we lost compatibility with filesystem-livesync or some.
It will be addressed soon. Please be patient if you are using filesystem-livesync with E2EE.
#### Minors
- 0.19.1
- Fixed: Fixed hidden file handling on Linux
- Improved: Now customization sync works more smoothly.
- 0.19.2
#### Version history
- 0.21.2
- IMPORTANT NOTICE: **0.21.1 CONTAINS A BUG WHILE REBUILDING THE DATABASE. IF YOU HAVE BEEN REBUILT, PLEASE MAKE SURE THAT ALL FILES ARE SANE.**
- This has been fixed in this version.
- Fixed:
- Fixed garbage collection error while unreferenced chunks exist many.
- Fixed filename validation on Linux.
- Improved:
- Showing status is now thinned for performance.
- Enhance caching while collecting chunks.
- 0.19.3
- Improved:
- Now replication will be paced by collecting chunks. If synchronisation has been deadlocked, please enable `Do not pace synchronization` once.
- 0.19.4
- Improved:
- Reduced remote database checking to improve speed and reduce bandwidth.
- No longer files are broken while rebuilding.
- Now, Large binary files can be written correctly on a mobile platform.
- Any decoding errors now make zero-byte files.
- Modified:
- All files are processed sequentially for each.
- 0.21.1
- Fixed:
- Chunks which previously misinterpreted are now interpreted correctly.
- No more missing chunks which not be found forever, except if it has been actually missing.
- Deleted file detection on hidden file synchronising now works fine.
- Now the Customisation sync is surely quiet while it has been disabled.
- No more infinity loops on larger files.
- Show message on decode error.
- Refactored:
- Fixed to avoid obsolete global variables.
- 0.21.0
- Changes and performance improvements:
- Now the saving files are processed by Blob.
- The V2-Format has been reverted.
- New encoding format has been enabled in default.
- WARNING: Since this version, the compatibilities with older Filesystem LiveSync have been lost.
... To continue on to `updates_old.md`.
... To continue on to `updates_old.md`.

260
updates_old.md
View File

@@ -1,3 +1,263 @@
## 0.20.0
At 0.20.0, Self-hosted LiveSync has changed the binary file format and encrypting format, for efficient synchronisation.
The dialogue will be shown and asks us to decide whether to keep v1 or use v2. Once we have enabled v2, all subsequent edits will be saved in v2. Therefore, devices running 0.19 or below cannot understand this and they might say that decryption error. Please update all devices.
Then we will have an impressive performance.
Of course, these are very impactful changes. If you have any questions or troubled things, please feel free to open an issue and mention me.
Note: if you want to roll it back to v1, please enable `Use binary and encryption version 1` on the `Hatch` pane and perform the `rebuild everything` once.
Extra but notable information:
This format change gives us the ability to detect some `marks` in the binary files as same as text files. Therefore, we can split binary files and some specific sort of them (i.e., PDF files) at the specific character. It means that editing the middle of files could be detected with marks.
Now only a few chunks are transferred, even if we add a comment to the PDF or put new files into the ZIP archives.
- 0.20.7
- Fixed
- To better replication, path obfuscation is now deterministic even if with E2EE.
Note: Compatible with previous database without any conversion. Only new files will be obfuscated in deterministic.
- 0.20.6
- Fixed
- Now empty file could be decoded.
- Local files are no longer pre-saved before fetching from a remote database.
- No longer deadlock while applying customisation sync.
- Configuration with multiple files is now able to be applied correctly.
- Deleting folder propagation now works without enabling the use of a trash bin.
- 0.20.5
- Fixed
- Now the files which having digit or character prefixes in the path will not be ignored.
- 0.20.4
- Fixed
- The text-input-dialogue is no longer broken.
- Finally, we can use the Setup URI again on mobile.
- 0.20.3
- New feature:
- We can launch Customization sync from the Ribbon if we enabled it.
- Fixed:
- Setup URI is now back to the previous spec; be encrypted by V1.
- It may avoid the trouble with iOS 17.
- The Settings dialogue is now registered at the beginning of the start-up process.
- We can change the configuration even though LiveSync could not be launched in normal.
- Improved:
- Enumerating documents has been faster.
- 0.20.2
- New feature:
- We can delete all data of customization sync from the `Delete all customization sync data` on the `Hatch` pane.
- Fixed:
- Prevent keep restarting on iOS by yielding microtasks.
- 0.20.1
- Fixed:
- No more UI freezing and keep restarting on iOS.
- Diff of Non-markdown documents are now shown correctly.
- Improved:
- Performance has been a bit improved.
- Customization sync has gotten faster.
- However, We lost forward compatibility again (only for this feature). Please update all devices.
- Misc
- Terser configuration has been more aggressive.
- 0.20.0
- Improved:
- A New binary file handling implemented
- A new encrypted format has been implemented
- Now the chunk sizes will be adjusted for efficient sync
- Fixed:
- levels of exception in some logs have been fixed
- Tidied:
- Some Lint warnings have been suppressed.
### 0.19.0
#### Customization sync
Since `Plugin and their settings` have been broken, so I tried to fix it, not just fix it, but fix it the way it should be.
Now, we have `Customization sync`.
It is a real shame that the compatibility between these features has been broken. However, this new feature is surely useful and I believe that worth getting over the pain.
We can use the new feature with the same configuration. Only the menu on the command palette has been changed. The dialog can be opened by `Show customization sync dialog`.
I hope you will give it a try.
#### Minors
- 0.19.1
- Fixed: Fixed hidden file handling on Linux
- Improved: Now customization sync works more smoothly.
- 0.19.2
- Fixed:
- Fixed garbage collection error while unreferenced chunks exist many.
- Fixed filename validation on Linux.
- Improved:
- Showing status is now thinned for performance.
- Enhance caching while collecting chunks.
- 0.19.3
- Improved:
- Now replication will be paced by collecting chunks. If synchronisation has been deadlocked, please enable `Do not pace synchronization` once.
- 0.19.4
- Improved:
- Reduced remote database checking to improve speed and reduce bandwidth.
- Fixed:
- Chunks which previously misinterpreted are now interpreted correctly.
- No more missing chunks which not be found forever, except if it has been actually missing.
- Deleted file detection on hidden file synchronising now works fine.
- Now the Customisation sync is surely quiet while it has been disabled.
- 0.19.5
- Fixed:
- Now hidden file synchronisation would not be hanged, even if so many files exist.
- Improved:
- Customisation sync works more smoothly.
- Note: Concurrent processing has been rollbacked into the original implementation. As a result, the total number of processes is no longer shown next to the hourglass icon. However, only the processes that are running concurrently are shown.
- 0.19.6
- Fixed:
- Logging has been tweaked.
- No more too many planes and rockets.
- The batch database update now surely only works in non-live mode.
- Internal things:
- Some frameworks has been upgraded.
- Import declaration has been fixed.
- Improved:
- The plug-in now asks to enable a new adaptor, when rebuilding, if it is not enabled yet.
- The setting dialogue refined.
- Configurations for compatibilities have been moved under the hatch.
- Made it clear that disabled is the default.
- Ambiguous names configuration have been renamed.
- Items that have no meaning in the settings are no longer displayed.
- Some items have been reordered for clarity.
- Each configuration has been grouped.
- 0.19.7
- Fixed:
- The initial pane of Setting dialogue is now changed to General Settings.
- The Setup Wizard is now able to flush existing settings and get into the mode again.
- 0.19.8
- New feature:
- Vault history: A tab has been implemented to give a birds-eye view of the changes that have occurred in the vault.
- Improved:
- Now the passphrases on the dialogue masked out. Thank you @antoKeinanen!
- Log dialogue is now shown as one of tabs.
- Fixed:
- Some minor issues has been fixed.
- 0.19.9
- New feature (For fixing a problem):
- We can fix the database obfuscated and plain paths that have been mixed up.
- Improvements
- Customisation Sync performance has been improved.
- 0.19.10
- Fixed
- Fixed the issue about fixing the database.
- 0.19.11
- Improvements:
- Hashing ChunkID has been improved.
- Logging keeps 400 lines now.
- Refactored:
- Import statement has been fixed about types.
- 0.19.12
- Improved:
- Boot-up performance has been improved.
- Customisation sync performance has been improved.
- Synchronising performance has been improved.
- 0.19.13
- Implemented:
- Database clean-up is now in beta 2!
We can shrink the remote database by deleting unused chunks, with keeping history.
Note: Local database is not cleaned up totally. We have to `Fetch` again to let it done.
**Note2**: Still in beta. Please back your vault up anything before.
- Fixed:
- The log updates are not thinned out now.
- 0.19.14
- Fixed:
- Internal documents are now ignored.
- Merge dialogue now respond immediately to button pressing.
- Periodic processing now works fine.
- The checking interval of detecting conflicted has got shorter.
- Replication is now cancelled while cleaning up.
- The database locking by the cleaning up is now carefully unlocked.
- Missing chunks message is correctly reported.
- New feature:
- Suspend database reflecting has been implemented.
- This can be disabled by `Fetch database with previous behaviour`.
- Now fetch suspends the reflecting database and storage changes temporarily to improve the performance.
- We can choose the action when the remote database has been cleaned
- Merge dialogue now show `↲` before the new line.
- Improved:
- Now progress is reported while the cleaning up and fetch process.
- Cancelled replication is now detected.
- 0.19.15
- Fixed:
- Now storing files after cleaning up is correct works.
- Improved:
- Cleaning the local database up got incredibly fastened.
Now we can clean instead of fetching again when synchronising with the remote which has been cleaned up.
- 0.19.16
- Many upgrades on this release. I have tried not to let that happen, if something got corrupted, please feel free to notify me.
- New feature:
- (Beta) ignore files handling
We can use `.gitignore`, `.dockerignore`, and anything you like to filter the synchronising files.
- Fixed:
- Buttons on lock-detected-dialogue now can be shown in narrow-width devices.
- Improved:
- Some constant has been flattened to be evaluated.
- The usage of the deprecated API of obsidian has been reduced.
- Now the indexedDB adapter will be enabled while the importing configuration.
- Misc:
- Compiler, framework, and dependencies have been upgraded.
- Due to standing for these impacts (especially in esbuild and svelte,) terser has been introduced.
Feel free to notify your opinion to me! I do not like to obfuscate the code too.
- 0.19.17
- Fixed:
- Now nested ignore files could be parsed correctly.
- The unexpected deletion of hidden files in some cases has been corrected.
- Hidden file change is no longer reflected on the device which has made the change itself.
- Behaviour changed:
- From this version, the file which has `:` in its name should be ignored even if on Linux devices.
- 0.19.18
- Fixed:
- Now the empty (or deleted) file could be conflict-resolved.
- 0.19.19
- Fixed:
- Resolving conflicted revision has become more robust.
- LiveSync now try to keep local changes when fetching from the rebuilt remote database.
Local changes now have been kept as a revision and fetched things will be new revisions.
- Now, all files will be restored after performing `fetch` immediately.
- 0.19.20
- New feature:
- `Sync on Editor save` has been implemented
- We can start synchronisation when we save from the Obsidian explicitly.
- Now we can use the `Hidden file sync` and the `Customization sync` cooperatively.
- We can exclude files from `Hidden file sync` which is already handled in Customization sync.
- We can ignore specific plugins in Customization sync.
- Now the message of leftover conflicted files accepts our click.
- We can open `Resolve all conflicted files` in an instant.
- Refactored:
- Parallelism functions made more explicit.
- Type errors have been reduced.
- Fixed:
- Now documents would not be overwritten if they are conflicted.
It will be saved as a new conflicted revision.
- Some error messages have been fixed.
- Missing dialogue titles have been shown now.
- We can click close buttons on mobile now.
- Conflicted Customisation sync files will be resolved automatically by their modified time.
- 0.19.21
- Fixed:
- Hidden files are no longer handled in the initial replication.
- Report from `Making report` fixed
- No longer contains customisation sync information.
- Version of LiveSync has been added.
- 0.19.22
- Fixed:
- Now the synchronisation will begin without our interaction.
- No longer puts the configuration of the remote database into the log while checking configuration.
- Some outdated description notes have been removed.
- Options that are meaningless depending on other settings configured are now hidden.
- Scan for hidden files before replication
- Scan customization periodically
- 0.19.23
-Improved:
- We can open the log pane also from the command palette now.
- Now, the hidden file scanning interval could be configured to 0.
- `Check database configuration` now points out that we do not have administrator permission.
### 0.18.0