public/obsidian-livesync
1
0
Fork 0
mirror of https://github.com/vrtmrz/obsidian-livesync.git synced 2026-02-22 20:18:48 +00:00
Code Issues Packages Projects Releases Wiki Activity

Compare commits

base: public:0.20.2
Branches Tags
public:main public:beta public:snyk-upgrade-7fd31e6f474568014502e7205847f257 public:snyk-fix-90f3b71450f7543a1c61724d8afa0e71 public:refactor_services_2 public:refactor_services public:dev public:snyk-upgrade-fe9625385aa4be849c7157033fd2abb0 public:snyk-upgrade-99a31e70c822894595544b0c9813b11a public:svelteui public:disenchant public:snyk-fix-61f76ef0bce2a276df9552c2fc80a14f public:snyk-upgrade-2baa7938ba318464c3bad5c3fd5cbda5 public:snyk-upgrade-7af62a575eab20e8fb4e9565bf1adcfc public:snyk-upgrade-b32f281b31a81483f126b8fb6c54a957 public:snyk-upgrade-bd61b1103442d3faecc9fc726c34e9a4 public:snyk-upgrade-5b2d6584430c061b5187611a5d7fb64a public:snyk-fix-0e9fbbcb374fcb8d5e8cb027d8414fb7 public:snyk-upgrade-5242daf91e706d1ae500c9ea6545636e public:snyk-upgrade-70434a6822fbad83424135dfad9df909 public:snyk-upgrade-8fa01ccad0352125f1866d41e57d35e1 public:snyk-upgrade-39eb97b4655e70091f718a88fccd57c1 public:snyk-upgrade-2b99d54d19e207887e36c3071b796925 public:forprivacy public:snyk-upgrade-dad9088d0a333677e0157e3deef7fc38 public:snyk-upgrade-b9a76d74b3e5723759af5b3319aabb80 public:snyk-upgrade-7da6459d8ff5d293d80de9d80061d912 public:snyk-upgrade-6a3786fc4cbe394542e9d1b6058c46e7 public:snyk-upgrade-8fc9e62771dc41b0f23dc0efefb5c949
public:0.25.43-patched-9 public:0.25.43-patched-8 public:0.25.43-patched-7 public:0.25.43-patched-6 public:0.25.43-patched-5 public:0.25.43-patched-4 public:0.25.43-patched-3 public:0.25.43-patched-2 public:0.25.43-patched-1 public:0.25.43 public:0.25.42 public:0.25.41-patched-3 public:0.25.41-patched-2 public:0.25.41-patched-1 public:0.25.41 public:0.25.40 public:0.25.39 public:0.25.38 public:0.25.37 public:0.25.36 public:0.25.35 public:0.25.34 public:0.25.33 public:0.25.32 public:0.25.31 public:0.25.30 public:0.25.30-dev public:0.25.29 public:0.25.28 public:0.25.27 public:0.25.26 public:0.25.25 public:0.25.24 public:0.25.24.beta3 public:0.25.24.beta2 public:0.25.24.beta1 public:0.25.23 public:0.25.23.beta1 public:0.25.22 public:0.25.21 public:0.25.21.beta2 public:0.25.21.beta1 public:0.25.20 public:0.25.19 public:0.25.18 public:0.25.17 public:0.25.16 public:0.25.15 public:0.25.14 public:0.25.13 public:0.25.12 public:0.25.11 public:0.25.10 public:0.25.9 public:0.25.8 public:0.25.7 public:0.25.6 public:0.25.5 public:0.25.4 public:0.25.3 public:0.25.2 public:0.25.1 public:0.25.0 public:0.25.0-beta1 public:0.24.31 public:0.24.30 public:0.24.29 public:0.24.28 public:0.24.27 public:0.24.26 public:0.24.25 public:0.24.24 public:0.24.23 public:0.24.22 public:0.24.21 public:0.24.20 public:0.24.19 public:0.24.18 public:0.24.17 public:0.24.16 public:0.24.15 public:0.24.14 public:0.24.13 public:0.24.12 public:0.24.11 public:0.24.10 public:0.24.9 public:0.24.8 public:0.24.7 public:0.24.6 public:0.24.5 public:0.24.4 public:0.24.3 public:0.24.2 public:0.24.1 public:0.24.0 public:0.24.0.dev-rc8 public:0.24.0.dev-rc7 public:0.24.0.dev-rc6 public:0.24.0.dev-rc5 public:0.24.0.dev-rc4 public:0.24.0.dev-rc3 public:0.24.0.dev-rc2 public:0.23.23 public:0.23.22 public:0.23.21 public:0.23.20 public:0.23.19 public:0.23.18 public:0.23.17 public:0.23.16 public:0.23.15 public:0.23.14 public:0.23.13 public:0.23.12 public:0.23.11 public:0.23.10 public:0.23.9 public:0.23.8 public:0.23.7 public:0.23.6 public:0.23.5 public:0.23.4 public:0.23.3 public:0.23.2 public:0.23.1 public:0.23.0 public:0.22.19 public:0.22.18 public:0.22.17 public:0.22.16 public:0.22.15 public:0.22.14 public:0.22.13 public:0.22.12 public:0.22.11 public:0.22.10 public:0.22.9 public:0.22.8 public:0.22.7 public:0.22.6 public:0.22.5 public:0.22.4 public:0.22.3 public:0.22.2 public:0.22.1 public:0.22.0 public:0.21.5 public:0.21.4 public:0.21.3 public:0.21.2 public:0.21.1 public:0.21.0 public:0.20.7 public:0.20.6 public:0.20.5 public:0.20.4 public:0.20.3 public:0.20.2 public:0.20.1 public:0.20.0 public:0.19.23 public:0.19.22 public:0.19.21 public:0.19.20 public:0.19.19 public:0.19.18 public:0.19.17 public:0.19.16 public:0.19.15 public:0.19.14 public:0.19.13 public:0.19.12 public:0.19.11 public:0.19.10 public:0.19.9 public:0.19.8 public:0.19.7 public:0.19.6 public:0.19.5 public:0.19.4 public:0.19.3 public:0.19.2 public:0.19.1 public:0.19.0 public:0.18.6 public:0.18.5 public:0.18.4 public:0.18.3 public:0.18.2 public:0.18.1 public:0.18.0 public:0.17.34 public:0.17.33 public:0.17.32 public:0.17.31 public:0.17.30 public:0.17.29 public:0.17.28 public:0.17.27 public:0.17.26 public:0.17.25 public:0.17.24 public:0.17.23 public:0.17.22 public:0.17.21 public:0.17.20 public:0.17.19 public:0.17.18 public:0.17.17 public:0.17.16 public:0.17.15 public:0.17.14 public:0.17.13 public:0.17.12 public:0.17.11 public:0.17.10 public:0.17.9 public:0.17.8 public:0.17.7 public:0.17.6 public:0.17.5 public:0.17.4 public:0.17.3 public:0.17.2 public:0.17.1 public:0.17.0 public:0.16.8 public:0.16.7 public:0.16.6 public:0.16.5 public:0.16.4 public:0.16.3 public:0.16.2 public:0.16.1 public:0.16.0 public:0.15.10 public:0.15.9 public:0.15.8 public:0.15.7 public:0.15.6 public:0.15.5 public:0.15.4 public:0.15.3 public:0.15.2 public:0.15.1 public:0.15.0 public:0.14.8 public:0.14.7 public:0.14.6 public:0.14.5 public:0.14.4 public:0.14.3 public:0.14.2 public:0.14.1 public:0.14.0 public:0.13.4 public:0.13.3 public:0.13.2 public:0.13.1 public:0.13.0 public:0.12.5 public:0.12.4 public:0.12.3 public:0.12.2 public:0.12.1 public:0.12.0 public:0.11.10 public:0.11.9 public:0.11.8 public:0.11.7 public:0.11.6 public:0.11.5 public:0.11.4 public:0.11.3 public:0.11.2 public:0.11.1 public:0.11.0 public:0.10.1 public:0.10.0 public:0.9.0 public:0.8.8 public:0.8.7 public:0.8.6 public:0.8.5 public:0.8.3 public:0.8.2 public:0.8.1 public:0.8.0 public:0.7.2 public:0.7.1 public:0.7.0 public:0.6.1 public:0.6.0 public:0.5.0 public:0.4.1 public:0.4.0 public:0.3.1 public:0.3.0 public:0.2.1 public:0.2.0 public:0.1.26 public:0.1.25 public:0.1.24 public:0.1.23 public:0.1.22 public:0.1.21 public:0.1.20 public:0.1.19 public:0.1.18 public:0.1.17 public:0.1.16 public:0.1.15 public:0.1.14 public:0.1.13 public:0.1.12 public:0.1.11 public:0.1.10 public:0.1.9 public:0.1.8 public:0.1.7 public:0.1.6 public:0.1.5 public:0.1.4 public:0.1.3 public:0.1.2 public:0.1.1 public:0.1.0 public:0.0.9 public:0.0.8 public:0.0.7 public:0.0.6
...
compare: public:snyk-fix-0e9fbbcb374fcb8d5e8cb027d8414fb7
Branches Tags
public:beta public:main public:snyk-upgrade-7fd31e6f474568014502e7205847f257 public:snyk-fix-90f3b71450f7543a1c61724d8afa0e71 public:refactor_services_2 public:refactor_services public:dev public:snyk-upgrade-fe9625385aa4be849c7157033fd2abb0 public:snyk-upgrade-99a31e70c822894595544b0c9813b11a public:svelteui public:disenchant public:snyk-fix-61f76ef0bce2a276df9552c2fc80a14f public:snyk-upgrade-2baa7938ba318464c3bad5c3fd5cbda5 public:snyk-upgrade-7af62a575eab20e8fb4e9565bf1adcfc public:snyk-upgrade-b32f281b31a81483f126b8fb6c54a957 public:snyk-upgrade-bd61b1103442d3faecc9fc726c34e9a4 public:snyk-upgrade-5b2d6584430c061b5187611a5d7fb64a public:snyk-fix-0e9fbbcb374fcb8d5e8cb027d8414fb7 public:snyk-upgrade-5242daf91e706d1ae500c9ea6545636e public:snyk-upgrade-70434a6822fbad83424135dfad9df909 public:snyk-upgrade-8fa01ccad0352125f1866d41e57d35e1 public:snyk-upgrade-39eb97b4655e70091f718a88fccd57c1 public:snyk-upgrade-2b99d54d19e207887e36c3071b796925 public:forprivacy public:snyk-upgrade-dad9088d0a333677e0157e3deef7fc38 public:snyk-upgrade-b9a76d74b3e5723759af5b3319aabb80 public:snyk-upgrade-7da6459d8ff5d293d80de9d80061d912 public:snyk-upgrade-6a3786fc4cbe394542e9d1b6058c46e7 public:snyk-upgrade-8fc9e62771dc41b0f23dc0efefb5c949
public:0.25.43-patched-9 public:0.25.43-patched-8 public:0.25.43-patched-7 public:0.25.43-patched-6 public:0.25.43-patched-5 public:0.25.43-patched-4 public:0.25.43-patched-3 public:0.25.43-patched-2 public:0.25.43-patched-1 public:0.25.43 public:0.25.42 public:0.25.41-patched-3 public:0.25.41-patched-2 public:0.25.41-patched-1 public:0.25.41 public:0.25.40 public:0.25.39 public:0.25.38 public:0.25.37 public:0.25.36 public:0.25.35 public:0.25.34 public:0.25.33 public:0.25.32 public:0.25.31 public:0.25.30 public:0.25.30-dev public:0.25.29 public:0.25.28 public:0.25.27 public:0.25.26 public:0.25.25 public:0.25.24 public:0.25.24.beta3 public:0.25.24.beta2 public:0.25.24.beta1 public:0.25.23 public:0.25.23.beta1 public:0.25.22 public:0.25.21 public:0.25.21.beta2 public:0.25.21.beta1 public:0.25.20 public:0.25.19 public:0.25.18 public:0.25.17 public:0.25.16 public:0.25.15 public:0.25.14 public:0.25.13 public:0.25.12 public:0.25.11 public:0.25.10 public:0.25.9 public:0.25.8 public:0.25.7 public:0.25.6 public:0.25.5 public:0.25.4 public:0.25.3 public:0.25.2 public:0.25.1 public:0.25.0 public:0.25.0-beta1 public:0.24.31 public:0.24.30 public:0.24.29 public:0.24.28 public:0.24.27 public:0.24.26 public:0.24.25 public:0.24.24 public:0.24.23 public:0.24.22 public:0.24.21 public:0.24.20 public:0.24.19 public:0.24.18 public:0.24.17 public:0.24.16 public:0.24.15 public:0.24.14 public:0.24.13 public:0.24.12 public:0.24.11 public:0.24.10 public:0.24.9 public:0.24.8 public:0.24.7 public:0.24.6 public:0.24.5 public:0.24.4 public:0.24.3 public:0.24.2 public:0.24.1 public:0.24.0 public:0.24.0.dev-rc8 public:0.24.0.dev-rc7 public:0.24.0.dev-rc6 public:0.24.0.dev-rc5 public:0.24.0.dev-rc4 public:0.24.0.dev-rc3 public:0.24.0.dev-rc2 public:0.23.23 public:0.23.22 public:0.23.21 public:0.23.20 public:0.23.19 public:0.23.18 public:0.23.17 public:0.23.16 public:0.23.15 public:0.23.14 public:0.23.13 public:0.23.12 public:0.23.11 public:0.23.10 public:0.23.9 public:0.23.8 public:0.23.7 public:0.23.6 public:0.23.5 public:0.23.4 public:0.23.3 public:0.23.2 public:0.23.1 public:0.23.0 public:0.22.19 public:0.22.18 public:0.22.17 public:0.22.16 public:0.22.15 public:0.22.14 public:0.22.13 public:0.22.12 public:0.22.11 public:0.22.10 public:0.22.9 public:0.22.8 public:0.22.7 public:0.22.6 public:0.22.5 public:0.22.4 public:0.22.3 public:0.22.2 public:0.22.1 public:0.22.0 public:0.21.5 public:0.21.4 public:0.21.3 public:0.21.2 public:0.21.1 public:0.21.0 public:0.20.7 public:0.20.6 public:0.20.5 public:0.20.4 public:0.20.3 public:0.20.2 public:0.20.1 public:0.20.0 public:0.19.23 public:0.19.22 public:0.19.21 public:0.19.20 public:0.19.19 public:0.19.18 public:0.19.17 public:0.19.16 public:0.19.15 public:0.19.14 public:0.19.13 public:0.19.12 public:0.19.11 public:0.19.10 public:0.19.9 public:0.19.8 public:0.19.7 public:0.19.6 public:0.19.5 public:0.19.4 public:0.19.3 public:0.19.2 public:0.19.1 public:0.19.0 public:0.18.6 public:0.18.5 public:0.18.4 public:0.18.3 public:0.18.2 public:0.18.1 public:0.18.0 public:0.17.34 public:0.17.33 public:0.17.32 public:0.17.31 public:0.17.30 public:0.17.29 public:0.17.28 public:0.17.27 public:0.17.26 public:0.17.25 public:0.17.24 public:0.17.23 public:0.17.22 public:0.17.21 public:0.17.20 public:0.17.19 public:0.17.18 public:0.17.17 public:0.17.16 public:0.17.15 public:0.17.14 public:0.17.13 public:0.17.12 public:0.17.11 public:0.17.10 public:0.17.9 public:0.17.8 public:0.17.7 public:0.17.6 public:0.17.5 public:0.17.4 public:0.17.3 public:0.17.2 public:0.17.1 public:0.17.0 public:0.16.8 public:0.16.7 public:0.16.6 public:0.16.5 public:0.16.4 public:0.16.3 public:0.16.2 public:0.16.1 public:0.16.0 public:0.15.10 public:0.15.9 public:0.15.8 public:0.15.7 public:0.15.6 public:0.15.5 public:0.15.4 public:0.15.3 public:0.15.2 public:0.15.1 public:0.15.0 public:0.14.8 public:0.14.7 public:0.14.6 public:0.14.5 public:0.14.4 public:0.14.3 public:0.14.2 public:0.14.1 public:0.14.0 public:0.13.4 public:0.13.3 public:0.13.2 public:0.13.1 public:0.13.0 public:0.12.5 public:0.12.4 public:0.12.3 public:0.12.2 public:0.12.1 public:0.12.0 public:0.11.10 public:0.11.9 public:0.11.8 public:0.11.7 public:0.11.6 public:0.11.5 public:0.11.4 public:0.11.3 public:0.11.2 public:0.11.1 public:0.11.0 public:0.10.1 public:0.10.0 public:0.9.0 public:0.8.8 public:0.8.7 public:0.8.6 public:0.8.5 public:0.8.3 public:0.8.2 public:0.8.1 public:0.8.0 public:0.7.2 public:0.7.1 public:0.7.0 public:0.6.1 public:0.6.0 public:0.5.0 public:0.4.1 public:0.4.0 public:0.3.1 public:0.3.0 public:0.2.1 public:0.2.0 public:0.1.26 public:0.1.25 public:0.1.24 public:0.1.23 public:0.1.22 public:0.1.21 public:0.1.20 public:0.1.19 public:0.1.18 public:0.1.17 public:0.1.16 public:0.1.15 public:0.1.14 public:0.1.13 public:0.1.12 public:0.1.11 public:0.1.10 public:0.1.9 public:0.1.8 public:0.1.7 public:0.1.6 public:0.1.5 public:0.1.4 public:0.1.3 public:0.1.2 public:0.1.1 public:0.1.0 public:0.0.9 public:0.0.8 public:0.0.7 public:0.0.6

151 Commits
0.20.2 ... snyk-fix-0

Author SHA1 Message Date
snyk-bot
d0548a280a fix: package.json & package-lock.json to reduce vulnerabilities 
The following vulnerabilities are fixed with an upgrade:
- https://snyk.io/vuln/SNYK-JS-FASTXMLPARSER-7573289
 2024-07-31 01:17:04 +00:00
vorotamoroz
86d5582f37 bump 2024-07-31 02:14:11 +01:00
vorotamoroz
697ee1855b Fixed: 
- Customisation Sync now checks the difference while storing or applying the configuration.
- Time difference in the dialogue has been fixed.
 2024-07-31 02:13:25 +01:00
vorotamoroz
b8edc85528 bump 2024-07-25 13:37:34 +01:00
vorotamoroz
e2740cbefe New feature: 
- Per-file-saved customization sync has been shipped.
- Customisation sync has got beta3.
Improved:
- Start-up speed has been improved.
Fixed:
- On the customisation sync dialogue, buttons are kept within the screen.
- No more unnecessary entries on `data.json` for customisation sync.
- Selections are no longer lost while updating customisation items.
Tidied on source codes:
- Many typos have been fixed.
- Some unnecessary type casting removed.
 2024-07-25 13:36:26 +01:00
vorotamoroz
a96e4e4472 bump 2024-07-12 10:13:04 +01:00
vorotamoroz
dd26bbfe64 Improved: 
- Overall performance has been improved by using PouchDB 9.0.0.
- Configuration mismatch detection is refined. We can resolve mismatches more smoothly and naturally.
Fixed:
- Customisation Sync will be disabled when a corrupted configuration is detected.
New feature:
- We can get a notification about the storage usage of the remote database.
 2024-07-12 10:11:16 +01:00
vorotamoroz
6b9bd473cf bump 2024-07-10 05:24:26 +01:00
vorotamoroz
4be4fa6cc7 Maintenance: 
- Library refining (Phase 1 - step 2). There are no significant changes on the user side.
 2024-07-10 05:23:34 +01:00
vorotamoroz
a9745e850e Improved: 
- The passphrase of the Setup URI is now automatically generated. (#426)
 2024-07-01 11:05:33 +01:00
vorotamoroz
7b9515a47e bump 2024-07-01 06:18:52 +01:00
vorotamoroz
220dce51f2 Dependency Update 2024-07-01 06:16:04 +01:00
vorotamoroz
a23fc866c0 Tidied: 
- Thinning of this repository through the creation of a library of universal functions
 2024-07-01 06:12:23 +01:00
vorotamoroz
5c86966d89 Bump 2024-06-14 12:36:18 +01:00
vorotamoroz
29ed4d2b95 Fixed: 
- No longer batch-saving ignores editor inputs.
- The file-watching and serialisation processes have been changed to the one which is similar to previous implementations.
- We can configure the settings (Especially about text-boxes) even if we have configured the device name.
Improved:
- We can configure the delay of batch-saving.
  - Default: 5 seconds, the same as the previous hard-coded value. (Note: also, the previous behaviour was not correct).
- Also, we can configure the limit of delaying batch-saving.
- The performance of showing status indicators has been improved.
 2024-06-14 12:35:56 +01:00
vorotamoroz
16c6c52128 bump 2024-06-04 11:35:36 +01:00
vorotamoroz
8b94a0b72e Fixed: 
- No longer files have been trimmed even delimiters have been continuous.
- Fixed the toggle title to `Do not split chunks in the background` from `Do not split chunks in the foreground`.
- Non-configured item mismatches are no longer detected.
 2024-06-04 11:34:42 +01:00
vorotamoroz
c5ac76d916 bump 2024-05-30 10:53:48 +01:00
vorotamoroz
b67a6db8a1 Improved: 
- Now notes will be split into chunks in the background thread to improve smoothness.
  - Default enabled, to disable, toggle `Do not split chunks in the foreground` on `Hatch` -> `Compatibility`.
  - If you want to process very small notes in the foreground, please enable `Process small files in the foreground` on `Hatch` -> `Compatibility`.
- We can use a `splitting-limit-capped chunk splitter`; which performs more simple and make less amount of chunks.
  - Default disabled, to enable, toggle `Use splitting-limit-capped chunk splitter` on `Sync settings` -> `Performance tweaks`
Tidied
  - Some files have been separated into multiple files to make them more explicit in what they are responsible for.
 2024-05-30 10:52:20 +01:00
vorotamoroz
d4202161e8 bump 2024-05-28 12:27:30 +01:00
vorotamoroz
2a2b39009c Fixed: 
- Now we *surely* can set the device name and enable customised synchronisation.
- Unnecessary dialogue update processes have been eliminated.
- Customisation sync no longer stores half-collected files.
- No longer hangs up when removing or renaming files with the `Sync on Save` toggle enabled.
Improved:
- Customisation sync now performs data deserialization more smoothly.
- New translations have been merged.
 2024-05-28 12:26:23 +01:00
vorotamoroz
bf3a6e7570 Add the documentation and new a build option (buildDev). 2024-05-28 08:56:26 +01:00
vorotamoroz
069b8513d1 bump 2024-05-27 12:21:08 +01:00
vorotamoroz
128b1843df Fixed: No longer configurations have been locked in the minimal setup. 2024-05-27 12:20:18 +01:00
vorotamoroz
fd722b1fe5 bump 2024-05-27 12:05:41 +01:00
vorotamoroz
0bf087dba0 Fixed: 
- No longer unexpected parallel replication is performed.
- Now we can set the device name and enable customised synchronisation again.
 2024-05-27 12:04:19 +01:00
vorotamoroz
3a4b59b998 Update troubleshooting.md 2024-05-27 12:12:37 +09:00
vorotamoroz
8fc9d51c45 Add Note. 2024-05-27 04:11:44 +01:00
vorotamoroz
35feb5bf93 bump 2024-05-22 14:05:15 +01:00
vorotamoroz
b3a85c5462 New feature: 
- Now we are ready for i18n.
- The setting dialogue has been refined. Very controllable, clearly displayed disabled items, and ready to i18n.
Fixed:
- Many memory leaks have been rescued.
- Chunk caches now work well.
- Many trivial but potential bugs are fixed.
- No longer error messages will be shown on retrieving checkpoint or server information.
- Now we can check and correct tweak mismatch during the setup
Improved:
- Customisation synchronisation has got more smoother.
Tidied
- Practically unused functions have been removed or are being prepared for removal.
- Many of the type-errors and lint errors have been corrected.
- Unused files have been removed.
Note:
- From this version, some test files have been included. However, they are not enabled and released in the release build.
 2024-05-22 14:04:22 +01:00
vorotamoroz
7b0ac22c3b Create terms.md 2024-05-13 14:04:02 +09:00
vorotamoroz
dca8e4b2a4 bump 2024-05-10 11:38:03 +01:00
vorotamoroz
89de2dcc37 Fixed: 
- No longer missing tasks which have queued as the same key (e.g., for the same operation to the same file).
- Some trivial issues have been fixed.
New feature:
- Reloading Obsidian can be scheduled until that file and database operations are stable.
 2024-05-10 11:33:59 +01:00
vorotamoroz
172b08dbb3 bump 2024-05-08 23:57:19 +09:00
vorotamoroz
d518a3fc1b Fixed: 
- Now the remote chunks could be decrypted even if we are using `Incubate chunks in Document`. (The note of 0.23.6 has been fixed).
- Chunk retrieving with `Incubate chunks in document` got more efficiently.
- No longer task processor misses the completed tasks.
- Replication is no longer started automatically during changes in window visibility (e.g., task switching on the desktop) when off-focused.
 2024-05-08 23:56:29 +09:00
vorotamoroz
c6ed867498 bump 2024-05-07 12:59:55 +01:00
vorotamoroz
4f4923e977 New feature: 
- Now we can check configuration mismatching between clients before synchronisation.
- Now we can perform remote database compaction from the `Maintenance` pane.
Fixed:
- We can detect the bucket could not be reachable.
 2024-05-07 12:55:48 +01:00
vorotamoroz
a5ebf29b3d Merge pull request #417 from MichaelBrunn3r/translation 
fix: Grammar issues in settings page
 2024-05-07 20:26:59 +09:00
vorotamoroz
ee465184c8 bump 2024-05-05 23:46:16 +09:00
vorotamoroz
d7d4f1e6f2 New feature: 
- We can now use `Incubate Chunks in Document` to reduce non-well-formed chunks.
Fixed:
- No longer experimental configuration is shown on the Minimal Setup.
 2024-05-05 23:45:45 +09:00
Michael Brunner
cbf5023593 fix: Grammar issues in settings page 2024-05-04 12:34:53 +02:00
vorotamoroz
3925052f92 Add the design document of planned improving. 2024-05-04 02:59:24 +09:00
vorotamoroz
1934418258 Add the design document of planned improving. 2024-05-04 01:57:03 +09:00
vorotamoroz
2ae018b2bd Refactor: 
- Files have been categorised for clarity. The deliverables are not affected.
 2024-05-02 04:07:36 +01:00
vorotamoroz
8474497985 bump 2024-05-01 02:24:08 +09:00
vorotamoroz
b5714cc83b Fixed: 
- No longer unwanted `\f` in journal sync.
 2024-05-01 02:22:30 +09:00
vorotamoroz
133f5a7109 bump 2024-04-30 11:49:16 +01:00
vorotamoroz
daa3feebf1 Fixed: 
- Journal Sync will not hang up during big replication, especially the initial one.
- All changes which have been replicated while rebuilding will not be postponed (Previous behaviour).
Improved:
- Now Journal Sync works efficiently in download and parse, or pack and upload.
- Less server storage and faster packing/unpacking usage by the new chunk format.
 2024-04-30 11:48:27 +01:00
vorotamoroz
7b5f7d0fbf bump 2024-04-30 01:40:01 +09:00
vorotamoroz
29532193cb - Fixed: 
- Now journal synchronisation considers untransferred each from sent and received.
  - Journal sync now handles retrying.
  - Journal synchronisation no longer considers the synchronisation of chunks as revision updates (Simply ignored).
  - Journal sync now splits the journal pack to prevent mobile device rebooting.
  - Maintenance menus which had been on the command palette are now back in the maintain pane on the setting dialogue.
- Improved:
  - Now all changes which have been replicated while rebuilding will be postponed.
 2024-04-30 01:39:09 +09:00
vorotamoroz
5b4309c09d For the future. Because of a good opportunity. 2024-04-29 02:01:27 +09:00
vorotamoroz
16ef582453 Update: wrote about the new Remote Type. 2024-04-28 23:37:26 +09:00
vorotamoroz
3e22f70c7a Update README.md 2024-04-28 17:49:50 +09:00
vorotamoroz
0a8dbe097e bump 2024-04-27 03:35:32 +09:00
vorotamoroz
2c0fcf74d0 New feature: Object storage support 2024-04-27 03:33:59 +09:00
vorotamoroz
a1ab1efd5d Update README.md 2024-04-20 21:45:21 +09:00
vorotamoroz
c8fcf2d0d5 Bump 2024-04-19 12:06:09 +01:00
vorotamoroz
c384e2f7fb Fixed: 
- No longer data corrupting due to false BASE64 detections.
 2024-04-19 12:04:14 +01:00
vorotamoroz
99c1c7dc1a bump 2024-04-18 12:37:49 +01:00
vorotamoroz
84adec4b1a New feature: Automatic data compression to reduce amount of traffic and the usage of remote database. 2024-04-18 12:30:29 +01:00
vorotamoroz
f0b202bd91 bump 2024-04-12 01:32:03 +09:00
vorotamoroz
d54b7e2d93 - Fixed: 
- Error handling on booting now works fine.
  - Replication is now started automatically in LiveSync mode.
  - Batch database update is now disabled in LiveSync mode.
  - No longer automatically reconnection while off-focused.
  - Status saves are thinned out.
  - Now Self-hosted LiveSync waits for all files between the local database and storage to be surely checked.
- Improved:
  - The job scheduler is now more robust and stable.
  - The status indicator no longer flickers and keeps zero for a while.
  - No longer meaningless frequent updates of status indicators.
  - Now we can configure regular expression filters in handy UI. Thank you so much, @eth-p!
  - `Fetch` or `Rebuild everything` is now more safely performed.
- Minor things
  - Some utility function has been added.
  - Customisation sync now less wrong messages.
  - Digging the weeds for eradication of type errors.
 2024-04-12 01:30:35 +09:00
vorotamoroz
6952ef37f5 Update quick_setup.md 2024-04-09 13:10:31 +09:00
vorotamoroz
9630bcbae8 bump 2024-03-22 10:50:03 +01:00
vorotamoroz
c3f925ab9a Merge branch 'main' of https://github.com/vrtmrz/obsidian-livesync 2024-03-22 10:48:25 +01:00
vorotamoroz
034dc0538f - Fixed: 
- Fixed the issue that binary files were sometimes corrupted.
  - Fixed customisation sync data could be corrupted.
- Improved:
  - Now the remote database costs lower memory.
    - This release requires a brief wait on the first synchronisation, to track the latest changeset again.
  - Description added for the `Device name`.
- Refactored:
  - Many type-errors have been resolved.
  - Obsolete file has been deleted.
 2024-03-22 10:48:16 +01:00
vorotamoroz
b6136df836 Update quick_setup.md 2024-03-22 14:27:34 +09:00
vorotamoroz
24aacdc2a1 bump 2024-03-22 04:07:17 +01:00
vorotamoroz
f91109b1ad - Improved: 
- Faster start-up by removing too many logs which indicates normality
  - By streamlined scanning of customised synchronisation extra phases have been deleted.
 2024-03-22 04:07:07 +01:00
vorotamoroz
e76e7ae8ea bump 2024-03-19 17:59:38 +01:00
vorotamoroz
f7fbe85d65 - New feature: 
- We can disable the status bar in the setting dialogue.
- Improved:
  - Now some files are handled as correct data type.
  - Customisation sync now uses the digest of each file for better performance.
  - The status in the Editor now works performant.
- Refactored:
  - Common functions have been ready and the codebase has been organised.
  - Stricter type checking following TypeScript updates.
  - Remove old iOS workaround for simplicity and performance.
 2024-03-19 17:58:55 +01:00
vorotamoroz
0313443b29 Merge pull request #389 from Seeker0472/fix-command 
Fixed docker-compose command in docs
 2024-03-19 14:06:23 +09:00
seeker0472
755c30f468 fix docker-compose command 2024-03-17 14:30:35 +08:00
vorotamoroz
b00b0cc5e5 bump 2024-03-15 10:37:15 +01:00
vorotamoroz
d7985a6b41 Improved: 
- Now using HTTP for the remote database URI warns of an error (on mobile) or notice (on desktop).
 2024-03-15 10:36:00 +01:00
vorotamoroz
486e816902 Update dependencies 2024-03-15 10:35:41 +01:00
vorotamoroz
ef9b19c24b bump 2024-03-04 04:07:51 +00:00
vorotamoroz
4ed9494176 Changed: 
- The default settings has been changed.
Improved:
- Default and preferred settings are applied on completion of the wizard.
Fixed:
- Now Initialisation `Fetch` will be performed smoothly and there will be fewer conflicts.
- No longer stuck while Handling transferred or initialised documents.
 2024-03-04 04:07:11 +00:00
vorotamoroz
fcd56d59d5 bump 2024-03-01 08:33:37 +00:00
vorotamoroz
1cabfcfd19 Fixed: 
- `Verify and repair all files` is no longer broken.
New feature::
- Now `Verify and repair all files` can restore or show history
Improved:
- Performance improved
 2024-03-01 08:32:48 +00:00
vorotamoroz
37a18dbfef bump 2024-03-01 03:28:46 +00:00
vorotamoroz
e7edf88713 Fixed 
- No longer unchanged hidden files and customisations are saved and transferred now.
- File integrity of vault history indicates the integrity correctly.
Improved
- In the report, the schema of the remote database URI is now printed.
 2024-03-01 03:28:06 +00:00
vorotamoroz
90ff75ab35 add notes. 2024-02-29 00:30:07 +00:00
vorotamoroz
bff1d661f5 Update troubleshooting.md 
Fix grammar
 2024-02-29 00:42:28 +09:00
vorotamoroz
6b59c14774 Update doc 2024-02-28 08:29:06 +00:00
vorotamoroz
8249274eac bump 2024-02-28 08:28:07 +00:00
vorotamoroz
3c6dae7814 - Fixed: 
- Fixed a bug on `fetch chunks on demand` that could not fetch the chunks on demand.
- Improved:
  - `fetch chunks on demand` works more smoothly.
  - Initialisation `Fetch` is now more efficient.
- Tidied:
  - Removed some meaningless codes.
 2024-02-28 08:27:17 +00:00
vorotamoroz
60cf8fe640 bump 2024-02-27 08:36:37 +00:00
vorotamoroz
3d89b3863f Fixed: 
- Now fetch and unlock the locked remote database works well again.
- No longer crash on symbolic links inside hidden folders.
Improved:
- Chunks are now created more efficiently.
- Better performance in saving notes.
- Network activities are indicated as an icon.
- Less memory used for binary processing.
Tidied:
- Cleaned unused functions up.
- Sorting out the codes that have become nonsense.
Changed:
- Now no longer `fetch chunks on demand` needs `Pacing replication`
 2024-02-27 08:35:46 +00:00
vorotamoroz
ee9364310d bump 2024-02-20 09:36:43 +00:00
vorotamoroz
86b9695bc2 Fixed: 
- No longer deleted hidden files were ignored.
- The document history dialogue is now able to process the deleted revisions.
- Deletion of a hidden file is now surely performed even if the file is already conflicted.
 2024-02-20 09:32:48 +00:00
vorotamoroz
e05f8771b9 bump 2024-02-20 05:15:48 +00:00
vorotamoroz
65619c2478 Fixed: 
- Fixed a problem with synchronisation taking a long time to start in some cases.
- Now we can disable E2EE encryption.
Improved:
- `Setup Wizard` is now more clear.
- `Minimal Setup` is now more simple.
- Self-hosted LiveSync now be able to use even if there are vaults with the same name.
- Now Self-hosted LiveSync waits until set-up is complete.
- Show reload prompts when possibly recommended while settings.
New feature:
- A guidance dialogue prompting for settings will be shown after the installation.
Changed
- Some setting names has been changed
 2024-02-20 05:13:53 +00:00
vorotamoroz
1552fa9d9e Merge branch 'main' of https://github.com/vrtmrz/obsidian-livesync 2024-02-15 08:36:49 +00:00
vorotamoroz
767f12b52f Rewrite 2024-02-15 08:35:12 +00:00
vorotamoroz
4071ba120e Update troubleshooting.md 2024-02-15 13:56:15 +09:00
vorotamoroz
2c0e3ba01c update faq 2024-02-15 04:38:31 +00:00
vorotamoroz
90adf06830 Merge branch 'main' of https://github.com/vrtmrz/obsidian-livesync 2024-02-15 04:04:06 +00:00
vorotamoroz
cf8e7ff6ca bump 2024-02-15 04:04:01 +00:00
vorotamoroz
95c3ff5043 Update common lib. 2024-02-15 04:02:28 +00:00
vorotamoroz
7ea3515801 Fixed: 
- Some description of settings have been refined
New feature:
- TroubleShooting is now shown in the setting dialogue.
 2024-02-15 04:02:20 +00:00
vorotamoroz
f866981a8a Update troubleshooting.md 2024-02-14 17:38:52 +09:00
vorotamoroz
8f36d6f893 fix minor layout 2024-02-09 03:20:31 +00:00
vorotamoroz
6dd86e9392 modify minor layouts. 2024-02-09 03:16:33 +00:00
vorotamoroz
d22716bef0 make more clear documents. 2024-02-09 03:13:03 +00:00
vorotamoroz
5d9baec5e4 update colab note 2024-02-07 10:17:45 +00:00
vorotamoroz
27d71ca2fb New utilities. 2024-02-06 11:03:51 +00:00
vorotamoroz
c024ed13d3 Refining the content. 2024-02-06 11:02:48 +00:00
vorotamoroz
b9527ccab0 bump 2024-01-30 17:31:52 +00:00
vorotamoroz
fa3aa2702c Fixed: 
- Now the result of conflict resolution could be surely written into the storage.
- Deleted files can be handled correctly again in the history dialogue and conflict dialogue.
- Some wrong log messages were fixed.
- Change handling now has become more stable.
- Some event handling became to be safer.
Improved:
- Dumping document information shows conflicts and revisions.
- The timestamp-only differences can be surely cached.
- Timestamp difference detection can be rounded by two seconds.
Refactored:
- A bit of organisation to write the test.
 2024-01-30 17:31:02 +00:00
vorotamoroz
93e7cbb133 bump. 2024-01-29 08:41:03 +00:00
vorotamoroz
716ae32e02 Fixed: 
- Deletion of files is now reliably synchronised.
 2024-01-29 08:40:41 +00:00
vorotamoroz
d6d8cbcf5a bump 2024-01-29 07:57:02 +00:00
vorotamoroz
efd348b266 Fixed: 
- No longer detects storage changes which have been caused by Self-hosted LiveSync itself.
- Setting sync file will be detected only if it has been configured now.
  - And its log will be shown only while the verbose log is enabled.
- Customisation file enumeration has got less blingy.
Fixed and improved:
- In-editor-status is now shown in the following areas:
  - Note editing pane (Source mode and live-preview mode).
  - New tab pane.
  - Canvas pane.
 2024-01-29 07:56:02 +00:00
vorotamoroz
8969b1800a bump 2024-01-24 08:53:00 +00:00
vorotamoroz
2c8e026e29 Fixed: 
- Now the results of resolving conflicts are surely synchronised.
Modified:
- Some setting items got new clear names.
New feature:
- We can limit the synchronising files by their size.
- Now the settings could be stored in a specific markdown file to synchronise or switch it
- Customisation of the obsoleted device is now able to be deleted at once.
 2024-01-24 08:52:47 +00:00
vorotamoroz
a6c27eab3d Merge pull request #367 from calvinbui/patch-1 
Skip workspace-mobile.json for cross-platform sync
 2024-01-24 15:55:50 +09:00
vorotamoroz
9b5c57d540 Merge pull request #336 from toon159/patch-1 
Lower payload size limit and batch limit from 10 to 2
 2024-01-22 13:03:29 +09:00
Calvin Bui
c251c596e8 Skip workspace-mobile.json for cross-platform sync 2024-01-18 13:23:48 +11:00
vorotamoroz
61188cfaef bump 2024-01-16 08:36:37 +00:00
vorotamoroz
97d944fd75 New feature: 
- We can perform automatic conflict resolution for inactive files, and postpone only manual ones by `Postpone manual resolution of inactive files`.
- Now we can see the image in the document history dialogue.
  - We can see the difference of the image, in the document history dialogue.
	- And also we can highlight differences.

Improved:
- Hidden file sync has been stabilised.
- Now automatically reloads the conflict-resolution dialogue when new conflicted revisions have arrived.

Fixed:
- No longer periodic process runs after unloading the plug-in.
- Now the modification of binary files is surely stored in the storage.
 2024-01-16 08:32:43 +00:00
vorotamoroz
d3dc1e7328 Minor fix and refine the readme 2024-01-12 10:29:18 +00:00
vorotamoroz
45304af369 bump 2024-01-12 09:38:57 +00:00
vorotamoroz
7f422d58f2 - Refined: 
- Task scheduling logics has been rewritten.
  - Possibly many bugs and fragile behaviour has been fixed
- Fixed:
  - Remote-chunk-fetching now works with keeping request intervals
- New feature:
  - We can show only the icons in the editor.
 2024-01-12 09:36:49 +00:00
vorotamoroz
c2491fdfad bump 2023-12-11 12:55:01 +09:00
vorotamoroz
06a6e391e8 Fixed for change detection bug. 2023-12-11 12:53:50 +09:00
vorotamoroz
f99475f6b7 bump 2023-12-11 12:46:23 +09:00
vorotamoroz
109fc00b9d Fixed 
- Now ID of the documents is shown in the log with the first 8 letters.
 2023-12-11 12:45:40 +09:00
vorotamoroz
c071d822e1 - Improved: 
- Now all revisions will be shown only its first a few letters.
- Fixed:
  - Check before modifying files has been implemented.
  - Content change detection has been improved.
 2023-12-11 12:22:17 +09:00
vorotamoroz
d2de5b4710 bump 2023-12-04 19:39:47 +09:00
vorotamoroz
cf5ecd8922 Implemented: 
- Now we can use SHA1 for hash function as fallback.
 2023-12-04 19:39:04 +09:00
vorotamoroz
b337a05b5a bump 2023-11-27 07:13:15 +00:00
vorotamoroz
9ea6bee9d1 - Fixed: 
- No longer files are broken while rebuilding.
    - Now, Large binary files can be written correctly on a mobile platform.
    - Any decoding errors now make zero-byte files.
  - Modified:
    - All files are processed sequentially for each.
 2023-11-27 06:55:55 +00:00
vorotamoroz
9747c26d50 bump 2023-11-25 02:22:26 +09:00
vorotamoroz
bb4b764586 - Fixed: 
- No more infinity loops on larger files.
    - Show message on decode error.
  - Refactored:
    - Fixed to avoid obsolete global variables.
 2023-11-25 02:21:44 +09:00
vorotamoroz
279b4b41e5 bump 2023-11-24 10:32:46 +00:00
vorotamoroz
b644fb791d - Changes and performance improvements: 
- Now the saving files are processed by Blob.
    - The V2-Format has been reverted.
    - New encoding format has been enabled in default.
    - WARNING: Since this version, the compatibilities with older Filesystem LiveSync have been lost.
 2023-11-24 10:31:58 +00:00
Vichaya Raksakunpanich
5802ed31be Update ObsidianLiveSyncSettingTab.ts 
Lower payload size limit and batch limit to 2 due to IBM Cloudant read/write limitation. Hope it will fix the "Replication error".
 2023-11-24 16:25:16 +07:00
vorotamoroz
ac9428e96b Fixed 
- To better replication, path obfuscation is now deterministic even if with E2EE.
 2023-11-15 08:44:03 +00:00
vorotamoroz
280d9e1dd9 Fixed: Fixed the issue of TOML editing. 2023-11-07 01:07:58 +00:00
vorotamoroz
f7209e566c bump 2023-10-24 10:07:29 +01:00
vorotamoroz
4a9ab2d1de Fixed: 
- No longer enumerating file names is broken.
 2023-10-24 10:07:17 +01:00
vorotamoroz
cb74b5ee93 - Fixed 
- Now empty file could be decoded.
    - Local files are no longer pre-saved before fetching from a remote database.
    - No longer deadlock while applying customisation sync.
    - Configuration with multiple files is now able to be applied correctly.
    - Deleting folder propagation now works without enabling the use of a trash bin.
 2023-10-24 09:54:56 +01:00
vorotamoroz
60eecd7001 bump 2023-10-17 12:00:59 +09:00
vorotamoroz
4bd7b54bcd Fixed: 
- Now the files which having digit or character prefixes in the path will not be ignored.
 2023-10-17 12:00:19 +09:00
vorotamoroz
8923c73d1b bump 2023-10-14 23:08:34 +09:00
vorotamoroz
11e64b13e2 The text-input-dialogue is no longer broken. 2023-10-14 23:07:51 +09:00
vorotamoroz
983d9248ed bump 2023-10-13 04:23:59 +01:00
vorotamoroz
7240e84328 - New feature: 
- We can launch Customization sync from the Ribbon if we enable it.
- Fixed:
  - Setup URI is now back to the previous spec; be encrypted by V1.
  - The Settings dialogue is now registered at the beginning of the start-up process.
- Improved:
  - Enumerating documents has been faster.
 2023-10-13 04:22:24 +01:00
vorotamoroz
0d55ae2532 Merge pull request #298 from LiamSwayne/patch-1 
Grammar fix
 2023-10-04 13:07:42 +09:00
Liam Swayne
dbd284f5dd grammar fix 2023-10-03 22:26:30 -04:00
79 changed files with 15860 additions and 8651 deletions
Expand all files Collapse all files

116
README.md
View File

@@ -1,95 +1,85 @@
<!-- For translation: 20240227r0 -->
# Self-hosted LiveSync
[Japanese docs](./README_ja.md) - [Chinese docs](./README_cn.md).
Self-hosted LiveSync is a community-implemented synchronization plugin.
A self-hosted or purchased CouchDB acts as the intermediate server. Available on every obsidian-compatible platform.
Note: It has no compatibility with the official "Obsidian Sync".
Self-hosted LiveSync is a community-implemented synchronization plugin, available on every obsidian-compatible platform and using CouchDB or Object Storage (e.g., MinIO, S3, R2, etc.) as the server.
![obsidian_live_sync_demo](https://user-images.githubusercontent.com/45774780/137355323-f57a8b09-abf2-4501-836c-8cb7d2ff24a3.gif)
Before installing or upgrading LiveSync, please back your vault up.
Note: This plugin cannot synchronise with the official "Obsidian Sync".
## Features
- Visual conflict resolver included.
- Bidirectional synchronization between devices nearly in real-time
- You can use CouchDB or its compatibles like IBM Cloudant.
- End-to-End encryption is supported.
- Plugin synchronization(Beta)
- Receive WebClip from [obsidian-livesync-webclip](https://chrome.google.com/webstore/detail/obsidian-livesync-webclip/jfpaflmpckblieefkegjncjoceapakdf) (End-to-End encryption will not be applicable.)
- Synchronize vaults very efficiently with less traffic.
- Good at conflicted modification.
- Automatic merging for simple conflicts.
- Using OSS solution for the server.
- Compatible solutions can be used.
- Supporting End-to-end encryption.
- Synchronisation of settings, snippets, themes, and plug-ins, via [Customization sync(Beta)](#customization-sync) or [Hidden File Sync](#hiddenfilesync)
- WebClip from [obsidian-livesync-webclip](https://chrome.google.com/webstore/detail/obsidian-livesync-webclip/jfpaflmpckblieefkegjncjoceapakdf)
Useful for researchers, engineers and developers with a need to keep their notes fully self-hosted for security reasons. Or just anyone who would like the peace of mind of knowing that their notes are fully private.
This plug-in might be useful for researchers, engineers, and developers with a need to keep their notes fully self-hosted for security reasons. Or just anyone who would like the peace of mind of knowing that their notes are fully private.
## IMPORTANT NOTICE
- Do not enable this plugin with another synchronization solution at the same time (including iCloud and Obsidian Sync). Before enabling this plugin, make sure to disable all the other synchronization methods to avoid content corruption or duplication. If you want to synchronize to two or more services, do them one by one and never enable two synchronization methods at the same time.
This includes not putting your vault inside a cloud-synchronized folder (eg. an iCloud folder or Dropbox folder)
- This is a synchronization plugin. Not a backup solution. Do not rely on this for backup.
- If the device's storage runs out, database corruption may happen.
- Hidden files or any other invisible files wouldn't be kept in the database, and thus won't be synchronized. (**and may also get deleted**)
>[!IMPORTANT]
> - Before installing or upgrading this plug-in, please back your vault up.
> - Do not enable this plugin with another synchronization solution at the same time (including iCloud and Obsidian Sync).
> - This is a synchronization plugin. Not a backup solution. Do not rely on this for backup.
## How to use
### Get your database ready.
### 3-minute setup - CouchDB on fly.io
First, get your database ready. fly.io is preferred for testing. Or you can use your own server with CouchDB. For more information, refer below:
1. [Setup fly.io](docs/setup_flyio.md)
2. [Setup IBM Cloudant](docs/setup_cloudant.md)
3. [Setup your CouchDB](docs/setup_own_server.md)
**Recommended for beginners**
### Configure the plugin
[![LiveSync Setup onto Fly.io SpeedRun 2024 using Google Colab](https://img.youtube.com/vi/7sa_I1832Xc/0.jpg)](https://www.youtube.com/watch?v=7sa_I1832Xc)
See [Quick setup guide](doccs/../docs/quick_setup.md)
1. [Setup CouchDB on fly.io](docs/setup_flyio.md)
2. Configure plug-in in [Quick Setup](docs/quick_setup.md)
## Something looks corrupted...
### Manually Setup
Please open the configuration link again and Answer below:
- If your local database looks corrupted (in other words, when your Obsidian getting weird even standalone.)
- Answer `No` to `Keep local DB?`
- If your remote database looks corrupted (in other words, when something happens while replicating)
- Answer `No` to `Keep remote DB?`
1. Setup the server
1. [Setup CouchDB on fly.io](docs/setup_flyio.md)
2. [Setup your CouchDB](docs/setup_own_server.md)
2. Configure plug-in in [Quick Setup](docs/quick_setup.md)
> [!TIP]
> Now, fly.io has become not free. Fortunately, even though there are some issues, we are still able to use IBM Cloudant. Here is [Setup IBM Cloudant](docs/setup_cloudant.md). It will be updated soon!
If you answered `No` to both, your databases will be rebuilt by the content on your device. And the remote database will lock out other devices. You have to synchronize all your devices again. (When this time, almost all your files should be synchronized with a timestamp. So you can use an existing vault).
## Information in StatusBar
Synchronization status is shown in statusbar.
Synchronization status is shown in the status bar with the following icons.
- Activity Indicator
- 📲 Network request
- Status
- ⏹️ Stopped
- 💤 LiveSync enabled. Waiting for changes.
- ⚡️ Synchronization in progress.
- ⚠ An error occurred.
- ↑ Uploaded chunks and metadata
- ↓ Downloaded chunks and metadata
- ⏳ Number of pending processes
- 🧩 Number of files waiting for their chunks.
If you have deleted or renamed files, please wait until ⏳ icon disappeared.
- 💤 LiveSync enabled. Waiting for changes
- ⚡️ Synchronization in progress
- ⚠ An error occurred
- Statistical indicator
- ↑ Uploaded chunks and metadata
- ↓ Downloaded chunks and metadata
- Progress indicator
- 📥 Unprocessed transferred items
- 📄 Working database operation
- 💾 Working write storage processes
- ⏳ Working read storage processes
- 🛫 Pending read storage processes
- 📬 Batched read storage processes
- ⚙️ Working or pending storage processes of hidden files
- 🧩 Waiting chunks
- 🔌 Working Customisation items (Configuration, snippets, and plug-ins)
To prevent file and database corruption, please wait to stop Obsidian until all progress indicators have disappeared as possible (The plugin will also try to resume, though). Especially in case of if you have deleted or renamed files.
## Hints
- If a folder becomes empty after a replication, it will be deleted by default. But you can toggle this behaviour. Check the [Settings](docs/settings.md).
- LiveSync mode drains more batteries in mobile devices. Periodic sync with some automatic sync is recommended.
- Mobile Obsidian can not connect to non-secure (HTTP) or locally-signed servers, even if the root certificate is installed on the device.
- There are no 'exclude_folders' like configurations.
- While synchronizing, files are compared by their modification time and the older ones will be overwritten by the newer ones. Then plugin checks for conflicts and if a merge is needed, a dialog will open.
- Rarely, a file in the database could be corrupted. The plugin will not write to local storage when a file looks corrupted. If a local version of the file is on your device, the corruption could be fixed by editing the local file and synchronizing it. But if the file does not exist on any of your devices, then it can not be rescued. In this case, you can delete these items from the settings dialog.
- To stop the boot-up sequence (eg. for fixing problems on databases), you can put a `redflag.md` file (or directory) at the root of your vault.
Tip for iOS: a redflag directory can be created at the root of the vault using the File application.
- Also, with `redflag2.md` placed, we can automatically rebuild both the local and the remote databases during the boot-up sequence. With `redflag3.md`, we can discard only the local database and fetch from the remote again.
- Q: The database is growing, how can I shrink it down?
A: each of the docs is saved with their past 100 revisions for detecting and resolving conflicts. Picturing that one device has been offline for a while, and comes online again. The device has to compare its notes with the remotely saved ones. If there exists a historic revision in which the note used to be identical, it could be updated safely (like git fast-forward). Even if that is not in revision histories, we only have to check the differences after the revision that both devices commonly have. This is like git's conflict-resolving method. So, We have to make the database again like an enlarged git repo if you want to solve the root of the problem.
- And more technical Information is in the [Technical Information](docs/tech_info.md)
- If you want to synchronize files without obsidian, you can use [filesystem-livesync](https://github.com/vrtmrz/filesystem-livesync).
- WebClipper is also available on Chrome Web Store:[obsidian-livesync-webclip](https://chrome.google.com/webstore/detail/obsidian-livesync-webclip/jfpaflmpckblieefkegjncjoceapakdf)
Repo is here: [obsidian-livesync-webclip](https://github.com/vrtmrz/obsidian-livesync-webclip). (Docs are a work in progress.)
## Troubleshooting
If you are having problems getting the plugin working see: [Troubleshooting](docs/troubleshooting.md)
## Tips and Troubleshooting
If you are having problems getting the plugin working see: [Tips and Troubleshooting](docs/troubleshooting.md)
## License
The source code is licensed under the MIT License.
Licensed under the MIT License.

153
README_ja.md
View File

@@ -1,84 +1,85 @@
<!-- For translation: 20240227r0 -->
# Self-hosted LiveSync
[英語版ドキュメント](./README.md) - [中国語版ドキュメント](./README_cn.md).
**旧): obsidian-livesync**
Obsidianで利用可能なすべてのプラットフォームで使える、CouchDBをサーバに使用する、コミュニティ版の同期プラグイン
セルフホストしたデータベースを使って、双方向のライブシンクするObsidianのプラグイン。
**公式のSyncとは互換性はありません**
![obsidian_live_sync_demo](https://user-images.githubusercontent.com/45774780/137355323-f57a8b09-abf2-4501-836c-8cb7d2ff24a3.gif)
**インストールする前に、Vaultのバックアップを確実に取得してください**
[英語版](./README.md)
## こんなことができるプラグインです。
- Windows, Mac, iPad, iPhone, Android, Chromebookで動く
- セルフホストしたデータベースに同期して
- 複数端末で同時にその変更をほぼリアルタイムで配信し
- さらに、他の端末での変更も別の端末に配信する、双方向リアルタイムなLiveSyncを実現でき、
- 発生した変更の衝突はその場で解決できます。
- 同期先のホストにはCouchDBまたはその互換DBaaSのIBM Cloudantをサーバーに使用できます。あなたのデータは、あなたのものです。
- もちろんLiveではない同期もできます。
- 万が一のために、サーバーに送る内容を暗号化できます(betaです)。
- [Webクリッパー](https://chrome.google.com/webstore/detail/obsidian-livesync-webclip/jfpaflmpckblieefkegjncjoceapakdf) もあります(End-to-End暗号化対象外です)
NDAや類似の契約や義務、倫理を守る必要のある、研究者、設計者、開発者のような方に特にオススメです。
特にエンタープライズでは、たとえEnd to Endの暗号化が行われていても、管理下にあるサーバーにのみデータを格納することが求められる場合があります。
# 重要なお知らせ
- ❌ファイルの重複や破損を避けるため、複数の同期手段を同時に使用しないでください。
これは、Vaultをクラウド管理下のフォルダに置くことも含みます。(例えば、iCloudの管理フォルダ内に入れたり)。
- ⚠️このプラグインは、端末間でのノートの反映を目的として作成されました。バックアップ等が目的ではありません。そのため、バックアップは必ず別のソリューションで行うようにしてください。
- ストレージの空き容量が枯渇した場合、データベースが破損することがあります。
# このプラグインの使い方
1. Community Pluginsから、Self-holsted LiveSyncと検索しインストールするか、このリポジトリのReleasesから`main.js`, `manifest.json`, `style.css` をダウンロードしvaultの中の`.obsidian/plugins/obsidian-livesync`に入れて、Obsidianを再起動してください。
2. サーバーをセットアップします。IBM Cloudantがお手軽かつ堅牢で便利です。完全にセルフホストする際にはお持ちのサーバーにCouchDBをインストールする必要があります。詳しくは下記を参照してください
1. [IBM Cloudantのセットアップ](docs/setup_cloudant_ja.md)
2. [独自のCouchDBのセットアップ](docs/setup_own_server_ja.md)
備考： IBM Cloudantのアカウント登録が出来ないケースがあるようです。代替を探していて、今 [using fly.io](https://github.com/vrtmrz/obsidian-livesync/discussions/85)を検討しています。
1. [Quick setup](docs/quick_setup_ja.md)から、セットアップウィザード使ってセットアップしてください。
# テストサーバー
もし、CouchDBをインストールしたり、Cloudantのインスタンスをセットアップしたりするのに気が引ける場合、[Self-hosted LiveSyncのテストサーバー](https://olstaste.vrtmrz.net/)を作りましたので、使ってみてください。
備考: 制限事項をよく確認して使用してください。くれぐれも、本当に使用している自分のVaultを同期しないようにしてください。
# WebClipperあります
Self-hosted LiveSync用にWebClipperも作りました。Chrome Web Storeからダウンロードできます。
[obsidian-livesync-webclip](https://chrome.google.com/webstore/detail/obsidian-livesync-webclip/jfpaflmpckblieefkegjncjoceapakdf)
リポジトリはこちらです: [obsidian-livesync-webclip](https://github.com/vrtmrz/obsidian-livesync-webclip)。
相変わらずドキュメントは間に合っていません。
# ステータスバーの情報
右下のステータスバーに、同期の状態が表示されます
- 同期状態
- ⏹️ 同期は停止しています
- 💤 同期はLiveSync中で、なにか起こるのを待っています
- ⚡️ 同期中です
- ⚠ エラーが発生しています
- ↑ 送信したデータ数
- ↓ 受信したデータ数
- ⏳ 保留している処理の数です
ファイルを削除したりリネームした場合、この表示が消えるまでお待ちください。
# さらなる補足
- ファイルは同期された後、タイムスタンプを比較して新しければいったん新しい方で上書きされます。その後、衝突が発生したかによって、マージが行われます。
- まれにファイルが破損することがあります。破損したファイルに関してはディスクへの反映を試みないため、実際には使用しているデバイスには少し古いファイルが残っていることが多いです。そのファイルを再度更新してもらうと、データベースが更新されて問題なくなるケースがあります。ファイルがどの端末にも存在しない場合は、設定画面から、削除できます。
- データベースの復旧中に再起動した場合など、うまくローカルデータベースを修正できない際には、Vaultのトップに`redflag.md`というファイルを置いてください。起動時のシーケンスがスキップされます。
- データベースが大きくなってきてるんだけど、小さくできる？→各ノートは、それぞれの古い100リビジョンとともに保存されています。例えば、しばらくオフラインだったあるデバイスが、久しぶりに同期したと想定してみてください。そのとき、そのデバイスは最新とは少し異なるリビジョンを持ってるはずです。その場合でも、リモートのリビジョン履歴にリモートのものが存在した場合、安全にマージできます。もしリビジョン履歴に存在しなかった場合、確認しなければいけない差分も、対象を存在して持っている共通のリビジョン以降のみに絞れます。ちょうどGitのような方法で、衝突を解決している形になるのです。そのため、肥大化したリポジトリの解消と同様に、本質的にデータベースを小さくしたい場合は、データベースの作り直しが必要です。
- その他の技術的なお話は、[技術的な内容](docs/tech_info_ja.md)に書いてあります。
※公式のSyncと同期することはできません。
# ライセンス
## 機能
- 高効率・低トラフィックでVault同士を同期
- 競合解決がいい感じ
- 単純な競合なら自動マージします
- OSSソリューションを同期サーバに使用
- 互換ソリューションも使用可能です
- End-to-End暗号化実装済み
- 設定・スニペット・テーマ、プラグインの同期が可能
- [Webクリッパー](https://chrome.google.com/webstore/detail/obsidian-livesync-webclip/jfpaflmpckblieefkegjncjoceapakdf) もあります
The source code is licensed MIT.
NDAや類似の契約や義務、倫理を守る必要のある、研究者、設計者、開発者のような方に特にオススメです。
>[!IMPORTANT]
> - インストール・アップデート前には必ずVaultをバックアップしてください
> - 複数の同期ソリューションを同時に有効にしないでください（これはiCloudや公式のSyncも含みます）
> - このプラグインは同期プラグインです。バックアップとして使用しないでください
## このプラグインの使い方
### 3分セットアップ - CouchDB on fly.io
**はじめての方におすすめ**
[![LiveSync Setup onto Fly.io SpeedRun 2024 using Google Colab](https://img.youtube.com/vi/7sa_I1832Xc/0.jpg)](https://www.youtube.com/watch?v=7sa_I1832Xc)
1. [Fly.ioにCouchDBをセットアップする](docs/setup_flyio.md)
2. [Quick Setup](docs/quick_setup_ja.md)でプラグインを設定する
### Manually Setup
1. サーバのセットアップ
1. [Fly.ioにCouchDBをセットアップする](docs/setup_flyio.md)
2. [CouchDBをセットアップする](docs/setup_own_server_ja.md)
2. [Quick Setup](docs/quick_setup_ja.md)でプラグインを設定する
> [!TIP]
> IBM Cloudantもまだ使用できますが、いくつかの理由で現在はおすすめしていません。[IBM Cloudantのセットアップ](docs/setup_cloudant_ja.md)はまだあります。
## ステータスバーの説明
同期ステータスはステータスバーに、下記のアイコンとともに表示されます
- アクティビティー
- 📲 ネットワーク接続中
- 同期ステータス
- ⏹️ 停止中
- 💤 変更待ち（LiveSync中）
- ⚡️ 同期の進行中
- ⚠ エラー
- 統計情報
- ↑ アップロードしたチャンクとメタデータ数
- ↓ ダウンロードしたチャンクとメタデータ数
- 進捗情報
- 📥 転送後、未処理の項目数
- 📄 稼働中データベース操作数
- 💾 稼働中のストレージ書き込み数操作数
- ⏳ 稼働中のストレージ読み込み数操作数
- 🛫 待機中のストレージ読み込み数操作数
- ⚙️ 隠しファイルの操作数（待機・稼働中合計）
- 🧩 取得待ちを行っているチャンク数
- 🔌 設定同期関連の操作数
データベースやファイルの破損を避けるため、Obsidianの終了は進捗情報が表示されなくなるまで待ってください（プラグインも復帰を試みますが）。特にファイルを削除やリネームした場合は気をつけてください。
## Tips and Troubleshooting
何かこまったら、[Tips and Troubleshooting](docs/troubleshooting.md)をご参照ください。
## License
Licensed under the MIT License.

322
deploy_couchdb_to_flyio_v2_with_swap.ipynb
View File

@@ -1,322 +0,0 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/gist/vrtmrz/37c3efd7842e49947aaaa7f665e5020a/deploy_couchdb_to_flyio_v2_with_swap.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "markdown",
"source": [
"History:\n",
"- 18, May, 2023: Initial.\n",
"- 19, Jun., 2023: Patched for enabling swap.\n",
"- 22, Aug., 2023: Generating Setup-URI implemented."
],
"metadata": {
"id": "HiRV7G8Gk1Rs"
}
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "2Vh0mEQEZuAK"
},
"outputs": [],
"source": [
"# Configurations\n",
"import os\n",
"os.environ['region']=\"nrt\"\n",
"os.environ['couchUser']=\"alkcsa93\"\n",
"os.environ['couchPwd']=\"c349usdfnv48fsasd\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "SPmbB0jZauQ1"
},
"outputs": [],
"source": [
"# Delete once\n",
"!rm ./fly.toml"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Nze7QoxLZ7Yx"
},
"outputs": [],
"source": [
"# Installation\n",
"# You have to set up your account in here.\n",
"!curl -L https://fly.io/install.sh | sh\n",
"!/root/.fly/bin/flyctl auth signup"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "MVJwsIYrbgtx"
},
"outputs": [],
"source": [
"# Generate server\n",
"!/root/.fly/bin/flyctl launch --auto-confirm --generate-name --detach --no-deploy --region ${region}\n",
"!/root/.fly/bin/fly volumes create --region ${region} couchdata --size 2 --yes"
]
},
{
"cell_type": "code",
"source": [
"# Check the toml once.\n",
"!cat fly.toml"
],
"metadata": {
"id": "2RSoO9o-i2TT"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "zUtPZLVnbvdQ"
},
"outputs": [],
"source": [
"# Modify fly.toml\n",
"## Port modification\n",
"!sed -i 's/8080/5984/g' fly.toml\n",
"## Add user into.\n",
"!echo -e \"\\n[env]\\n COUCHDB_USER = \\\"${couchUser}\\\"\" >> ./fly.toml\n",
"## Set the location of an ini file which to save configurations persistently via erlang flags.\n",
"!echo -e \"\\nERL_FLAGS=\\\"-couch_ini /opt/couchdb/etc/default.ini /opt/couchdb/etc/default.d/ /opt/couchdb/etc/local.d /opt/couchdb/etc/local.ini /opt/couchdb/data/persistence.ini\\\"\" >> ./fly.toml\n",
"## Mounting volumes to store data and ini file.\n",
"!echo -e \"\\n[mounts]\\n source=\\\"couchdata\\\"\\n destination=\\\"/opt/couchdb/data\\\"\" >> ./fly.toml\n",
"!cat fly.toml"
]
},
{
"cell_type": "code",
"source": [
"# Make the Dockerfile to modify the permission of the ini file. If you want to use a specific version, you should change `latest` here.\n",
"!echo -e \"\\n[build]\\n dockerfile = \\\"./Dockerfile\\\"\" >> ./fly.toml"
],
"metadata": {
"id": "LQPsZ_dYxkTu"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"!echo -e \"FROM couchdb:latest\\nRUN sed -i '2itouch /opt/couchdb/data/persistence.ini && chmod +w /opt/couchdb/data/persistence.ini && fallocate -l 512M /swapfile && chmod 0600 /swapfile && mkswap /swapfile && echo 10 > /proc/sys/vm/swappiness && swapon /swapfile && echo 1 > /proc/sys/vm/overcommit_memory' /docker-entrypoint.sh\" > ./Dockerfile"
],
"metadata": {
"id": "44cBeGJ9on5i"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Check dockerfile\n",
"!cat ./Dockerfile"
],
"metadata": {
"id": "ai2R3BbpxRSe"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "xWdsTCI6bzk2"
},
"outputs": [],
"source": [
"# Configure password\n",
"!/root/.fly/bin/flyctl secrets set COUCHDB_PASSWORD=${couchPwd}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "k0WIQlShcXGa"
},
"outputs": [],
"source": [
"# Deploy server\n",
"# Be sure to shutdown after the test.\n",
"!/root/.fly/bin/flyctl deploy --detach --remote-only\n",
"!/root/.fly/bin/flyctl status"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "0ySggkdlfq7M"
},
"outputs": [],
"source": [
"import subprocess, json\n",
"result = subprocess.run([\"/root/.fly/bin/flyctl\",\"status\",\"-j\"], capture_output=True, text=True)\n",
"if result.returncode==0:\n",
" hostname = json.loads(result.stdout)[\"Hostname\"]\n",
" os.environ['couchHost']=\"https://%s\" % (hostname)\n",
" print(\"Your couchDB server is https://%s/\" % (hostname))\n",
"else:\n",
" print(\"Something occured.\")\n"
]
},
{
"cell_type": "code",
"source": [
"# Finish setting up the CouchDB\n",
"# Please repeat until the request is completed without error messages\n",
"# i.e., You have to redo this block while \"curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to xxxx\" is showing.\n",
"!curl -X POST \"${couchHost}/_cluster_setup\" -H \"Content-Type: application/json\" -d \"{\\\"action\\\":\\\"enable_single_node\\\",\\\"username\\\":\\\"${couchUser}\\\",\\\"password\\\":\\\"${couchPwd}\\\",\\\"bind_address\\\":\\\"0.0.0.0\\\",\\\"port\\\":5984,\\\"singlenode\\\":true}\" --user \"${couchUser}:${couchPwd}\""
],
"metadata": {
"id": "cGlSzVqlQG_z"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Please repeat until all lines are completed without error messages\n",
"!curl -X PUT \"${couchHost}/_node/nonode@nohost/_config/chttpd/require_valid_user\" -H \"Content-Type: application/json\" -d '\"true\"' --user \"${couchUser}:${couchPwd}\"\n",
"!curl -X PUT \"${couchHost}/_node/nonode@nohost/_config/chttpd_auth/require_valid_user\" -H \"Content-Type: application/json\" -d '\"true\"' --user \"${couchUser}:${couchPwd}\"\n",
"!curl -X PUT \"${couchHost}/_node/nonode@nohost/_config/httpd/WWW-Authenticate\" -H \"Content-Type: application/json\" -d '\"Basic realm=\\\"couchdb\\\"\"' --user \"${couchUser}:${couchPwd}\"\n",
"!curl -X PUT \"${couchHost}/_node/nonode@nohost/_config/httpd/enable_cors\" -H \"Content-Type: application/json\" -d '\"true\"' --user \"${couchUser}:${couchPwd}\"\n",
"!curl -X PUT \"${couchHost}/_node/nonode@nohost/_config/chttpd/enable_cors\" -H \"Content-Type: application/json\" -d '\"true\"' --user \"${couchUser}:${couchPwd}\"\n",
"!curl -X PUT \"${couchHost}/_node/nonode@nohost/_config/chttpd/max_http_request_size\" -H \"Content-Type: application/json\" -d '\"4294967296\"' --user \"${couchUser}:${couchPwd}\"\n",
"!curl -X PUT \"${couchHost}/_node/nonode@nohost/_config/couchdb/max_document_size\" -H \"Content-Type: application/json\" -d '\"50000000\"' --user \"${couchUser}:${couchPwd}\"\n",
"!curl -X PUT \"${couchHost}/_node/nonode@nohost/_config/cors/credentials\" -H \"Content-Type: application/json\" -d '\"true\"' --user \"${couchUser}:${couchPwd}\"\n",
"!curl -X PUT \"${couchHost}/_node/nonode@nohost/_config/cors/origins\" -H \"Content-Type: application/json\" -d '\"app://obsidian.md,capacitor://localhost,http://localhost\"' --user \"${couchUser}:${couchPwd}\""
],
"metadata": {
"id": "JePzrsHypY18"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"Now, our CouchDB has been surely installed and configured. Cheers!\n",
"\n",
"In the steps that follow, create a setup-URI.\n",
"\n",
"This URI could be imported directly into Self-hosted LiveSync, to configure the use of the CouchDB which we configured now."
],
"metadata": {
"id": "YfSOomsoXbGS"
}
},
{
"cell_type": "code",
"source": [
"# Database config\n",
"import random, string\n",
"\n",
"def randomname(n):\n",
" return ''.join(random.choices(string.ascii_letters + string.digits, k=n))\n",
"\n",
"# The database name\n",
"os.environ['database']=\"obsidiannote\"\n",
"# The passphrase to E2EE\n",
"os.environ['passphrase']=randomname(20)\n",
"\n",
"print(\"Your database:\"+os.environ['database'])\n",
"print(\"Your passphrase:\"+os.environ['passphrase'])"
],
"metadata": {
"id": "416YncOqXdNn"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Install deno for make setup uri\n",
"!curl -fsSL https://deno.land/x/install/install.sh | sh"
],
"metadata": {
"id": "C4d7C0HAXgsr"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Fetch module for encrypting a Setup URI\n",
"!curl -o encrypt.ts https://gist.githubusercontent.com/vrtmrz/f9d1d95ee2ca3afa1a924a2c6759b854/raw/d7a070d864a6f61403d8dc74208238d5741aeb5a/encrypt.ts"
],
"metadata": {
"id": "hQL_Dx-PXise"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Make buttons!\n",
"from IPython.display import HTML\n",
"result = subprocess.run([\"/root/.deno/bin/deno\",\"run\",\"-A\",\"encrypt.ts\"], capture_output=True, text=True)\n",
"text=\"\"\n",
"if result.returncode==0:\n",
" text = result.stdout.strip()\n",
" result = HTML(f\"<button onclick=navigator.clipboard.writeText('{text}')>Copy setup uri</button><br>Importing passphrase is `welcome`. <br>If you want to synchronise in live mode, please apply a preset after setup.)\")\n",
"else:\n",
" result = \"Failed to encrypt the setup URI\"\n",
"result"
],
"metadata": {
"id": "o0gX_thFXlIZ"
},
"execution_count": null,
"outputs": []
}
],
"metadata": {
"colab": {
"provenance": [],
"private_outputs": true,
"include_colab_link": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": "python"
},
"gpuClass": "standard"
},
"nbformat": 4,
"nbformat_minor": 0
}

34
docs/adding_translations.md Normal file
View File

@@ -0,0 +1,34 @@
# How to add translations
## Getting ready
1. Clone this repository recursively.
```sh
git clone --recursive https://github.com/vrtmrz/obsidian-livesync
```
2. Make `ls-debug` folder under your vault's `.obsidian` folder (as like `.../dev/.obsidian/ls-debug`).
## Add translations for already defined terms
1. Install dependencies, and build the plug-in as dev. build.
```sh
cd obsidian-livesync
npm i -D
npm run buildDev
```
2. Copy the `main.js` to `.obsidian/plugins/obsidian-livesync` folder of your vault, and run Obsidian-Self-hosted LiveSync.
3. You will get the `missing-translation-yyyy-mm-dd.jsonl`, please fill in new translations.
4. Build the plug-in again, and confirm that displayed things were expected.
5. Merge them into `rosetta.ts`, and make the PR to `https://github.com/vrtmrz/livesync-commonlib`.
## Make messages to be translated
1. Find the message that you want to be translated.
2. Change the literal to a tagged template literal using `$f`, like below.
```diff
- Logger("Could not determine passphrase to save data.json! You probably make the configuration sure again!", LOG_LEVEL_URGENT);
+ Logger($f`Could not determine passphrase to save data.json! You probably make the configuration sure again!`, LOG_LEVEL_URGENT);
```
3. Make the PR to `https://github.com/vrtmrz/obsidian-livesync`.
4. Follow the steps of "Add translations for already defined terms" to add the translations.

50
docs/design_docs_of_journalsync.md Normal file
View File

@@ -0,0 +1,50 @@
## The design document of the journal sync
Original title: Synchronise without CouchDB
### Goal
- Synchronise vaults without CouchDB
### Motivation
- Serving CouchDB is not pretty easy.
- Full spec DBaaS (Paid IBM Cloudant) is a bit expensive and lacking of alternatives.
- Securing alternatives, from just one protocol.
### Prerequisite
- We should have multiple implementations of the server software.
- We should also be able to use SaaS, with a choice of options.
- We should require them a reasonable sense of cost, ideally free of charge for trials.
- We should be able to serve some instance of the server software, as OSS — with transparency, availability of auditing, and the fact that they actually took place.
### Methods and implementations
Ordinarily, local pouchDB and the remote CouchDB are synchronised by sending each missing document through several conversations in their replication protocol. However, to achieve this plan, we cannot rely on CouchDB and its protocols. This limitation is so harsh. However, Overcoming this means gaining new possibilities. After some trials, It was concluded that synchronisation could be completed even if the actions that could be performed were limited to uploading, downloading and retrieving the list. This means we can use any old-fashioned WebDAV server, and Sophisticated “Object storages” such as Self-hosted MinIO, S3, and R2 or any we like. This is realised by sharing and complementing the differences of the journal by each client. Therefore, The focus is therefore on how to identify which are the differences and send them without dynamic communication.
All clients manage their data in PouchDB. I know this is probably known information, but it has its own journal.
First, all clients should record to what point in the journal they sent themselves last time. The client then packs from the previous point to the latest when sending and also updates their record. This pack is uploaded to the server with the name starting with the timestamp of its creation. This is the send operation.
Conversely, when receiving, the packs uploaded to the server that have not yet been received are received in order. This is easy as their names are in date order. When the process is successfully completed, the names of the files received are recorded. The journals from this pack are then reflected in their own database. Conflict resolution is left to PouchDB, so the client only needs to do the work of applying any differences. And here is the key: the client records the ID and revision of the document that was in the journal and applied.
This key works when creating a pack. When creating a pack, the client omits this 'document recorded as received and used'. This is because received and applied means that it has already been sent by another client and exists on the server. This ensures that unnecessary transmissions do not take place.
Synchronisation is then always started by receiving. This is a little trick to avoid including unnecessary documents in the pack.
These behaviours allow clients to voluntarily send and receive only the missing parts of the journal that are not stored on the server, without having to communicate with each other, and still keep a single, consistent journal on the server.
Source codes actually implemented this is already committed into the repository.
### Test strategy
This implementation replaces the synchronisation performed by CouchDB. Therefore, testing was simply done by comparing the same changes to the same vault, replicated in CouchDB, with those done by this implementation.
### Documentation strategy
- Documentation should be done in a quick setup, at least.
- As several server implementations can be selected, the description is omitted with regard to specific configuration values.
- A MinIO set-up might be nice to have. However, it is not considered essential.
- It would be a good opportunity to also publish these design documents.
### Consideration and Conclusion
This design offers a novel approach to journal synchronisation without relying on CouchDB. It leverages PouchDB's journaling capabilities and leverages simple server-side storage for efficient data exchange. Hence, the new design could be said to have gotten a broader outlook.

81
docs/design_docs_of_keep_newborn_chunks.md Normal file
View File

@@ -0,0 +1,81 @@
# Keep newborn chunks in Eden.
NOTE: This is the planned feature design document. This is planned, but not be implemented now (v0.23.3). This has not reached the design freeze and will be added to from time to time.
## Goal
Reduce the number of chunks which in volatile, and reduce the usage of storage of the remote database in middle or long term.
## Motivation
- In the current implementation, Self-hosted LiveSync splits documents into metadata and multiple chunks. In particular, chunks are split so that they do not exceed a certain length.
- This is to optimise the transfer and take advantage of the properties of CouchDB. This also complies with the restriction of IBM Cloudant on the size of a single document.
- However, creating chunks halfway through each editing operation increases the number of unnecessary chunks.
- Chunks are shared by several documents. For this reason, it is not clear whether these chunks are needed or not unless all revisions of all documents are checked. This makes it difficult to remove unnecessary data.
- On the other hand, chunks are done in units that can be neatly divided as markdown to ensure relatively accurate de-duplication, even if they are created simultaneously on multiple terminals. Therefore, it is unlikely that the data in the editing process will be reused.
- For this reason, we have made features such as Batch save available, but they are not a fundamental solution.
- As a result, there is a large amount of data that cannot be erased and is probably unused. Therefore, `Fetch chunks on demand` is currently performed for optimal communication.
- If the generation of unnecessary chunks is sufficiently reduced, this function will become unnecessary.
- The problem is that this unnecessary chunking slows down both local and remote operations.
## Prerequisite
- The implementation must be able to control the size of the document appropriately so that it does not become non-transferable (1).
- The implementation must be such that data corruption can be avoided even if forward compatibility is not maintained; due to the nature of Self-hosted LiveSync, backward version connexions are expected.
- Viewed as a feature:
- This feature should be disabled for migration users.
- This feature should be enabled for new users and after rebuilds of migrated users.
- Therefore, back into the implementation view, Ideally, the implementation should be such that data recovery can be achieved by immediately upgrading after replication.
## Outlined methods and implementation plans
### Abstract
To store and transfer only stable chunks independently and share them from multiple documents after stabilisation, new chunks, i.e. chunks that are considered non-stable, are modified to be stored in the document and transferred with the document. In this case, care should be taken not to exceed prerequisite (1).
If this is achieved, the non-leaf document will not be transferred, and even if it is, the chunk will be stored in the document, so that the size can be reduced by the compaction.
Details are given below.
1. The document will henceforth have the property eden.
```typescript
// Paritally Type
type EntryWithEden = {
eden: {
[key: DocumentID]: {
data: string,
epoch: number, // The document revision which this chunk has been born.
}
}
}
```
2. The following configuration items are added:
Note: These configurations should be shared as `Tweaks value` between each client.
- useEden : boolean
- Max chunks in eden : number
- Max Total chunk lengths in eden: number
- Max age while in eden: number
3. In the document saving operation, chunks are added to Eden within each document, having the revision number of the existing document. And if some chunks in eden are not used in the operating revision, they would be removed.
Then after being so chosen, a few chunks are also chosen to be graduated as an independent `chunk` in following rules, and they would be left the eden:
- Those that have already been confirmed to exist as independent chunks.
- This confirmation of existence may ideally be determined by a fast first-order determination, e.g. by a Bloom filter.
- Those whose length exceeds the configured maximum length.
- Those have aged over the configured value, since epoch at the operating revision.
- Those whose total length, when added up when they are arranged in reverse order of the revision in which they were generated, is after the point at which they exceed the max length in the configuration. Or, those after the configured maximum items.
4. In the document loading operation, chunks are firstly read from these eden.
5. In End-to-End Encryption, property `eden` of documents will also be encrypted.
### Note
- When this feature has been enabled, forward compatibility is temporarily lost. However, it is detected as missing chunks, and this data is not reflected in the storage in the old version. Therefore, no data loss will occur.
## Test strategy
1. Confirm that synchronisation with the previous version is possible with this feature disabled.
2. With this feature enabled, connect from the previous version and confirm that errors are detected in the previous version but the files are not corrupted.
3. Ensure that the two versions with this feature enabled can withstand normal use.
## Documentation strategy
- This document is published, and will be referred from the release note.
- Indeed, we lack a fulfilled configuration table. Efforts will be made and, if they can be produced, this document will then be referenced. But not required while in the experimental or beta feature.
- However, this might be an essential feature. Further efforts are desired.
### Consideration and Conclusion
To be described after implemented, tested, and, released.

55
docs/design_docs_of_sharing_tweak_value.md Normal file
View File

@@ -0,0 +1,55 @@
# Sharing `Tweak values`
NOTE: This is the planned feature design document. This is planned, but not be implemented now (v0.23.3). This has not reached the design freeze and will be added to from time to time.
## Goal
Share `Tweak values` between clients to match the chunk lengths, and match per-server configurations for better performance.
## Motivation
- In the current implementation, Self-hosted LiveSync splits documents into metadata and multiple chunks. In particular, chunks are split so that they do not exceed a certain length.
- This is to optimise the transfer and take advantage of the properties of CouchDB. This also complies with the restriction of IBM Cloudant on the size of a single document.
- The length of this chunk is adjusted according to a configured factor. Therefore, if this is inconsistent between clients, de-duplication will not work. This is because, in fact, they point to the same content in total, but are split in different places. This results in unnecessary transfers or storage consumption.
- The same applies to hash algorithms.
- There are more configurations which `preferred to be matched`, even if it is not required. such as the maximum size of files to be handled and the interval between requests to the remote database, unless there are specific circumstances.
- To avoid the tragedy of "Too many toggles", "Unexpected transfer amount", or "Poor performance" at once, the plug-in should know these problems or potential problems and be able to let us know.
## Prerequisite
- We must be informed of a discrepancy in a configured value that is required to be absolutely consistent and be able to make a decision on the spot.
- We should be able to see on the configuration dialogue, that there is a discrepancy between configured values that should be matched, and it should be possible to adjust them to a specific one of them (or default).
- We must not be exposed to unexpected; such as leaking credentials or their secrets.
## Outlined methods and implementation plans
### Abstract
- In the current implementation, each client checks the remote database for the existence of their node information, to detect whether the remote database accepts them.
- This is what 'Lock' is all about.
- To achieve this feature, the client will also send each configuration value. However, the configuration contains credentials and/or secret values. Hence we cannot send all of them.
- With a favourable prediction, Self-hosted LiveSync will continue to increase in feature. Each time this happens, the number of configuration values to be kept secret will also increase. Therefore, they must be handled by an allow-list.
- This allow-listed configuration are the `Tweak values`.
- If the plug-in detects mismatched `Tweak values` on checking the remote database, the plug-in will ask us to decide which is win (Mine, or theirs).
- Node information is one of the documents. Therefore, it will be replicated and saved locally. While showing dialogue, show the notice on each `Match preferred` configuration.
## Note
This feature should be mostly harmless. We will not be able to disable this.
## Test strategy
A: During synchronisation.
1. No message shall be displayed with all settings matched.
2. Message shall be displayed when there are mismatched, required match items.
1. The setting values can be changed according to the message.
2. The message can be ignored.
3. The message shall not be displayed even if there are mismatched items which is recommended to be matched.
B: On the setting dialogue.
1. All mismatched items shall be highlighted in some way.
## Documentation strategy
- This document is published, and will be referred from the release note.
- Indeed, we lack a fulfilled configuration table. Efforts will be made and, if they can be produced, this document will then be referenced. But not required while in the experimental or beta feature.
- However, this might be an essential feature. Further efforts are desired.
### Consideration and Conclusion
To be described after implemented, tested, and, released.

120
docs/quick_setup.md
View File

@@ -2,48 +2,81 @@
[Japanese docs](./quick_setup_ja.md) - [Chinese docs](./quick_setup_cn.md).
The plugin has so many configuration options to deal with different circumstances. However, there are not so many settings that are actually used. Therefore, `The Setup wizard` has been implemented to simplify the initial setup.
Note: Subsequent devices are recommended to be set up using the `Copy setup URI` and `Open setup URI`.
## The Setup wizard
Open the `🧙‍♂️ Setup wizard` in the settings dialogue. If the plugin has not been configured before, it should already be open.
The plugin has so many configuration options to deal with different circumstances. However, only a few settings are required in the normal cases. Therefore, `The Setup wizard` has been implemented to simplify the setup.
![](../images/quick_setup_1.png)
- Discard the existing configuration and set up
If you have changed any settings, this button allows you to discard all changes before setting up.
There are three methods to set up Self-hosted LiveSync.
- Do not discard the existing configuration and set up
Simply reconfigure. Be careful. In wizard mode, you cannot see all configuration items, even if they have been configured.
1. [Using setup URIs](#1-using-setup-uris) *(Recommended)*
2. [Minimal setup](#2-minimal-setup)
3. [Full manually setup the and Enable on this dialogue](#3-manually-setup)
Pressing `Next` on one of the above options will put the configuration dialog into wizard mode.
## At the first device
### Wizard mode
### 1. Using setup URIs
> [!TIP]
> What is the setup URI? Why is it required?
> The setup URI is the encrypted representation of Self-hosted LiveSync configuration as a URI. This starts `obsidian://setuplivesync?settings=`. This is encrypted with a passphrase, so that it can be shared relatively securely between devices. It is a bit long, but it is one line. This allows a series of settings to be set at once without any inconsistencies.
>
> If you have configured the remote database by [Automated setup on Fly.io](./setup_flyio.md#a-very-automated-setup) or [set up your server with the tool](./setup_own_server.md#1-generate-the-setup-uri-on-a-desktop-device-or-server), **you should have one of them**
In this procedure, [this video](https://youtu.be/7sa_I1832Xc?t=146) may help us.
1. Click `Use` button (Or launch `Use the copied setup URI` from Command palette).
2. Paste the Setup URI into the dialogue
3. Type the passphrase of the Setup URI
4. Answer `yes` for `Importing LiveSync's conf, OK?`.
5. Answer `Set it up as secondary or subsequent device` for `How would you like to set it up?`.
6. Initialisation will begin, please hold a while.
7. You will asked about the hidden file synchronisation, answer as you like.
1. If you are new to Self-hosted LiveSync, we can configure it later so leave it once.
8. Synchronisation has been started! `Reload app without saving` is recommended after the indicators of Self-hosted LiveSync disappear.
OK, we can proceed the [next step](#).
### 2. Minimal setup
If you do not have any setup URI, Press the `start` button. The setting dialogue turns into the wizard mode and will display only minimal items.
>[!TIP]
> We can generate the setup URI with the tool in any time. Please use [this tool](./setup_own_server.md#1-generate-the-setup-uri-on-a-desktop-device-or-server).
![](../images/quick_setup_2.png)
Let's see how to use it step-by-step.
## Remote Database configuration
#### Select the remote type
### Remote database configuration
1. Select the Remote Type from dropdown list.
We now have a choice between CouchDB (and its compatibles) and object storage (MinIO, S3, R2). CouchDB is the first choice and is also recommended. And supporting Object Storage is an experimental feature.
Enter the information for the database we have set up.
#### Remote configuration
##### CouchDB
Enter the information for the database we have set up.
![](../images/quick_setup_3.png)
##### Object Storage
#### Test database connection and Check database configuration
1. Enter the information for the S3 API and bucket.
![](../images/quick_setup_3b.png)
Note 1: if you use S3, you can leave the Endpoint URL empty.
Note 2: if your Object Storage cannot configure the CORS setting fully, you may able to connect to the server by enabling the `Use Custom HTTP Handler` toggle.
2. Press `Test` of `Test Connection` once and ensure you can connect to the Object Storage.
#### Only CouchDB: Test database connection and Check database configuration
We can check the connectivity to the database, and the database settings.
![](../images/quick_setup_5.png)
#### Test Database Connection
Check whether we can connect to the database. If it fails, there are several possible reasons, but first attempt the `Check database configuration` check to see if it fails there too.
#### Check database configuration
#### Only CouchDB: Check and Fix database configuration
Check the database settings and fix any problems on the spot.
@@ -52,40 +85,45 @@ Check the database settings and fix any problems on the spot.
This item may vary depending on the connection. In the above case, press all three Fix buttons.
If the Fix buttons disappear and all become check marks, we are done.
### Confidentiality configuration
#### Confidentiality configuration (Optional but very preferred)
![](../images/quick_setup_4.png)
Encrypt your database in case of unintended database exposure; enable End to End encryption and the contents of your notes will be encrypted at the moment it leaves the device. We strongly recommend enabling it. And `Path Obfuscation` also obfuscates filenames. Now stable and recommended.
Encryption is based on 256-bit AES-GCM.
Enable End-to-end encryption and the contents of your notes will be encrypted at the moment it leaves the device. We strongly recommend enabling it. And `Path Obfuscation` also obfuscates filenames. Now stable and recommended.
These setting can be disabled if you are inside a closed network and it is clear that you will not be accessed by third parties.
![](../images/quick_setup_7.png)
> [!TIP]
> Encryption is based on 256-bit AES-GCM.
#### Next
Go to the Sync Settings.
We should proceed to the Next step.
#### Discard existing database and proceed
Discard the contents of the Remote database and go to the Sync Settings.
### Sync Settings
#### Sync Settings
Finally, finish the wizard by selecting a preset for synchronisation.
Note: If you are going to use Object Storage, you cannot select `LiveSync`.
![](../images/quick_setup_9_1.png)
Select any synchronisation methods we want to use and `Apply` to initialise and build the local and remote databases as required. If `All done!` is displayed, we are done. Automatically, `Copy setup URI` will open and we will be asked for a passphrase to encrypt the `Setup URI`.
Select any synchronisation methods we want to use and `Apply`. If database initialisation is required, it will be performed at this time. When `All done!` is displayed, we are ready to synchronise.
The dialogue of `Copy settings as a new setup URI` will be open automatically. Please input a passphrase to encrypt the new `Setup URI`. (This passphrase is to encrypt the setup URI, not the vault).
![](../images/quick_setup_10.png)
Set the passphrase as you like.
The Setup URI will be copied to the clipboard, which you can then transfer to the second and subsequent devices in some way.
The Setup URI will be copied to the clipboard, please make a note(Not in Obsidian) of this.
# How to set up the second and subsequent units
After installing Self-hosted LiveSync on the first device, select `Open setup URI` from the command palette and enter the setup URI you transferred. Afterwards, enter your passphrase and a setup wizard will open.
Answer the following.
>[!TIP]
We can copy this in any time by `Copy current settings as a new setup URI`.
- `Yes` to `Importing LiveSync's conf, OK?`
- `Set it up as secondary or subsequent device` to `How would you like to set it up?`.
### 3. Manually setup
Then, The configuration will take effect and replication will start. Your files will be synchronised soon! You may need to close the settings dialog and reopen it to see the settings fields populated properly, but they will be set.
It is strongly recommended to perform a "minimal set-up" first and set up the other contents after making sure has been synchronised.
However, if you have some specific reasons to configure it manually, please click the `Enable` button of `Enable LiveSync on this device as the set-up was completed manually`.
And, please copy the setup URI by `Copy current settings as a new setup URI` and make a note(Not in Obsidian) of this.
## At the subsequent device
After installing Self-hosted LiveSync on the first device, we should have a setup URI. **The first choice is to use it**. Please share it with the device you want to setup.
It is completely same as [Using setup URIs on the first device](#1-using-setup-uris). Please refer it.

237
docs/setup_flyio.md
View File

@@ -1,24 +1,53 @@
<!-- For translation: 20240209r0 -->
# Setup CouchDB on fly.io
In some cases, the use of IBM Cloudant was found to be hard. We looked for alternatives, but there were no services available. Therefore, we have to build our own servers, which is quite a challenge. In this situation, with fly.io, most of the processes are simplified and only CouchDB needs to be configured.
This is how to configure fly.io and CouchDB on it for Self-hosted LiveSync.
It generally falls within the `Free Allowances` range in most cases.
> [!WARNING]
> It is **your** instance. In Obsidian, we have files locally. Hence, do not hesitate to destroy the remote database if you feel something have got weird. We can launch and switch to the new CouchDB instance anytime[^1].
>
[^1]: Actually, I am always building the database for reproduction of the issue like so.
**[Automatic setup using Colaboratory](#automatic-setup-using-colaboratory) is recommended, after reading this document through. It is reproducible and hard to fail.**
> [!NOTE]
> **What and why is the Fly.io?**
> At some point, we started to experience problems related to our IBM Cloudant account. At the same time, Self-hosted LiveSync started to improve its functionality, requiring CouchDB in a more natural state to use all its features.
>
> Then we found Fly.io. Fly.io is the PaaS Platform, which can be useable for a very reasonable price. It generally falls within the `Free Allowances` range in most cases.
## Required materials
- A valid credit or debit card.
## Warning
## Setup CouchDB instance
- It will be `your` instance. Check the log regularly.
### A. Very automated setup
## Prerequisites
[![LiveSync Setup onto Fly.io SpeedRun 2024 using Google Colab](https://img.youtube.com/vi/7sa_I1832Xc/0.jpg)](https://www.youtube.com/watch?v=7sa_I1832Xc)
For simplicity, the following description assumes that the settings are as shown in the table below. Please read it in accordance with the actual settings you wish to make.
1. Open [setup-flyio-on-the-fly-v2.ipynb](../setup-flyio-on-the-fly-v2.ipynb).
2. Press the `Open in Colab` button.
3. Choose a region and run all blocks (Refer to video).
1. If you do not have the account yet, the sign-up page will be shown, please follow the instructions. The [Official document is here](https://fly.io/docs/hands-on/sign-up/).
4. Copy the Setup-URI and Use it in the Obsidian.
5. You have been synchronised. Use the Setup-URI in subsequent devices.
Steps 4 and 5 are detailed in the [Quick Setup](./quick_setup.md#1-using-setup-uris).
> [!NOTE]
> Your automatically configured configurations will be shown on the result in the Colab note like below, and **it will not be saved**. Please make a note of it somewhere.
> ```
> -- YOUR CONFIGURATION --
> URL : https://billowing-dawn-6619.fly.dev
> username: billowing-cherry-22580
> password: misty-dew-13571
> region : nrt
> ```
### B. Scripted Setup
Please refer to the document of [deploy-server.sh](../utils/readme.md#deploy-serversh).
### C. Manual Setup
| Used in the text | Meaning and where to use | Memo |
| ---------------- | --------------------------- | ------------------------------------------------------------------------ |
@@ -26,11 +55,7 @@ For simplicity, the following description assumes that the settings are as shown
| dfusiuada9suy | Password | |
| nrt | Region to make the instance | We can use any [region](https://fly.io/docs/reference/regions/) near us. |
## Steps with your computer
If you want to avoid installing anything, please skip to [Automatic setup using Colaboratory](#automatic-setup-using-colaboratory).
### 1. Install flyctl
#### 1. Install flyctl
- Mac or Linux
@@ -44,7 +69,7 @@ $ curl -L https://fly.io/install.sh | sh
$ iwr https://fly.io/install.ps1 -useb | iex
```
### 2. Sign up or Sign in to fly.io
#### 2. Sign up or Sign in to fly.io
- Sign up
@@ -58,148 +83,90 @@ $ fly auth signup
$ fly auth login
```
For more information, please refer [Sign up](https://fly.io/docs/hands-on/sign-up/) and [Sign in](https://fly.io/docs/hands-on/sign-in/).
For more information, please refer to [Sign up](https://fly.io/docs/hands-on/sign-up/) and [Sign in](https://fly.io/docs/hands-on/sign-in/).
### 3. Make configuration files
#### 3. Make a configuration file
Please be careful, `nrt` is the region where near to Japan. Please use your preferred region.
1. Make `fly.toml` from template `fly.template.toml`.
We can simply copy and rename the file. The template is on [utils/flyio/fly.template.toml](../utils/flyio/fly.template.toml)
2. Decide the instance name, initialize the App, and set credentials.
1. Make fly.toml
>[!TIP]
> - The name `billowing-dawn-6619` is randomly decided name, and it will be a part of the CouchDB URL. It should be globally unique. Therefore, it is recommended to use something random for this name.
> - Explicit naming is very good for humans. However, we do not often get the chance to actually enter this manually (have designed so). This database may contain important information for you. The needle should be hidden in the haystack.
```bash
$ fly launch --name=billowing-dawn-6619 --env="COUCHDB_USER=campanella" --copy-config=true --detach --no-deploy --region nrt --yes
$ fly secrets set COUCHDB_PASSWORD=dfusiuada9suy
```
#### 4. Deploy
```
$ flyctl launch --generate-name --detach --no-deploy --region nrt
Creating app in /home/vrtmrz/dev/fly/demo
Scanning source code
Could not find a Dockerfile, nor detect a runtime or framework from source code. Continuing with a blank app.
automatically selected personal organization: vorotamoroz
App will use 'nrt' region as primary
$ flyctl deploy
An existing fly.toml file was found
Using build strategies '[the "couchdb:latest" docker image]'. Remove [build] from fly.toml to force a rescan
Creating app in /home/vorotamoroz/dev/obsidian-livesync/utils/flyio
We're about to launch your app on Fly.io. Here's what you're getting:
Organization: vorotamoroz (fly launch defaults to the personal org)
Name: billowing-dawn-6619 (specified on the command line)
Region: Tokyo, Japan (specified on the command line)
App Machines: shared-cpu-1x, 256MB RAM (specified on the command line)
Postgres: <none> (not requested)
Redis: <none> (not requested)
Created app 'billowing-dawn-6619' in organization 'personal'
Admin URL: https://fly.io/apps/billowing-dawn-6619
Hostname: billowing-dawn-6619.fly.dev
Wrote config file fly.toml
```
`billowing-dawn-6619` is an automatically generated name. It is used as the hostname. Please note it in something.
Note: we can specify this without `--generate-name`, but does not recommend in the trial phases.
1. Make volume
```
$ flyctl volumes create --region nrt couchdata --size 2 --yes
ID: vol_g67340kxgmmvydxw
Name: couchdata
App: billowing-dawn-6619
Region: nrt
Zone: 35b7
Size GB: 2
Encrypted: true
Created at: 02 Jun 23 01:19 UTC
```
3. Edit fly.toml
Changes:
- Change exposing port from `8080` to `5984`
- Mounting the volume `couchdata` created in step 2 under `/opt/couchdb/data`
- Set `campanella` for the administrator of CouchDB
- Customise CouchDB to use persistent ini-file; which is located under the data directory.
- To use Dockerfile
```diff
# fly.toml app configuration file generated for billowing-dawn-6619 on 2023-06-02T10:18:59+09:00
#
# See https://fly.io/docs/reference/configuration/ for information about how to use this file.
#
app = "billowing-dawn-6619"
primary_region = "nrt"
[http_service]
- internal_port = 8080
+ internal_port = 5984
force_https = true
auto_stop_machines = true
auto_start_machines = true
min_machines_running = 0
+[mounts]
+ source="couchdata"
+ destination="/opt/couchdb/data"
+
+[env]
+ COUCHDB_USER = "campanella"
+ ERL_FLAGS="-couch_ini /opt/couchdb/etc/default.ini /opt/couchdb/etc/default.d/ /opt/couchdb/etc/local.d /opt/couchdb/etc/local.ini /opt/couchdb/data/persistence.ini"
+
+[build]
+ dockerfile = "./Dockerfile"
```
4. Make `Dockerfile`
Create a Dockerfile that patches the start-up script to fix ini file permissions.
```dockerfile
FROM couchdb:latest
RUN sed -i '2itouch /opt/couchdb/data/persistence.ini && chmod +w /opt/couchdb/data/persistence.ini' /docker-entrypoint.sh
```
5. Set credential
```
flyctl secrets set COUCHDB_PASSWORD=dfusiuada9suy
```
### 4. Deploy
```
$ flyctl deploy --detach --remote-only
Validating /home/vorotamoroz/dev/obsidian-livesync/utils/flyio/fly.toml
Platform: machines
✓ Configuration is valid
Your app is ready! Deploy with `flyctl deploy`
Secrets are staged for the first deployment
==> Verifying app config
Validating /home/vrtmrz/dev/fly/demo/fly.toml
Validating /home/vorotamoroz/dev/obsidian-livesync/utils/flyio/fly.toml
Platform: machines
✓ Configuration is valid
--> Verified app config
==> Building image
Remote builder fly-builder-bold-sky-4515 ready
==> Creating build context
--> Creating build context done
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
Searching for image 'couchdb:latest' remotely...
image found: img_ox20prk63084j1zq
-------------:SNIPPED:-------------
Watch your app at https://fly.io/apps/billowing-dawn-6619/monitoring
Watch your deployment at https://fly.io/apps/billowing-dawn-6619/monitoring
Provisioning ips for billowing-dawn-6619
Dedicated ipv6: 2a09:8280:1::2d:240f
Shared ipv4: 66.241.125.213
Dedicated ipv6: 2a09:8280:1::37:fde9
Shared ipv4: 66.241.124.163
Add a dedicated ipv4 with: fly ips allocate-v4
Creating a 1 GB volume named 'couchdata' for process group 'app'. Use 'fly vol extend' to increase its size
This deployment will:
* create 1 "app" machine
No machines in group app, launching a new machine
Machine e7845d1f297183 [app] has state: started
WARNING The app is not listening on the expected address and will not be reachable by fly-proxy.
You can fix this by configuring your app to listen on the following addresses:
- 0.0.0.0:5984
Found these processes inside the machine with open listening sockets:
PROCESS | ADDRESSES
-----------------*---------------------------------------
/.fly/hallpass | [fdaa:0:73b9:a7b:22e:3851:7f28:2]:22
Finished launching new machines
NOTE: The machines for [app] have services with 'auto_stop_machines = true' that will be stopped when idling
-------
Checking DNS configuration for billowing-dawn-6619.fly.dev
Visit your newly deployed app at https://billowing-dawn-6619.fly.dev/
```
Now your CouchDB has been launched. (Do not forget to delete it if no longer need).
If failed, please check by `flyctl doctor`. Failure of remote build may be resolved by `flyctl` wireguard reset` or something.
```
$ flyctl status
App
Name = billowing-dawn-6619
Owner = personal
Hostname = billowing-dawn-6619.fly.dev
Image = billowing-dawn-6619:deployment-01H1WWB3CK5Z9ZX71KHBSDGHF1
Platform = machines
Machines
PROCESS ID VERSION REGION STATE CHECKS LAST UPDATED
app e7845d1f297183 1 nrt started 2023-06-02T01:43:34Z
```
### 5. Apply CouchDB configuration
#### 5. Apply CouchDB configuration
After the initial setup, CouchDB needs some more customisations to be used from Self-hosted LiveSync. It can be configured in browsers or by HTTP-REST APIs.
@@ -273,15 +240,13 @@ iwr -UseBasicParsing -Method 'PUT' -ContentType 'application/json; charset=utf-8
Note: Each of these should also be repeated until finished in 200.
### 6. Use it from Self-hosted LiveSync
#### 6. Use it from Self-hosted LiveSync
Now the CouchDB is ready to use from Self-hosted LiveSync. We can use `https://billowing-dawn-6619.fly.dev` in URI, `campanella` in `Username` and `dfusiuada9suy` in `Password` on Self-hosted LiveSync. `Database name` could be anything you want.
`Enhance chunk size` could be up to around `100`.
Now the CouchDB is ready to use from Self-hosted LiveSync. We can use `https://billowing-dawn-6619.fly.dev` in URI, `campanella` in `Username` and `dfusiuada9suy` in `Password` on Self-hosted LiveSync. The `Database name` could be anything you want.
Please refer to the [Minimal Setup of the Quick Setup](./quick_setup.md#2-minimal-setup).
## Automatic setup using Colaboratory
## Delete the Instance
We can perform all these steps by using [this Colaboratory notebook](/deploy_couchdb_to_flyio_v2_with_swap.ipynb) without installing anything.
If you want to delete the CouchDB instance, you can do that in [fly.io Dashboard](https://fly.io/dashboard/personal)
## After testing / before creating a new instance
**Be sure to delete the instance. We can check instances on the [Dashboard](https://fly.io/dashboard/personal)**
If you have done with [B. Scripted Setup](#b-scripted-setup), we can use [delete-server.sh](../utils/readme.md#delete-serversh).

211
docs/setup_own_server.md
View File

@@ -1,138 +1,131 @@
# Setup a CouchDB server
## Table of Contents
- [Configure](#configure)
- [Run](#run)
- [Docker CLI](#docker-cli)
- [Docker Compose](#docker-compose)
- [Access from a mobile device](#access-from-a-mobile-device)
- [Testing from a mobile](#testing-from-a-mobile)
- [Setting up your domain](#setting-up-your-domain)
- [Reverse Proxies](#reverse-proxies)
- [Traefik](#traefik)
- [Setup a CouchDB server](#setup-a-couchdb-server)
- [Table of Contents](#table-of-contents)
- [1. Prepare CouchDB](#1-prepare-couchdb)
- [A. Using Docker container](#a-using-docker-container)
- [1. Prepare](#1-prepare)
- [2. Run docker container](#2-run-docker-container)
- [B. Install CouchDB directly](#b-install-couchdb-directly)
- [2. Run couchdb-init.sh for initialise](#2-run-couchdb-initsh-for-initialise)
- [3. Expose CouchDB to the Internet](#3-expose-couchdb-to-the-internet)
- [4. Client Setup](#4-client-setup)
- [1. Generate the setup URI on a desktop device or server](#1-generate-the-setup-uri-on-a-desktop-device-or-server)
- [2. Setup Self-hosted LiveSync to Obsidian](#2-setup-self-hosted-livesync-to-obsidian)
- [Manual setup information](#manual-setup-information)
- [Setting up your domain](#setting-up-your-domain)
- [Reverse Proxies](#reverse-proxies)
- [Traefik](#traefik)
---
## Configure
## 1. Prepare CouchDB
### A. Using Docker container
The easiest way to set up a CouchDB instance is using the official [docker image](https://hub.docker.com/_/couchdb).
#### 1. Prepare
```bash
Some initial configuration is required. Create a `local.ini` to use Self-hosted LiveSync as follows ([CouchDB has to be version 3.2 or higher](https://docs.couchdb.org/en/latest/config/http.html#chttpd/enable_cors), if lower `enable_cors = true` has to be under section `[httpd]` ):
# Prepare environment variables.
export hostname=localhost:5984
export username=goojdasjdas #Please change as you like.
export password=kpkdasdosakpdsa #Please change as you like
```ini
[couchdb]
single_node=true
max_document_size = 50000000
[chttpd]
require_valid_user = true
max_http_request_size = 4294967296
enable_cors = true
[chttpd_auth]
require_valid_user = true
authentication_redirect = /_utils/session.html
[httpd]
WWW-Authenticate = Basic realm="couchdb"
bind_address = 0.0.0.0
[cors]
origins = app://obsidian.md, capacitor://localhost, http://localhost
credentials = true
headers = accept, authorization, content-type, origin, referer
methods = GET,PUT,POST,HEAD,DELETE
max_age = 3600
# Prepare directories which saving data and configurations.
mkdir couchdb-data
mkdir couchdb-etc
```
## Run
#### 2. Run docker container
### Docker CLI
1. Boot Check.
```
$ docker run --name couchdb-for-ols --rm -it -e COUCHDB_USER=${username} -e COUCHDB_PASSWORD=${password} -v ${PWD}/couchdb-data:/opt/couchdb/data -v ${PWD}/couchdb-etc:/opt/couchdb/etc/local.d -p 5984:5984 couchdb
```
If your container has been exited, please check the permission of couchdb-data, and couchdb-etc.
Once CouchDB run, these directories will be owned by uid:`5984`. Please chown it for you again.
You can launch CouchDB using your `local.ini` like this:
2. Enable it in background
```
$ docker run --rm -it -e COUCHDB_USER=admin -e COUCHDB_PASSWORD=password -v /path/to/local.ini:/opt/couchdb/etc/local.ini -p 5984:5984 couchdb
$ docker run --name couchdb-for-ols -d --restart always -e COUCHDB_USER=${username} -e COUCHDB_PASSWORD=${password} -v ${PWD}/couchdb-data:/opt/couchdb/data -v ${PWD}/couchdb-etc:/opt/couchdb/etc/local.d -p 5984:5984 couchdb
```
*Remember to replace the path with the path to your local.ini*
### B. Install CouchDB directly
Please refer the [official document](https://docs.couchdb.org/en/stable/install/index.html). However, we do not have to configure it fully. Just administrator needs to be configured.
Run in detached mode:
## 2. Run couchdb-init.sh for initialise
```
$ docker run -d --restart always -e COUCHDB_USER=admin -e COUCHDB_PASSWORD=password -v /path/to/local.ini:/opt/couchdb/etc/local.ini -p 5984:5984 couchdb
```
*Remember to replace the path with the path to your local.ini*
### Docker Compose
Create a directory, place your `local.ini` within it, and create a `docker-compose.yml` alongside it. Make sure to have write permissions for `local.ini` and the about to be created `data` folder after the container start. The directory structure should look similar to this:
```
obsidian-livesync
├── docker-compose.yml
└── local.ini
$ curl -s https://raw.githubusercontent.com/vrtmrz/obsidian-livesync/main/utils/couchdb/couchdb-init.sh | bash
```
A good place to start for `docker-compose.yml`:
```yaml
version: "2.1"
services:
couchdb:
image: couchdb
container_name: obsidian-livesync
user: 1000:1000
environment:
- COUCHDB_USER=admin
- COUCHDB_PASSWORD=password
volumes:
- ./data:/opt/couchdb/data
- ./local.ini:/opt/couchdb/etc/local.ini
ports:
- 5984:5984
restart: unless-stopped
If it results like following:
```
-- Configuring CouchDB by REST APIs... -->
{"ok":true}
""
""
""
""
""
""
""
""
""
<-- Configuring CouchDB by REST APIs Done!
```
And finally launch the container
Your CouchDB has been initialised successfully. If you want this manually, please read the script.
## 3. Expose CouchDB to the Internet
- You can skip this instruction if you using only in intranet and only with desktop devices.
- For mobile devices, Obsidian requires a valid SSL certificate. Usually, it needs exposing the internet.
Whatever solutions we can use. For the simplicity, following sample uses Cloudflare Zero Trust for testing.
```
# -d will launch detached so the container runs in background
docker compose up -d
$ cloudflared tunnel --url http://localhost:5984
2024-02-14T10:35:25Z INF Thank you for trying Cloudflare Tunnel. Doing so, without a Cloudflare account, is a quick way to experiment and try it out. However, be aware that these account-less Tunnels have no uptime guarantee. If you intend to use Tunnels in production you should use a pre-created named tunnel by following: https://developers.cloudflare.com/cloudflare-one/connections/connect-apps
2024-02-14T10:35:25Z INF Requesting new quick Tunnel on trycloudflare.com...
2024-02-14T10:35:26Z INF +--------------------------------------------------------------------------------------------+
2024-02-14T10:35:26Z INF | Your quick Tunnel has been created! Visit it at (it may take some time to be reachable): |
2024-02-14T10:35:26Z INF | https://tiles-photograph-routine-groundwater.trycloudflare.com |
2024-02-14T10:35:26Z INF +--------------------------------------------------------------------------------------------+
:
:
:
```
Now `https://tiles-photograph-routine-groundwater.trycloudflare.com` is our server. Make it into background once please.
## 4. Client Setup
> [!TIP]
> Now manually configuration is not recommended for some reasons. However, if you want to do so, please use `Setup wizard`. The recommended extra configurations will be also set.
### 1. Generate the setup URI on a desktop device or server
```bash
$ export hostname=https://tiles-photograph-routine-groundwater.trycloudflare.com #Point to your vault
$ export database=obsidiannotes #Please change as you like
$ export passphrase=dfsapkdjaskdjasdas #Please change as you like
$ deno run -A https://raw.githubusercontent.com/vrtmrz/obsidian-livesync/main/utils/flyio/generate_setupuri.ts
obsidian://setuplivesync?settings=%5B%22tm2DpsOE74nJAryprZO2M93wF%2Fvg.......4b26ed33230729%22%5D
Your passphrase of Setup-URI is: patient-haze
This passphrase is never shown again, so please note it in a safe place.
```
## Access from a mobile device
If you want to access Self-hosted LiveSync from mobile devices, you need a valid SSL certificate.
Please keep your passphrase of Setup-URI.
### Testing from a mobile
In the testing phase, [localhost.run](https://localhost.run/) or something like services is very useful.
### 2. Setup Self-hosted LiveSync to Obsidian
[This video](https://youtu.be/7sa_I1832Xc?t=146) may help us.
1. Install Self-hosted LiveSync
2. Choose `Use the copied setup URI` from the command palette and paste the setup URI. (obsidian://setuplivesync?settings=.....).
3. Type the previously displayed passphrase (`patient-haze`) for setup-uri passphrase.
4. Answer `yes` and `Set it up...`, and finish the first dialogue with `Keep them disabled`.
5. `Reload app without save` once.
example using localhost.run:
```
$ ssh -R 80:localhost:5984 nokey@localhost.run
Warning: Permanently added the RSA host key for IP address '35.171.254.69' to the list of known hosts.
---
===============================================================================
Welcome to localhost.run!
Follow your favourite reverse tunnel at [https://twitter.com/localhost_run].
**You need a SSH key to access this service.**
If you get a permission denied follow Gitlab's most excellent howto:
https://docs.gitlab.com/ee/ssh/
*Only rsa and ed25519 keys are supported*
To set up and manage custom domains go to https://admin.localhost.run/
More details on custom domains (and how to enable subdomains of your custom
domain) at https://localhost.run/docs/custom-domains
To explore using localhost.run visit the documentation site:
https://localhost.run/docs/
===============================================================================
** your connection id is xxxxxxxxxxxxxxxxxxxxxxxxxxxx, please mention it if you send me a message about an issue. **
xxxxxxxx.localhost.run tunneled with tls termination, https://xxxxxxxx.localhost.run
Connection to localhost.run closed by remote host.
Connection to localhost.run closed.
```
https://xxxxxxxx.localhost.run is the temporary server address.
## Manual setup information
### Setting up your domain

2
docs/setup_own_server_cn.md
View File

@@ -91,7 +91,7 @@ services:
最后, 创建并启动容器:
```
# -d will launch detached so the container runs in background
docker compose up -d
docker-compose up -d
```
## 创建数据库

10
docs/terms.md Normal file
View File

@@ -0,0 +1,10 @@
# Terms used in this project
## Terms
### Chunks
<!-- TBW, sorry for the draft! -->
<!-- Please feel free to write any terms that should be mentioned. And please make pull request. I would love to fill the rest. -->
<!-- ### Chunks -->

156
docs/troubleshooting.md
View File

@@ -1,12 +1,148 @@
# Info
In this document some tips for solving issues will be given.
<!-- 2024-02-15 -->
# Tips and Troubleshooting
# Issue with syncing with mobile device
- If you are getting problem like could not sync files when doing initial sync. Try lower batch size
- Open setting
- Open Live Sync settings
- Go to sync settings
- Go down to Advanced settings.
- Lower "Batch size" and "Batch limit" and try to sync again.
- If you keep getting error keep lowering until you find the sweet spot.
- [Tips and Troubleshooting](#tips-and-troubleshooting)
- [Notable bugs and fixes](#notable-bugs-and-fixes)
- [Binary files get bigger on iOS](#binary-files-get-bigger-on-ios)
- [Some setting name has been changed](#some-setting-name-has-been-changed)
- [FAQ](#faq)
- [Why `Use an old adapter for compatibility` is somehow enabled in my vault?](#why-use-an-old-adapter-for-compatibility-is-somehow-enabled-in-my-vault)
- [ZIP (or any extensions) files were not synchronised. Why?](#zip-or-any-extensions-files-were-not-synchronised-why)
- [I hope to report the issue, but you said you needs `Report`. How to make it?](#i-hope-to-report-the-issue-but-you-said-you-needs-report-how-to-make-it)
- [Where can I check the log?](#where-can-i-check-the-log)
- [Why are the logs volatile and ephemeral?](#why-are-the-logs-volatile-and-ephemeral)
- [Some network logs are not written into the file.](#some-network-logs-are-not-written-into-the-file)
- [If a file were deleted or trimmed, the capacity of the database should be reduced, right?](#if-a-file-were-deleted-or-trimmed-the-capacity-of-the-database-should-be-reduced-right)
- [How can I use the DevTools?](#how-can-i-use-the-devtools)
- [Checking the network log](#checking-the-network-log)
- [Troubleshooting](#troubleshooting)
- [On the mobile device, cannot synchronise on the local network!](#on-the-mobile-device-cannot-synchronise-on-the-local-network)
- [I think that something bad happening on the vault...](#i-think-that-something-bad-happening-on-the-vault)
- [Tips](#tips)
- [How to resolve `Tweaks Mismatched of Changed`](#how-to-resolve-tweaks-mismatched-of-changed)
- [Old tips](#old-tips)
<!-- - -->
## Notable bugs and fixes
### Binary files get bigger on iOS
- Reported at: v0.20.x
- Fixed at: v0.21.2 (Fixed but not reviewed)
- Required action: larger files will not be fixed automatically, please perform `Verify and repair all files`. If our local database and storage are not matched, we will be asked to apply which one.
### Some setting name has been changed
- Fixed at: v0.22.6
| Previous name | New name |
| ---------------------------- | ---------------------------------------- |
| Open setup URI | Use the copied setup URI |
| Copy setup URI | Copy current settings as a new setup URI |
| Setup Wizard | Minimal Setup |
| Check database configuration | Check and Fix database configuration |
## FAQ
### Why `Use an old adapter for compatibility` is somehow enabled in my vault?
Because you are a compassionate and experienced user. Before v0.17.16, we used an old adapter for the local database. At that time, current default adapter has not been stable.
The new adapter has better performance and has a new feature like purging. Therefore, we should use new adapters and current default is so.
However, when switching from an old adapter to a new adapter, some converting or local database rebuilding is required, and it takes a few time. It was a long time ago now, but we once inconvenienced everyone in a hurry when we changed the format of our database.
For these reasons, this toggle is automatically on if we have upgraded from vault which using an old adapter.
When you rebuild everything or fetch from the remote again, you will be asked to switch this.
Therefore, experienced users (especially those stable enough not to have to rebuild the database) may have this toggle enabled in their Vault.
Please disable it when you have enough time.
### ZIP (or any extensions) files were not synchronised. Why?
It depends on Obsidian detects. May toggling `Detect all extensions` of `File and links` (setting of Obsidian) will help us.
### I hope to report the issue, but you said you needs `Report`. How to make it?
We can copy the report to the clipboard, by pressing the `Make report` button on the `Hatch` pane.
![Screenshot](../images/hatch.png)
### Where can I check the log?
We can launch the log pane by `Show log` on the command palette.
And if you have troubled something, please enable the `Verbose Log` on the `General Setting` pane.
However, the logs would not be kept so long and cleared when restarted. If you want to check the logs, please enable `Write logs into the file` temporarily.
![ScreenShot](../images/write_logs_into_the_file.png)
> [!IMPORTANT]
> - Writing logs into the file will impact the performance.
> - Please make sure that you have erased all your confidential information before reporting issue.
### Why are the logs volatile and ephemeral?
To avoid unexpected exposure to our confidential things.
### Some network logs are not written into the file.
Especially the CORS error will be reported as a general error to the plug-in for security reasons. So we cannot detect and log it. We are only able to investigate them by [Checking the network log](#checking-the-network-log).
### If a file were deleted or trimmed, the capacity of the database should be reduced, right?
No, even though if files were deleted, chunks were not deleted.
Self-hosted LiveSync splits the files into multiple chunks and transfers only newly created. This behaviour enables us to less traffic. And, the chunks will be shared between the files to reduce the total usage of the database.
And one more thing, we can handle the conflicts on any device even though it has happened on other devices. This means that conflicts will happen in the past, after the time we have synchronised. Hence we cannot collect and delete the unused chunks even though if we are not currently referenced.
To shrink the database size, `Rebuild everything` only reliably and effectively. But do not worry, if we have synchronised well. We have the actual and real files. Only it takes a bit of time and traffics.
### How can I use the DevTools?
#### Checking the network log
1. Open the network pane.
2. Find the requests marked in red.
![Errored](../images/devtools1.png)
3. Capture the `Headers`, `Payload`, and, `Response`. **Please be sure to keep important information confidential**. If the `Response` contains secrets, you can omitted that.
Note: Headers contains a some credentials. **The path of the request URL, Remote Address, authority, and authorization must be concealed.**
![Concealed sample](../images/devtools2.png)
## Troubleshooting
<!-- Add here -->
### On the mobile device, cannot synchronise on the local network!
Obsidian mobile is not able to connect to the non-secure end-point, such as starting with `http://`. Make sure your URI of CouchDB. Also not able to use a self-signed certificate.
### I think that something bad happening on the vault...
Place `redflag.md` on top of the vault, and restart Obsidian. The most simple way is to create a new note and rename it to `redflag`. Of course, we can put it without Obsidian.
If there is `redflag.md`, Self-hosted LiveSync suspends all database and storage processes.
## Tips
### How to resolve `Tweaks Mismatched of Changed`
(Since v0.23.17)
If you have changed some configurations or tweaks which should be unified between the devices, you will be asked how to reflect (or not) other devices at the next synchronisation. It also occurs on the device itself, where changes are made, to prevent unexpected configuration changes from unwanted propagation.
(We may thank this behaviour if we have synchronised or backed up and restored Self-hosted LiveSync. At least, for me so).
Following dialogue will be shown:
![Dialogue](tweak_mismatch_dialogue.png)
- If we want to propagate the setting of the device, we should choose `Update with mine`.
- On other devices, we should choose `Use configured` to accept and use the configured configuration.
- `Dismiss` can postpone a decision. However, we cannot synchronise until we have decided.
Rest assured that in most cases we can choose `Use configured`. (Unless you are certain that you have not changed the configuration).
If we see it for the first time, it reflects the settings of the device that has been synchronised with the remote for the first time since the upgrade. Probably, we can accept that.
<!-- Add here -->
### Old tips
- Rarely, a file in the database could be corrupted. The plugin will not write to local storage when a file looks corrupted. If a local version of the file is on your device, the corruption could be fixed by editing the local file and synchronizing it. But if the file does not exist on any of your devices, then it can not be rescued. In this case, you can delete these items from the settings dialog.
- To stop the boot-up sequence (eg. for fixing problems on databases), you can put a `redflag.md` file (or directory) at the root of your vault.
Tip for iOS: a redflag directory can be created at the root of the vault using the File application.
- Also, with `redflag2.md` placed, we can automatically rebuild both the local and the remote databases during the boot-up sequence. With `redflag3.md`, we can discard only the local database and fetch from the remote again.
- Q: The database is growing, how can I shrink it down?
A: each of the docs is saved with their past 100 revisions for detecting and resolving conflicts. Picturing that one device has been offline for a while, and comes online again. The device has to compare its notes with the remotely saved ones. If there exists a historic revision in which the note used to be identical, it could be updated safely (like git fast-forward). Even if that is not in revision histories, we only have to check the differences after the revision that both devices commonly have. This is like git's conflict-resolving method. So, We have to make the database again like an enlarged git repo if you want to solve the root of the problem.
- And more technical Information is in the [Technical Information](tech_info.md)
- If you want to synchronize files without obsidian, you can use [filesystem-livesync](https://github.com/vrtmrz/filesystem-livesync).
- WebClipper is also available on Chrome Web Store:[obsidian-livesync-webclip](https://chrome.google.com/webstore/detail/obsidian-livesync-webclip/jfpaflmpckblieefkegjncjoceapakdf)
Repo is here: [obsidian-livesync-webclip](https://github.com/vrtmrz/obsidian-livesync-webclip). (Docs are a work in progress.)

BIN
docs/tweak_mismatch_dialogue.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 46 KiB

119
esbuild.config.mjs
View File

@@ -8,26 +8,31 @@ import sveltePreprocess from "svelte-preprocess";
import fs from "node:fs";
// import terser from "terser";
import { minify } from "terser";
import inlineWorkerPlugin from "esbuild-plugin-inline-worker";
const banner = `/*
THIS IS A GENERATED/BUNDLED FILE BY ESBUILD AND TERSER
if you want to view the source, please visit the github repository of this plugin
*/
`;
const prod = process.argv[2] === "production";
const dev = process.argv[2] === "dev";
const keepTest = !prod || dev;
const terserOpt = {
sourceMap: (!prod ? {
url: "inline"
} : {}),
sourceMap: !prod
? {
url: "inline",
}
: {},
format: {
indent_level: 2,
beautify: true,
comments: "some",
ecma: 2018,
preamble: banner,
webkit: true
webkit: true,
},
parse: {
// parse options
@@ -52,16 +57,6 @@ const terserOpt = {
ecma: 2018,
unused: true,
},
// mangle: {
// // mangle options
// keep_classnames: true,
// keep_fnames: true,
// properties: {
// // mangle property options
// }
// },
ecma: 2018, // specify one of: 5, 2015, 2016, etc.
enclose: false, // or specify true, or "args:values"
@@ -71,42 +66,43 @@ const terserOpt = {
module: false,
// nameCache: null, // or specify a name cache object
safari10: false,
toplevel: false
}
toplevel: false,
};
const manifestJson = JSON.parse(fs.readFileSync("./manifest.json") + "");
const packageJson = JSON.parse(fs.readFileSync("./package.json") + "");
const updateInfo = JSON.stringify(fs.readFileSync("./updates.md") + "");
/** @type esbuild.Plugin[] */
const plugins = [{
name: 'my-plugin',
setup(build) {
let count = 0;
build.onEnd(async result => {
if (count++ === 0) {
console.log('first build:', result);
} else {
console.log('subsequent build:');
}
if (prod) {
console.log("Performing terser");
const src = fs.readFileSync("./main_org.js").toString();
// @ts-ignore
const ret = await minify(src, terserOpt);
if (ret && ret.code) {
fs.writeFileSync("./main.js", ret.code);
const plugins = [
{
name: "my-plugin",
setup(build) {
let count = 0;
build.onEnd(async (result) => {
if (count++ === 0) {
console.log("first build:", result);
} else {
console.log("subsequent build:");
}
console.log("Finished terser");
} else {
fs.copyFileSync("./main_org.js", "./main.js");
}
});
if (prod) {
console.log("Performing terser");
const src = fs.readFileSync("./main_org.js").toString();
// @ts-ignore
const ret = await minify(src, terserOpt);
if (ret && ret.code) {
fs.writeFileSync("./main.js", ret.code);
}
console.log("Finished terser");
} else {
fs.copyFileSync("./main_org.js", "./main.js");
}
});
},
},
}];
];
const externals = ["obsidian", "electron", "crypto", "@codemirror/autocomplete", "@codemirror/collab", "@codemirror/commands", "@codemirror/language", "@codemirror/lint", "@codemirror/search", "@codemirror/state", "@codemirror/view", "@lezer/common", "@lezer/highlight", "@lezer/lr"];
const context = await esbuild.context({
banner: {
js: banner,
@@ -114,26 +110,12 @@ const context = await esbuild.context({
entryPoints: ["src/main.ts"],
bundle: true,
define: {
"MANIFEST_VERSION": `"${manifestJson.version}"`,
"PACKAGE_VERSION": `"${packageJson.version}"`,
"UPDATE_INFO": `${updateInfo}`,
"global": "window",
MANIFEST_VERSION: `"${manifestJson.version}"`,
PACKAGE_VERSION: `"${packageJson.version}"`,
UPDATE_INFO: `${updateInfo}`,
global: "window",
},
external: [
"obsidian",
"electron",
"crypto",
"@codemirror/autocomplete",
"@codemirror/collab",
"@codemirror/commands",
"@codemirror/language",
"@codemirror/lint",
"@codemirror/search",
"@codemirror/state",
"@codemirror/view",
"@lezer/common",
"@lezer/highlight",
"@lezer/lr"],
external: externals,
// minifyWhitespace: true,
format: "cjs",
target: "es2018",
@@ -142,22 +124,27 @@ const context = await esbuild.context({
sourcemap: prod ? false : "inline",
treeShaking: true,
outfile: "main_org.js",
mainFields: ["browser", "module", "main"],
minifyWhitespace: false,
minifySyntax: false,
minifyIdentifiers: false,
minify: false,
dropLabels: prod && !keepTest ? ["TEST", "DEV"] : [],
// keepNames: true,
plugins: [
inlineWorkerPlugin({
external: externals,
treeShaking: true,
}),
sveltePlugin({
preprocess: sveltePreprocess(),
compilerOptions: { css: true, preserveComments: true },
compilerOptions: { css: "injected", preserveComments: false },
}),
...plugins
...plugins,
],
})
});
if (prod) {
if (prod || dev) {
await context.rebuild();
process.exit(0);
} else {

BIN
images/devtools1.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 11 KiB

BIN
images/devtools2.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 41 KiB

BIN
images/hatch.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 13 KiB

BIN
images/quick_setup_1.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 25 KiB

After

Width:  |  Height:  |  Size: 20 KiB

BIN
images/quick_setup_3b.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 74 KiB

BIN
images/quick_setup_5.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 36 KiB

After

Width:  |  Height:  |  Size: 5.7 KiB

BIN
images/quick_setup_7.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 10 KiB

BIN
images/write_logs_into_the_file.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 17 KiB

4
manifest.json
View File

@@ -1,10 +1,10 @@
{
"id": "obsidian-livesync",
"name": "Self-hosted LiveSync",
"version": "0.20.2",
"version": "0.23.19",
"minAppVersion": "0.9.12",
"description": "Community implementation of self-hosted livesync. Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.",
"author": "vorotamoroz",
"authorUrl": "https://github.com/vrtmrz",
"isDesktopOnly": false
}
}

7473
package-lock.json generated
View File

File diff suppressed because it is too large Load Diff

88
package.json
View File

@@ -1,63 +1,71 @@
{
"name": "obsidian-livesync",
"version": "0.20.2",
"version": "0.23.19",
"description": "Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.",
"main": "main.js",
"type": "module",
"scripts": {
"dev": "node esbuild.config.mjs",
"build": "node esbuild.config.mjs production",
"buildDev": "node esbuild.config.mjs dev",
"lint": "eslint src"
},
"keywords": [],
"author": "vorotamoroz",
"license": "MIT",
"devDependencies": {
"@tsconfig/svelte": "^5.0.0",
"@types/diff-match-patch": "^1.0.32",
"@types/node": "^20.2.5",
"@types/pouchdb": "^6.4.0",
"@types/pouchdb-browser": "^6.1.3",
"@types/pouchdb-adapter-http": "^6.1.3",
"@types/pouchdb-adapter-idb": "^6.1.4",
"@types/pouchdb-core": "^7.0.11",
"@types/pouchdb-mapreduce": "^6.1.7",
"@types/pouchdb-replication": "^6.4.4",
"@types/transform-pouch": "^1.0.2",
"@typescript-eslint/eslint-plugin": "^6.2.1",
"@typescript-eslint/parser": "^6.2.1",
"builtin-modules": "^3.3.0",
"esbuild": "0.18.17",
"esbuild-svelte": "^0.7.4",
"eslint": "^8.46.0",
"@tsconfig/svelte": "^5.0.4",
"@types/diff-match-patch": "^1.0.36",
"@types/node": "^20.14.10",
"@types/pouchdb": "^6.4.2",
"@types/pouchdb-adapter-http": "^6.1.6",
"@types/pouchdb-adapter-idb": "^6.1.7",
"@types/pouchdb-browser": "^6.1.5",
"@types/pouchdb-core": "^7.0.14",
"@types/pouchdb-mapreduce": "^6.1.10",
"@types/pouchdb-replication": "^6.4.7",
"@types/transform-pouch": "^1.0.6",
"@typescript-eslint/eslint-plugin": "^7.16.0",
"@typescript-eslint/parser": "^7.16.0",
"builtin-modules": "^4.0.0",
"esbuild": "0.23.0",
"esbuild-svelte": "^0.8.1",
"eslint": "^8.57.0",
"eslint-config-airbnb-base": "^15.0.0",
"eslint-plugin-import": "^2.28.0",
"eslint-plugin-import": "^2.29.1",
"events": "^3.3.0",
"obsidian": "^1.3.5",
"postcss": "^8.4.27",
"postcss-load-config": "^4.0.1",
"pouchdb-adapter-http": "^8.0.1",
"pouchdb-adapter-idb": "^8.0.1",
"pouchdb-adapter-indexeddb": "^8.0.1",
"pouchdb-core": "^8.0.1",
"pouchdb-errors": "^8.0.1",
"pouchdb-find": "^8.0.1",
"pouchdb-mapreduce": "^8.0.1",
"pouchdb-merge": "^8.0.1",
"pouchdb-replication": "^8.0.1",
"pouchdb-utils": "^8.0.1",
"svelte": "^4.1.2",
"svelte-preprocess": "^5.0.4",
"terser": "^5.19.2",
"obsidian": "^1.5.7",
"postcss": "^8.4.39",
"postcss-load-config": "^6.0.1",
"pouchdb-adapter-http": "^9.0.0",
"pouchdb-adapter-idb": "^9.0.0",
"pouchdb-adapter-indexeddb": "^9.0.0",
"pouchdb-core": "^9.0.0",
"pouchdb-errors": "^9.0.0",
"pouchdb-find": "^9.0.0",
"pouchdb-mapreduce": "^9.0.0",
"pouchdb-merge": "^9.0.0",
"pouchdb-replication": "^9.0.0",
"pouchdb-utils": "^9.0.0",
"svelte": "^4.2.18",
"svelte-preprocess": "^6.0.2",
"terser": "^5.31.2",
"transform-pouch": "^2.0.0",
"tslib": "^2.6.1",
"typescript": "^5.1.6"
"tslib": "^2.6.3",
"typescript": "^5.5.3"
},
"dependencies": {
"@aws-sdk/client-s3": "^3.621.0",
"@smithy/fetch-http-handler": "^3.2.1",
"@smithy/protocol-http": "^4.0.3",
"@smithy/querystring-builder": "^3.0.3",
"diff-match-patch": "^1.0.5",
"idb": "^7.1.1",
"minimatch": "^9.0.3",
"esbuild-plugin-inline-worker": "^0.1.1",
"fflate": "^0.8.2",
"idb": "^8.0.0",
"minimatch": "^10.0.1",
"octagonal-wheels": "^0.1.13",
"xxhash-wasm": "0.4.2",
"xxhash-wasm-102": "npm:xxhash-wasm@^1.0.2"
}
}
}

151
setup-flyio-on-the-fly-v2.ipynb Normal file
View File

@@ -0,0 +1,151 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "view-in-github"
},
"source": [
"<a href=\"https://colab.research.google.com/gist/vrtmrz/9402b101746e08e969b1a4f5f0deb465/setup-flyio-on-the-fly-v2.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "AzLlAcLFRO5A"
},
"source": [
"- Initial version 7th Feb. 2024"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "z1x8DQpa9opC"
},
"outputs": [],
"source": [
"# Install prerequesties\n",
"!curl -L https://fly.io/install.sh | sh\n",
"!curl -fsSL https://deno.land/x/install/install.sh | sh\n",
"!apt update && apt -y install jq\n",
"import os\n",
"%env PATH=/root/.fly/bin:/root/.deno/bin/:{os.environ[\"PATH\"]}\n",
"!git clone --recursive https://github.com/vrtmrz/obsidian-livesync"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "mGN08BaFDviy"
},
"outputs": [],
"source": [
"# Login up sign up\n",
"!flyctl auth signup"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BBFTFOP6vA8m"
},
"source": [
"Select a region and execute the block."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "TNl0A603EF9E"
},
"outputs": [],
"source": [
"# see https://fly.io/docs/reference/regions/\n",
"region = \"nrt/Tokyo, Japan\" #@param [\"ams/Amsterdam, Netherlands\",\"arn/Stockholm, Sweden\",\"atl/Atlanta, Georgia (US)\",\"bog/Bogotá, Colombia\",\"bos/Boston, Massachusetts (US)\",\"cdg/Paris, France\",\"den/Denver, Colorado (US)\",\"dfw/Dallas, Texas (US)\",\"ewr/Secaucus, NJ (US)\",\"eze/Ezeiza, Argentina\",\"gdl/Guadalajara, Mexico\",\"gig/Rio de Janeiro, Brazil\",\"gru/Sao Paulo, Brazil\",\"hkg/Hong Kong, Hong Kong\",\"iad/Ashburn, Virginia (US)\",\"jnb/Johannesburg, South Africa\",\"lax/Los Angeles, California (US)\",\"lhr/London, United Kingdom\",\"mad/Madrid, Spain\",\"mia/Miami, Florida (US)\",\"nrt/Tokyo, Japan\",\"ord/Chicago, Illinois (US)\",\"otp/Bucharest, Romania\",\"phx/Phoenix, Arizona (US)\",\"qro/Querétaro, Mexico\",\"scl/Santiago, Chile\",\"sea/Seattle, Washington (US)\",\"sin/Singapore, Singapore\",\"sjc/San Jose, California (US)\",\"syd/Sydney, Australia\",\"waw/Warsaw, Poland\",\"yul/Montreal, Canada\",\"yyz/Toronto, Canada\" ] {allow-input: true}\n",
"%env region={region.split(\"/\")[0]}\n",
"#%env appame=\n",
"#%env username=\n",
"#%env password=\n",
"#%env database=\n",
"#%env passphrase=\n",
"\n",
"# automatic setup leave it -->\n",
"%cd obsidian-livesync/utils/flyio\n",
"!./deploy-server.sh | tee deploy-result.txt\n",
"\n",
"## Show result button\n",
"from IPython.display import HTML\n",
"last_line=\"\"\n",
"with open('deploy-result.txt', 'r') as f:\n",
" last_line = f.readlines()[-1]\n",
" last_line = str.strip(last_line)\n",
"\n",
"if last_line.startswith(\"obsidian://\"):\n",
" result = HTML(f\"Copy your setup-URI with this button! -&gt; <button onclick=\\\"navigator.clipboard.writeText('{last_line}')\\\">Copy setup uri</button><br>Importing passphrase is displayed one. <br>If you want to synchronise in live mode, please apply a preset after ensuring the imported configuration works.\")\n",
"else:\n",
" result = \"Failed to encrypt the setup URI\"\n",
"result"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "oeIzExnEKhFp"
},
"source": [
"If you see the `Copy setup URI` button, Congratulations! Your CouchDB is ready to use! Please click the button. And open this on Obsidian.\n",
"\n",
"And, you should keep the output to your secret memo.\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sdQrqOjERN3K"
},
"source": [
"\n",
"\n",
"---\n",
"\n",
"\n",
"If you want to delete this CouchDB instance, you can do it by executing next cell. \n",
"If your fly.toml has been gone, access https://fly.io/dashboard and check the existing app."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "7JMSkNvVIIfg"
},
"outputs": [],
"source": [
"!./delete-server.sh"
]
}
],
"metadata": {
"colab": {
"authorship_tag": "ABX9TyMexQ5pErH5LBG2tENtEVWf",
"include_colab_link": true,
"private_outputs": true,
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 0
}

805
src/CmdConfigSync.ts
View File

@@ -1,805 +0,0 @@
import { writable } from 'svelte/store';
import { Notice, type PluginManifest, parseYaml, normalizePath } from "./deps";
import type { EntryDoc, LoadedEntry, InternalFileEntry, FilePathWithPrefix, FilePath, DocumentID, AnyEntry } from "./lib/src/types";
import { LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE, MODE_SELECTIVE } from "./lib/src/types";
import { ICXHeader, PERIODIC_PLUGIN_SWEEP, } from "./types";
import { delay, getDocData } from "./lib/src/utils";
import { Logger } from "./lib/src/logger";
import { WrappedNotice } from "./lib/src/wrapper";
import { readString, crc32CKHash, decodeBinary, encodeBinary } from "./lib/src/strbin";
import { serialized } from "./lib/src/lock";
import { LiveSyncCommands } from "./LiveSyncCommands";
import { stripAllPrefixes } from "./lib/src/path";
import { PeriodicProcessor, askYesNo, disposeMemoObject, memoIfNotExist, memoObject, retrieveMemoObject, scheduleTask } from "./utils";
import { PluginDialogModal } from "./dialogs";
import { JsonResolveModal } from "./JsonResolveModal";
import { pipeGeneratorToGenerator, processAllGeneratorTasksWithConcurrencyLimit } from './lib/src/task';
const d = "\u200b";
const d2 = "\n";
function serialize(data: PluginDataEx): string {
// For higher performance, create custom plug-in data strings.
// Self-hosted LiveSync uses `\n` to split chunks. Therefore, grouping together those with similar entropy would work nicely.
let ret = "";
ret += ":";
ret += data.category + d + data.name + d + data.term + d2;
ret += (data.version ?? "") + d2;
ret += data.mtime + d2;
for (const file of data.files) {
ret += file.filename + d + (file.displayName ?? "") + d + (file.version ?? "") + d2;
ret += file.mtime + d + file.size + d2;
for (const data of file.data ?? []) {
ret += data + d
}
ret += d2;
}
return ret;
}
function fetchToken(source: string, from: number): [next: number, token: string] {
const limitIdx = source.indexOf(d2, from);
const limit = limitIdx == -1 ? source.length : limitIdx;
const delimiterIdx = source.indexOf(d, from);
const delimiter = delimiterIdx == -1 ? source.length : delimiterIdx;
const tokenEnd = Math.min(limit, delimiter);
let next = tokenEnd;
if (limit < delimiter) {
next = tokenEnd;
} else {
next = tokenEnd + 1
}
return [next, source.substring(from, tokenEnd)];
}
function getTokenizer(source: string) {
const t = {
pos: 1,
next() {
const [next, token] = fetchToken(source, this.pos);
this.pos = next;
return token;
},
nextLine() {
const nextPos = source.indexOf(d2, this.pos);
if (nextPos == -1) {
this.pos = source.length;
} else {
this.pos = nextPos + 1;
}
}
}
return t;
}
function deserialize2(str: string): PluginDataEx {
const tokens = getTokenizer(str);
const ret = {} as PluginDataEx;
const category = tokens.next();
const name = tokens.next();
const term = tokens.next();
tokens.nextLine();
const version = tokens.next();
tokens.nextLine();
const mtime = Number(tokens.next());
tokens.nextLine();
const result: PluginDataEx = Object.assign(ret,
{ category, name, term, version, mtime, files: [] as PluginDataExFile[] })
let filename = "";
do {
filename = tokens.next();
if (!filename) break;
const displayName = tokens.next();
const version = tokens.next();
tokens.nextLine();
const mtime = Number(tokens.next());
const size = Number(tokens.next());
tokens.nextLine();
const data = [] as string[];
let piece = "";
do {
piece = tokens.next();
if (piece == "") break;
data.push(piece);
} while (piece != "");
result.files.push(
{
filename,
displayName,
version,
mtime,
size,
data
}
)
} while (filename);
return result;
}
function deserialize<T>(str: string, def: T) {
try {
if (str[0] == ":") return deserialize2(str);
return JSON.parse(str) as T;
} catch (ex) {
try {
return parseYaml(str);
} catch (ex) {
return def;
}
}
}
export const pluginList = writable([] as PluginDataExDisplay[]);
export const pluginIsEnumerating = writable(false);
export type PluginDataExFile = {
filename: string,
data?: string[],
mtime: number,
size: number,
version?: string,
displayName?: string,
}
export type PluginDataExDisplay = {
documentPath: FilePathWithPrefix,
category: string,
name: string,
term: string,
displayName?: string,
files: PluginDataExFile[],
version?: string,
mtime: number,
}
export type PluginDataEx = {
documentPath?: FilePathWithPrefix,
category: string,
name: string,
displayName?: string,
term: string,
files: PluginDataExFile[],
version?: string,
mtime: number,
};
export class ConfigSync extends LiveSyncCommands {
confirmPopup: WrappedNotice = null;
get kvDB() {
return this.plugin.kvDB;
}
ensureDirectoryEx(fullPath: string) {
return this.plugin.ensureDirectoryEx(fullPath);
}
pluginDialog: PluginDialogModal = null;
periodicPluginSweepProcessor = new PeriodicProcessor(this.plugin, async () => await this.scanAllConfigFiles(false));
pluginList: PluginDataExDisplay[] = [];
showPluginSyncModal() {
if (!this.settings.usePluginSync) {
return;
}
if (this.pluginDialog != null) {
this.pluginDialog.open();
} else {
this.pluginDialog = new PluginDialogModal(this.app, this.plugin);
this.pluginDialog.open();
}
}
hidePluginSyncModal() {
if (this.pluginDialog != null) {
this.pluginDialog.close();
this.pluginDialog = null;
}
}
onunload() {
this.hidePluginSyncModal();
this.periodicPluginSweepProcessor?.disable();
}
onload() {
this.plugin.addCommand({
id: "livesync-plugin-dialog-ex",
name: "Show customization sync dialog",
callback: () => {
this.showPluginSyncModal();
},
});
}
getFileCategory(filePath: string): "CONFIG" | "THEME" | "SNIPPET" | "PLUGIN_MAIN" | "PLUGIN_ETC" | "PLUGIN_DATA" | "" {
if (filePath.split("/").length == 2 && filePath.endsWith(".json")) return "CONFIG";
if (filePath.split("/").length == 4 && filePath.startsWith(`${this.app.vault.configDir}/themes/`)) return "THEME";
if (filePath.startsWith(`${this.app.vault.configDir}/snippets/`) && filePath.endsWith(".css")) return "SNIPPET";
if (filePath.startsWith(`${this.app.vault.configDir}/plugins/`)) {
if (filePath.endsWith("/styles.css") || filePath.endsWith("/manifest.json") || filePath.endsWith("/main.js")) {
return "PLUGIN_MAIN";
} else if (filePath.endsWith("/data.json")) {
return "PLUGIN_DATA";
} else {
//TODO: to be configurable.
// With algorithm which implemented at v0.19.0, is too heavy.
return "";
// return "PLUGIN_ETC";
}
// return "PLUGIN";
}
return "";
}
isTargetPath(filePath: string): boolean {
if (!filePath.startsWith(this.app.vault.configDir)) return false;
// Idea non-filter option?
return this.getFileCategory(filePath) != "";
}
async onInitializeDatabase(showNotice: boolean) {
if (this.settings.usePluginSync) {
try {
Logger("Scanning customizations...");
await this.scanAllConfigFiles(showNotice);
Logger("Scanning customizations : done");
} catch (ex) {
Logger("Scanning customizations : failed");
Logger(ex, LOG_LEVEL_VERBOSE);
}
}
}
async beforeReplicate(showNotice: boolean) {
if (this.settings.autoSweepPlugins && this.settings.usePluginSync) {
await this.scanAllConfigFiles(showNotice);
}
}
async onResume() {
if (this.plugin.suspended) {
return;
}
if (this.settings.autoSweepPlugins && this.settings.usePluginSync) {
await this.scanAllConfigFiles(false);
}
this.periodicPluginSweepProcessor.enable(this.settings.autoSweepPluginsPeriodic && !this.settings.watchInternalFileChanges ? (PERIODIC_PLUGIN_SWEEP * 1000) : 0);
}
async reloadPluginList(showMessage: boolean) {
this.pluginList = [];
pluginList.set(this.pluginList)
await this.updatePluginList(showMessage);
}
async loadPluginData(path: FilePathWithPrefix): Promise<PluginDataExDisplay | false> {
const wx = await this.localDatabase.getDBEntry(path, null, false, false);
if (wx) {
const data = deserialize(getDocData(wx.data), {}) as PluginDataEx;
const xFiles = [] as PluginDataExFile[];
for (const file of data.files) {
const work = { ...file };
const tempStr = getDocData(work.data);
work.data = [crc32CKHash(tempStr)];
xFiles.push(work);
}
return ({
...data,
documentPath: this.getPath(wx),
files: xFiles
}) as PluginDataExDisplay;
}
return false;
}
createMissingConfigurationEntry() {
let saveRequired = false;
for (const v of this.pluginList) {
const key = `${v.category}/${v.name}`;
if (!(key in this.plugin.settings.pluginSyncExtendedSetting)) {
this.plugin.settings.pluginSyncExtendedSetting[key] = {
key,
mode: MODE_SELECTIVE,
files: []
}
}
if (this.plugin.settings.pluginSyncExtendedSetting[key].files.sort().join(",").toLowerCase() !=
v.files.map(e => e.filename).sort().join(",").toLowerCase()) {
this.plugin.settings.pluginSyncExtendedSetting[key].files = v.files.map(e => e.filename).sort();
saveRequired = true;
}
}
if (saveRequired) {
this.plugin.saveSettingData();
}
}
async updatePluginList(showMessage: boolean, updatedDocumentPath?: FilePathWithPrefix): Promise<void> {
const logLevel = showMessage ? LOG_LEVEL_NOTICE : LOG_LEVEL_INFO;
// pluginList.set([]);
if (!this.settings.usePluginSync) {
this.pluginList = [];
pluginList.set(this.pluginList)
return;
}
await Promise.resolve(); // Just to prevent warning.
scheduleTask("update-plugin-list-task", 200, async () => {
await serialized("update-plugin-list", async () => {
try {
const updatedDocumentId = updatedDocumentPath ? await this.path2id(updatedDocumentPath) : "";
const plugins = updatedDocumentPath ?
this.localDatabase.findEntries(updatedDocumentId, updatedDocumentId + "\u{10ffff}", { include_docs: true, key: updatedDocumentId, limit: 1 }) :
this.localDatabase.findEntries(ICXHeader + "", `${ICXHeader}\u{10ffff}`, { include_docs: true });
let count = 0;
pluginIsEnumerating.set(true);
for await (const v of processAllGeneratorTasksWithConcurrencyLimit(20, pipeGeneratorToGenerator(plugins, async plugin => {
const path = plugin.path || this.getPath(plugin);
if (updatedDocumentPath && updatedDocumentPath != path) {
return false;
}
const oldEntry = (this.pluginList.find(e => e.documentPath == path));
if (oldEntry && oldEntry.mtime == plugin.mtime) return false;
try {
count++;
if (count % 10 == 0) Logger(`Enumerating files... ${count}`, logLevel, "get-plugins");
Logger(`plugin-${path}`, LOG_LEVEL_VERBOSE);
return this.loadPluginData(path);
// return entries;
} catch (ex) {
//TODO
Logger(`Something happened at enumerating customization :${path}`, LOG_LEVEL_NOTICE);
console.warn(ex);
}
return false;
}))) {
if ("ok" in v) {
if (v.ok !== false) {
let newList = [...this.pluginList];
const item = v.ok;
newList = newList.filter(x => x.documentPath != item.documentPath);
newList.push(item)
if (updatedDocumentPath != "") newList = newList.filter(e => e.documentPath != updatedDocumentPath);
this.pluginList = newList;
pluginList.set(newList);
}
}
}
Logger(`All files enumerated`, logLevel, "get-plugins");
this.createMissingConfigurationEntry();
} finally {
pluginIsEnumerating.set(false);
}
});
});
// return entries;
}
async compareUsingDisplayData(dataA: PluginDataExDisplay, dataB: PluginDataExDisplay) {
const docA = await this.localDatabase.getDBEntry(dataA.documentPath);
const docB = await this.localDatabase.getDBEntry(dataB.documentPath);
if (docA && docB) {
const pluginDataA = deserialize(getDocData(docA.data), {}) as PluginDataEx;
pluginDataA.documentPath = dataA.documentPath;
const pluginDataB = deserialize(getDocData(docB.data), {}) as PluginDataEx;
pluginDataB.documentPath = dataB.documentPath;
// Use outer structure to wrap each data.
return await this.showJSONMergeDialogAndMerge(docA, docB, pluginDataA, pluginDataB);
}
return false;
}
showJSONMergeDialogAndMerge(docA: LoadedEntry, docB: LoadedEntry, pluginDataA: PluginDataEx, pluginDataB: PluginDataEx): Promise<boolean> {
const fileA = { ...pluginDataA.files[0], ctime: pluginDataA.files[0].mtime, _id: `${pluginDataA.documentPath}` as DocumentID };
const fileB = pluginDataB.files[0];
const docAx = { ...docA, ...fileA } as LoadedEntry, docBx = { ...docB, ...fileB } as LoadedEntry
return serialized("config:merge-data", () => new Promise((res) => {
Logger("Opening data-merging dialog", LOG_LEVEL_VERBOSE);
// const docs = [docA, docB];
const path = stripAllPrefixes(docAx.path.split("/").slice(-1).join("/") as FilePath);
const modal = new JsonResolveModal(this.app, path, [docAx, docBx], async (keep, result) => {
if (result == null) return res(false);
try {
res(await this.applyData(pluginDataA, result));
} catch (ex) {
Logger("Could not apply merged file");
Logger(ex, LOG_LEVEL_VERBOSE);
res(false);
}
}, "📡", "🛰️", "B");
modal.open();
}));
}
async applyData(data: PluginDataEx, content?: string): Promise<boolean> {
Logger(`Applying ${data.displayName || data.name}..`);
const baseDir = this.app.vault.configDir;
try {
if (!data.documentPath) throw "InternalError: Document path not exist";
const dx = await this.localDatabase.getDBEntry(data.documentPath);
if (dx == false) {
throw "Not found on database"
}
const loadedData = deserialize(getDocData(dx.data), {}) as PluginDataEx;
for (const f of loadedData.files) {
Logger(`Applying ${f.filename} of ${data.displayName || data.name}..`);
try {
// console.dir(f);
const path = `${baseDir}/${f.filename}`;
await this.ensureDirectoryEx(path);
if (!content) {
const dt = decodeBinary(f.data);
await this.app.vault.adapter.writeBinary(path, dt);
} else {
await this.app.vault.adapter.write(path, content);
}
Logger(`Applying ${f.filename} of ${data.displayName || data.name}.. Done`);
} catch (ex) {
Logger(`Applying ${f.filename} of ${data.displayName || data.name}.. Failed`);
Logger(ex, LOG_LEVEL_VERBOSE);
}
}
const uPath = `${baseDir}/${loadedData.files[0].filename}` as FilePath;
await this.storeCustomizationFiles(uPath);
await this.updatePluginList(true, uPath);
await delay(100);
Logger(`Config ${data.displayName || data.name} has been applied`, LOG_LEVEL_NOTICE);
if (data.category == "PLUGIN_DATA" || data.category == "PLUGIN_MAIN") {
//@ts-ignore
const manifests = Object.values(this.app.plugins.manifests) as any as PluginManifest[];
//@ts-ignore
const enabledPlugins = this.app.plugins.enabledPlugins as Set<string>;
const pluginManifest = manifests.find((manifest) => enabledPlugins.has(manifest.id) && manifest.dir == `${baseDir}/plugins/${data.name}`);
if (pluginManifest) {
Logger(`Unloading plugin: ${pluginManifest.name}`, LOG_LEVEL_NOTICE, "plugin-reload-" + pluginManifest.id);
// @ts-ignore
await this.app.plugins.unloadPlugin(pluginManifest.id);
// @ts-ignore
await this.app.plugins.loadPlugin(pluginManifest.id);
Logger(`Plugin reloaded: ${pluginManifest.name}`, LOG_LEVEL_NOTICE, "plugin-reload-" + pluginManifest.id);
}
} else if (data.category == "CONFIG") {
scheduleTask("configReload", 250, async () => {
if (await askYesNo(this.app, "Do you want to restart and reload Obsidian now?") == "yes") {
// @ts-ignore
this.app.commands.executeCommandById("app:reload")
}
})
}
return true;
} catch (ex) {
Logger(`Applying ${data.displayName || data.name}.. Failed`);
Logger(ex, LOG_LEVEL_VERBOSE);
return false;
}
}
async deleteData(data: PluginDataEx): Promise<boolean> {
try {
if (data.documentPath) {
await this.deleteConfigOnDatabase(data.documentPath);
await this.updatePluginList(false, data.documentPath);
Logger(`Delete: ${data.documentPath}`, LOG_LEVEL_NOTICE);
}
return true;
} catch (ex) {
Logger(`Failed to delete: ${data.documentPath}`, LOG_LEVEL_NOTICE);
return false;
}
}
async parseReplicationResultItem(docs: PouchDB.Core.ExistingDocument<EntryDoc>) {
if (docs._id.startsWith(ICXHeader)) {
if (this.plugin.settings.usePluginSync) {
await this.updatePluginList(false, (docs as AnyEntry).path ? (docs as AnyEntry).path : this.getPath((docs as AnyEntry)));
}
if (this.plugin.settings.usePluginSync && this.plugin.settings.notifyPluginOrSettingUpdated) {
if (!this.pluginDialog || (this.pluginDialog && !this.pluginDialog.isOpened())) {
const fragment = createFragment((doc) => {
doc.createEl("span", null, (a) => {
a.appendText(`Some configuration has been arrived, Press `);
a.appendChild(a.createEl("a", null, (anchor) => {
anchor.text = "HERE";
anchor.addEventListener("click", () => {
this.showPluginSyncModal();
});
}));
a.appendText(` to open the config sync dialog , or press elsewhere to dismiss this message.`);
});
});
const updatedPluginKey = "popupUpdated-plugins";
scheduleTask(updatedPluginKey, 1000, async () => {
const popup = await memoIfNotExist(updatedPluginKey, () => new Notice(fragment, 0));
//@ts-ignore
const isShown = popup?.noticeEl?.isShown();
if (!isShown) {
memoObject(updatedPluginKey, new Notice(fragment, 0));
}
scheduleTask(updatedPluginKey + "-close", 20000, () => {
const popup = retrieveMemoObject<Notice>(updatedPluginKey);
if (!popup)
return;
//@ts-ignore
if (popup?.noticeEl?.isShown()) {
popup.hide();
}
disposeMemoObject(updatedPluginKey);
});
});
}
}
return true;
}
return false;
}
async realizeSettingSyncMode(): Promise<void> {
this.periodicPluginSweepProcessor?.disable();
if (this.plugin.suspended)
return;
if (!this.settings.usePluginSync) {
return;
}
if (this.settings.autoSweepPlugins) {
await this.scanAllConfigFiles(false);
}
this.periodicPluginSweepProcessor.enable(this.settings.autoSweepPluginsPeriodic && !this.settings.watchInternalFileChanges ? (PERIODIC_PLUGIN_SWEEP * 1000) : 0);
return;
}
recentProcessedInternalFiles = [] as string[];
async makeEntryFromFile(path: FilePath): Promise<false | PluginDataExFile> {
const stat = await this.app.vault.adapter.stat(path);
let version: string | undefined;
let displayName: string | undefined;
if (!stat) {
return false;
}
const contentBin = await this.app.vault.adapter.readBinary(path);
let content: string[];
try {
content = await encodeBinary(contentBin, this.settings.useV1);
if (path.toLowerCase().endsWith("/manifest.json")) {
const v = readString(new Uint8Array(contentBin));
try {
const json = JSON.parse(v);
if ("version" in json) {
version = `${json.version}`;
}
if ("name" in json) {
displayName = `${json.name}`;
}
} catch (ex) {
Logger(`Configuration sync data: ${path} looks like manifest, but could not read the version`, LOG_LEVEL_INFO);
}
}
} catch (ex) {
Logger(`The file ${path} could not be encoded`);
Logger(ex, LOG_LEVEL_VERBOSE);
return false;
}
const mtime = stat.mtime;
return {
filename: path.substring(this.app.vault.configDir.length + 1),
data: content,
mtime,
size: stat.size,
version,
displayName: displayName,
}
}
filenameToUnifiedKey(path: string, termOverRide?: string) {
const term = termOverRide || this.plugin.deviceAndVaultName;
const category = this.getFileCategory(path);
const name = (category == "CONFIG" || category == "SNIPPET") ?
(path.split("/").slice(-1)[0]) :
(category == "PLUGIN_ETC" ?
path.split("/").slice(-2).join("/") :
path.split("/").slice(-2)[0]);
return `${ICXHeader}${term}/${category}/${name}.md` as FilePathWithPrefix
}
async storeCustomizationFiles(path: FilePath, termOverRide?: string) {
const term = termOverRide || this.plugin.deviceAndVaultName;
if (term == "") {
Logger("We have to configure the device name", LOG_LEVEL_NOTICE);
return;
}
const vf = this.filenameToUnifiedKey(path, term);
return await serialized(`plugin-${vf}`, async () => {
const category = this.getFileCategory(path);
let mtime = 0;
let fileTargets = [] as FilePath[];
// let savePath = "";
const name = (category == "CONFIG" || category == "SNIPPET") ?
(path.split("/").reverse()[0]) :
(path.split("/").reverse()[1]);
const parentPath = path.split("/").slice(0, -1).join("/");
const prefixedFileName = this.filenameToUnifiedKey(path, term);
const id = await this.path2id(prefixedFileName);
const dt: PluginDataEx = {
category: category,
files: [],
name: name,
mtime: 0,
term: term
}
// let scheduleKey = "";
if (category == "CONFIG" || category == "SNIPPET" || category == "PLUGIN_ETC" || category == "PLUGIN_DATA") {
fileTargets = [path];
if (category == "PLUGIN_ETC") {
dt.displayName = path.split("/").slice(-1).join("/");
}
} else if (category == "PLUGIN_MAIN") {
fileTargets = ["manifest.json", "main.js", "styles.css"].map(e => `${parentPath}/${e}` as FilePath);
} else if (category == "THEME") {
fileTargets = ["manifest.json", "theme.css"].map(e => `${parentPath}/${e}` as FilePath);
}
for (const target of fileTargets) {
const data = await this.makeEntryFromFile(target);
if (data == false) {
// Logger(`Config: skipped: ${target} `, LOG_LEVEL_VERBOSE);
continue;
}
if (data.version) {
dt.version = data.version;
}
if (data.displayName) {
dt.displayName = data.displayName;
}
// Use average for total modified time.
mtime = mtime == 0 ? data.mtime : ((data.mtime + mtime) / 2);
dt.files.push(data);
}
dt.mtime = mtime;
// Logger(`Configuration saving: ${prefixedFileName}`);
if (dt.files.length == 0) {
Logger(`Nothing left: deleting.. ${path}`);
await this.deleteConfigOnDatabase(prefixedFileName);
await this.updatePluginList(false, prefixedFileName);
return
}
const content = serialize(dt);
try {
const old = await this.localDatabase.getDBEntryMeta(prefixedFileName, null, false);
let saveData: LoadedEntry;
if (old === false) {
saveData = {
_id: id,
path: prefixedFileName,
data: content,
mtime,
ctime: mtime,
datatype: "newnote",
size: content.length,
children: [],
deleted: false,
type: "newnote",
};
} else {
if (old.mtime == mtime) {
// Logger(`STORAGE --> DB:${file.path}: (hidden) Not changed`, LOG_LEVEL_VERBOSE);
return true;
}
saveData =
{
...old,
data: content,
mtime,
size: content.length,
datatype: "newnote",
children: [],
deleted: false,
type: "newnote",
};
}
const ret = await this.localDatabase.putDBEntry(saveData);
await this.updatePluginList(false, saveData.path);
Logger(`STORAGE --> DB:${prefixedFileName}: (config) Done`);
return ret;
} catch (ex) {
Logger(`STORAGE --> DB:${prefixedFileName}: (config) Failed`);
Logger(ex, LOG_LEVEL_VERBOSE);
return false;
}
})
}
async watchVaultRawEventsAsync(path: FilePath) {
if (!this.settings.usePluginSync) return false;
if (!this.isTargetPath(path)) return false;
const stat = await this.app.vault.adapter.stat(path);
// Make sure that target is a file.
if (stat && stat.type != "file")
return false;
const configDir = normalizePath(this.app.vault.configDir);
const synchronisedInConfigSync = Object.values(this.settings.pluginSyncExtendedSetting).filter(e => e.mode != MODE_SELECTIVE).map(e => e.files).flat().map(e => `${configDir}/${e}`.toLowerCase());
if (synchronisedInConfigSync.some(e => e.startsWith(path.toLowerCase()))) {
Logger(`Customization file skipped: ${path}`, LOG_LEVEL_VERBOSE);
return;
}
const storageMTime = ~~((stat && stat.mtime || 0) / 1000);
const key = `${path}-${storageMTime}`;
if (this.recentProcessedInternalFiles.contains(key)) {
// If recently processed, it may caused by self.
return true;
}
this.recentProcessedInternalFiles = [key, ...this.recentProcessedInternalFiles].slice(0, 100);
this.storeCustomizationFiles(path).then(() => {/* Fire and forget */ });
}
async scanAllConfigFiles(showMessage: boolean) {
const logLevel = showMessage ? LOG_LEVEL_NOTICE : LOG_LEVEL_INFO;
Logger("Scanning customizing files.", logLevel, "scan-all-config");
const term = this.plugin.deviceAndVaultName;
if (term == "") {
Logger("We have to configure the device name", LOG_LEVEL_NOTICE);
return;
}
const filesAll = await this.scanInternalFiles();
const files = filesAll.filter(e => this.isTargetPath(e)).map(e => ({ key: this.filenameToUnifiedKey(e), file: e }));
const virtualPathsOfLocalFiles = [...new Set(files.map(e => e.key))];
const filesOnDB = ((await this.localDatabase.allDocsRaw({ startkey: ICXHeader + "", endkey: `${ICXHeader}\u{10ffff}`, include_docs: true })).rows.map(e => e.doc) as InternalFileEntry[]).filter(e => !e.deleted);
let deleteCandidate = filesOnDB.map(e => this.getPath(e)).filter(e => e.startsWith(`${ICXHeader}${term}/`));
for (const vp of virtualPathsOfLocalFiles) {
const p = files.find(e => e.key == vp).file;
await this.storeCustomizationFiles(p);
deleteCandidate = deleteCandidate.filter(e => e != vp);
}
for (const vp of deleteCandidate) {
await this.deleteConfigOnDatabase(vp);
}
this.updatePluginList(false).then(/* fire and forget */);
}
async deleteConfigOnDatabase(prefixedFileName: FilePathWithPrefix, forceWrite = false) {
// const id = await this.path2id(prefixedFileName);
const mtime = new Date().getTime();
await serialized("file-x-" + prefixedFileName, async () => {
try {
const old = await this.localDatabase.getDBEntryMeta(prefixedFileName, null, false) as InternalFileEntry | false;
let saveData: InternalFileEntry;
if (old === false) {
Logger(`STORAGE -x> DB:${prefixedFileName}: (config) already deleted (Not found on database)`);
} else {
if (old.deleted) {
Logger(`STORAGE -x> DB:${prefixedFileName}: (config) already deleted`);
return;
}
saveData =
{
...old,
mtime,
size: 0,
children: [],
deleted: true,
type: "newnote",
};
}
await this.localDatabase.putRaw(saveData);
await this.updatePluginList(false, prefixedFileName);
Logger(`STORAGE -x> DB:${prefixedFileName}: (config) Done`);
} catch (ex) {
Logger(`STORAGE -x> DB:${prefixedFileName}: (config) Failed`);
Logger(ex, LOG_LEVEL_VERBOSE);
return false;
}
});
}
async scanInternalFiles(): Promise<FilePath[]> {
const filenames = (await this.getFiles(this.app.vault.configDir, 2)).filter(e => e.startsWith(".")).filter(e => !e.startsWith(".trash"));
return filenames as FilePath[];
}
async getFiles(
path: string,
lastDepth: number
) {
if (lastDepth == -1) return [];
const w = await this.app.vault.adapter.list(path);
let files = [
...w.files
];
for (const v of w.folders) {
files = files.concat(await this.getFiles(v, lastDepth - 1));
}
return files;
}
}

316
src/CmdPluginAndTheirSettings.ts
View File

@@ -1,316 +0,0 @@
import { normalizePath, type PluginManifest } from "./deps";
import type { DocumentID, EntryDoc, FilePathWithPrefix, LoadedEntry } from "./lib/src/types";
import { LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE } from "./lib/src/types";
import { type PluginDataEntry, PERIODIC_PLUGIN_SWEEP, type PluginList, type DevicePluginList, PSCHeader, PSCHeaderEnd } from "./types";
import { getDocData, isDocContentSame } from "./lib/src/utils";
import { Logger } from "./lib/src/logger";
import { PouchDB } from "./lib/src/pouchdb-browser.js";
import { isPluginMetadata, PeriodicProcessor } from "./utils";
import { PluginDialogModal } from "./dialogs";
import { NewNotice } from "./lib/src/wrapper";
import { versionNumberString2Number } from "./lib/src/strbin";
import { serialized, skipIfDuplicated } from "./lib/src/lock";
import { LiveSyncCommands } from "./LiveSyncCommands";
export class PluginAndTheirSettings extends LiveSyncCommands {
get deviceAndVaultName() {
return this.plugin.deviceAndVaultName;
}
pluginDialog: PluginDialogModal = null;
periodicPluginSweepProcessor = new PeriodicProcessor(this.plugin, async () => await this.sweepPlugin(false));
showPluginSyncModal() {
if (this.pluginDialog != null) {
this.pluginDialog.open();
} else {
this.pluginDialog = new PluginDialogModal(this.app, this.plugin);
this.pluginDialog.open();
}
}
hidePluginSyncModal() {
if (this.pluginDialog != null) {
this.pluginDialog.close();
this.pluginDialog = null;
}
}
onload(): void | Promise<void> {
this.plugin.addCommand({
id: "livesync-plugin-dialog",
name: "Show Plugins and their settings",
callback: () => {
this.showPluginSyncModal();
},
});
this.showPluginSyncModal();
}
onunload() {
this.hidePluginSyncModal();
this.periodicPluginSweepProcessor?.disable();
}
parseReplicationResultItem(doc: PouchDB.Core.ExistingDocument<EntryDoc>) {
if (isPluginMetadata(doc._id)) {
if (this.settings.notifyPluginOrSettingUpdated) {
this.triggerCheckPluginUpdate();
return true;
}
}
return false;
}
async beforeReplicate(showMessage: boolean) {
if (this.settings.autoSweepPlugins) {
await this.sweepPlugin(showMessage);
}
}
async onResume() {
if (this.plugin.suspended)
return;
if (this.settings.autoSweepPlugins) {
await this.sweepPlugin(false);
}
this.periodicPluginSweepProcessor.enable(this.settings.autoSweepPluginsPeriodic && !this.settings.watchInternalFileChanges ? (PERIODIC_PLUGIN_SWEEP * 1000) : 0);
}
async onInitializeDatabase(showNotice: boolean) {
if (this.settings.usePluginSync) {
try {
Logger("Scanning plugins...");
await this.sweepPlugin(showNotice);
Logger("Scanning plugins done");
} catch (ex) {
Logger("Scanning plugins failed");
Logger(ex, LOG_LEVEL_VERBOSE);
}
}
}
async realizeSettingSyncMode() {
this.periodicPluginSweepProcessor?.disable();
if (this.plugin.suspended)
return;
if (this.settings.autoSweepPlugins) {
await this.sweepPlugin(false);
}
this.periodicPluginSweepProcessor.enable(this.settings.autoSweepPluginsPeriodic && !this.settings.watchInternalFileChanges ? (PERIODIC_PLUGIN_SWEEP * 1000) : 0);
}
triggerCheckPluginUpdate() {
(async () => await this.checkPluginUpdate())();
}
async getPluginList(): Promise<{ plugins: PluginList; allPlugins: DevicePluginList; thisDevicePlugins: DevicePluginList; }> {
const docList = await this.localDatabase.allDocsRaw<PluginDataEntry>({ startkey: PSCHeader, endkey: PSCHeaderEnd, include_docs: false });
const oldDocs: PluginDataEntry[] = ((await Promise.all(docList.rows.map(async (e) => await this.localDatabase.getDBEntry(e.id as FilePathWithPrefix /* WARN!! THIS SHOULD BE WRAPPED */)))).filter((e) => e !== false) as LoadedEntry[]).map((e) => JSON.parse(getDocData(e.data)));
const plugins: { [key: string]: PluginDataEntry[]; } = {};
const allPlugins: { [key: string]: PluginDataEntry; } = {};
const thisDevicePlugins: { [key: string]: PluginDataEntry; } = {};
for (const v of oldDocs) {
if (typeof plugins[v.deviceVaultName] === "undefined") {
plugins[v.deviceVaultName] = [];
}
plugins[v.deviceVaultName].push(v);
allPlugins[v._id] = v;
if (v.deviceVaultName == this.deviceAndVaultName) {
thisDevicePlugins[v.manifest.id] = v;
}
}
return { plugins, allPlugins, thisDevicePlugins };
}
async checkPluginUpdate() {
if (!this.plugin.settings.usePluginSync)
return;
await this.sweepPlugin(false);
const { allPlugins, thisDevicePlugins } = await this.getPluginList();
const arrPlugins = Object.values(allPlugins);
let updateFound = false;
for (const plugin of arrPlugins) {
const ownPlugin = thisDevicePlugins[plugin.manifest.id];
if (ownPlugin) {
const remoteVersion = versionNumberString2Number(plugin.manifest.version);
const ownVersion = versionNumberString2Number(ownPlugin.manifest.version);
if (remoteVersion > ownVersion) {
updateFound = true;
}
if (((plugin.mtime / 1000) | 0) > ((ownPlugin.mtime / 1000) | 0) && (plugin.dataJson ?? "") != (ownPlugin.dataJson ?? "")) {
updateFound = true;
}
}
}
if (updateFound) {
const fragment = createFragment((doc) => {
doc.createEl("a", null, (a) => {
a.text = "There're some new plugins or their settings";
a.addEventListener("click", () => this.showPluginSyncModal());
});
});
NewNotice(fragment, 10000);
} else {
Logger("Everything is up to date.", LOG_LEVEL_NOTICE);
}
}
async sweepPlugin(showMessage = false, specificPluginPath = "") {
if (!this.settings.usePluginSync)
return;
if (!this.localDatabase.isReady)
return;
// @ts-ignore
const pl = this.app.plugins;
const manifests: PluginManifest[] = Object.values(pl.manifests);
let specificPlugin = "";
if (specificPluginPath != "") {
specificPlugin = manifests.find(e => e.dir.endsWith("/" + specificPluginPath))?.id ?? "";
}
await skipIfDuplicated("sweepplugin", async () => {
const logLevel = showMessage ? LOG_LEVEL_NOTICE : LOG_LEVEL_INFO;
if (!this.deviceAndVaultName) {
Logger("You have to set your device name.", LOG_LEVEL_NOTICE);
return;
}
Logger("Scanning plugins", logLevel);
const oldDocs = await this.localDatabase.allDocsRaw<EntryDoc>({
startkey: `ps:${this.deviceAndVaultName}-${specificPlugin}`,
endkey: `ps:${this.deviceAndVaultName}-${specificPlugin}\u{10ffff}`,
include_docs: true,
});
// Logger("OLD DOCS.", LOG_LEVEL_VERBOSE);
// sweep current plugin.
const procs = manifests.map(async (m) => {
const pluginDataEntryID = `ps:${this.deviceAndVaultName}-${m.id}` as DocumentID;
try {
if (specificPlugin && m.id != specificPlugin) {
return;
}
Logger(`Reading plugin:${m.name}(${m.id})`, LOG_LEVEL_VERBOSE);
const path = normalizePath(m.dir) + "/";
const adapter = this.app.vault.adapter;
const files = ["manifest.json", "main.js", "styles.css", "data.json"];
const pluginData: { [key: string]: string; } = {};
for (const file of files) {
const thePath = path + file;
if (await adapter.exists(thePath)) {
pluginData[file] = await adapter.read(thePath);
}
}
let mtime = 0;
if (await adapter.exists(path + "/data.json")) {
mtime = (await adapter.stat(path + "/data.json")).mtime;
}
const p: PluginDataEntry = {
_id: pluginDataEntryID,
dataJson: pluginData["data.json"],
deviceVaultName: this.deviceAndVaultName,
mainJs: pluginData["main.js"],
styleCss: pluginData["styles.css"],
manifest: m,
manifestJson: pluginData["manifest.json"],
mtime: mtime,
type: "plugin",
};
const d: LoadedEntry = {
_id: p._id,
path: p._id as string as FilePathWithPrefix,
data: JSON.stringify(p),
ctime: mtime,
mtime: mtime,
size: 0,
children: [],
datatype: "plain",
type: "plain"
};
Logger(`check diff:${m.name}(${m.id})`, LOG_LEVEL_VERBOSE);
await serialized("plugin-" + m.id, async () => {
const old = await this.localDatabase.getDBEntry(p._id as string as FilePathWithPrefix /* This also should be explained */, null, false, false);
if (old !== false) {
const oldData = { data: old.data, deleted: old._deleted };
const newData = { data: d.data, deleted: d._deleted };
if (isDocContentSame(oldData.data, newData.data) && oldData.deleted == newData.deleted) {
Logger(`Nothing changed:${m.name}`);
return;
}
}
await this.localDatabase.putDBEntry(d);
Logger(`Plugin saved:${m.name}`, logLevel);
});
} catch (ex) {
Logger(`Plugin save failed:${m.name}`, LOG_LEVEL_NOTICE);
} finally {
oldDocs.rows = oldDocs.rows.filter((e) => e.id != pluginDataEntryID);
}
//remove saved plugin data.
}
);
await Promise.all(procs);
const delDocs = oldDocs.rows.map((e) => {
// e.doc._deleted = true;
if (e.doc.type == "newnote" || e.doc.type == "plain") {
e.doc.deleted = true;
if (this.settings.deleteMetadataOfDeletedFiles) {
e.doc._deleted = true;
}
} else {
e.doc._deleted = true;
}
return e.doc;
});
Logger(`Deleting old plugin:(${delDocs.length})`, LOG_LEVEL_VERBOSE);
await this.localDatabase.bulkDocsRaw(delDocs);
Logger(`Scan plugin done.`, logLevel);
});
}
async applyPluginData(plugin: PluginDataEntry) {
await serialized("plugin-" + plugin.manifest.id, async () => {
const pluginTargetFolderPath = normalizePath(plugin.manifest.dir) + "/";
const adapter = this.app.vault.adapter;
// @ts-ignore
const stat = this.app.plugins.enabledPlugins.has(plugin.manifest.id) == true;
if (stat) {
// @ts-ignore
await this.app.plugins.unloadPlugin(plugin.manifest.id);
Logger(`Unload plugin:${plugin.manifest.id}`, LOG_LEVEL_NOTICE);
}
if (plugin.dataJson)
await adapter.write(pluginTargetFolderPath + "data.json", plugin.dataJson);
Logger("wrote:" + pluginTargetFolderPath + "data.json", LOG_LEVEL_NOTICE);
if (stat) {
// @ts-ignore
await this.app.plugins.loadPlugin(plugin.manifest.id);
Logger(`Load plugin:${plugin.manifest.id}`, LOG_LEVEL_NOTICE);
}
});
}
async applyPlugin(plugin: PluginDataEntry) {
await serialized("plugin-" + plugin.manifest.id, async () => {
// @ts-ignore
const stat = this.app.plugins.enabledPlugins.has(plugin.manifest.id) == true;
if (stat) {
// @ts-ignore
await this.app.plugins.unloadPlugin(plugin.manifest.id);
Logger(`Unload plugin:${plugin.manifest.id}`, LOG_LEVEL_NOTICE);
}
const pluginTargetFolderPath = normalizePath(plugin.manifest.dir) + "/";
const adapter = this.app.vault.adapter;
if ((await adapter.exists(pluginTargetFolderPath)) === false) {
await adapter.mkdir(pluginTargetFolderPath);
}
await adapter.write(pluginTargetFolderPath + "main.js", plugin.mainJs);
await adapter.write(pluginTargetFolderPath + "manifest.json", plugin.manifestJson);
if (plugin.styleCss)
await adapter.write(pluginTargetFolderPath + "styles.css", plugin.styleCss);
if (stat) {
// @ts-ignore
await this.app.plugins.loadPlugin(plugin.manifest.id);
Logger(`Load plugin:${plugin.manifest.id}`, LOG_LEVEL_NOTICE);
}
});
}
}

85
src/ConflictResolveModal.ts
View File

@@ -1,85 +0,0 @@
import { App, Modal } from "./deps";
import { DIFF_DELETE, DIFF_EQUAL, DIFF_INSERT } from "diff-match-patch";
import { type diff_result } from "./lib/src/types";
import { escapeStringToHTML } from "./lib/src/strbin";
export class ConflictResolveModal extends Modal {
// result: Array<[number, string]>;
result: diff_result;
filename: string;
callback: (remove_rev: string) => Promise<void>;
constructor(app: App, filename: string, diff: diff_result, callback: (remove_rev: string) => Promise<void>) {
super(app);
this.result = diff;
this.callback = callback;
this.filename = filename;
}
onOpen() {
const { contentEl } = this;
this.titleEl.setText("Conflicting changes");
contentEl.empty();
contentEl.createEl("span", { text: this.filename });
const div = contentEl.createDiv("");
div.addClass("op-scrollable");
let diff = "";
for (const v of this.result.diff) {
const x1 = v[0];
const x2 = v[1];
if (x1 == DIFF_DELETE) {
diff += "<span class='deleted'>" + escapeStringToHTML(x2).replace(/\n/g, "<span class='ls-mark-cr'></span>\n") + "</span>";
} else if (x1 == DIFF_EQUAL) {
diff += "<span class='normal'>" + escapeStringToHTML(x2).replace(/\n/g, "<span class='ls-mark-cr'></span>\n") + "</span>";
} else if (x1 == DIFF_INSERT) {
diff += "<span class='added'>" + escapeStringToHTML(x2).replace(/\n/g, "<span class='ls-mark-cr'></span>\n") + "</span>";
}
}
diff = diff.replace(/\n/g, "<br>");
div.innerHTML = diff;
const div2 = contentEl.createDiv("");
const date1 = new Date(this.result.left.mtime).toLocaleString() + (this.result.left.deleted ? " (Deleted)" : "");
const date2 = new Date(this.result.right.mtime).toLocaleString() + (this.result.right.deleted ? " (Deleted)" : "");
div2.innerHTML = `
<span class='deleted'>A:${date1}</span><br /><span class='added'>B:${date2}</span><br>
`;
contentEl.createEl("button", { text: "Keep A" }, (e) => {
e.addEventListener("click", async () => {
const callback = this.callback;
this.callback = null;
this.close();
await callback(this.result.right.rev);
});
});
contentEl.createEl("button", { text: "Keep B" }, (e) => {
e.addEventListener("click", async () => {
const callback = this.callback;
this.callback = null;
this.close();
await callback(this.result.left.rev);
});
});
contentEl.createEl("button", { text: "Concat both" }, (e) => {
e.addEventListener("click", async () => {
const callback = this.callback;
this.callback = null;
this.close();
await callback("");
});
});
contentEl.createEl("button", { text: "Not now" }, (e) => {
e.addEventListener("click", () => {
this.close();
});
});
}
onClose() {
const { contentEl } = this;
contentEl.empty();
if (this.callback != null) {
this.callback(null);
}
}
}

64
src/JsonResolveModal.ts
View File

@@ -1,64 +0,0 @@
import { App, Modal } from "./deps";
import { type FilePath, type LoadedEntry } from "./lib/src/types";
import JsonResolvePane from "./JsonResolvePane.svelte";
export class JsonResolveModal extends Modal {
// result: Array<[number, string]>;
filename: FilePath;
callback: (keepRev: string, mergedStr?: string) => Promise<void>;
docs: LoadedEntry[];
component: JsonResolvePane;
nameA: string;
nameB: string;
defaultSelect: string;
constructor(app: App, filename: FilePath, docs: LoadedEntry[], callback: (keepRev: string, mergedStr?: string) => Promise<void>, nameA?: string, nameB?: string, defaultSelect?: string) {
super(app);
this.callback = callback;
this.filename = filename;
this.docs = docs;
this.nameA = nameA;
this.nameB = nameB;
this.defaultSelect = defaultSelect;
}
async UICallback(keepRev: string, mergedStr?: string) {
this.close();
await this.callback(keepRev, mergedStr);
this.callback = null;
}
onOpen() {
const { contentEl } = this;
this.titleEl.setText("Conflicted Setting");
contentEl.empty();
if (this.component == null) {
this.component = new JsonResolvePane({
target: contentEl,
props: {
docs: this.docs,
filename: this.filename,
nameA: this.nameA,
nameB: this.nameB,
defaultSelect: this.defaultSelect,
callback: (keepRev: string, mergedStr: string) => this.UICallback(keepRev, mergedStr),
},
});
}
return;
}
onClose() {
const { contentEl } = this;
contentEl.empty();
// contentEl.empty();
if (this.callback != null) {
this.callback(null);
}
if (this.component != null) {
this.component.$destroy();
this.component = null;
}
}
}

38
src/LogDisplayModal.ts
View File

@@ -1,38 +0,0 @@
import { App, Modal } from "./deps";
import { logMessageStore } from "./lib/src/stores";
import { escapeStringToHTML } from "./lib/src/strbin";
import ObsidianLiveSyncPlugin from "./main";
export class LogDisplayModal extends Modal {
plugin: ObsidianLiveSyncPlugin;
logEl: HTMLDivElement;
unsubscribe: () => void;
constructor(app: App, plugin: ObsidianLiveSyncPlugin) {
super(app);
this.plugin = plugin;
}
onOpen() {
const { contentEl } = this;
this.titleEl.setText("Sync status");
contentEl.empty();
const div = contentEl.createDiv("");
div.addClass("op-scrollable");
div.addClass("op-pre");
this.logEl = div;
this.unsubscribe = logMessageStore.observe((e) => {
let msg = "";
for (const v of e) {
msg += escapeStringToHTML(v) + "<br>";
}
this.logEl.innerHTML = msg;
})
logMessageStore.invalidate();
}
onClose() {
const { contentEl } = this;
contentEl.empty();
if (this.unsubscribe) this.unsubscribe();
}
}

2050
src/ObsidianLiveSyncSettingTab.ts
View File

File diff suppressed because it is too large Load Diff

426
src/PluginPane.svelte
View File

@@ -1,426 +0,0 @@
<script lang="ts">
import { onMount } from "svelte";
import ObsidianLiveSyncPlugin from "./main";
import { type PluginDataExDisplay, pluginIsEnumerating, pluginList } from "./CmdConfigSync";
import PluginCombo from "./PluginCombo.svelte";
import { Menu } from "obsidian";
import { unique } from "./lib/src/utils";
import { MODE_SELECTIVE, MODE_AUTOMATIC, MODE_PAUSED, type SYNC_MODE, type PluginSyncSettingEntry } from "./lib/src/types";
import { normalizePath } from "./deps";
export let plugin: ObsidianLiveSyncPlugin;
$: hideNotApplicable = false;
$: thisTerm = plugin.deviceAndVaultName;
const addOn = plugin.addOnConfigSync;
let list: PluginDataExDisplay[] = [];
let selectNewestPulse = 0;
let hideEven = false;
let loading = false;
let applyAllPluse = 0;
let isMaintenanceMode = false;
async function requestUpdate() {
await addOn.updatePluginList(true);
}
async function requestReload() {
await addOn.reloadPluginList(true);
}
pluginList.subscribe((e) => {
list = e;
});
pluginIsEnumerating.subscribe((e) => {
loading = e;
});
onMount(async () => {
requestUpdate();
});
function filterList(list: PluginDataExDisplay[], categories: string[]) {
const w = list.filter((e) => categories.indexOf(e.category) !== -1);
return w.sort((a, b) => `${a.category}-${a.name}`.localeCompare(`${b.category}-${b.name}`));
}
function groupBy(items: PluginDataExDisplay[], key: string) {
let ret = {} as Record<string, PluginDataExDisplay[]>;
for (const v of items) {
//@ts-ignore
const k = (key in v ? v[key] : "") as string;
ret[k] = ret[k] || [];
ret[k].push(v);
}
for (const k in ret) {
ret[k] = ret[k].sort((a, b) => `${a.category}-${a.name}`.localeCompare(`${b.category}-${b.name}`));
}
const w = Object.entries(ret);
return w.sort(([a], [b]) => `${a}`.localeCompare(`${b}`));
}
const displays = {
CONFIG: "Configuration",
THEME: "Themes",
SNIPPET: "Snippets",
};
async function scanAgain() {
await addOn.scanAllConfigFiles(true);
await requestUpdate();
}
async function replicate() {
await plugin.replicate(true);
}
function selectAllNewest() {
selectNewestPulse++;
}
function applyAll() {
applyAllPluse++;
}
async function applyData(data: PluginDataExDisplay): Promise<boolean> {
return await addOn.applyData(data);
}
async function compareData(docA: PluginDataExDisplay, docB: PluginDataExDisplay): Promise<boolean> {
return await addOn.compareUsingDisplayData(docA, docB);
}
async function deleteData(data: PluginDataExDisplay): Promise<boolean> {
return await addOn.deleteData(data);
}
function askMode(evt: MouseEvent, title: string, key: string) {
const menu = new Menu();
menu.addItem((item) => item.setTitle(title).setIsLabel(true));
menu.addSeparator();
const prevMode = automaticList.get(key) ?? MODE_SELECTIVE;
for (const mode of [MODE_SELECTIVE, MODE_AUTOMATIC, MODE_PAUSED]) {
menu.addItem((item) => {
item.setTitle(`${getIcon(mode as SYNC_MODE)}:${TITLES[mode]}`)
.onClick((e) => {
if (mode === MODE_AUTOMATIC) {
askOverwriteModeForAutomatic(evt, key);
} else {
setMode(key, mode as SYNC_MODE);
}
})
.setChecked(prevMode == mode)
.setDisabled(prevMode == mode);
});
}
menu.showAtMouseEvent(evt);
}
function applyAutomaticSync(key: string, direction: "pushForce" | "pullForce" | "safe") {
setMode(key, MODE_AUTOMATIC);
const configDir = normalizePath(plugin.app.vault.configDir);
const files = (plugin.settings.pluginSyncExtendedSetting[key]?.files ?? []).map((e) => `${configDir}/${e}`);
plugin.addOnHiddenFileSync.syncInternalFilesAndDatabase(direction, true, false, files);
}
function askOverwriteModeForAutomatic(evt: MouseEvent, key: string) {
const menu = new Menu();
menu.addItem((item) => item.setTitle("Initial Action").setIsLabel(true));
menu.addSeparator();
menu.addItem((item) => {
item.setTitle(`↑: Overwrite Remote`).onClick((e) => {
applyAutomaticSync(key, "pushForce");
});
})
.addItem((item) => {
item.setTitle(`↓: Overwrite Local`).onClick((e) => {
applyAutomaticSync(key, "pullForce");
});
})
.addItem((item) => {
item.setTitle(`⇅: Use newer`).onClick((e) => {
applyAutomaticSync(key, "safe");
});
});
menu.showAtMouseEvent(evt);
}
$: options = {
thisTerm,
hideNotApplicable,
selectNewest: selectNewestPulse,
applyAllPluse,
applyData,
compareData,
deleteData,
plugin,
isMaintenanceMode,
};
const ICON_EMOJI_PAUSED = `⛔`;
const ICON_EMOJI_AUTOMATIC = `✨`;
const ICON_EMOJI_SELECTIVE = `🔀`;
const ICONS: { [key: number]: string } = {
[MODE_SELECTIVE]: ICON_EMOJI_SELECTIVE,
[MODE_PAUSED]: ICON_EMOJI_PAUSED,
[MODE_AUTOMATIC]: ICON_EMOJI_AUTOMATIC,
};
const TITLES: { [key: number]: string } = {
[MODE_SELECTIVE]: "Selective",
[MODE_PAUSED]: "Ignore",
[MODE_AUTOMATIC]: "Automatic",
};
const PREFIX_PLUGIN_ALL = "PLUGIN_ALL";
const PREFIX_PLUGIN_DATA = "PLUGIN_DATA";
const PREFIX_PLUGIN_MAIN = "PLUGIN_MAIN";
function setMode(key: string, mode: SYNC_MODE) {
if (key.startsWith(PREFIX_PLUGIN_ALL + "/")) {
setMode(PREFIX_PLUGIN_DATA + key.substring(PREFIX_PLUGIN_ALL.length), mode);
setMode(PREFIX_PLUGIN_MAIN + key.substring(PREFIX_PLUGIN_ALL.length), mode);
}
const files = unique(
list
.filter((e) => `${e.category}/${e.name}` == key)
.map((e) => e.files)
.flat()
.map((e) => e.filename)
);
automaticList.set(key, mode);
automaticListDisp = automaticList;
if (!(key in plugin.settings.pluginSyncExtendedSetting)) {
plugin.settings.pluginSyncExtendedSetting[key] = {
key,
mode,
files: [],
};
}
plugin.settings.pluginSyncExtendedSetting[key].files = files;
plugin.settings.pluginSyncExtendedSetting[key].mode = mode;
plugin.saveSettingData();
}
function getIcon(mode: SYNC_MODE) {
if (mode in ICONS) {
return ICONS[mode];
} else {
("");
}
}
let automaticList = new Map<string, SYNC_MODE>();
let automaticListDisp = new Map<string, SYNC_MODE>();
// apply current configuration to the dialogue
for (const { key, mode } of Object.values(plugin.settings.pluginSyncExtendedSetting)) {
automaticList.set(key, mode);
}
automaticListDisp = automaticList;
let displayKeys: Record<string, string[]> = {};
$: {
const extraKeys = Object.keys(plugin.settings.pluginSyncExtendedSetting);
displayKeys = [
...list,
...extraKeys
.map((e) => `${e}///`.split("/"))
.filter((e) => e[0] && e[1])
.map((e) => ({ category: e[0], name: e[1], displayName: e[1] })),
]
.sort((a, b) => (a.displayName ?? a.name).localeCompare(b.displayName ?? b.name))
.reduce((p, c) => ({ ...p, [c.category]: unique(c.category in p ? [...p[c.category], c.displayName ?? c.name] : [c.displayName ?? c.name]) }), {} as Record<string, string[]>);
}
</script>
<div>
<div>
<div class="buttons">
<button on:click={() => scanAgain()}>Scan changes</button>
<button on:click={() => replicate()}>Sync once</button>
<button on:click={() => requestUpdate()}>Refresh</button>
{#if isMaintenanceMode}
<button on:click={() => requestReload()}>Reload</button>
{/if}
<button on:click={() => selectAllNewest()}>Select All Shiny</button>
</div>
<div class="buttons">
<button on:click={() => applyAll()}>Apply All</button>
</div>
</div>
{#if loading}
<div>
<span>Updating list...</span>
</div>
{/if}
<div class="list">
{#if list.length == 0}
<div class="center">No Items.</div>
{:else}
{#each Object.entries(displays).filter(([key, _]) => key in displayKeys) as [key, label]}
<div>
<h3>{label}</h3>
{#each displayKeys[key] as name}
{@const bindKey = `${key}/${name}`}
{@const mode = automaticListDisp.get(bindKey) ?? MODE_SELECTIVE}
<div class="labelrow {hideEven ? 'hideeven' : ''}">
<div class="title">
<button class="status" on:click={(evt) => askMode(evt, `${key}/${name}`, bindKey)}>
{getIcon(mode)}
</button>
<span class="name">{name}</span>
</div>
{#if mode == MODE_SELECTIVE}
<PluginCombo {...options} list={list.filter((e) => e.category == key && e.name == name)} hidden={false} />
{:else}
<div class="statusnote">{TITLES[mode]}</div>
{/if}
</div>
{/each}
</div>
{/each}
<div>
<h3>Plugins</h3>
{#each groupBy(filterList(list, ["PLUGIN_MAIN", "PLUGIN_DATA", "PLUGIN_ETC"]), "name") as [name, listX]}
{@const bindKeyAll = `${PREFIX_PLUGIN_ALL}/${name}`}
{@const modeAll = automaticListDisp.get(bindKeyAll) ?? MODE_SELECTIVE}
{@const bindKeyMain = `${PREFIX_PLUGIN_MAIN}/${name}`}
{@const modeMain = automaticListDisp.get(bindKeyMain) ?? MODE_SELECTIVE}
{@const bindKeyData = `${PREFIX_PLUGIN_DATA}/${name}`}
{@const modeData = automaticListDisp.get(bindKeyData) ?? MODE_SELECTIVE}
<div class="labelrow {hideEven ? 'hideeven' : ''}">
<div class="title">
<button class="status" on:click={(evt) => askMode(evt, `${PREFIX_PLUGIN_ALL}/${name}`, bindKeyAll)}>
{getIcon(modeAll)}
</button>
<span class="name">{name}</span>
</div>
{#if modeAll == MODE_SELECTIVE}
<PluginCombo {...options} list={listX} hidden={true} />
{/if}
</div>
{#if modeAll == MODE_SELECTIVE}
<div class="filerow {hideEven ? 'hideeven' : ''}">
<div class="filetitle">
<button class="status" on:click={(evt) => askMode(evt, `${PREFIX_PLUGIN_MAIN}/${name}/MAIN`, bindKeyMain)}>
{getIcon(modeMain)}
</button>
<span class="name">MAIN</span>
</div>
{#if modeMain == MODE_SELECTIVE}
<PluginCombo {...options} list={filterList(listX, ["PLUGIN_MAIN"])} hidden={false} />
{:else}
<div class="statusnote">{TITLES[modeMain]}</div>
{/if}
</div>
<div class="filerow {hideEven ? 'hideeven' : ''}">
<div class="filetitle">
<button class="status" on:click={(evt) => askMode(evt, `${PREFIX_PLUGIN_DATA}/${name}`, bindKeyData)}>
{getIcon(modeData)}
</button>
<span class="name">DATA</span>
</div>
{#if modeData == MODE_SELECTIVE}
<PluginCombo {...options} list={filterList(listX, ["PLUGIN_DATA"])} hidden={false} />
{:else}
<div class="statusnote">{TITLES[modeData]}</div>
{/if}
</div>
{:else}
<div class="noterow">
<div class="statusnote">{TITLES[modeAll]}</div>
</div>
{/if}
{/each}
</div>
{/if}
</div>
<div class="buttons">
<label><span>Hide not applicable items</span><input type="checkbox" bind:checked={hideEven} /></label>
</div>
<div class="buttons">
<label><span>Maintenance mode</span><input type="checkbox" bind:checked={isMaintenanceMode} /></label>
</div>
</div>
<style>
span.spacer {
min-width: 1px;
flex-grow: 1;
}
h3 {
position: sticky;
top: 0;
background-color: var(--modal-background);
}
.labelrow {
margin-left: 0.4em;
display: flex;
justify-content: flex-start;
align-items: center;
border-top: 1px solid var(--background-modifier-border);
padding: 4px;
flex-wrap: wrap;
}
.filerow {
margin-left: 1.25em;
display: flex;
justify-content: flex-start;
align-items: center;
padding-right: 4px;
flex-wrap: wrap;
}
.filerow.hideeven:has(.even),
.labelrow.hideeven:has(.even) {
display: none;
}
.noterow {
min-height: 2em;
display: flex;
}
button.status {
flex-grow: 0;
margin: 2px 4px;
min-width: 3em;
max-width: 4em;
}
.statusnote {
display: flex;
justify-content: flex-end;
padding-right: var(--size-4-12);
align-items: center;
min-width: 10em;
flex-grow: 1;
}
.title {
color: var(--text-normal);
font-size: var(--font-ui-medium);
line-height: var(--line-height-tight);
margin-right: auto;
}
.filetitle {
color: var(--text-normal);
font-size: var(--font-ui-medium);
line-height: var(--line-height-tight);
margin-right: auto;
}
.buttons {
display: flex;
flex-direction: row;
justify-content: flex-end;
margin-top: 8px;
flex-wrap: wrap;
}
.buttons > button {
margin-left: 4px;
width: auto;
}
label {
display: flex;
justify-content: center;
align-items: center;
}
label > span {
margin-right: 0.25em;
}
:global(.is-mobile) .title,
:global(.is-mobile) .filetitle {
width: 100%;
}
.center {
display: flex;
justify-content: center;
align-items: center;
min-height: 3em;
}
</style>

181
src/StorageEventManager.ts
View File

@@ -1,181 +0,0 @@
import { Plugin, TAbstractFile, TFile, TFolder } from "./deps";
import { isPlainText, shouldBeIgnored } from "./lib/src/path";
import { getGlobalStore } from "./lib/src/store";
import { type FilePath, type ObsidianLiveSyncSettings } from "./lib/src/types";
import { type FileEventItem, type FileEventType, type FileInfo, type InternalFileInfo, type queueItem } from "./types";
import { recentlyTouched } from "./utils";
export abstract class StorageEventManager {
abstract fetchEvent(): FileEventItem | false;
abstract cancelRelativeEvent(item: FileEventItem): void;
abstract getQueueLength(): number;
}
type LiveSyncForStorageEventManager = Plugin &
{
settings: ObsidianLiveSyncSettings
ignoreFiles: string[],
} & {
isTargetFile: (file: string | TAbstractFile) => Promise<boolean>,
procFileEvent: (applyBatch?: boolean) => Promise<any>,
};
export class StorageEventManagerObsidian extends StorageEventManager {
plugin: LiveSyncForStorageEventManager;
queuedFilesStore = getGlobalStore("queuedFiles", { queuedItems: [] as queueItem[], fileEventItems: [] as FileEventItem[] });
watchedFileEventQueue = [] as FileEventItem[];
constructor(plugin: LiveSyncForStorageEventManager) {
super();
this.plugin = plugin;
this.watchVaultChange = this.watchVaultChange.bind(this);
this.watchVaultCreate = this.watchVaultCreate.bind(this);
this.watchVaultDelete = this.watchVaultDelete.bind(this);
this.watchVaultRename = this.watchVaultRename.bind(this);
this.watchVaultRawEvents = this.watchVaultRawEvents.bind(this);
plugin.registerEvent(plugin.app.vault.on("modify", this.watchVaultChange));
plugin.registerEvent(plugin.app.vault.on("delete", this.watchVaultDelete));
plugin.registerEvent(plugin.app.vault.on("rename", this.watchVaultRename));
plugin.registerEvent(plugin.app.vault.on("create", this.watchVaultCreate));
//@ts-ignore : Internal API
plugin.registerEvent(plugin.app.vault.on("raw", this.watchVaultRawEvents));
}
watchVaultCreate(file: TAbstractFile, ctx?: any) {
this.appendWatchEvent([{ type: "CREATE", file }], ctx);
}
watchVaultChange(file: TAbstractFile, ctx?: any) {
this.appendWatchEvent([{ type: "CHANGED", file }], ctx);
}
watchVaultDelete(file: TAbstractFile, ctx?: any) {
this.appendWatchEvent([{ type: "DELETE", file }], ctx);
}
watchVaultRename(file: TAbstractFile, oldFile: string, ctx?: any) {
if (file instanceof TFile) {
this.appendWatchEvent([
{ type: "DELETE", file: { path: oldFile as FilePath, mtime: file.stat.mtime, ctime: file.stat.ctime, size: file.stat.size, deleted: true } },
{ type: "CREATE", file },
], ctx);
}
}
// Watch raw events (Internal API)
watchVaultRawEvents(path: FilePath) {
if (this.plugin.settings.useIgnoreFiles && this.plugin.ignoreFiles.some(e => path.endsWith(e.trim()))) {
// If it is one of ignore files, refresh the cached one.
this.plugin.isTargetFile(path).then(() => this._watchVaultRawEvents(path));
} else {
this._watchVaultRawEvents(path);
}
}
_watchVaultRawEvents(path: FilePath) {
if (!this.plugin.settings.syncInternalFiles && !this.plugin.settings.usePluginSync) return;
if (!this.plugin.settings.watchInternalFileChanges) return;
if (!path.startsWith(this.plugin.app.vault.configDir)) return;
const ignorePatterns = this.plugin.settings.syncInternalFilesIgnorePatterns
.replace(/\n| /g, "")
.split(",").filter(e => e).map(e => new RegExp(e, "i"));
if (ignorePatterns.some(e => path.match(e))) return;
this.appendWatchEvent(
[{
type: "INTERNAL",
file: { path, mtime: 0, ctime: 0, size: 0 }
}], null);
}
// Cache file and waiting to can be proceed.
async appendWatchEvent(params: { type: FileEventType, file: TAbstractFile | InternalFileInfo, oldPath?: string }[], ctx?: any) {
let forcePerform = false;
for (const param of params) {
if (shouldBeIgnored(param.file.path)) {
continue;
}
const atomicKey = [0, 0, 0, 0, 0, 0].map(e => `${Math.floor(Math.random() * 100000)}`).join("-");
const type = param.type;
const file = param.file;
const oldPath = param.oldPath;
if (file instanceof TFolder) continue;
if (!await this.plugin.isTargetFile(file.path)) continue;
if (this.plugin.settings.suspendFileWatching) continue;
let cache: null | string | ArrayBuffer;
// new file or something changed, cache the changes.
if (file instanceof TFile && (type == "CREATE" || type == "CHANGED")) {
if (recentlyTouched(file)) {
continue;
}
if (!isPlainText(file.name)) {
cache = await this.plugin.app.vault.readBinary(file);
} else {
// cache = await this.app.vault.read(file);
cache = await this.plugin.app.vault.cachedRead(file);
if (!cache) cache = await this.plugin.app.vault.read(file);
}
}
if (type == "DELETE" || type == "RENAME") {
forcePerform = true;
}
if (this.plugin.settings.batchSave && !this.plugin.settings.liveSync) {
// if the latest event is the same type, omit that
// a.md MODIFY <- this should be cancelled when a.md MODIFIED
// b.md MODIFY <- this should be cancelled when b.md MODIFIED
// a.md MODIFY
// a.md CREATE
// :
let i = this.watchedFileEventQueue.length;
L1:
while (i >= 0) {
i--;
if (i < 0) break L1;
if (this.watchedFileEventQueue[i].args.file.path != file.path) {
continue L1;
}
if (this.watchedFileEventQueue[i].type != type) break L1;
this.watchedFileEventQueue.remove(this.watchedFileEventQueue[i]);
//this.queuedFilesStore.set({ queuedItems: this.queuedFiles, fileEventItems: this.watchedFileEventQueue });
this.queuedFilesStore.apply((value) => ({ ...value, fileEventItems: this.watchedFileEventQueue }));
}
}
const fileInfo = file instanceof TFile ? {
ctime: file.stat.ctime,
mtime: file.stat.mtime,
file: file,
path: file.path,
size: file.stat.size
} as FileInfo : file as InternalFileInfo;
this.watchedFileEventQueue.push({
type,
args: {
file: fileInfo,
oldPath,
cache,
ctx
},
key: atomicKey
})
}
// this.queuedFilesStore.set({ queuedItems: this.queuedFiles, fileEventItems: this.watchedFileEventQueue });
this.queuedFilesStore.apply((value) => ({ ...value, fileEventItems: this.watchedFileEventQueue }));
this.plugin.procFileEvent(forcePerform);
}
fetchEvent(): FileEventItem | false {
if (this.watchedFileEventQueue.length == 0) return false;
const item = this.watchedFileEventQueue.shift();
this.queuedFilesStore.apply((value) => ({ ...value, fileEventItems: this.watchedFileEventQueue }));
return item;
}
cancelRelativeEvent(item: FileEventItem) {
this.watchedFileEventQueue = [...this.watchedFileEventQueue].filter(e => e.key != item.key);
this.queuedFilesStore.apply((value) => ({ ...value, fileEventItems: this.watchedFileEventQueue }));
}
getQueueLength() {
return this.watchedFileEventQueue.length;
}
}

31
src/KeyValueDB.ts → src/common/KeyValueDB.ts
View File

@@ -1,12 +1,12 @@
import { deleteDB, type IDBPDatabase, openDB } from "idb";
export interface KeyValueDatabase {
get<T>(key: string): Promise<T>;
set<T>(key: string, value: T): Promise<IDBValidKey>;
del(key: string): Promise<void>;
get<T>(key: IDBValidKey): Promise<T>;
set<T>(key: IDBValidKey, value: T): Promise<IDBValidKey>;
del(key: IDBValidKey): Promise<void>;
clear(): Promise<void>;
keys(query?: IDBValidKey | IDBKeyRange, count?: number): Promise<IDBValidKey[]>;
close(): void;
destroy(): void;
destroy(): Promise<void>;
}
const databaseCache: { [key: string]: IDBPDatabase<any> } = {};
export const OpenKeyValueDatabase = async (dbKey: string): Promise<KeyValueDatabase> => {
@@ -20,24 +20,23 @@ export const OpenKeyValueDatabase = async (dbKey: string): Promise<KeyValueDatab
db.createObjectStore(storeKey);
},
});
let db: IDBPDatabase<any> = null;
db = await dbPromise;
const db = await dbPromise;
databaseCache[dbKey] = db;
return {
get<T>(key: string): Promise<T> {
return db.get(storeKey, key);
async get<T>(key: IDBValidKey): Promise<T> {
return await db.get(storeKey, key);
},
set<T>(key: string, value: T) {
return db.put(storeKey, value, key);
async set<T>(key: IDBValidKey, value: T) {
return await db.put(storeKey, value, key);
},
del(key: string) {
return db.delete(storeKey, key);
async del(key: IDBValidKey) {
return await db.delete(storeKey, key);
},
clear() {
return db.clear(storeKey);
async clear() {
return await db.clear(storeKey);
},
keys(query?: IDBValidKey | IDBKeyRange, count?: number) {
return db.getAllKeys(storeKey, query, count);
async keys(query?: IDBValidKey | IDBKeyRange, count?: number) {
return await db.getAllKeys(storeKey, query, count);
},
close() {
delete databaseCache[dbKey];

134
src/common/ObsHttpHandler.ts Normal file
View File

@@ -0,0 +1,134 @@
// This file is based on a file that was published by the @remotely-save, under the Apache 2 License.
// I would love to express my deepest gratitude to the original authors for their hard work and dedication. Without their contributions, this project would not have been possible.
//
// Original Implementation is here: https://github.com/remotely-save/remotely-save/blob/28b99557a864ef59c19d2ad96101196e401718f0/src/remoteForS3.ts
import {
FetchHttpHandler,
type FetchHttpHandlerOptions,
} from "@smithy/fetch-http-handler";
import { HttpRequest, HttpResponse, type HttpHandlerOptions } from "@smithy/protocol-http";
//@ts-ignore
import { requestTimeout } from "@smithy/fetch-http-handler/dist-es/request-timeout";
import { buildQueryString } from "@smithy/querystring-builder";
import { requestUrl, type RequestUrlParam } from "../deps.ts";
////////////////////////////////////////////////////////////////////////////////
// special handler using Obsidian requestUrl
////////////////////////////////////////////////////////////////////////////////
/**
* This is close to origin implementation of FetchHttpHandler
* https://github.com/aws/aws-sdk-js-v3/blob/main/packages/fetch-http-handler/src/fetch-http-handler.ts
* that is released under Apache 2 License.
* But this uses Obsidian requestUrl instead.
*/
export class ObsHttpHandler extends FetchHttpHandler {
requestTimeoutInMs: number | undefined;
reverseProxyNoSignUrl: string | undefined;
constructor(
options?: FetchHttpHandlerOptions,
reverseProxyNoSignUrl?: string
) {
super(options);
this.requestTimeoutInMs =
options === undefined ? undefined : options.requestTimeout;
this.reverseProxyNoSignUrl = reverseProxyNoSignUrl;
}
// eslint-disable-next-line require-await
async handle(
request: HttpRequest,
{ abortSignal }: HttpHandlerOptions = {}
): Promise<{ response: HttpResponse }> {
if (abortSignal?.aborted) {
const abortError = new Error("Request aborted");
abortError.name = "AbortError";
return Promise.reject(abortError);
}
let path = request.path;
if (request.query) {
const queryString = buildQueryString(request.query);
if (queryString) {
path += `?${queryString}`;
}
}
const { port, method } = request;
let url = `${request.protocol}//${request.hostname}${port ? `:${port}` : ""
}${path}`;
if (
this.reverseProxyNoSignUrl !== undefined &&
this.reverseProxyNoSignUrl !== ""
) {
const urlObj = new URL(url);
urlObj.host = this.reverseProxyNoSignUrl;
url = urlObj.href;
}
const body =
method === "GET" || method === "HEAD" ? undefined : request.body;
const transformedHeaders: Record<string, string> = {};
for (const key of Object.keys(request.headers)) {
const keyLower = key.toLowerCase();
if (keyLower === "host" || keyLower === "content-length") {
continue;
}
transformedHeaders[keyLower] = request.headers[key];
}
let contentType: string | undefined = undefined;
if (transformedHeaders["content-type"] !== undefined) {
contentType = transformedHeaders["content-type"];
}
let transformedBody: any = body;
if (ArrayBuffer.isView(body)) {
transformedBody = new Uint8Array(body.buffer).buffer;
}
const param: RequestUrlParam = {
body: transformedBody,
headers: transformedHeaders,
method: method,
url: url,
contentType: contentType,
};
const raceOfPromises = [
requestUrl(param).then((rsp) => {
const headers = rsp.headers;
const headersLower: Record<string, string> = {};
for (const key of Object.keys(headers)) {
headersLower[key.toLowerCase()] = headers[key];
}
const stream = new ReadableStream<Uint8Array>({
start(controller) {
controller.enqueue(new Uint8Array(rsp.arrayBuffer));
controller.close();
},
});
return {
response: new HttpResponse({
headers: headersLower,
statusCode: rsp.status,
body: stream,
}),
};
}),
requestTimeout(this.requestTimeoutInMs),
];
if (abortSignal) {
raceOfPromises.push(
new Promise<never>((resolve, reject) => {
abortSignal.onabort = () => {
const abortError = new Error("Request aborted");
abortError.name = "AbortError";
reject(abortError);
};
})
);
}
return Promise.race(raceOfPromises);
}
}

47
src/dialogs.ts → src/common/dialogs.ts
View File

@@ -1,16 +1,15 @@
import { ButtonComponent } from "obsidian";
import { App, FuzzySuggestModal, MarkdownRenderer, Modal, Plugin, Setting } from "./deps";
import ObsidianLiveSyncPlugin from "./main";
import { App, FuzzySuggestModal, MarkdownRenderer, Modal, Plugin, Setting } from "../deps.ts";
import ObsidianLiveSyncPlugin from "../main.ts";
//@ts-ignore
import PluginPane from "./PluginPane.svelte";
import PluginPane from "../ui/PluginPane.svelte";
export class PluginDialogModal extends Modal {
plugin: ObsidianLiveSyncPlugin;
logEl: HTMLDivElement;
component: PluginPane = null;
component: PluginPane | undefined;
isOpened() {
return this.component != null;
return this.component != undefined;
}
constructor(app: App, plugin: ObsidianLiveSyncPlugin) {
@@ -20,8 +19,11 @@ export class PluginDialogModal extends Modal {
onOpen() {
const { contentEl } = this;
this.titleEl.setText("Customization Sync (Beta2)")
if (this.component == null) {
this.contentEl.style.overflow = "auto";
this.contentEl.style.display = "flex";
this.contentEl.style.flexDirection = "column";
this.titleEl.setText("Customization Sync (Beta3)")
if (!this.component) {
this.component = new PluginPane({
target: contentEl,
props: { plugin: this.plugin },
@@ -30,9 +32,9 @@ export class PluginDialogModal extends Modal {
}
onClose() {
if (this.component != null) {
if (this.component) {
this.component.$destroy();
this.component = null;
this.component = undefined;
}
}
}
@@ -58,8 +60,7 @@ export class InputStringDialog extends Modal {
onOpen() {
const { contentEl } = this;
this.titleEl.setText(this.title);
// For enter to submit
const formEl = contentEl.createEl("form");
const formEl = contentEl.createDiv();
new Setting(formEl).setName(this.key).setClass(this.isPassword ? "password-input" : "normal-input").addText((text) =>
text.onChange((value) => {
this.result = value;
@@ -95,13 +96,13 @@ export class InputStringDialog extends Modal {
}
export class PopoverSelectString extends FuzzySuggestModal<string> {
app: App;
callback: (e: string) => void = () => { };
callback: ((e: string) => void) | undefined = () => { };
getItemsFun: () => string[] = () => {
return ["yes", "no"];
}
constructor(app: App, note: string, placeholder: string | null, getItemsFun: () => string[], callback: (e: string) => void) {
constructor(app: App, note: string, placeholder: string | undefined, getItemsFun: (() => string[]) | undefined, callback: (e: string) => void) {
super(app);
this.app = app;
this.setPlaceholder((placeholder ?? "y/n) ") + note);
@@ -119,13 +120,14 @@ export class PopoverSelectString extends FuzzySuggestModal<string> {
onChooseItem(item: string, evt: MouseEvent | KeyboardEvent): void {
// debugger;
this.callback(item);
this.callback = null;
this.callback?.(item);
this.callback = undefined;
}
onClose(): void {
setTimeout(() => {
if (this.callback != null) {
if (this.callback) {
this.callback("");
this.callback = undefined;
}
}, 100);
}
@@ -137,16 +139,16 @@ export class MessageBox extends Modal {
title: string;
contentMd: string;
buttons: string[];
result: string;
result: string | false = false;
isManuallyClosed = false;
defaultAction: string | undefined;
timeout: number | undefined;
timer: ReturnType<typeof setInterval> = undefined;
timer: ReturnType<typeof setInterval> | undefined = undefined;
defaultButtonComponent: ButtonComponent | undefined;
onSubmit: (result: string | false) => void;
constructor(plugin: Plugin, title: string, contentMd: string, buttons: string[], defaultAction: (typeof buttons)[number], timeout: number, onSubmit: (result: (typeof buttons)[number] | false) => void) {
constructor(plugin: Plugin, title: string, contentMd: string, buttons: string[], defaultAction: (typeof buttons)[number], timeout: number | undefined, onSubmit: (result: (typeof buttons)[number] | false) => void) {
super(plugin.app);
this.plugin = plugin;
this.title = title;
@@ -157,6 +159,7 @@ export class MessageBox extends Modal {
this.timeout = timeout;
if (this.timeout) {
this.timer = setInterval(() => {
if (this.timeout === undefined) return;
this.timeout--;
if (this.timeout < 0) {
if (this.timer) {
@@ -167,7 +170,7 @@ export class MessageBox extends Modal {
this.isManuallyClosed = true;
this.close();
} else {
this.defaultButtonComponent.setButtonText(`( ${this.timeout} ) ${defaultAction}`);
this.defaultButtonComponent?.setButtonText(`( ${this.timeout} ) ${defaultAction}`);
}
}, 1000);
}
@@ -224,7 +227,7 @@ export class MessageBox extends Modal {
}
export function confirmWithMessage(plugin: Plugin, title: string, contentMd: string, buttons: string[], defaultAction?: (typeof buttons)[number], timeout?: number): Promise<(typeof buttons)[number] | false> {
export function confirmWithMessage(plugin: Plugin, title: string, contentMd: string, buttons: string[], defaultAction: (typeof buttons)[number], timeout?: number): Promise<(typeof buttons)[number] | false> {
return new Promise((res) => {
const dialog = new MessageBox(plugin, title, contentMd, buttons, defaultAction, timeout, (result) => res(result));
dialog.open();

7
src/common/stores.ts Normal file
View File

@@ -0,0 +1,7 @@
import { PersistentMap } from "../lib/src/dataobject/PersistentMap.ts";
export let sameChangePairs: PersistentMap<number[]>;
export function initializeStores(vaultName: string) {
sameChangePairs = new PersistentMap<number[]>(`ls-persist-same-changes-${vaultName}`);
}

7
src/types.ts → src/common/types.ts
View File

@@ -1,5 +1,5 @@
import { type PluginManifest, TFile } from "./deps";
import { type DatabaseEntry, type EntryBody, type FilePath } from "./lib/src/types";
import { type PluginManifest, TFile } from "../deps.ts";
import { type DatabaseEntry, type EntryBody, type FilePath } from "../lib/src/common/types.ts";
export interface PluginDataEntry extends DatabaseEntry {
deviceVaultName: string;
@@ -60,6 +60,9 @@ export type FileEventItem = {
type: FileEventType,
args: FileEventArgs,
key: string,
skipBatchWait?: boolean,
cancelled?: boolean,
batched?: boolean
}
// Hidden items (Now means `chunk`)

193
src/utils.ts → src/common/utils.ts
View File

@@ -1,12 +1,16 @@
import { type DataWriteOptions, normalizePath, TFile, Platform, TAbstractFile, App, Plugin, type RequestUrlParam, requestUrl } from "./deps";
import { path2id_base, id2path_base, isValidFilenameInLinux, isValidFilenameInDarwin, isValidFilenameInWidows, isValidFilenameInAndroid, stripAllPrefixes } from "./lib/src/path";
import { normalizePath, Platform, TAbstractFile, App, type RequestUrlParam, requestUrl, TFile } from "../deps.ts";
import { path2id_base, id2path_base, isValidFilenameInLinux, isValidFilenameInDarwin, isValidFilenameInWidows, isValidFilenameInAndroid, stripAllPrefixes } from "../lib/src/string_and_binary/path.ts";
import { Logger } from "./lib/src/logger";
import { LOG_LEVEL_VERBOSE, type AnyEntry, type DocumentID, type EntryHasPath, type FilePath, type FilePathWithPrefix } from "./lib/src/types";
import { CHeader, ICHeader, ICHeaderLength, ICXHeader, PSCHeader } from "./types";
import { InputStringDialog, PopoverSelectString } from "./dialogs";
import ObsidianLiveSyncPlugin from "./main";
import { writeString } from "./lib/src/strbin";
import { Logger } from "../lib/src/common/logger.ts";
import { LOG_LEVEL_VERBOSE, type AnyEntry, type DocumentID, type EntryHasPath, type FilePath, type FilePathWithPrefix } from "../lib/src/common/types.ts";
import { CHeader, ICHeader, ICHeaderLength, ICXHeader, PSCHeader } from "./types.ts";
import { InputStringDialog, PopoverSelectString } from "./dialogs.ts";
import type ObsidianLiveSyncPlugin from "../main.ts";
import { writeString } from "../lib/src/string_and_binary/convert.ts";
import { fireAndForget } from "../lib/src/common/utils.ts";
import { sameChangePairs } from "./stores.ts";
export { scheduleTask, setPeriodicTask, cancelTask, cancelAllTasks, cancelPeriodicTask, cancelAllPeriodicTask, } from "../lib/src/concurrency/task.ts";
// For backward compatibility, using the path for determining id.
// Only CouchDB unacceptable ID (that starts with an underscore) has been prefixed with "/".
@@ -43,49 +47,6 @@ export function getPathFromTFile(file: TAbstractFile) {
return file.path as FilePath;
}
const tasks: { [key: string]: ReturnType<typeof setTimeout> } = {};
export function scheduleTask(key: string, timeout: number, proc: (() => Promise<any> | void), skipIfTaskExist?: boolean) {
if (skipIfTaskExist && key in tasks) {
return;
}
cancelTask(key);
tasks[key] = setTimeout(async () => {
delete tasks[key];
await proc();
}, timeout);
}
export function cancelTask(key: string) {
if (key in tasks) {
clearTimeout(tasks[key]);
delete tasks[key];
}
}
export function cancelAllTasks() {
for (const v in tasks) {
clearTimeout(tasks[v]);
delete tasks[v];
}
}
const intervals: { [key: string]: ReturnType<typeof setInterval> } = {};
export function setPeriodicTask(key: string, timeout: number, proc: (() => Promise<any> | void)) {
cancelPeriodicTask(key);
intervals[key] = setInterval(async () => {
delete intervals[key];
await proc();
}, timeout);
}
export function cancelPeriodicTask(key: string) {
if (key in intervals) {
clearInterval(intervals[key]);
delete intervals[key];
}
}
export function cancelAllPeriodicTask() {
for (const v in intervals) {
clearInterval(intervals[v]);
delete intervals[v];
}
}
const memos: { [key: string]: any } = {};
export function memoObject<T>(key: string, obj: T): T {
@@ -281,14 +242,11 @@ export function mergeObject(
ret[key] = v;
}
}
const retSorted = Object.fromEntries(Object.entries(ret).sort((a, b) => a[0] < b[0] ? -1 : a[0] > b[0] ? 1 : 0));
if (Array.isArray(objA) && Array.isArray(objB)) {
return Object.values(Object.entries(ret)
.sort()
.reduce((p, [key, value]) => ({ ...p, [key]: value }), {}));
return Object.values(retSorted);
}
return Object.entries(ret)
.sort()
.reduce((p, [key, value]) => ({ ...p, [key]: value }), {});
return retSorted;
}
export function flattenObject(obj: Record<string | number | symbol, any>, path: string[] = []): [string, any][] {
@@ -303,20 +261,6 @@ export function flattenObject(obj: Record<string | number | symbol, any>, path:
return ret;
}
export function modifyFile(file: TFile, data: string | ArrayBuffer, options?: DataWriteOptions) {
if (typeof (data) === "string") {
return app.vault.modify(file, data, options);
} else {
return app.vault.modifyBinary(file, data, options);
}
}
export function createFile(path: string, data: string | ArrayBuffer, options?: DataWriteOptions): Promise<TFile> {
if (typeof (data) === "string") {
return app.vault.create(path, data, options);
} else {
return app.vault.createBinary(path, data, options);
}
}
export function isValidPath(filename: string) {
if (Platform.isDesktop) {
@@ -332,38 +276,10 @@ export function isValidPath(filename: string) {
return isValidFilenameInWidows(filename);
}
let touchedFiles: string[] = [];
export function getAbstractFileByPath(path: FilePath): TAbstractFile | null {
// Disabled temporary.
return app.vault.getAbstractFileByPath(path);
// // Hidden API but so useful.
// // @ts-ignore
// if ("getAbstractFileByPathInsensitive" in app.vault && (app.vault.adapter?.insensitive ?? false)) {
// // @ts-ignore
// return app.vault.getAbstractFileByPathInsensitive(path);
// } else {
// return app.vault.getAbstractFileByPath(path);
// }
}
export function trimPrefix(target: string, prefix: string) {
return target.startsWith(prefix) ? target.substring(prefix.length) : target;
}
export function touch(file: TFile | FilePath) {
const f = file instanceof TFile ? file : getAbstractFileByPath(file) as TFile;
const key = `${f.path}-${f.stat.mtime}-${f.stat.size}`;
touchedFiles.unshift(key);
touchedFiles = touchedFiles.slice(0, 100);
}
export function recentlyTouched(file: TFile) {
const key = `${file.path}-${file.stat.mtime}-${file.stat.size}`;
if (touchedFiles.indexOf(key) == -1) return false;
return true;
}
export function clearTouched() {
touchedFiles = [];
}
/**
* returns is internal chunk of file
@@ -394,7 +310,7 @@ export function isCustomisationSyncMetadata(str: string): boolean {
export const askYesNo = (app: App, message: string): Promise<"yes" | "no"> => {
return new Promise((res) => {
const popover = new PopoverSelectString(app, message, null, null, (result) => res(result as "yes" | "no"));
const popover = new PopoverSelectString(app, message, undefined, undefined, (result) => res(result as "yes" | "no"));
popover.open();
});
};
@@ -408,7 +324,7 @@ export const askSelectString = (app: App, message: string, items: string[]): Pro
};
export const askString = (app: App, title: string, key: string, placeholder: string, isPassword?: boolean): Promise<string | false> => {
export const askString = (app: App, title: string, key: string, placeholder: string, isPassword: boolean = false): Promise<string | false> => {
return new Promise((res) => {
const dialog = new InputStringDialog(app, title, key, placeholder, isPassword, (result) => res(result));
dialog.open();
@@ -419,8 +335,8 @@ export const askString = (app: App, title: string, key: string, placeholder: str
export class PeriodicProcessor {
_process: () => Promise<any>;
_timer?: number;
_plugin: Plugin;
constructor(plugin: Plugin, process: () => Promise<any>) {
_plugin: ObsidianLiveSyncPlugin;
constructor(plugin: ObsidianLiveSyncPlugin, process: () => Promise<any>) {
this._plugin = plugin;
this._process = process;
}
@@ -434,12 +350,19 @@ export class PeriodicProcessor {
enable(interval: number) {
this.disable();
if (interval == 0) return;
this._timer = window.setInterval(() => this.process().then(() => { }), interval);
this._timer = window.setInterval(() => fireAndForget(async () => {
await this.process();
if (this._plugin._unloaded) {
this.disable();
}
}), interval);
this._plugin.registerInterval(this._timer);
}
disable() {
if (this._timer !== undefined) window.clearInterval(this._timer);
this._timer = undefined;
if (this._timer !== undefined) {
window.clearInterval(this._timer);
this._timer = undefined;
}
}
}
@@ -474,14 +397,17 @@ export const _requestToCouchDB = async (baseUri: string, username: string, passw
};
return await requestUrl(requestParam);
}
export const requestToCouchDB = async (baseUri: string, username: string, password: string, origin: string, key?: string, body?: string, method?: string) => {
export const requestToCouchDB = async (baseUri: string, username: string, password: string, origin: string = "", key?: string, body?: string, method?: string) => {
const uri = `_node/_local/_config${key ? "/" + key : ""}`;
return await _requestToCouchDB(baseUri, username, password, origin, uri, body, method);
};
export async function performRebuildDB(plugin: ObsidianLiveSyncPlugin, method: "localOnly" | "remoteOnly" | "rebuildBothByThisDevice") {
export async function performRebuildDB(plugin: ObsidianLiveSyncPlugin, method: "localOnly" | "remoteOnly" | "rebuildBothByThisDevice" | "localOnlyWithChunks") {
if (method == "localOnly") {
await plugin.addOnSetup.fetchLocalWithKeepLocal();
await plugin.addOnSetup.fetchLocal();
}
if (method == "localOnlyWithChunks") {
await plugin.addOnSetup.fetchLocal(true);
}
if (method == "remoteOnly") {
await plugin.addOnSetup.rebuildRemote();
@@ -490,3 +416,52 @@ export async function performRebuildDB(plugin: ObsidianLiveSyncPlugin, method: "
await plugin.addOnSetup.rebuildEverything();
}
}
export const BASE_IS_NEW = Symbol("base");
export const TARGET_IS_NEW = Symbol("target");
export const EVEN = Symbol("even");
// Why 2000? : ZIP FILE Does not have enough resolution.
const resolution = 2000;
export function compareMTime(baseMTime: number, targetMTime: number): typeof BASE_IS_NEW | typeof TARGET_IS_NEW | typeof EVEN {
const truncatedBaseMTime = (~~(baseMTime / resolution)) * resolution;
const truncatedTargetMTime = (~~(targetMTime / resolution)) * resolution;
// Logger(`Resolution MTime ${truncatedBaseMTime} and ${truncatedTargetMTime} `, LOG_LEVEL_VERBOSE);
if (truncatedBaseMTime == truncatedTargetMTime) return EVEN;
if (truncatedBaseMTime > truncatedTargetMTime) return BASE_IS_NEW;
if (truncatedBaseMTime < truncatedTargetMTime) return TARGET_IS_NEW;
throw new Error("Unexpected error");
}
export function markChangesAreSame(file: TFile | AnyEntry | string, mtime1: number, mtime2: number) {
if (mtime1 === mtime2) return true;
const key = typeof file == "string" ? file : file instanceof TFile ? file.path : file.path ?? file._id;
const pairs = sameChangePairs.get(key, []) || [];
if (pairs.some(e => e == mtime1 || e == mtime2)) {
sameChangePairs.set(key, [...new Set([...pairs, mtime1, mtime2])]);
} else {
sameChangePairs.set(key, [mtime1, mtime2]);
}
}
export function isMarkedAsSameChanges(file: TFile | AnyEntry | string, mtimes: number[]) {
const key = typeof file == "string" ? file : file instanceof TFile ? file.path : file.path ?? file._id;
const pairs = sameChangePairs.get(key, []) || [];
if (mtimes.every(e => pairs.indexOf(e) !== -1)) {
return EVEN;
}
}
export function compareFileFreshness(baseFile: TFile | AnyEntry | undefined, checkTarget: TFile | AnyEntry | undefined): typeof BASE_IS_NEW | typeof TARGET_IS_NEW | typeof EVEN {
if (baseFile === undefined && checkTarget == undefined) return EVEN;
if (baseFile == undefined) return TARGET_IS_NEW;
if (checkTarget == undefined) return BASE_IS_NEW;
const modifiedBase = baseFile instanceof TFile ? baseFile?.stat?.mtime ?? 0 : baseFile?.mtime ?? 0;
const modifiedTarget = checkTarget instanceof TFile ? checkTarget?.stat?.mtime ?? 0 : checkTarget?.mtime ?? 0;
if (modifiedBase && modifiedTarget && isMarkedAsSameChanges(baseFile, [modifiedBase, modifiedTarget])) {
return EVEN;
}
return compareMTime(modifiedBase, modifiedTarget);
}

4
src/deps.ts
View File

@@ -1,10 +1,10 @@
import { type FilePath } from "./lib/src/types";
import { type FilePath } from "./lib/src/common/types.ts";
export {
addIcon, App, debounce, Editor, FuzzySuggestModal, MarkdownRenderer, MarkdownView, Modal, Notice, Platform, Plugin, PluginSettingTab, requestUrl, sanitizeHTMLToDom, Setting, stringifyYaml, TAbstractFile, TextAreaComponent, TFile, TFolder,
parseYaml, ItemView, WorkspaceLeaf
} from "obsidian";
export type { DataWriteOptions, PluginManifest, RequestUrlParam, RequestUrlResponse, MarkdownFileInfo } from "obsidian";
export type { DataWriteOptions, PluginManifest, RequestUrlParam, RequestUrlResponse, MarkdownFileInfo, ListedFiles } from "obsidian";
import {
normalizePath as normalizePath_
} from "obsidian";

1476
src/features/CmdConfigSync.ts Normal file
View File

File diff suppressed because it is too large Load Diff

348
src/CmdHiddenFileSync.ts → src/features/CmdHiddenFileSync.ts
View File

@@ -1,26 +1,23 @@
import { normalizePath, type PluginManifest } from "./deps";
import { type EntryDoc, type LoadedEntry, type InternalFileEntry, type FilePathWithPrefix, type FilePath, LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE, MODE_SELECTIVE, MODE_PAUSED } from "./lib/src/types";
import { type InternalFileInfo, ICHeader, ICHeaderEnd } from "./types";
import { Parallels, delay, isDocContentSame } from "./lib/src/utils";
import { Logger } from "./lib/src/logger";
import { PouchDB } from "./lib/src/pouchdb-browser.js";
import { scheduleTask, isInternalMetadata, PeriodicProcessor } from "./utils";
import { WrappedNotice } from "./lib/src/wrapper";
import { decodeBinary, encodeBinary } from "./lib/src/strbin";
import { serialized } from "./lib/src/lock";
import { JsonResolveModal } from "./JsonResolveModal";
import { LiveSyncCommands } from "./LiveSyncCommands";
import { addPrefix, stripAllPrefixes } from "./lib/src/path";
import { normalizePath, type PluginManifest, type ListedFiles } from "../deps.ts";
import { type EntryDoc, type LoadedEntry, type InternalFileEntry, type FilePathWithPrefix, type FilePath, LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE, MODE_SELECTIVE, MODE_PAUSED, type SavingEntry, type DocumentID } from "../lib/src/common/types.ts";
import { type InternalFileInfo, ICHeader, ICHeaderEnd } from "../common/types.ts";
import { readAsBlob, isDocContentSame, sendSignal, readContent, createBlob } from "../lib/src/common/utils.ts";
import { Logger } from "../lib/src/common/logger.ts";
import { PouchDB } from "../lib/src/pouchdb/pouchdb-browser.js";
import { isInternalMetadata, PeriodicProcessor } from "../common/utils.ts";
import { serialized } from "../lib/src/concurrency/lock.ts";
import { JsonResolveModal } from "../ui/JsonResolveModal.ts";
import { LiveSyncCommands } from "./LiveSyncCommands.ts";
import { addPrefix, stripAllPrefixes } from "../lib/src/string_and_binary/path.ts";
import { QueueProcessor } from "../lib/src/concurrency/processor.ts";
import { hiddenFilesEventCount, hiddenFilesProcessingCount } from "../lib/src/mock_and_interop/stores.ts";
export class HiddenFileSync extends LiveSyncCommands {
periodicInternalFileScanProcessor: PeriodicProcessor = new PeriodicProcessor(this.plugin, async () => this.settings.syncInternalFiles && this.localDatabase.isReady && await this.syncInternalFilesAndDatabase("push", false));
confirmPopup: WrappedNotice = null;
get kvDB() {
return this.plugin.kvDB;
}
ensureDirectoryEx(fullPath: string) {
return this.plugin.ensureDirectoryEx(fullPath);
}
getConflictedDoc(path: FilePathWithPrefix, rev: string) {
return this.plugin.getConflictedDoc(path, rev);
}
@@ -68,29 +65,24 @@ export class HiddenFileSync extends LiveSyncCommands {
realizeSettingSyncMode(): Promise<void> {
this.periodicInternalFileScanProcessor?.disable();
if (this.plugin.suspended)
return;
return Promise.resolve();
if (!this.plugin.isReady)
return;
return Promise.resolve();
this.periodicInternalFileScanProcessor.enable(this.settings.syncInternalFiles && this.settings.syncInternalFilesInterval ? (this.settings.syncInternalFilesInterval * 1000) : 0);
return;
return Promise.resolve();
}
procInternalFiles: string[] = [];
async execInternalFile() {
await serialized("execInternal", async () => {
const w = [...this.procInternalFiles];
this.procInternalFiles = [];
Logger(`Applying hidden ${w.length} files change...`);
await this.syncInternalFilesAndDatabase("pull", false, false, w);
Logger(`Applying hidden ${w.length} files changed`);
});
}
procInternalFile(filename: string) {
this.procInternalFiles.push(filename);
scheduleTask("procInternal", 500, async () => {
await this.execInternalFile();
});
this.internalFileProcessor.enqueue(filename);
}
internalFileProcessor = new QueueProcessor<string, any>(
async (filenames) => {
Logger(`START :Applying hidden ${filenames.length} files change`, LOG_LEVEL_VERBOSE);
await this.syncInternalFilesAndDatabase("pull", false, false, filenames);
Logger(`DONE :Applying hidden ${filenames.length} files change`, LOG_LEVEL_VERBOSE);
return;
}, { batchSize: 100, concurrentLimit: 1, delay: 10, yieldThreshold: 100, suspended: false, totalRemainingReactiveSource: hiddenFilesEventCount }
);
recentProcessedInternalFiles = [] as string[];
async watchVaultRawEventsAsync(path: FilePath) {
@@ -103,13 +95,15 @@ export class HiddenFileSync extends LiveSyncCommands {
Logger(`Hidden file skipped: ${path} is synchronized in customization sync.`, LOG_LEVEL_VERBOSE);
return;
}
const stat = await this.app.vault.adapter.stat(path);
const stat = await this.vaultAccess.adapterStat(path);
// sometimes folder is coming.
if (stat && stat.type != "file")
if (stat != null && stat.type != "file") {
return;
const storageMTime = ~~((stat && stat.mtime || 0) / 1000);
}
const mtime = stat == null ? 0 : stat?.mtime ?? 0;
const storageMTime = ~~((mtime) / 1000);
const key = `${path}-${storageMTime}`;
if (this.recentProcessedInternalFiles.contains(key)) {
if (mtime != 0 && this.recentProcessedInternalFiles.contains(key)) {
//If recently processed, it may caused by self.
return;
}
@@ -129,7 +123,7 @@ export class HiddenFileSync extends LiveSyncCommands {
if (storageMTime == 0) {
await this.deleteInternalFileOnDatabase(path);
} else {
await this.storeInternalFileToDatabase({ path: path, ...stat });
await this.storeInternalFileToDatabase({ path: path, mtime, ctime: stat?.ctime ?? mtime, size: stat?.size ?? 0 });
}
}
@@ -137,25 +131,56 @@ export class HiddenFileSync extends LiveSyncCommands {
async resolveConflictOnInternalFiles() {
// Scan all conflicted internal files
const conflicted = this.localDatabase.findEntries(ICHeader, ICHeaderEnd, { conflicts: true });
for await (const doc of conflicted) {
if (!("_conflicts" in doc))
continue;
if (isInternalMetadata(doc._id)) {
await this.resolveConflictOnInternalFile(doc.path);
this.conflictResolutionProcessor.suspend();
try {
for await (const doc of conflicted) {
if (!("_conflicts" in doc))
continue;
if (isInternalMetadata(doc._id)) {
this.conflictResolutionProcessor.enqueue(doc.path);
}
}
} catch (ex) {
Logger("something went wrong on resolving all conflicted internal files");
Logger(ex, LOG_LEVEL_VERBOSE);
}
await this.conflictResolutionProcessor.startPipeline().waitForAllProcessed();
}
async resolveConflictOnInternalFile(path: FilePathWithPrefix): Promise<boolean> {
async resolveByNewerEntry(id: DocumentID, path: FilePathWithPrefix, currentDoc: EntryDoc, currentRev: string, conflictedRev: string) {
const conflictedDoc = await this.localDatabase.getRaw(id, { rev: conflictedRev });
// determine which revision should been deleted.
// simply check modified time
const mtimeCurrent = ("mtime" in currentDoc && currentDoc.mtime) || 0;
const mtimeConflicted = ("mtime" in conflictedDoc && conflictedDoc.mtime) || 0;
// Logger(`Revisions:${new Date(mtimeA).toLocaleString} and ${new Date(mtimeB).toLocaleString}`);
// console.log(`mtime:${mtimeA} - ${mtimeB}`);
const delRev = mtimeCurrent < mtimeConflicted ? currentRev : conflictedRev;
// delete older one.
await this.localDatabase.removeRevision(id, delRev);
Logger(`Older one has been deleted:${path}`);
const cc = await this.localDatabase.getRaw(id, { conflicts: true });
if (cc._conflicts?.length === 0) {
await this.extractInternalFileFromDatabase(stripAllPrefixes(path))
} else {
this.conflictResolutionProcessor.enqueue(path);
}
// check the file again
}
conflictResolutionProcessor = new QueueProcessor(async (paths: FilePathWithPrefix[]) => {
const path = paths[0];
sendSignal(`cancel-internal-conflict:${path}`);
try {
// Retrieve data
const id = await this.path2id(path, ICHeader);
const doc = await this.localDatabase.getRaw(id, { conflicts: true });
// If there is no conflict, return with false.
if (!("_conflicts" in doc))
return false;
// if (!("_conflicts" in doc)){
// return [];
// }
if (doc._conflicts === undefined) return [];
if (doc._conflicts.length == 0)
return false;
return [];
Logger(`Hidden file conflicted:${path}`);
const conflicts = doc._conflicts.sort((a, b) => Number(a.split("-")[0]) - Number(b.split("-")[0]));
const revA = doc._rev;
@@ -166,54 +191,59 @@ export class HiddenFileSync extends LiveSyncCommands {
const conflictedRevNo = Number(conflictedRev.split("-")[0]);
//Search
const revFrom = (await this.localDatabase.getRaw<EntryDoc>(id, { revs_info: true }));
const commonBase = revFrom._revs_info.filter(e => e.status == "available" && Number(e.rev.split("-")[0]) < conflictedRevNo).first()?.rev ?? "";
const commonBase = revFrom._revs_info?.filter(e => e.status == "available" && Number(e.rev.split("-")[0]) < conflictedRevNo).first()?.rev ?? "";
const result = await this.plugin.mergeObject(path, commonBase, doc._rev, conflictedRev);
if (result) {
Logger(`Object merge:${path}`, LOG_LEVEL_INFO);
const filename = stripAllPrefixes(path);
const isExists = await this.app.vault.adapter.exists(filename);
const isExists = await this.plugin.vaultAccess.adapterExists(filename);
if (!isExists) {
await this.ensureDirectoryEx(filename);
await this.vaultAccess.ensureDirectory(filename);
}
await this.plugin.vaultAccess.adapterWrite(filename, result);
const stat = await this.vaultAccess.adapterStat(filename);
if (!stat) {
throw new Error(`conflictResolutionProcessor: Failed to stat file ${filename}`);
}
await this.app.vault.adapter.write(filename, result);
const stat = await this.app.vault.adapter.stat(filename);
await this.storeInternalFileToDatabase({ path: filename, ...stat });
await this.extractInternalFileFromDatabase(filename);
await this.localDatabase.removeRaw(id, revB);
return this.resolveConflictOnInternalFile(path);
await this.localDatabase.removeRevision(id, revB);
this.conflictResolutionProcessor.enqueue(path);
return [];
} else {
Logger(`Object merge is not applicable.`, LOG_LEVEL_VERBOSE);
}
const docAMerge = await this.localDatabase.getDBEntry(path, { rev: revA });
const docBMerge = await this.localDatabase.getDBEntry(path, { rev: revB });
if (docAMerge != false && docBMerge != false) {
if (await this.showJSONMergeDialogAndMerge(docAMerge, docBMerge)) {
await delay(200);
// Again for other conflicted revisions.
return this.resolveConflictOnInternalFile(path);
}
return false;
}
return [{ path, revA, revB, id, doc }];
}
const revBDoc = await this.localDatabase.getRaw(id, { rev: revB });
// determine which revision should been deleted.
// simply check modified time
const mtimeA = ("mtime" in doc && doc.mtime) || 0;
const mtimeB = ("mtime" in revBDoc && revBDoc.mtime) || 0;
// Logger(`Revisions:${new Date(mtimeA).toLocaleString} and ${new Date(mtimeB).toLocaleString}`);
// console.log(`mtime:${mtimeA} - ${mtimeB}`);
const delRev = mtimeA < mtimeB ? revA : revB;
// delete older one.
await this.localDatabase.removeRaw(id, delRev);
Logger(`Older one has been deleted:${path}`);
// check the file again
return this.resolveConflictOnInternalFile(path);
// When not JSON file, resolve conflicts by choosing a newer one.
await this.resolveByNewerEntry(id, path, doc, revA, revB);
return [];
} catch (ex) {
Logger(`Failed to resolve conflict (Hidden): ${path}`);
Logger(ex, LOG_LEVEL_VERBOSE);
return false;
return [];
}
}, {
suspended: false, batchSize: 1, concurrentLimit: 5, delay: 10, keepResultUntilDownstreamConnected: true, yieldThreshold: 10,
pipeTo: new QueueProcessor(async (results) => {
const { id, doc, path, revA, revB } = results[0];
const docAMerge = await this.localDatabase.getDBEntry(path, { rev: revA });
const docBMerge = await this.localDatabase.getDBEntry(path, { rev: revB });
if (docAMerge != false && docBMerge != false) {
if (await this.showJSONMergeDialogAndMerge(docAMerge, docBMerge)) {
// Again for other conflicted revisions.
this.conflictResolutionProcessor.enqueue(path);
}
return;
} else {
// If either revision could not read, force resolving by the newer one.
await this.resolveByNewerEntry(id, path, doc, revA, revB);
}
}, { suspended: false, batchSize: 1, concurrentLimit: 1, delay: 10, keepResultUntilDownstreamConnected: false, yieldThreshold: 10 })
})
queueConflictCheck(path: FilePathWithPrefix) {
this.conflictResolutionProcessor.enqueue(path);
}
//TODO: Tidy up. Even though it is experimental feature, So dirty...
@@ -278,28 +308,38 @@ export class HiddenFileSync extends LiveSyncCommands {
acc[stripAllPrefixes(this.getPath(cur))] = cur;
return acc;
}, {} as { [key: string]: InternalFileEntry; });
const para = Parallels();
for (const filename of allFileNames) {
await new QueueProcessor(async (filenames: FilePath[]) => {
const filename = filenames[0];
processed++;
if (processed % 100 == 0) {
Logger(`Hidden file: ${processed}/${fileCount}`, logLevel, "sync_internal");
}
if (!filename) continue;
if (!filename) return [];
if (ignorePatterns.some(e => filename.match(e)))
continue;
return [];
if (await this.plugin.isIgnoredByIgnoreFiles(filename)) {
continue;
return [];
}
const fileOnStorage = filename in filesMap ? filesMap[filename] : undefined;
const fileOnDatabase = filename in filesOnDBMap ? filesOnDBMap[filename] : undefined;
const cache = filename in caches ? caches[filename] : { storageMtime: 0, docMtime: 0 };
await para.wait(5);
const proc = (async (xFileOnStorage: InternalFileInfo, xFileOnDatabase: InternalFileEntry) => {
return [{
filename,
fileOnStorage,
fileOnDatabase,
}]
}, { suspended: true, batchSize: 1, concurrentLimit: 10, delay: 0, totalRemainingReactiveSource: hiddenFilesProcessingCount })
.pipeTo(new QueueProcessor(async (params) => {
const
{
filename,
fileOnStorage: xFileOnStorage,
fileOnDatabase: xFileOnDatabase
} = params[0];
if (xFileOnStorage && xFileOnDatabase) {
const cache = filename in caches ? caches[filename] : { storageMtime: 0, docMtime: 0 };
// Both => Synchronize
if ((direction != "pullForce" && direction != "pushForce") && xFileOnDatabase.mtime == cache.docMtime && xFileOnStorage.mtime == cache.storageMtime) {
return;
@@ -335,16 +375,21 @@ export class HiddenFileSync extends LiveSyncCommands {
}
}
} else if (xFileOnStorage && !xFileOnDatabase) {
await this.storeInternalFileToDatabase(xFileOnStorage);
if (direction == "push" || direction == "pushForce" || direction == "safe") {
await this.storeInternalFileToDatabase(xFileOnStorage);
} else {
await this.extractInternalFileFromDatabase(xFileOnStorage.path);
}
} else {
throw new Error("Invalid state on hidden file sync");
// Something corrupted?
}
return;
}, { suspended: true, batchSize: 1, concurrentLimit: 5, delay: 0 }))
.root
.enqueueAll(allFileNames)
.startPipeline().waitForAllDoneAndTerminate();
});
para.add(proc(fileOnStorage, fileOnDatabase))
}
await para.all();
await this.kvDB.set("diff-caches-internal", caches);
// When files has been retrieved from the database. they must be reloaded.
@@ -360,7 +405,7 @@ export class HiddenFileSync extends LiveSyncCommands {
const enabledPlugins = this.app.plugins.enabledPlugins as Set<string>;
const enabledPluginManifests = manifests.filter(e => enabledPlugins.has(e.id));
for (const manifest of enabledPluginManifests) {
if (manifest.dir in updatedFolders) {
if (manifest.dir && manifest.dir in updatedFolders) {
// If notified about plug-ins, reloading Obsidian may not be necessary.
updatedCount -= updatedFolders[manifest.dir];
const updatePluginId = manifest.id;
@@ -387,13 +432,14 @@ export class HiddenFileSync extends LiveSyncCommands {
// If something changes left, notify for reloading Obsidian.
if (updatedCount != 0) {
this.plugin.askInPopup(`updated-any-hidden`, `Hidden files have been synchronized, Press {HERE} to reload Obsidian, or press elsewhere to dismiss this message.`, (anchor) => {
anchor.text = "HERE";
anchor.addEventListener("click", () => {
// @ts-ignore
this.app.commands.executeCommandById("app:reload");
if (!this.plugin.isReloadingScheduled) {
this.plugin.askInPopup(`updated-any-hidden`, `Hidden files have been synchronised, Press {HERE} to schedule a reload of Obsidian, or press elsewhere to dismiss this message.`, (anchor) => {
anchor.text = "HERE";
anchor.addEventListener("click", () => {
this.plugin.scheduleAppReload();
});
});
});
}
}
}
}
@@ -408,20 +454,12 @@ export class HiddenFileSync extends LiveSyncCommands {
const id = await this.path2id(file.path, ICHeader);
const prefixedFileName = addPrefix(file.path, ICHeader);
const contentBin = await this.app.vault.adapter.readBinary(file.path);
let content: string[];
try {
content = await encodeBinary(contentBin, this.settings.useV1);
} catch (ex) {
Logger(`The file ${file.path} could not be encoded`);
Logger(ex, LOG_LEVEL_VERBOSE);
return false;
}
const content = createBlob(await this.plugin.vaultAccess.adapterReadAuto(file.path));
const mtime = file.mtime;
return await serialized("file-" + prefixedFileName, async () => {
try {
const old = await this.localDatabase.getDBEntry(prefixedFileName, null, false, false);
let saveData: LoadedEntry;
const old = await this.localDatabase.getDBEntry(prefixedFileName, undefined, false, false);
let saveData: SavingEntry;
if (old === false) {
saveData = {
_id: id,
@@ -434,9 +472,10 @@ export class HiddenFileSync extends LiveSyncCommands {
children: [],
deleted: false,
type: "newnote",
eden: {},
};
} else {
if (isDocContentSame(old.data, content) && !forceWrite) {
if (await isDocContentSame(readAsBlob(old), content) && !forceWrite) {
// Logger(`STORAGE --> DB:${file.path}: (hidden) Not changed`, LOG_LEVEL_VERBOSE);
return;
}
@@ -446,13 +485,13 @@ export class HiddenFileSync extends LiveSyncCommands {
data: content,
mtime,
size: file.size,
datatype: "newnote",
datatype: old.datatype,
children: [],
deleted: false,
type: "newnote",
type: old.datatype,
};
}
const ret = await this.localDatabase.putDBEntry(saveData, true);
const ret = await this.localDatabase.putDBEntry(saveData);
Logger(`STORAGE --> DB:${file.path}: (hidden) Done`);
return ret;
} catch (ex) {
@@ -472,7 +511,7 @@ export class HiddenFileSync extends LiveSyncCommands {
}
await serialized("file-" + prefixedFileName, async () => {
try {
const old = await this.localDatabase.getDBEntryMeta(prefixedFileName, null, true) as InternalFileEntry | false;
const old = await this.localDatabase.getDBEntryMeta(prefixedFileName, undefined, true) as InternalFileEntry | false;
let saveData: InternalFileEntry;
if (old === false) {
saveData = {
@@ -484,8 +523,17 @@ export class HiddenFileSync extends LiveSyncCommands {
children: [],
deleted: true,
type: "newnote",
eden: {}
};
} else {
// Remove all conflicted before deleting.
const conflicts = await this.localDatabase.getRaw(old._id, { conflicts: true });
if (conflicts._conflicts !== undefined) {
for (const conflictRev of conflicts._conflicts) {
await this.localDatabase.removeRevision(old._id, conflictRev);
Logger(`STORAGE -x> DB:${filename}: (hidden) conflict removed ${old._rev} => ${conflictRev}`, LOG_LEVEL_VERBOSE);
}
}
if (old.deleted) {
Logger(`STORAGE -x> DB:${filename}: (hidden) already deleted`);
return;
@@ -511,7 +559,7 @@ export class HiddenFileSync extends LiveSyncCommands {
}
async extractInternalFileFromDatabase(filename: FilePath, force = false) {
const isExists = await this.app.vault.adapter.exists(filename);
const isExists = await this.plugin.vaultAccess.adapterExists(filename);
const prefixedFileName = addPrefix(filename, ICHeader);
if (await this.plugin.isIgnoredByIgnoreFiles(filename)) {
return;
@@ -519,8 +567,7 @@ export class HiddenFileSync extends LiveSyncCommands {
return await serialized("file-" + prefixedFileName, async () => {
try {
// Check conflicted status
//TODO option
const fileOnDB = await this.localDatabase.getDBEntry(prefixedFileName, { conflicts: true }, false, true);
const fileOnDB = await this.localDatabase.getDBEntry(prefixedFileName, { conflicts: true }, false, true, true);
if (fileOnDB === false)
throw new Error(`File not found on database.:${filename}`);
// Prevent overwrite for Prevent overwriting while some conflicted revision exists.
@@ -528,13 +575,13 @@ export class HiddenFileSync extends LiveSyncCommands {
Logger(`Hidden file ${filename} has conflicted revisions, to keep in safe, writing to storage has been prevented`, LOG_LEVEL_INFO);
return;
}
const deleted = "deleted" in fileOnDB ? fileOnDB.deleted : false;
const deleted = fileOnDB.deleted || fileOnDB._deleted || false;
if (deleted) {
if (!isExists) {
Logger(`STORAGE <x- DB:${filename}: deleted (hidden) Deleted on DB, but the file is already not found on storage.`);
Logger(`STORAGE <x- DB:${filename}: deleted (hidden) Deleted on DB, but the file is already not found on storage.`);
} else {
Logger(`STORAGE <x- DB:${filename}: deleted (hidden).`);
await this.app.vault.adapter.remove(filename);
await this.plugin.vaultAccess.adapterRemove(filename);
try {
//@ts-ignore internalAPI
await this.app.vault.adapter.reconcileInternalFile(filename);
@@ -546,8 +593,8 @@ export class HiddenFileSync extends LiveSyncCommands {
return true;
}
if (!isExists) {
await this.ensureDirectoryEx(filename);
await this.app.vault.adapter.writeBinary(filename, decodeBinary(fileOnDB.data), { mtime: fileOnDB.mtime, ctime: fileOnDB.ctime });
await this.vaultAccess.ensureDirectory(filename);
await this.plugin.vaultAccess.adapterWrite(filename, readContent(fileOnDB), { mtime: fileOnDB.mtime, ctime: fileOnDB.ctime });
try {
//@ts-ignore internalAPI
await this.app.vault.adapter.reconcileInternalFile(filename);
@@ -558,13 +605,13 @@ export class HiddenFileSync extends LiveSyncCommands {
Logger(`STORAGE <-- DB:${filename}: written (hidden,new${force ? ", force" : ""})`);
return true;
} else {
const contentBin = await this.app.vault.adapter.readBinary(filename);
const content = await encodeBinary(contentBin, this.settings.useV1);
if (isDocContentSame(content, fileOnDB.data) && !force) {
const content = await this.plugin.vaultAccess.adapterReadAuto(filename);
const docContent = readContent(fileOnDB);
if (await isDocContentSame(content, docContent) && !force) {
// Logger(`STORAGE <-- DB:${filename}: skipped (hidden) Not changed`, LOG_LEVEL_VERBOSE);
return true;
}
await this.app.vault.adapter.writeBinary(filename, decodeBinary(fileOnDB.data), { mtime: fileOnDB.mtime, ctime: fileOnDB.ctime });
await this.plugin.vaultAccess.adapterWrite(filename, docContent, { mtime: fileOnDB.mtime, ctime: fileOnDB.ctime });
try {
//@ts-ignore internalAPI
await this.app.vault.adapter.reconcileInternalFile(filename);
@@ -587,7 +634,7 @@ export class HiddenFileSync extends LiveSyncCommands {
showJSONMergeDialogAndMerge(docA: LoadedEntry, docB: LoadedEntry): Promise<boolean> {
return serialized("conflict:merge-data", () => new Promise((res) => {
return new Promise((res) => {
Logger("Opening data-merging dialog", LOG_LEVEL_VERBOSE);
const docs = [docA, docB];
const path = stripAllPrefixes(docA.path);
@@ -613,13 +660,17 @@ export class HiddenFileSync extends LiveSyncCommands {
}
}
if (!keep && result) {
const isExists = await this.app.vault.adapter.exists(filename);
const isExists = await this.plugin.vaultAccess.adapterExists(filename);
if (!isExists) {
await this.ensureDirectoryEx(filename);
await this.vaultAccess.ensureDirectory(filename);
}
await this.app.vault.adapter.write(filename, result);
const stat = await this.app.vault.adapter.stat(filename);
await this.storeInternalFileToDatabase({ path: filename, ...stat }, true);
await this.plugin.vaultAccess.adapterWrite(filename, result);
const stat = await this.plugin.vaultAccess.adapterStat(filename);
if (!stat) {
throw new Error("Stat failed");
}
const mtime = stat?.mtime ?? 0;
await this.storeInternalFileToDatabase({ path: filename, mtime, ctime: stat?.ctime ?? mtime, size: stat?.size ?? 0 }, true);
try {
//@ts-ignore internalAPI
await this.app.vault.adapter.reconcileInternalFile(filename);
@@ -641,7 +692,7 @@ export class HiddenFileSync extends LiveSyncCommands {
}
});
modal.open();
}));
});
}
async scanInternalFiles(): Promise<InternalFileInfo[]> {
@@ -653,11 +704,11 @@ export class HiddenFileSync extends LiveSyncCommands {
const root = this.app.vault.getRoot();
const findRoot = root.path;
const filenames = (await this.getFiles(findRoot, [], null, ignoreFilter)).filter(e => e.startsWith(".")).filter(e => !e.startsWith(".trash"));
const filenames = (await this.getFiles(findRoot, [], undefined, ignoreFilter)).filter(e => e.startsWith(".")).filter(e => !e.startsWith(".trash"));
const files = filenames.filter(path => synchronisedInConfigSync.every(filterFile => !path.toLowerCase().startsWith(filterFile))).map(async (e) => {
return {
path: e as FilePath,
stat: await this.app.vault.adapter.stat(e)
stat: await this.plugin.vaultAccess.adapterStat(e)
};
});
const result: InternalFileInfo[] = [];
@@ -666,9 +717,12 @@ export class HiddenFileSync extends LiveSyncCommands {
if (await this.plugin.isIgnoredByIgnoreFiles(w.path)) {
continue
}
const mtime = w.stat?.mtime ?? 0
const ctime = w.stat?.ctime ?? mtime;
const size = w.stat?.size ?? 0;
result.push({
...w,
...w.stat
mtime, ctime, size
});
}
return result;
@@ -679,11 +733,17 @@ export class HiddenFileSync extends LiveSyncCommands {
async getFiles(
path: string,
ignoreList: string[],
filter: RegExp[],
ignoreFilter: RegExp[]
filter?: RegExp[],
ignoreFilter?: RegExp[]
) {
const w = await this.app.vault.adapter.list(path);
let w: ListedFiles;
try {
w = await this.app.vault.adapter.list(path);
} catch (ex) {
Logger(`Could not traverse(HiddenSync):${path}`, LOG_LEVEL_INFO);
Logger(ex, LOG_LEVEL_VERBOSE);
return [];
}
const filesSrc = [
...w.files
.filter((e) => !ignoreList.some((ee) => e.endsWith(ee)))

108
src/CmdSetupLiveSync.ts → src/features/CmdSetupLiveSync.ts
View File

@@ -1,14 +1,15 @@
import { type EntryDoc, type ObsidianLiveSyncSettings, DEFAULT_SETTINGS, LOG_LEVEL_NOTICE } from "./lib/src/types";
import { configURIBase } from "./types";
import { Logger } from "./lib/src/logger";
import { PouchDB } from "./lib/src/pouchdb-browser.js";
import { askSelectString, askYesNo, askString } from "./utils";
import { decrypt, encrypt } from "./lib/src/e2ee_v2";
import { LiveSyncCommands } from "./LiveSyncCommands";
import { delay } from "./lib/src/utils";
import { confirmWithMessage } from "./dialogs";
import { Platform } from "./deps";
import { fetchAllUsedChunks } from "./lib/src/utils_couchdb";
import { type EntryDoc, type ObsidianLiveSyncSettings, DEFAULT_SETTINGS, LOG_LEVEL_NOTICE, REMOTE_COUCHDB, REMOTE_MINIO } from "../lib/src/common/types.ts";
import { configURIBase } from "../common/types.ts";
import { Logger } from "../lib/src/common/logger.ts";
import { PouchDB } from "../lib/src/pouchdb/pouchdb-browser.js";
import { askSelectString, askYesNo, askString } from "../common/utils.ts";
import { decrypt, encrypt } from "../lib/src/encryption/e2ee_v2.ts";
import { LiveSyncCommands } from "./LiveSyncCommands.ts";
import { delay, fireAndForget } from "../lib/src/common/utils.ts";
import { confirmWithMessage } from "../common/dialogs.ts";
import { Platform } from "../deps.ts";
import { fetchAllUsedChunks } from "../lib/src/pouchdb/utils_couchdb.ts";
import type { LiveSyncCouchDBReplicator } from "../lib/src/replication/couchdb/LiveSyncReplicator.js";
export class SetupLiveSync extends LiveSyncCommands {
onunload() { }
@@ -17,25 +18,25 @@ export class SetupLiveSync extends LiveSyncCommands {
this.plugin.addCommand({
id: "livesync-copysetupuri",
name: "Copy the setup URI",
callback: this.command_copySetupURI.bind(this),
name: "Copy settings as a new setup URI",
callback: () => fireAndForget(this.command_copySetupURI()),
});
this.plugin.addCommand({
id: "livesync-copysetupuri-short",
name: "Copy the setup URI (With customization sync)",
callback: this.command_copySetupURIWithSync.bind(this),
name: "Copy settings as a new setup URI (With customization sync)",
callback: () => fireAndForget(this.command_copySetupURIWithSync()),
});
this.plugin.addCommand({
id: "livesync-copysetupurifull",
name: "Copy the setup URI (Full)",
callback: this.command_copySetupURIFull.bind(this),
name: "Copy settings as a new setup URI (Full)",
callback: () => fireAndForget(this.command_copySetupURIFull()),
});
this.plugin.addCommand({
id: "livesync-opensetupuri",
name: "Open the setup URI",
callback: this.command_openSetupURI.bind(this),
name: "Use the copied setup URI (Formerly Open setup URI)",
callback: () => fireAndForget(this.command_openSetupURI()),
});
}
onInitializeDatabase(showNotice: boolean) { }
@@ -50,7 +51,7 @@ export class SetupLiveSync extends LiveSyncCommands {
const encryptingPassphrase = await askString(this.app, "Encrypt your settings", "The passphrase to encrypt the setup URI", "", true);
if (encryptingPassphrase === false)
return;
const setting = { ...this.settings, configPassphraseStore: "", encryptedCouchDBConnection: "", encryptedPassphrase: "" };
const setting = { ...this.settings, configPassphraseStore: "", encryptedCouchDBConnection: "", encryptedPassphrase: "" } as Partial<ObsidianLiveSyncSettings>;
if (stripExtra) {
delete setting.pluginSyncExtendedSetting;
}
@@ -60,7 +61,7 @@ export class SetupLiveSync extends LiveSyncCommands {
delete setting[k];
}
}
const encryptedSetting = encodeURIComponent(await encrypt(JSON.stringify(setting), encryptingPassphrase, false, false));
const encryptedSetting = encodeURIComponent(await encrypt(JSON.stringify(setting), encryptingPassphrase, false));
const uri = `${configURIBase}${encryptedSetting}`;
await navigator.clipboard.writeText(uri);
Logger("Setup URI copied to clipboard", LOG_LEVEL_NOTICE);
@@ -70,13 +71,13 @@ export class SetupLiveSync extends LiveSyncCommands {
if (encryptingPassphrase === false)
return;
const setting = { ...this.settings, configPassphraseStore: "", encryptedCouchDBConnection: "", encryptedPassphrase: "" };
const encryptedSetting = encodeURIComponent(await encrypt(JSON.stringify(setting), encryptingPassphrase, false, false));
const encryptedSetting = encodeURIComponent(await encrypt(JSON.stringify(setting), encryptingPassphrase, false));
const uri = `${configURIBase}${encryptedSetting}`;
await navigator.clipboard.writeText(uri);
Logger("Setup URI copied to clipboard", LOG_LEVEL_NOTICE);
}
async command_copySetupURIWithSync() {
this.command_copySetupURI(false);
await this.command_copySetupURI(false);
}
async command_openSetupURI() {
const setupURI = await askString(this.app, "Easy setup", "Set up URI", `${configURIBase}aaaaa`);
@@ -108,6 +109,7 @@ export class SetupLiveSync extends LiveSyncCommands {
newSettingW.configPassphraseStore = "";
newSettingW.encryptedPassphrase = "";
newSettingW.encryptedCouchDBConnection = "";
newSettingW.additionalSuffixOfDatabaseName = `${("appId" in this.app ? this.app.appId : "")}`
const setupJustImport = "Just import setting";
const setupAsNew = "Set it up as secondary or subsequent device";
const setupAsMerge = "Secondary device but try keeping local changes";
@@ -115,6 +117,7 @@ export class SetupLiveSync extends LiveSyncCommands {
const setupManually = "Leave everything to me";
newSettingW.syncInternalFiles = false;
newSettingW.usePluginSync = false;
newSettingW.isConfigured = true;
// Migrate completely obsoleted configuration.
if (!newSettingW.useIndexedDBAdapter) {
newSettingW.useIndexedDBAdapter = true;
@@ -132,7 +135,7 @@ export class SetupLiveSync extends LiveSyncCommands {
} else if (setupType == setupAsMerge) {
this.plugin.settings = newSettingW;
this.plugin.usedPassphrase = "";
await this.fetchLocalWithKeepLocal();
await this.fetchLocalWithRebuild();
} else if (setupType == setupAgain) {
const confirm = "I know this operation will rebuild all my databases with files on this device, and files that are on the remote database and I didn't synchronize to any other devices will be lost and want to proceed indeed.";
if (await askSelectString(this.app, "Do you really want to do this?", ["Cancel", confirm]) != confirm) {
@@ -309,6 +312,7 @@ Of course, we are able to disable these features.`
}
async suspendReflectingDatabase() {
if (this.plugin.settings.doNotSuspendOnFetching) return;
if (this.plugin.settings.remoteType == REMOTE_MINIO) return;
Logger(`Suspending reflection: Database and storage changes will not be reflected in each other until completely finished the fetching.`, LOG_LEVEL_NOTICE);
this.plugin.settings.suspendParseReplicationResult = true;
this.plugin.settings.suspendFileWatching = true;
@@ -316,12 +320,12 @@ Of course, we are able to disable these features.`
}
async resumeReflectingDatabase() {
if (this.plugin.settings.doNotSuspendOnFetching) return;
if (this.plugin.settings.remoteType == REMOTE_MINIO) return;
Logger(`Database and storage reflection has been resumed!`, LOG_LEVEL_NOTICE);
this.plugin.settings.suspendParseReplicationResult = false;
this.plugin.settings.suspendFileWatching = false;
await this.plugin.syncAllFiles(true);
await this.plugin.loadQueuedFiles();
this.plugin.procQueuedFiles();
await this.plugin.saveSettings();
}
@@ -334,14 +338,23 @@ Of course, we are able to disable these features.`
const ret = await confirmWithMessage(this.plugin, "Database adapter", message, choices, CHOICE_YES, 10);
if (ret == CHOICE_YES) {
this.plugin.settings.useIndexedDBAdapter = false;
this.plugin.settings.useIndexedDBAdapter = true;
}
}
}
async resetLocalDatabase() {
if (this.plugin.settings.isConfigured && this.plugin.settings.additionalSuffixOfDatabaseName == "") {
// Discard the non-suffixed database
await this.plugin.resetLocalDatabase();
}
this.plugin.settings.additionalSuffixOfDatabaseName = `${("appId" in this.app ? this.app.appId : "")}`
await this.plugin.resetLocalDatabase();
}
async fetchRemoteChunks() {
if (!this.plugin.settings.doNotSuspendOnFetching && this.plugin.settings.readChunksOnline) {
if (!this.plugin.settings.doNotSuspendOnFetching && this.plugin.settings.readChunksOnline && this.plugin.settings.remoteType == REMOTE_COUCHDB) {
Logger(`Fetching chunks`, LOG_LEVEL_NOTICE);
const remoteDB = await this.plugin.getReplicator().connectRemoteCouchDBWithSetting(this.settings, this.plugin.getIsMobile(), true);
const replicator = this.plugin.getReplicator() as LiveSyncCouchDBReplicator;
const remoteDB = await replicator.connectRemoteCouchDBWithSetting(this.settings, this.plugin.getIsMobile(), true);
if (typeof remoteDB == "string") {
Logger(remoteDB, LOG_LEVEL_NOTICE);
} else {
@@ -350,45 +363,33 @@ Of course, we are able to disable these features.`
Logger(`Fetching chunks done`, LOG_LEVEL_NOTICE);
}
}
async fetchLocal() {
async fetchLocal(makeLocalChunkBeforeSync?: boolean) {
this.suspendExtraSync();
this.askUseNewAdapter();
await this.askUseNewAdapter();
this.plugin.settings.isConfigured = true;
await this.suspendReflectingDatabase();
await this.plugin.realizeSettingSyncMode();
await this.plugin.resetLocalDatabase();
await this.resetLocalDatabase();
await delay(1000);
await this.plugin.markRemoteResolved();
await this.plugin.openDatabase();
this.plugin.isReady = true;
if (makeLocalChunkBeforeSync) {
await this.plugin.initializeDatabase(true);
}
await this.plugin.markRemoteResolved();
await delay(500);
await this.plugin.replicateAllFromServer(true);
await delay(1000);
await this.plugin.replicateAllFromServer(true);
await this.fetchRemoteChunks();
await this.resumeReflectingDatabase();
await this.askHiddenFileConfiguration({ enableFetch: true });
}
async fetchLocalWithKeepLocal() {
this.suspendExtraSync();
this.askUseNewAdapter();
await this.suspendReflectingDatabase();
await this.plugin.realizeSettingSyncMode();
await this.plugin.resetLocalDatabase();
await delay(1000);
await this.plugin.initializeDatabase(true);
await this.plugin.markRemoteResolved();
await this.plugin.openDatabase();
this.plugin.isReady = true;
await delay(500);
await this.plugin.replicateAllFromServer(true);
await delay(1000);
await this.plugin.replicateAllFromServer(true);
await this.fetchRemoteChunks();
await this.resumeReflectingDatabase();
await this.askHiddenFileConfiguration({ enableFetch: true });
async fetchLocalWithRebuild() {
return await this.fetchLocal(true);
}
async rebuildRemote() {
this.suspendExtraSync();
this.plugin.settings.isConfigured = true;
await this.plugin.realizeSettingSyncMode();
await this.plugin.markRemoteLocked();
await this.plugin.tryResetRemoteDatabase();
@@ -402,9 +403,10 @@ Of course, we are able to disable these features.`
}
async rebuildEverything() {
this.suspendExtraSync();
this.askUseNewAdapter();
await this.askUseNewAdapter();
this.plugin.settings.isConfigured = true;
await this.plugin.realizeSettingSyncMode();
await this.plugin.resetLocalDatabase();
await this.resetLocalDatabase();
await delay(1000);
await this.plugin.initializeDatabase(true);
await this.plugin.markRemoteLocked();

9
src/LiveSyncCommands.ts → src/features/LiveSyncCommands.ts
View File

@@ -1,6 +1,6 @@
import { type AnyEntry, type DocumentID, type EntryDoc, type EntryHasPath, type FilePath, type FilePathWithPrefix } from "./lib/src/types";
import { PouchDB } from "./lib/src/pouchdb-browser.js";
import type ObsidianLiveSyncPlugin from "./main";
import { type AnyEntry, type DocumentID, type EntryDoc, type EntryHasPath, type FilePath, type FilePathWithPrefix } from "../lib/src/common/types.ts";
import { PouchDB } from "../lib/src/pouchdb/pouchdb-browser.js";
import type ObsidianLiveSyncPlugin from "../main.ts";
export abstract class LiveSyncCommands {
@@ -14,6 +14,9 @@ export abstract class LiveSyncCommands {
get localDatabase() {
return this.plugin.localDatabase;
}
get vaultAccess() {
return this.plugin.vaultAccess;
}
id2path(id: DocumentID, entry?: EntryHasPath, stripPrefix?: boolean): FilePathWithPrefix {
return this.plugin.id2path(id, entry, stripPrefix);
}

2
src/lib

Submodule src/lib updated: 609c7aecf3...f0253a8548

2729
src/main.ts
View File

File diff suppressed because it is too large Load Diff

196
src/storages/SerializedFileAccess.ts Normal file
View File

@@ -0,0 +1,196 @@
import { type App, TFile, type DataWriteOptions, TFolder, TAbstractFile } from "../deps.ts";
import { serialized } from "../lib/src/concurrency/lock.ts";
import { Logger } from "../lib/src/common/logger.ts";
import { isPlainText } from "../lib/src/string_and_binary/path.ts";
import type { FilePath } from "../lib/src/common/types.ts";
import { createBinaryBlob, isDocContentSame } from "../lib/src/common/utils.ts";
import type { InternalFileInfo } from "../common/types.ts";
import { markChangesAreSame } from "../common/utils.ts";
function getFileLockKey(file: TFile | TFolder | string) {
return `fl:${typeof (file) == "string" ? file : file.path}`;
}
function toArrayBuffer(arr: Uint8Array | ArrayBuffer | DataView): ArrayBufferLike {
if (arr instanceof Uint8Array) {
return arr.buffer;
}
if (arr instanceof DataView) {
return arr.buffer;
}
return arr;
}
async function processReadFile<T>(file: TFile | TFolder | string, proc: () => Promise<T>) {
const ret = await serialized(getFileLockKey(file), () => proc());
return ret;
}
async function processWriteFile<T>(file: TFile | TFolder | string, proc: () => Promise<T>) {
const ret = await serialized(getFileLockKey(file), () => proc());
return ret;
}
export class SerializedFileAccess {
app: App
constructor(app: App) {
this.app = app;
}
async adapterStat(file: TFile | string) {
const path = file instanceof TFile ? file.path : file;
return await processReadFile(file, () => this.app.vault.adapter.stat(path));
}
async adapterExists(file: TFile | string) {
const path = file instanceof TFile ? file.path : file;
return await processReadFile(file, () => this.app.vault.adapter.exists(path));
}
async adapterRemove(file: TFile | string) {
const path = file instanceof TFile ? file.path : file;
return await processReadFile(file, () => this.app.vault.adapter.remove(path));
}
async adapterRead(file: TFile | string) {
const path = file instanceof TFile ? file.path : file;
return await processReadFile(file, () => this.app.vault.adapter.read(path));
}
async adapterReadBinary(file: TFile | string) {
const path = file instanceof TFile ? file.path : file;
return await processReadFile(file, () => this.app.vault.adapter.readBinary(path));
}
async adapterReadAuto(file: TFile | string) {
const path = file instanceof TFile ? file.path : file;
if (isPlainText(path)) return await processReadFile(file, () => this.app.vault.adapter.read(path));
return await processReadFile(file, () => this.app.vault.adapter.readBinary(path));
}
async adapterWrite(file: TFile | string, data: string | ArrayBuffer | Uint8Array, options?: DataWriteOptions) {
const path = file instanceof TFile ? file.path : file;
if (typeof (data) === "string") {
return await processWriteFile(file, () => this.app.vault.adapter.write(path, data, options));
} else {
return await processWriteFile(file, () => this.app.vault.adapter.writeBinary(path, toArrayBuffer(data), options));
}
}
async vaultCacheRead(file: TFile) {
return await processReadFile(file, () => this.app.vault.cachedRead(file));
}
async vaultRead(file: TFile) {
return await processReadFile(file, () => this.app.vault.read(file));
}
async vaultReadBinary(file: TFile) {
return await processReadFile(file, () => this.app.vault.readBinary(file));
}
async vaultReadAuto(file: TFile) {
const path = file.path;
if (isPlainText(path)) return await processReadFile(file, () => this.app.vault.read(file));
return await processReadFile(file, () => this.app.vault.readBinary(file));
}
async vaultModify(file: TFile, data: string | ArrayBuffer | Uint8Array, options?: DataWriteOptions) {
if (typeof (data) === "string") {
return await processWriteFile(file, async () => {
const oldData = await this.app.vault.read(file);
if (data === oldData) {
if (options && options.mtime) markChangesAreSame(file, file.stat.mtime, options.mtime);
return false
}
await this.app.vault.modify(file, data, options)
return true;
}
);
} else {
return await processWriteFile(file, async () => {
const oldData = await this.app.vault.readBinary(file);
if (await isDocContentSame(createBinaryBlob(oldData), createBinaryBlob(data))) {
if (options && options.mtime) markChangesAreSame(file, file.stat.mtime, options.mtime);
return false;
}
await this.app.vault.modifyBinary(file, toArrayBuffer(data), options)
return true;
});
}
}
async vaultCreate(path: string, data: string | ArrayBuffer | Uint8Array, options?: DataWriteOptions): Promise<TFile> {
if (typeof (data) === "string") {
return await processWriteFile(path, () => this.app.vault.create(path, data, options));
} else {
return await processWriteFile(path, () => this.app.vault.createBinary(path, toArrayBuffer(data), options));
}
}
trigger(name: string, ...data: any[]) {
return this.app.vault.trigger(name, ...data);
}
async adapterAppend(normalizedPath: string, data: string, options?: DataWriteOptions) {
return await this.app.vault.adapter.append(normalizedPath, data, options)
}
async delete(file: TFile | TFolder, force = false) {
return await processWriteFile(file, () => this.app.vault.delete(file, force));
}
async trash(file: TFile | TFolder, force = false) {
return await processWriteFile(file, () => this.app.vault.trash(file, force));
}
getAbstractFileByPath(path: FilePath | string): TAbstractFile | null {
// Disabled temporary.
return this.app.vault.getAbstractFileByPath(path);
// // Hidden API but so useful.
// // @ts-ignore
// if ("getAbstractFileByPathInsensitive" in app.vault && (app.vault.adapter?.insensitive ?? false)) {
// // @ts-ignore
// return app.vault.getAbstractFileByPathInsensitive(path);
// } else {
// return app.vault.getAbstractFileByPath(path);
// }
}
getFiles() {
return this.app.vault.getFiles();
}
async ensureDirectory(fullPath: string) {
const pathElements = fullPath.split("/");
pathElements.pop();
let c = "";
for (const v of pathElements) {
c += v;
try {
await this.app.vault.adapter.mkdir(c);
} catch (ex: any) {
if (ex?.message == "Folder already exists.") {
// Skip if already exists.
} else {
Logger("Folder Create Error");
Logger(ex);
}
}
c += "/";
}
}
touchedFiles: string[] = [];
touch(file: TFile | FilePath) {
const f = file instanceof TFile ? file : this.getAbstractFileByPath(file) as TFile;
const key = `${f.path}-${f.stat.mtime}-${f.stat.size}`;
this.touchedFiles.unshift(key);
this.touchedFiles = this.touchedFiles.slice(0, 100);
}
recentlyTouched(file: TFile | InternalFileInfo) {
const key = file instanceof TFile ? `${file.path}-${file.stat.mtime}-${file.stat.size}` : `${file.path}-${file.mtime}-${file.size}`;
if (this.touchedFiles.indexOf(key) == -1) return false;
return true;
}
clearTouched() {
this.touchedFiles = [];
}
}

307
src/storages/StorageEventManager.ts Normal file
View File

@@ -0,0 +1,307 @@
import type { SerializedFileAccess } from "./SerializedFileAccess.ts";
import { Plugin, TAbstractFile, TFile, TFolder } from "../deps.ts";
import { Logger } from "../lib/src/common/logger.ts";
import { shouldBeIgnored } from "../lib/src/string_and_binary/path.ts";
import { LOG_LEVEL_DEBUG, LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, type FilePath, type ObsidianLiveSyncSettings } from "../lib/src/common/types.ts";
import { delay, fireAndForget } from "../lib/src/common/utils.ts";
import { type FileEventItem, type FileEventType, type FileInfo, type InternalFileInfo } from "../common/types.ts";
import { skipIfDuplicated } from "../lib/src/concurrency/lock.ts";
import { finishAllWaitingForTimeout, finishWaitingForTimeout, isWaitingForTimeout, waitForTimeout } from "../lib/src/concurrency/task.ts";
import { reactiveSource, type ReactiveSource } from "../lib/src/dataobject/reactive.ts";
import { Semaphore } from "../lib/src/concurrency/semaphore.ts";
export type FileEvent = {
type: FileEventType;
file: TAbstractFile | InternalFileInfo;
oldPath?: string;
cachedData?: string;
skipBatchWait?: boolean;
};
export abstract class StorageEventManager {
abstract beginWatch(): void;
abstract flushQueue(): void;
abstract appendQueue(items: FileEvent[], ctx?: any): void;
abstract cancelQueue(key: string): void;
abstract isWaiting(filename: FilePath): boolean;
abstract totalQueued: ReactiveSource<number>;
abstract batched: ReactiveSource<number>;
abstract processing: ReactiveSource<number>;
}
type LiveSyncForStorageEventManager = Plugin &
{
settings: ObsidianLiveSyncSettings
ignoreFiles: string[],
vaultAccess: SerializedFileAccess
shouldBatchSave: boolean
batchSaveMinimumDelay: number;
batchSaveMaximumDelay: number;
} & {
isTargetFile: (file: string | TAbstractFile) => Promise<boolean>,
// fileEventQueue: QueueProcessor<FileEventItem, any>,
handleFileEvent: (queue: FileEventItem) => Promise<any>,
isFileSizeExceeded: (size: number) => boolean;
};
export class StorageEventManagerObsidian extends StorageEventManager {
totalQueued = reactiveSource(0);
batched = reactiveSource(0);
processing = reactiveSource(0);
plugin: LiveSyncForStorageEventManager;
get shouldBatchSave() {
return this.plugin.shouldBatchSave;
}
get batchSaveMinimumDelay(): number {
return this.plugin.batchSaveMinimumDelay;
}
get batchSaveMaximumDelay(): number {
return this.plugin.batchSaveMaximumDelay
}
constructor(plugin: LiveSyncForStorageEventManager) {
super();
this.plugin = plugin;
}
beginWatch() {
const plugin = this.plugin;
this.watchVaultChange = this.watchVaultChange.bind(this);
this.watchVaultCreate = this.watchVaultCreate.bind(this);
this.watchVaultDelete = this.watchVaultDelete.bind(this);
this.watchVaultRename = this.watchVaultRename.bind(this);
this.watchVaultRawEvents = this.watchVaultRawEvents.bind(this);
plugin.registerEvent(plugin.app.vault.on("modify", this.watchVaultChange));
plugin.registerEvent(plugin.app.vault.on("delete", this.watchVaultDelete));
plugin.registerEvent(plugin.app.vault.on("rename", this.watchVaultRename));
plugin.registerEvent(plugin.app.vault.on("create", this.watchVaultCreate));
//@ts-ignore : Internal API
plugin.registerEvent(plugin.app.vault.on("raw", this.watchVaultRawEvents));
// plugin.fileEventQueue.startPipeline();
}
watchVaultCreate(file: TAbstractFile, ctx?: any) {
this.appendQueue([{ type: "CREATE", file }], ctx);
}
watchVaultChange(file: TAbstractFile, ctx?: any) {
this.appendQueue([{ type: "CHANGED", file }], ctx);
}
watchVaultDelete(file: TAbstractFile, ctx?: any) {
this.appendQueue([{ type: "DELETE", file }], ctx);
}
watchVaultRename(file: TAbstractFile, oldFile: string, ctx?: any) {
if (file instanceof TFile) {
this.appendQueue([
{ type: "DELETE", file: { path: oldFile as FilePath, mtime: file.stat.mtime, ctime: file.stat.ctime, size: file.stat.size, deleted: true }, skipBatchWait: true },
{ type: "CREATE", file, skipBatchWait: true },
], ctx);
}
}
// Watch raw events (Internal API)
watchVaultRawEvents(path: FilePath) {
if (this.plugin.settings.useIgnoreFiles && this.plugin.ignoreFiles.some(e => path.endsWith(e.trim()))) {
// If it is one of ignore files, refresh the cached one.
this.plugin.isTargetFile(path).then(() => this._watchVaultRawEvents(path));
} else {
this._watchVaultRawEvents(path);
}
}
_watchVaultRawEvents(path: FilePath) {
if (!this.plugin.settings.syncInternalFiles && !this.plugin.settings.usePluginSync) return;
if (!this.plugin.settings.watchInternalFileChanges) return;
if (!path.startsWith(this.plugin.app.vault.configDir)) return;
const ignorePatterns = this.plugin.settings.syncInternalFilesIgnorePatterns
.replace(/\n| /g, "")
.split(",").filter(e => e).map(e => new RegExp(e, "i"));
if (ignorePatterns.some(e => path.match(e))) return;
this.appendQueue(
[{
type: "INTERNAL",
file: { path, mtime: 0, ctime: 0, size: 0 }
}], null);
}
// Cache file and waiting to can be proceed.
async appendQueue(params: FileEvent[], ctx?: any) {
if (!this.plugin.settings.isConfigured) return;
if (this.plugin.settings.suspendFileWatching) return;
const processFiles = new Set<FilePath>();
for (const param of params) {
if (shouldBeIgnored(param.file.path)) {
continue;
}
const atomicKey = [0, 0, 0, 0, 0, 0].map(e => `${Math.floor(Math.random() * 100000)}`).join("-");
const type = param.type;
const file = param.file;
const oldPath = param.oldPath;
const size = file instanceof TFile ? file.stat.size : (file as InternalFileInfo)?.size ?? 0;
if (this.plugin.isFileSizeExceeded(size) && (type == "CREATE" || type == "CHANGED")) {
Logger(`The storage file has been changed but exceeds the maximum size. Skipping: ${param.file.path}`, LOG_LEVEL_NOTICE);
continue;
}
if (file instanceof TFolder) continue;
if (!await this.plugin.isTargetFile(file.path)) continue;
// Stop cache using to prevent the corruption;
// let cache: null | string | ArrayBuffer;
// new file or something changed, cache the changes.
if (file instanceof TFile && (type == "CREATE" || type == "CHANGED")) {
// Wait for a bit while to let the writer has marked `touched` at the file.
await delay(10);
if (this.plugin.vaultAccess.recentlyTouched(file)) {
continue;
}
}
const fileInfo = file instanceof TFile ? {
ctime: file.stat.ctime,
mtime: file.stat.mtime,
file: file,
path: file.path,
size: file.stat.size
} as FileInfo : file as InternalFileInfo;
let cache: string | undefined = undefined;
if (param.cachedData) {
cache = param.cachedData
}
this.enqueue({
type,
args: {
file: fileInfo,
oldPath,
cache,
ctx,
},
skipBatchWait: param.skipBatchWait,
key: atomicKey
})
processFiles.add(file.path as FilePath);
if (oldPath) {
processFiles.add(oldPath as FilePath);
}
}
for (const path of processFiles) {
fireAndForget(() => this.startStandingBy(path));
}
}
bufferedQueuedItems = [] as FileEventItem[];
enqueue(newItem: FileEventItem) {
const filename = newItem.args.file.path;
if (this.shouldBatchSave) {
Logger(`Request cancel for waiting of previous ${filename}`, LOG_LEVEL_DEBUG);
finishWaitingForTimeout(`storage-event-manager-batchsave-${filename}`);
}
this.bufferedQueuedItems.push(newItem);
// When deleting or renaming, the queue must be flushed once before processing subsequent processes to prevent unexpected race condition.
if (newItem.type == "DELETE" || newItem.type == "RENAME") {
return this.flushQueue();
}
}
concurrentProcessing = Semaphore(5);
waitedSince = new Map<FilePath, number>();
async startStandingBy(filename: FilePath) {
// If waited, cancel previous waiting.
await skipIfDuplicated(`storage-event-manager-${filename}`, async () => {
Logger(`Processing ${filename}: Starting`, LOG_LEVEL_DEBUG);
const release = await this.concurrentProcessing.acquire();
try {
Logger(`Processing ${filename}: Started`, LOG_LEVEL_DEBUG);
let noMoreFiles = false;
do {
const target = this.bufferedQueuedItems.find(e => e.args.file.path == filename);
if (target === undefined) {
noMoreFiles = true;
break;
}
const operationType = target.type;
// if (target.waitedFrom + this.batchSaveMaximumDelay > now) {
// this.requestProcessQueue(target);
// continue;
// }
const type = target.type;
if (target.cancelled) {
Logger(`Processing ${filename}: Cancelled (scheduled): ${operationType}`, LOG_LEVEL_DEBUG)
this.cancelStandingBy(target);
continue;
}
if (!target.skipBatchWait) {
if (this.shouldBatchSave && (type == "CREATE" || type == "CHANGED")) {
const waitedSince = this.waitedSince.get(filename);
let canWait = true;
const now = Date.now();
if (waitedSince !== undefined) {
if (waitedSince + (this.batchSaveMaximumDelay * 1000) < now) {
Logger(`Processing ${filename}: Could not wait no more: ${operationType}`, LOG_LEVEL_INFO)
canWait = false;
}
}
if (canWait) {
if (waitedSince === undefined) this.waitedSince.set(filename, now)
target.batched = true
Logger(`Processing ${filename}: Waiting for batch save delay: ${operationType}`, LOG_LEVEL_DEBUG)
this.updateStatus();
const result = await waitForTimeout(`storage-event-manager-batchsave-${filename}`, this.batchSaveMinimumDelay * 1000);
if (!result) {
Logger(`Processing ${filename}: Cancelled by new queue: ${operationType}`, LOG_LEVEL_DEBUG)
// If could not wait for the timeout, possibly we got a new queue. therefore, currently processing one should be cancelled
this.cancelStandingBy(target);
continue;
}
}
}
} else {
Logger(`Processing ${filename}:Requested to perform immediately ${filename}: ${operationType}`, LOG_LEVEL_DEBUG)
}
Logger(`Processing ${filename}: Request main to process: ${operationType}`, LOG_LEVEL_DEBUG)
this.requestProcessQueue(target);
} while (!noMoreFiles)
} finally {
release()
}
Logger(`Processing ${filename}: Finished`, LOG_LEVEL_DEBUG);
})
}
cancelStandingBy(fei: FileEventItem) {
this.bufferedQueuedItems.remove(fei);
this.updateStatus();
}
processingCount = 0;
async requestProcessQueue(fei: FileEventItem) {
try {
this.processingCount++;
this.bufferedQueuedItems.remove(fei);
this.updateStatus()
this.waitedSince.delete(fei.args.file.path);
await this.plugin.handleFileEvent(fei);
} finally {
this.processingCount--;
this.updateStatus()
}
}
isWaiting(filename: FilePath) {
return isWaitingForTimeout(`storage-event-manager-batchsave-${filename}`);
}
flushQueue() {
this.bufferedQueuedItems.forEach(e => e.skipBatchWait = true)
finishAllWaitingForTimeout("storage-event-manager-batchsave-", true);
}
cancelQueue(key: string) {
this.bufferedQueuedItems.forEach(e => {
if (e.key === key) e.skipBatchWait = true
})
}
updateStatus() {
const allItems = this.bufferedQueuedItems.filter(e => !e.cancelled)
this.batched.value = allItems.filter(e => e.batched && !e.skipBatchWait).length;
this.processing.value = this.processingCount;
this.totalQueued.value = allItems.length - this.batched.value;
}
}

50
src/tests/TestPane.svelte Normal file
View File

@@ -0,0 +1,50 @@
<script lang="ts">
import { onDestroy, onMount } from "svelte";
import type ObsidianLiveSyncPlugin from "../main";
import { perf_trench } from "./tests";
import { MarkdownRenderer } from "../deps";
export let plugin: ObsidianLiveSyncPlugin;
let performanceTestResult = "";
let functionCheckResult = "";
let testRunning = false;
let prefTestResultEl: HTMLDivElement;
let isReady = false;
$: {
if (performanceTestResult != "" && isReady) {
MarkdownRenderer.render(plugin.app, performanceTestResult, prefTestResultEl, "/", plugin);
}
}
async function performTest() {
try {
testRunning = true;
performanceTestResult = await perf_trench(plugin);
} finally {
testRunning = false;
}
}
function clearPerfTestResult() {
prefTestResultEl.empty();
}
onMount(() => {
isReady = true;
// performTest();
});
</script>
<h2>TESTBENCH: Self-hosted LiveSync</h2>
<h3>Function check</h3>
<pre>{functionCheckResult}</pre>
<h3>Performance test</h3>
<button on:click={() => performTest()} disabled={testRunning}>Test!</button>
<button on:click={() => clearPerfTestResult()}>Clear</button>
<div bind:this={prefTestResultEl}></div>
<style>
* {
box-sizing: border-box;
}
</style>

49
src/tests/TestPaneView.ts Normal file
View File

@@ -0,0 +1,49 @@
import {
ItemView,
WorkspaceLeaf
} from "obsidian";
import TestPaneComponent from "./TestPane.svelte"
import type ObsidianLiveSyncPlugin from "../main"
export const VIEW_TYPE_TEST = "ols-pane-test";
//Log view
export class TestPaneView extends ItemView {
component?: TestPaneComponent;
plugin: ObsidianLiveSyncPlugin;
icon = "view-log";
title: string = "Self-hosted LiveSync Test and Results"
navigation = true;
getIcon(): string {
return "view-log";
}
constructor(leaf: WorkspaceLeaf, plugin: ObsidianLiveSyncPlugin) {
super(leaf);
this.plugin = plugin;
}
getViewType() {
return VIEW_TYPE_TEST;
}
getDisplayText() {
return "Self-hosted LiveSync Test and Results";
}
// eslint-disable-next-line require-await
async onOpen() {
this.component = new TestPaneComponent({
target: this.contentEl,
props: {
plugin: this.plugin
},
});
}
// eslint-disable-next-line require-await
async onClose() {
this.component?.$destroy();
}
}

45
src/tests/testUtils.ts Normal file
View File

@@ -0,0 +1,45 @@
import { fireAndForget } from "src/lib/src/common/utils";
import { serialized } from "src/lib/src/concurrency/lock";
import type ObsidianLiveSyncPlugin from "src/main";
let plugin: ObsidianLiveSyncPlugin;
export function enableTestFunction(plugin_: ObsidianLiveSyncPlugin) {
plugin = plugin_;
}
export function addDebugFileLog(message: any, stackLog = false) {
fireAndForget(serialized("debug-log", async () => {
const now = new Date();
const filename = `debug-log`
const time = now.toISOString().split("T")[0];
const outFile = `${filename}${time}.jsonl`;
// const messageContent = typeof message == "string" ? message : message instanceof Error ? `${message.name}:${message.message}` : JSON.stringify(message, null, 2);
const timestamp = now.toLocaleString();
const timestampEpoch = now;
let out = { "timestamp": timestamp, epoch: timestampEpoch, } as Record<string, any>;
if (message instanceof Error) {
// debugger;
// console.dir(message.stack);
out = { ...out, message };
} else if (stackLog) {
if (stackLog) {
const stackE = new Error();
const stack = stackE.stack;
out = { ...out, stack }
}
}
if (typeof message == "object") {
out = { ...out, ...message, }
} else {
out = {
result: message
}
}
// const out = "--" + timestamp + "--\n" + messageContent + " " + (stack || "");
// const out
try {
await plugin.vaultAccess.adapterAppend(plugin.app.vault.configDir + "/ls-debug/" + outFile, JSON.stringify(out) + "\n")
} catch (ex) {
//NO OP
}
}));
}

70
src/tests/tests.ts Normal file
View File

@@ -0,0 +1,70 @@
import { Trench } from "../lib/src/memory/memutil.ts";
import type ObsidianLiveSyncPlugin from "../main.ts";
type MeasureResult = [times: number, spent: number];
type NamedMeasureResult = [name: string, result: MeasureResult];
const measures = new Map<string, MeasureResult>();
function clearResult(name: string) {
measures.set(name, [0, 0]);
}
async function measureEach(name: string, proc: () => (void | Promise<void>)) {
const [times, spent] = measures.get(name) ?? [0, 0];
const start = performance.now();
const result = proc();
if (result instanceof Promise) await result;
const end = performance.now();
measures.set(name, [times + 1, spent + (end - start)]);
}
function formatNumber(num: number) {
return num.toLocaleString('en-US', { maximumFractionDigits: 2 });
}
async function measure(name: string, proc: () => (void | Promise<void>), times: number = 10000, duration: number = 1000): Promise<NamedMeasureResult> {
const from = Date.now();
let last = times;
clearResult(name);
do {
await measureEach(name, proc);
} while (last-- > 0 && (Date.now() - from) < duration)
return [name, measures.get(name) as MeasureResult];
}
// eslint-disable-next-line require-await
async function formatPerfResults(items: NamedMeasureResult[]) {
return `| Name | Runs | Each | Total |\n| --- | --- | --- | --- | \n` + items.map(e => `| ${e[0]} | ${e[1][0]} | ${e[1][0] != 0 ? formatNumber(e[1][1] / e[1][0]) : "-"} | ${formatNumber(e[1][0])} |`).join("\n");
}
export async function perf_trench(plugin: ObsidianLiveSyncPlugin) {
clearResult("trench");
const trench = new Trench(plugin.simpleStore);
const result = [] as NamedMeasureResult[];
result.push(await measure("trench-short-string", async () => {
const p = trench.evacuate("string");
await p();
}));
{
const testBinary = await plugin.vaultAccess.adapterReadBinary("testdata/10kb.png");
const uint8Array = new Uint8Array(testBinary);
result.push(await measure("trench-binary-10kb", async () => {
const p = trench.evacuate(uint8Array);
await p();
}));
}
{
const testBinary = await plugin.vaultAccess.adapterReadBinary("testdata/100kb.jpeg");
const uint8Array = new Uint8Array(testBinary);
result.push(await measure("trench-binary-100kb", async () => {
const p = trench.evacuate(uint8Array);
await p();
}));
}
{
const testBinary = await plugin.vaultAccess.adapterReadBinary("testdata/1mb.png");
const uint8Array = new Uint8Array(testBinary);
result.push(await measure("trench-binary-1mb", async () => {
const p = trench.evacuate(uint8Array);
await p();
}));
}
return formatPerfResults(result);
}

107
src/ui/ConflictResolveModal.ts Normal file
View File

@@ -0,0 +1,107 @@
import { App, Modal } from "../deps.ts";
import { DIFF_DELETE, DIFF_EQUAL, DIFF_INSERT } from "diff-match-patch";
import { CANCELLED, LEAVE_TO_SUBSEQUENT, RESULT_TIMED_OUT, type diff_result } from "../lib/src/common/types.ts";
import { escapeStringToHTML } from "../lib/src/string_and_binary/convert.ts";
import { delay, sendValue, waitForValue } from "../lib/src/common/utils.ts";
export type MergeDialogResult = typeof LEAVE_TO_SUBSEQUENT | typeof CANCELLED | string;
export class ConflictResolveModal extends Modal {
result: diff_result;
filename: string;
response: MergeDialogResult = CANCELLED;
isClosed = false;
consumed = false;
title: string = "Conflicting changes";
pluginPickMode: boolean = false;
localName: string = "Keep A";
remoteName: string = "Keep B";
constructor(app: App, filename: string, diff: diff_result, pluginPickMode?: boolean, remoteName?: string) {
super(app);
this.result = diff;
this.filename = filename;
this.pluginPickMode = pluginPickMode || false;
if (this.pluginPickMode) {
this.title = "Pick a version";
this.remoteName = `Use ${remoteName || "Remote"}`;
this.localName = "Use Local"
}
// Send cancel signal for the previous merge dialogue
// if not there, simply be ignored.
// sendValue("close-resolve-conflict:" + this.filename, false);
sendValue("cancel-resolve-conflict:" + this.filename, true);
}
onOpen() {
const { contentEl } = this;
// Send cancel signal for the previous merge dialogue
// if not there, simply be ignored.
sendValue("cancel-resolve-conflict:" + this.filename, true);
setTimeout(async () => {
const forceClose = await waitForValue("cancel-resolve-conflict:" + this.filename);
// debugger;
if (forceClose) {
this.sendResponse(CANCELLED);
}
}, 10)
// sendValue("close-resolve-conflict:" + this.filename, false);
this.titleEl.setText(this.title);
contentEl.empty();
contentEl.createEl("span", { text: this.filename });
const div = contentEl.createDiv("");
div.addClass("op-scrollable");
let diff = "";
for (const v of this.result.diff) {
const x1 = v[0];
const x2 = v[1];
if (x1 == DIFF_DELETE) {
diff += "<span class='deleted'>" + escapeStringToHTML(x2).replace(/\n/g, "<span class='ls-mark-cr'></span>\n") + "</span>";
} else if (x1 == DIFF_EQUAL) {
diff += "<span class='normal'>" + escapeStringToHTML(x2).replace(/\n/g, "<span class='ls-mark-cr'></span>\n") + "</span>";
} else if (x1 == DIFF_INSERT) {
diff += "<span class='added'>" + escapeStringToHTML(x2).replace(/\n/g, "<span class='ls-mark-cr'></span>\n") + "</span>";
}
}
diff = diff.replace(/\n/g, "<br>");
div.innerHTML = diff;
const div2 = contentEl.createDiv("");
const date1 = new Date(this.result.left.mtime).toLocaleString() + (this.result.left.deleted ? " (Deleted)" : "");
const date2 = new Date(this.result.right.mtime).toLocaleString() + (this.result.right.deleted ? " (Deleted)" : "");
div2.innerHTML = `
<span class='deleted'>A:${date1}</span><br /><span class='added'>B:${date2}</span><br>
`;
contentEl.createEl("button", { text: this.localName }, (e) => e.addEventListener("click", () => this.sendResponse(this.result.right.rev))).style.marginRight = "4px";
contentEl.createEl("button", { text: this.remoteName }, (e) => e.addEventListener("click", () => this.sendResponse(this.result.left.rev))).style.marginRight = "4px";
if (!this.pluginPickMode) {
contentEl.createEl("button", { text: "Concat both" }, (e) => e.addEventListener("click", () => this.sendResponse(LEAVE_TO_SUBSEQUENT))).style.marginRight = "4px";
}
contentEl.createEl("button", { text: !this.pluginPickMode ? "Not now" : "Cancel" }, (e) => e.addEventListener("click", () => this.sendResponse(CANCELLED))).style.marginRight = "4px";
}
sendResponse(result: MergeDialogResult) {
this.response = result;
this.close();
}
onClose() {
const { contentEl } = this;
contentEl.empty();
if (this.consumed) {
return;
}
this.consumed = true;
sendValue("close-resolve-conflict:" + this.filename, this.response);
sendValue("cancel-resolve-conflict:" + this.filename, false);
}
async waitForResult(): Promise<MergeDialogResult> {
await delay(100);
const r = await waitForValue<MergeDialogResult>("close-resolve-conflict:" + this.filename);
if (r === RESULT_TIMED_OUT) return CANCELLED;
return r;
}
}

164
src/DocumentHistoryModal.ts → src/ui/DocumentHistoryModal.ts
View File

@@ -1,13 +1,40 @@
import { TFile, Modal, App, DIFF_DELETE, DIFF_EQUAL, DIFF_INSERT, diff_match_patch } from "./deps";
import { getPathFromTFile, isValidPath } from "./utils";
import { decodeBinary, escapeStringToHTML, readString } from "./lib/src/strbin";
import ObsidianLiveSyncPlugin from "./main";
import { type DocumentID, type FilePathWithPrefix, type LoadedEntry, LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE } from "./lib/src/types";
import { Logger } from "./lib/src/logger";
import { isErrorOfMissingDoc } from "./lib/src/utils_couchdb";
import { getDocData } from "./lib/src/utils";
import { stripPrefix } from "./lib/src/path";
import { TFile, Modal, App, DIFF_DELETE, DIFF_EQUAL, DIFF_INSERT, diff_match_patch } from "../deps.ts";
import { getPathFromTFile, isValidPath } from "../common/utils.ts";
import { decodeBinary, escapeStringToHTML, readString } from "../lib/src/string_and_binary/convert.ts";
import ObsidianLiveSyncPlugin from "../main.ts";
import { type DocumentID, type FilePathWithPrefix, type LoadedEntry, LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE } from "../lib/src/common/types.ts";
import { Logger } from "../lib/src/common/logger.ts";
import { isErrorOfMissingDoc } from "../lib/src/pouchdb/utils_couchdb.ts";
import { getDocData, readContent } from "../lib/src/common/utils.ts";
import { isPlainText, stripPrefix } from "../lib/src/string_and_binary/path.ts";
function isImage(path: string) {
const ext = path.split(".").splice(-1)[0].toLowerCase();
return ["png", "jpg", "jpeg", "gif", "bmp", "webp"].includes(ext);
}
function isComparableText(path: string) {
const ext = path.split(".").splice(-1)[0].toLowerCase();
return isPlainText(path) || ["md", "mdx", "txt", "json"].includes(ext);
}
function isComparableTextDecode(path: string) {
const ext = path.split(".").splice(-1)[0].toLowerCase();
return ["json"].includes(ext)
}
function readDocument(w: LoadedEntry) {
if (w.data.length == 0) return "";
if (isImage(w.path)) {
return new Uint8Array(decodeBinary(w.data));
}
if (w.type == "plain" || w.datatype == "plain") return getDocData(w.data);
if (isComparableTextDecode(w.path)) return readString(new Uint8Array(decodeBinary(w.data)));
if (isComparableText(w.path)) return getDocData(w.data);
try {
return readString(new Uint8Array(decodeBinary(w.data)));
} catch (ex) {
// NO OP.
}
return getDocData(w.data);
}
export class DocumentHistoryModal extends Modal {
plugin: ObsidianLiveSyncPlugin;
range!: HTMLInputElement;
@@ -39,13 +66,13 @@ export class DocumentHistoryModal extends Modal {
}
}
async loadFile(initialRev: string) {
async loadFile(initialRev?: string) {
if (!this.id) {
this.id = await this.plugin.path2id(this.file);
}
const db = this.plugin.localDatabase;
try {
const w = await db.localDatabase.get(this.id, { revs_info: true });
const w = await db.getRaw(this.id, { revs_info: true });
this.revs_info = w._revs_info?.filter((e) => e?.status == "available") ?? [];
this.range.max = `${Math.max(this.revs_info.length - 1, 0)}`;
this.range.value = this.range.max;
@@ -56,7 +83,6 @@ export class DocumentHistoryModal extends Modal {
this.range.max = "0";
this.range.value = "";
this.range.disabled = true;
this.showDiff
this.contentView.setText(`History of this file was not recorded.`);
} else {
this.contentView.setText(`Error occurred.`);
@@ -76,6 +102,22 @@ export class DocumentHistoryModal extends Modal {
const rev = this.revs_info[index];
await this.showExactRev(rev.rev);
}
BlobURLs = new Map<string, string>();
revokeURL(key: string) {
const v = this.BlobURLs.get(key);
if (v) {
URL.revokeObjectURL(v);
}
this.BlobURLs.delete(key);
}
generateBlobURL(key: string, data: Uint8Array) {
this.revokeURL(key);
const v = URL.createObjectURL(new Blob([data], { endings: "transparent", type: "application/octet-stream" }));
this.BlobURLs.set(key, v);
return v;
}
async showExactRev(rev: string) {
const db = this.plugin.localDatabase;
const w = await db.getDBEntry(this.file, { rev: rev }, false, false, true);
@@ -88,42 +130,67 @@ export class DocumentHistoryModal extends Modal {
} else {
this.currentDoc = w;
this.info.innerHTML = `Modified:${new Date(w.mtime).toLocaleString()}`;
let result = "";
const w1data = w.datatype == "plain" ? getDocData(w.data) : readString(new Uint8Array(decodeBinary(w.data)));
let result = undefined;
const w1data = readDocument(w);
this.currentDeleted = !!w.deleted;
this.currentText = w1data;
// this.currentText = w1data;
if (this.showDiff) {
const prevRevIdx = this.revs_info.length - 1 - ((this.range.value as any) / 1 - 1);
if (prevRevIdx >= 0 && prevRevIdx < this.revs_info.length) {
const oldRev = this.revs_info[prevRevIdx].rev;
const w2 = await db.getDBEntry(this.file, { rev: oldRev }, false, false, true);
if (w2 != false) {
const dmp = new diff_match_patch();
const w2data = w2.datatype == "plain" ? getDocData(w2.data) : readString(new Uint8Array(decodeBinary(w2.data)));
const diff = dmp.diff_main(w2data, w1data);
dmp.diff_cleanupSemantic(diff);
for (const v of diff) {
const x1 = v[0];
const x2 = v[1];
if (x1 == DIFF_DELETE) {
result += "<span class='history-deleted'>" + escapeStringToHTML(x2) + "</span>";
} else if (x1 == DIFF_EQUAL) {
result += "<span class='history-normal'>" + escapeStringToHTML(x2) + "</span>";
} else if (x1 == DIFF_INSERT) {
result += "<span class='history-added'>" + escapeStringToHTML(x2) + "</span>";
if (typeof w1data == "string") {
result = "";
const dmp = new diff_match_patch();
const w2data = readDocument(w2) as string;
const diff = dmp.diff_main(w2data, w1data);
dmp.diff_cleanupSemantic(diff);
for (const v of diff) {
const x1 = v[0];
const x2 = v[1];
if (x1 == DIFF_DELETE) {
result += "<span class='history-deleted'>" + escapeStringToHTML(x2) + "</span>";
} else if (x1 == DIFF_EQUAL) {
result += "<span class='history-normal'>" + escapeStringToHTML(x2) + "</span>";
} else if (x1 == DIFF_INSERT) {
result += "<span class='history-added'>" + escapeStringToHTML(x2) + "</span>";
}
}
result = result.replace(/\n/g, "<br>");
} else if (isImage(this.file)) {
const src = this.generateBlobURL("base", w1data);
const overlay = this.generateBlobURL("overlay", readDocument(w2) as Uint8Array);
result =
`<div class='ls-imgdiff-wrap'>
<div class='overlay'>
<img class='img-base' src="${src}">
<img class='img-overlay' src='${overlay}'>
</div>
</div>`;
this.contentView.removeClass("op-pre");
}
}
}
result = result.replace(/\n/g, "<br>");
} else {
result = escapeStringToHTML(w1data);
}
if (result == undefined) {
if (typeof w1data != "string") {
if (isImage(this.file)) {
const src = this.generateBlobURL("base", w1data);
result =
`<div class='ls-imgdiff-wrap'>
<div class='overlay'>
<img class='img-base' src="${src}">
</div>
</div>`;
this.contentView.removeClass("op-pre");
}
} else {
result = escapeStringToHTML(w1data);
}
} else {
result = escapeStringToHTML(w1data);
}
if (result == undefined) result = typeof w1data == "string" ? escapeStringToHTML(w1data) : "Binary file";
this.contentView.innerHTML = (this.currentDeleted ? "(At this revision, the file has been deleted)\n" : "") + result;
}
}
@@ -180,12 +247,10 @@ export class DocumentHistoryModal extends Modal {
Logger(`Old content copied to clipboard`, LOG_LEVEL_NOTICE);
});
});
async function focusFile(path: string) {
const targetFile = app.vault
.getFiles()
.find((f) => f.path === path);
const focusFile = async (path: string) => {
const targetFile = this.plugin.app.vault.getFileByPath(path);
if (targetFile) {
const leaf = app.workspace.getLeaf(false);
const leaf = this.plugin.app.workspace.getLeaf(false);
await leaf.openFile(targetFile);
} else {
Logger("The file could not view on the editor", LOG_LEVEL_NOTICE)
@@ -198,24 +263,25 @@ export class DocumentHistoryModal extends Modal {
const pathToWrite = stripPrefix(this.file);
if (!isValidPath(pathToWrite)) {
Logger("Path is not valid to write content.", LOG_LEVEL_INFO);
return;
}
if (this.currentDoc?.datatype == "plain") {
await this.app.vault.adapter.write(pathToWrite, getDocData(this.currentDoc.data));
await focusFile(pathToWrite);
this.close();
} else if (this.currentDoc?.datatype == "newnote") {
await this.app.vault.adapter.writeBinary(pathToWrite, decodeBinary(this.currentDoc.data));
await focusFile(pathToWrite);
this.close();
} else {
Logger(`Could not parse entry`, LOG_LEVEL_NOTICE);
if (!this.currentDoc) {
Logger("No active file loaded.", LOG_LEVEL_INFO);
return;
}
const d = readContent(this.currentDoc);
await this.plugin.vaultAccess.adapterWrite(pathToWrite, d);
await focusFile(pathToWrite);
this.close();
});
});
}
onClose() {
const { contentEl } = this;
contentEl.empty();
this.BlobURLs.forEach(value => {
console.log(value);
if (value) URL.revokeObjectURL(value);
})
}
}

50
src/GlobalHistory.svelte → src/ui/GlobalHistory.svelte
View File

@@ -1,13 +1,12 @@
<script lang="ts">
import ObsidianLiveSyncPlugin from "./main";
import ObsidianLiveSyncPlugin from "../main";
import { onDestroy, onMount } from "svelte";
import type { AnyEntry, FilePathWithPrefix } from "./lib/src/types";
import { getDocData, isDocContentSame } from "./lib/src/utils";
import { diff_match_patch } from "./deps";
import type { AnyEntry, FilePathWithPrefix } from "../lib/src/common/types";
import { getDocData, isAnyNote, isDocContentSame, readAsBlob } from "../lib/src/common/utils";
import { diff_match_patch } from "../deps";
import { DocumentHistoryModal } from "./DocumentHistoryModal";
import { isPlainText, stripAllPrefixes } from "./lib/src/path";
import { TFile } from "./deps";
import { encodeBinary } from "./lib/src/strbin";
import { isPlainText, stripAllPrefixes } from "../lib/src/string_and_binary/path";
import { TFile } from "../deps";
export let plugin: ObsidianLiveSyncPlugin;
let showDiffInfo = false;
@@ -31,7 +30,7 @@
type HistoryData = {
id: string;
rev: string;
rev?: string;
path: string;
dirname: string;
filename: string;
@@ -54,12 +53,12 @@
if (docA.mtime < range_from_epoch) {
continue;
}
if (docA.type != "newnote" && docA.type != "plain") continue;
if (!isAnyNote(docA)) continue;
const path = plugin.getPath(docA as AnyEntry);
const isPlain = isPlainText(docA.path);
const revs = await db.getRaw(docA._id, { revs_info: true });
let p: string = undefined;
const reversedRevs = revs._revs_info.reverse();
let p: string | undefined = undefined;
const reversedRevs = (revs._revs_info ?? []).reverse();
const DIFF_DELETE = -1;
const DIFF_EQUAL = 0;
@@ -67,10 +66,7 @@
for (const revInfo of reversedRevs) {
if (revInfo.status == "available") {
const doc =
(!isPlain && showDiffInfo) || (checkStorageDiff && revInfo.rev == docA._rev)
? await db.getDBEntry(path, { rev: revInfo.rev }, false, false, true)
: await db.getDBEntryMeta(path, { rev: revInfo.rev }, true);
const doc = (!isPlain && showDiffInfo) || (checkStorageDiff && revInfo.rev == docA._rev) ? await db.getDBEntry(path, { rev: revInfo.rev }, false, false, true) : await db.getDBEntryMeta(path, { rev: revInfo.rev }, true);
if (doc === false) continue;
const rev = revInfo.rev;
@@ -108,17 +104,11 @@
}
if (rev == docA._rev) {
if (checkStorageDiff) {
const abs = plugin.app.vault.getAbstractFileByPath(stripAllPrefixes(plugin.getPath(docA)));
const abs = plugin.vaultAccess.getAbstractFileByPath(stripAllPrefixes(plugin.getPath(docA)));
if (abs instanceof TFile) {
let result = false;
if (isPlainText(docA.path)) {
const data = await plugin.app.vault.read(abs);
result = isDocContentSame(data, doc.data);
} else {
const data = await plugin.app.vault.readBinary(abs);
const dataEEncoded = await encodeBinary(data, plugin.settings.useV1);
result = isDocContentSame(dataEEncoded, doc.data);
}
const data = await plugin.vaultAccess.adapterReadAuto(abs);
const d = readAsBlob(doc);
const result = await isDocContentSame(data, d);
if (result) {
diffDetail += " ⚖️";
} else {
@@ -187,7 +177,7 @@
onDestroy(() => {});
function showHistory(file: string, rev: string) {
new DocumentHistoryModal(plugin.app, plugin, file as unknown as FilePathWithPrefix, null, rev).open();
new DocumentHistoryModal(plugin.app, plugin, file as unknown as FilePathWithPrefix, undefined, rev).open();
}
function openFile(file: string) {
plugin.app.workspace.openLinkText(file, file);
@@ -236,13 +226,19 @@
<td class="path">
<div class="filenames">
<span class="path">/{entry.dirname.split("/").join(`​/`)}</span>
<!-- svelte-ignore a11y-click-events-have-key-events -->
<!-- svelte-ignore a11y-no-static-element-interactions -->
<!-- svelte-ignore a11y-missing-attribute -->
<span class="filename"><a on:click={() => openFile(entry.path)}>{entry.filename}</a></span>
</div>
</td>
<td>
<span class="rev">
{#if entry.isPlain}
<a on:click={() => showHistory(entry.path, entry.rev)}>{entry.rev}</a>
<!-- svelte-ignore a11y-click-events-have-key-events -->
<!-- svelte-ignore a11y-no-static-element-interactions -->
<!-- svelte-ignore a11y-missing-attribute -->
<a on:click={() => showHistory(entry.path, entry?.rev || "")}>{entry.rev}</a>
{:else}
{entry.rev}
{/if}

14
src/GlobalHistoryView.ts → src/ui/GlobalHistoryView.ts
View File

@@ -1,18 +1,18 @@
import {
ItemView,
WorkspaceLeaf
} from "./deps";
} from "../deps.ts";
import GlobalHistoryComponent from "./GlobalHistory.svelte";
import type ObsidianLiveSyncPlugin from "./main";
import type ObsidianLiveSyncPlugin from "../main.ts";
export const VIEW_TYPE_GLOBAL_HISTORY = "global-history";
export class GlobalHistoryView extends ItemView {
component: GlobalHistoryComponent;
component?: GlobalHistoryComponent;
plugin: ObsidianLiveSyncPlugin;
icon: "clock";
title: string;
navigation: true;
icon = "clock";
title: string = "";
navigation = true;
getIcon(): string {
return "clock";
@@ -44,6 +44,6 @@ export class GlobalHistoryView extends ItemView {
// eslint-disable-next-line require-await
async onClose() {
this.component.$destroy();
this.component?.$destroy();
}
}

77
src/ui/JsonResolveModal.ts Normal file
View File

@@ -0,0 +1,77 @@
import { App, Modal } from "../deps.ts";
import { type FilePath, type LoadedEntry } from "../lib/src/common/types.ts";
import JsonResolvePane from "./JsonResolvePane.svelte";
import { waitForSignal } from "../lib/src/common/utils.ts";
export class JsonResolveModal extends Modal {
// result: Array<[number, string]>;
filename: FilePath;
callback?: (keepRev?: string, mergedStr?: string) => Promise<void>;
docs: LoadedEntry[];
component?: JsonResolvePane;
nameA: string;
nameB: string;
defaultSelect: string;
keepOrder: boolean;
hideLocal: boolean;
title: string = "Conflicted Setting";
constructor(app: App, filename: FilePath,
docs: LoadedEntry[], callback: (keepRev?: string, mergedStr?: string) => Promise<void>,
nameA?: string, nameB?: string, defaultSelect?: string,
keepOrder?: boolean, hideLocal?: boolean, title: string = "Conflicted Setting") {
super(app);
this.callback = callback;
this.filename = filename;
this.docs = docs;
this.nameA = nameA || "";
this.nameB = nameB || "";
this.keepOrder = keepOrder || false;
this.defaultSelect = defaultSelect || "";
this.title = title;
this.hideLocal = hideLocal ?? false;
waitForSignal(`cancel-internal-conflict:${filename}`).then(() => this.close());
}
async UICallback(keepRev?: string, mergedStr?: string) {
this.close();
await this.callback?.(keepRev, mergedStr);
this.callback = undefined;
}
onOpen() {
const { contentEl } = this;
this.titleEl.setText(this.title);
contentEl.empty();
if (this.component == undefined) {
this.component = new JsonResolvePane({
target: contentEl,
props: {
docs: this.docs,
filename: this.filename,
nameA: this.nameA,
nameB: this.nameB,
defaultSelect: this.defaultSelect,
keepOrder: this.keepOrder,
hideLocal: this.hideLocal,
callback: (keepRev: string | undefined, mergedStr: string | undefined) => this.UICallback(keepRev, mergedStr),
},
});
}
return;
}
onClose() {
const { contentEl } = this;
contentEl.empty();
// contentEl.empty();
if (this.callback != undefined) {
this.callback(undefined);
}
if (this.component != undefined) {
this.component.$destroy();
this.component = undefined;
}
}
}

108
src/JsonResolvePane.svelte → src/ui/JsonResolvePane.svelte
View File

@@ -1,20 +1,22 @@
<script lang="ts">
import { type Diff, DIFF_DELETE, DIFF_INSERT, diff_match_patch } from "./deps";
import type { FilePath, LoadedEntry } from "./lib/src/types";
import { decodeBinary, readString } from "./lib/src/strbin";
import { getDocData } from "./lib/src/utils";
import { mergeObject } from "./utils";
import { type Diff, DIFF_DELETE, DIFF_INSERT, diff_match_patch } from "../deps";
import type { FilePath, LoadedEntry } from "../lib/src/common/types";
import { decodeBinary, readString } from "../lib/src/string_and_binary/convert";
import { getDocData } from "../lib/src/common/utils";
import { mergeObject } from "../common/utils";
export let docs: LoadedEntry[] = [];
export let callback: (keepRev: string, mergedStr?: string) => Promise<void> = async (_, __) => {
export let callback: (keepRev?: string, mergedStr?: string) => Promise<void> = async (_, __) => {
Promise.resolve();
};
export let filename: FilePath = "" as FilePath;
export let nameA: string = "A";
export let nameB: string = "B";
export let defaultSelect: string = "";
let docA: LoadedEntry = undefined;
let docB: LoadedEntry = undefined;
export let keepOrder = false;
export let hideLocal: boolean = false;
let docA: LoadedEntry;
let docB: LoadedEntry;
let docAContent = "";
let docBContent = "";
let objA: any = {};
@@ -28,7 +30,8 @@
function docToString(doc: LoadedEntry) {
return doc.datatype == "plain" ? getDocData(doc.data) : readString(new Uint8Array(decodeBinary(doc.data)));
}
function revStringToRevNumber(rev: string) {
function revStringToRevNumber(rev?: string) {
if (!rev) return "";
return rev.split("-")[0];
}
@@ -44,19 +47,22 @@
}
function apply() {
if (docA._id == docB._id) {
if (mode == "A") return callback(docA._rev, null);
if (mode == "B") return callback(docB._rev, null);
if (mode == "A") return callback(docA._rev!, undefined);
if (mode == "B") return callback(docB._rev!, undefined);
} else {
if (mode == "A") return callback(null, docToString(docA));
if (mode == "B") return callback(null, docToString(docB));
if (mode == "A") return callback(undefined, docToString(docA));
if (mode == "B") return callback(undefined, docToString(docB));
}
if (mode == "BA") return callback(null, JSON.stringify(objBA, null, 2));
if (mode == "AB") return callback(null, JSON.stringify(objAB, null, 2));
callback(null, null);
if (mode == "BA") return callback(undefined, JSON.stringify(objBA, null, 2));
if (mode == "AB") return callback(undefined, JSON.stringify(objAB, null, 2));
callback(undefined, undefined);
}
function cancel() {
callback(undefined, undefined);
}
$: {
if (docs && docs.length >= 1) {
if (docs[0].mtime < docs[1].mtime) {
if (keepOrder || docs[0].mtime < docs[1].mtime) {
docA = docs[0];
docB = docs[1];
} else {
@@ -95,13 +101,19 @@
diffs = getJsonDiff(objA, selectedObj);
}
$: modes = [
["", "Not now"],
["A", nameA || "A"],
["B", nameB || "B"],
["AB", `${nameA || "A"} + ${nameB || "B"}`],
["BA", `${nameB || "B"} + ${nameA || "A"}`],
] as ["" | "A" | "B" | "AB" | "BA", string][];
let modes = [] as ["" | "A" | "B" | "AB" | "BA", string][];
$: {
let newModes = [] as typeof modes;
if (!hideLocal) {
newModes.push(["", "Not now"]);
newModes.push(["A", nameA || "A"]);
}
newModes.push(["B", nameB || "B"]);
newModes.push(["AB", `${nameA || "A"} + ${nameB || "B"}`]);
newModes.push(["BA", `${nameB || "B"} + ${nameA || "A"}`]);
modes = newModes;
}
</script>
<h2>{filename}</h2>
@@ -131,24 +143,54 @@
{:else}
NO PREVIEW
{/if}
<div>
{nameA}
{#if docA._id == docB._id} Rev:{revStringToRevNumber(docA._rev)} {/if} ,{new Date(docA.mtime).toLocaleString()}
{docAContent.length} letters
</div>
<div>
{nameB}
{#if docA._id == docB._id} Rev:{revStringToRevNumber(docB._rev)} {/if} ,{new Date(docB.mtime).toLocaleString()}
{docBContent.length} letters
<div class="infos">
<table>
<tr>
<th>{nameA}</th>
<td
>{#if docA._id == docB._id}
Rev:{revStringToRevNumber(docA._rev)}
{/if}
{new Date(docA.mtime).toLocaleString()}</td
>
<td>
{docAContent.length} letters
</td>
</tr>
<tr>
<th>{nameB}</th>
<td
>{#if docA._id == docB._id}
Rev:{revStringToRevNumber(docB._rev)}
{/if}
{new Date(docB.mtime).toLocaleString()}</td
>
<td>
{docBContent.length} letters
</td>
</tr>
</table>
</div>
<div class="buttons">
{#if hideLocal}
<button on:click={cancel}>Cancel</button>
{/if}
<button on:click={apply}>Apply</button>
</div>
{/if}
<style>
.spacer {
flex-grow: 1;
}
.infos {
display: flex;
justify-content: space-between;
margin: 4px 0.5em;
}
.deleted {
text-decoration: line-through;
}

29
src/LogPane.svelte → src/ui/LogPane.svelte
View File

@@ -1,26 +1,27 @@
<script lang="ts">
import { onDestroy, onMount } from "svelte";
import { logMessageStore } from "./lib/src/stores";
import { logMessages } from "../lib/src/mock_and_interop/stores";
import type { ReactiveInstance } from "../lib/src/dataobject/reactive";
import { Logger } from "../lib/src/common/logger";
let unsubscribe: () => void;
let messages = [] as string[];
let wrapRight = false;
let autoScroll = true;
let suspended = false;
function updateLog(logs: ReactiveInstance<string[]>) {
const e = logs.value;
if (!suspended) {
messages = [...e];
setTimeout(() => {
if (scroll) scroll.scrollTop = scroll.scrollHeight;
}, 10);
}
}
onMount(async () => {
unsubscribe = logMessageStore.observe((e) => {
if (!suspended) {
messages = [...e];
if (autoScroll) {
if (scroll) scroll.scrollTop = scroll.scrollHeight;
}
}
});
logMessageStore.invalidate();
setTimeout(() => {
if (scroll) scroll.scrollTop = scroll.scrollHeight;
}, 100);
logMessages.onChanged(updateLog);
Logger("Log window opened");
unsubscribe = () => logMessages.offChanged(updateLog);
});
onDestroy(() => {
if (unsubscribe) unsubscribe();

12
src/LogPaneView.ts → src/ui/LogPaneView.ts
View File

@@ -3,16 +3,16 @@ import {
WorkspaceLeaf
} from "obsidian";
import LogPaneComponent from "./LogPane.svelte";
import type ObsidianLiveSyncPlugin from "./main";
import type ObsidianLiveSyncPlugin from "../main.ts";
export const VIEW_TYPE_LOG = "log-log";
//Log view
export class LogPaneView extends ItemView {
component: LogPaneComponent;
component?: LogPaneComponent;
plugin: ObsidianLiveSyncPlugin;
icon: "view-log";
title: string;
navigation: true;
icon = "view-log";
title: string = "";
navigation = true;
getIcon(): string {
return "view-log";
@@ -43,6 +43,6 @@ export class LogPaneView extends ItemView {
// eslint-disable-next-line require-await
async onClose() {
this.component.$destroy();
this.component?.$destroy();
}
}

2534
src/ui/ObsidianLiveSyncSettingTab.ts Normal file
View File

File diff suppressed because it is too large Load Diff

578
src/ui/PluginPane.svelte Normal file
View File

@@ -0,0 +1,578 @@
<script lang="ts">
import { onMount } from "svelte";
import ObsidianLiveSyncPlugin from "../main";
import { type IPluginDataExDisplay, pluginIsEnumerating, pluginList, pluginManifestStore, pluginV2Progress } from "../features/CmdConfigSync";
import PluginCombo from "./components/PluginCombo.svelte";
import { Menu, type PluginManifest } from "obsidian";
import { unique } from "../lib/src/common/utils";
import { MODE_SELECTIVE, MODE_AUTOMATIC, MODE_PAUSED, type SYNC_MODE, type PluginSyncSettingEntry, MODE_SHINY } from "../lib/src/common/types";
import { normalizePath } from "../deps";
export let plugin: ObsidianLiveSyncPlugin;
$: hideNotApplicable = false;
$: thisTerm = plugin.deviceAndVaultName;
const addOn = plugin.addOnConfigSync;
let list: IPluginDataExDisplay[] = [];
let selectNewestPulse = 0;
let selectNewestStyle = 0;
let hideEven = false;
let loading = false;
let applyAllPluse = 0;
let isMaintenanceMode = false;
async function requestUpdate() {
await addOn.updatePluginList(true);
}
async function requestReload() {
await addOn.reloadPluginList(true);
}
let allTerms = [] as string[];
pluginList.subscribe((e) => {
list = e;
allTerms = unique(list.map((e) => e.term));
});
pluginIsEnumerating.subscribe((e) => {
loading = e;
});
onMount(async () => {
requestUpdate();
});
function filterList(list: IPluginDataExDisplay[], categories: string[]) {
const w = list.filter((e) => categories.indexOf(e.category) !== -1);
return w.sort((a, b) => `${a.category}-${a.name}`.localeCompare(`${b.category}-${b.name}`));
}
function groupBy(items: IPluginDataExDisplay[], key: string) {
let ret = {} as Record<string, IPluginDataExDisplay[]>;
for (const v of items) {
//@ts-ignore
const k = (key in v ? v[key] : "") as string;
ret[k] = ret[k] || [];
ret[k].push(v);
}
for (const k in ret) {
ret[k] = ret[k].sort((a, b) => `${a.category}-${a.name}`.localeCompare(`${b.category}-${b.name}`));
}
const w = Object.entries(ret);
return w.sort(([a], [b]) => `${a}`.localeCompare(`${b}`));
}
const displays = {
CONFIG: "Configuration",
THEME: "Themes",
SNIPPET: "Snippets",
};
async function scanAgain() {
await addOn.scanAllConfigFiles(true);
await requestUpdate();
}
async function replicate() {
await plugin.replicate(true);
}
function selectAllNewest(selectMode: boolean) {
selectNewestPulse++;
selectNewestStyle = selectMode ? 1 : 2;
}
function resetSelectNewest() {
selectNewestPulse++;
selectNewestStyle = 3;
}
function applyAll() {
applyAllPluse++;
}
async function applyData(data: IPluginDataExDisplay): Promise<boolean> {
return await addOn.applyData(data);
}
async function compareData(docA: IPluginDataExDisplay, docB: IPluginDataExDisplay, compareEach = false): Promise<boolean> {
return await addOn.compareUsingDisplayData(docA, docB, compareEach);
}
async function deleteData(data: IPluginDataExDisplay): Promise<boolean> {
return await addOn.deleteData(data);
}
function askMode(evt: MouseEvent, title: string, key: string) {
const menu = new Menu();
menu.addItem((item) => item.setTitle(title).setIsLabel(true));
menu.addSeparator();
const prevMode = automaticList.get(key) ?? MODE_SELECTIVE;
for (const mode of [MODE_SELECTIVE, MODE_AUTOMATIC, MODE_PAUSED, MODE_SHINY]) {
menu.addItem((item) => {
item.setTitle(`${getIcon(mode as SYNC_MODE)}:${TITLES[mode]}`)
.onClick((e) => {
if (mode === MODE_AUTOMATIC) {
askOverwriteModeForAutomatic(evt, key);
} else {
setMode(key, mode as SYNC_MODE);
}
})
.setChecked(prevMode == mode)
.setDisabled(prevMode == mode);
});
}
menu.showAtMouseEvent(evt);
}
function applyAutomaticSync(key: string, direction: "pushForce" | "pullForce" | "safe") {
setMode(key, MODE_AUTOMATIC);
const configDir = normalizePath(plugin.app.vault.configDir);
const files = (plugin.settings.pluginSyncExtendedSetting[key]?.files ?? []).map((e) => `${configDir}/${e}`);
plugin.addOnHiddenFileSync.syncInternalFilesAndDatabase(direction, true, false, files);
}
function askOverwriteModeForAutomatic(evt: MouseEvent, key: string) {
const menu = new Menu();
menu.addItem((item) => item.setTitle("Initial Action").setIsLabel(true));
menu.addSeparator();
menu.addItem((item) => {
item.setTitle(`↑: Overwrite Remote`).onClick((e) => {
applyAutomaticSync(key, "pushForce");
});
})
.addItem((item) => {
item.setTitle(`↓: Overwrite Local`).onClick((e) => {
applyAutomaticSync(key, "pullForce");
});
})
.addItem((item) => {
item.setTitle(`⇅: Use newer`).onClick((e) => {
applyAutomaticSync(key, "safe");
});
});
menu.showAtMouseEvent(evt);
}
$: options = {
thisTerm,
hideNotApplicable,
selectNewest: selectNewestPulse,
selectNewestStyle,
applyAllPluse,
applyData,
compareData,
deleteData,
plugin,
isMaintenanceMode,
};
const ICON_EMOJI_PAUSED = `⛔`;
const ICON_EMOJI_AUTOMATIC = `✨`;
const ICON_EMOJI_SELECTIVE = `🔀`;
const ICON_EMOJI_FLAGGED = `🚩`;
const ICONS: { [key: number]: string } = {
[MODE_SELECTIVE]: ICON_EMOJI_SELECTIVE,
[MODE_PAUSED]: ICON_EMOJI_PAUSED,
[MODE_AUTOMATIC]: ICON_EMOJI_AUTOMATIC,
[MODE_SHINY]: ICON_EMOJI_FLAGGED,
};
const TITLES: { [key: number]: string } = {
[MODE_SELECTIVE]: "Selective",
[MODE_PAUSED]: "Ignore",
[MODE_AUTOMATIC]: "Automatic",
[MODE_SHINY]: "Flagged Selective",
};
const PREFIX_PLUGIN_ALL = "PLUGIN_ALL";
const PREFIX_PLUGIN_DATA = "PLUGIN_DATA";
const PREFIX_PLUGIN_MAIN = "PLUGIN_MAIN";
const PREFIX_PLUGIN_ETC = "PLUGIN_ETC";
function setMode(key: string, mode: SYNC_MODE) {
if (key.startsWith(PREFIX_PLUGIN_ALL + "/")) {
setMode(PREFIX_PLUGIN_DATA + key.substring(PREFIX_PLUGIN_ALL.length), mode);
setMode(PREFIX_PLUGIN_MAIN + key.substring(PREFIX_PLUGIN_ALL.length), mode);
return;
}
const files = unique(
list
.filter((e) => `${e.category}/${e.name}` == key)
.map((e) => e.files)
.flat()
.map((e) => e.filename),
);
if (mode == MODE_SELECTIVE) {
automaticList.delete(key);
delete plugin.settings.pluginSyncExtendedSetting[key];
automaticListDisp = automaticList;
} else {
automaticList.set(key, mode);
automaticListDisp = automaticList;
if (!(key in plugin.settings.pluginSyncExtendedSetting)) {
plugin.settings.pluginSyncExtendedSetting[key] = {
key,
mode,
files: [],
};
}
plugin.settings.pluginSyncExtendedSetting[key].files = files;
plugin.settings.pluginSyncExtendedSetting[key].mode = mode;
}
plugin.saveSettingData();
}
function getIcon(mode: SYNC_MODE) {
if (mode in ICONS) {
return ICONS[mode];
} else {
("");
}
}
let automaticList = new Map<string, SYNC_MODE>();
let automaticListDisp = new Map<string, SYNC_MODE>();
// apply current configuration to the dialogue
for (const { key, mode } of Object.values(plugin.settings.pluginSyncExtendedSetting)) {
automaticList.set(key, mode);
}
automaticListDisp = automaticList;
let displayKeys: Record<string, string[]> = {};
function computeDisplayKeys(list: IPluginDataExDisplay[]) {
const extraKeys = Object.keys(plugin.settings.pluginSyncExtendedSetting);
return [
...list,
...extraKeys
.map((e) => `${e}///`.split("/"))
.filter((e) => e[0] && e[1])
.map((e) => ({ category: e[0], name: e[1], displayName: e[1] })),
]
.sort((a, b) => (a.displayName ?? a.name).localeCompare(b.displayName ?? b.name))
.reduce((p, c) => ({ ...p, [c.category]: unique(c.category in p ? [...p[c.category], c.displayName ?? c.name] : [c.displayName ?? c.name]) }), {} as Record<string, string[]>);
}
$: {
displayKeys = computeDisplayKeys(list);
}
let deleteTerm = "";
async function deleteAllItems(term: string) {
const deleteItems = list.filter((e) => e.term == term);
for (const item of deleteItems) {
await deleteData(item);
}
addOn.reloadPluginList(true);
}
let nameMap = new Map<string, string>();
function updateNameMap(e: Map<string, PluginManifest>) {
const items = [...e.entries()].map(([k, v]) => [k.split("/").slice(-2).join("/"), v.name] as [string, string]);
const newMap = new Map(items);
if (newMap.size == nameMap.size) {
let diff = false;
for (const [k, v] of newMap) {
if (nameMap.get(k) != v) {
diff = true;
break;
}
}
if (!diff) {
return;
}
}
nameMap = newMap;
}
$: updateNameMap($pluginManifestStore);
let displayEntries = [] as [string, string][];
$: {
displayEntries = Object.entries(displays).filter(([key, _]) => key in displayKeys);
}
let pluginEntries = [] as [string, IPluginDataExDisplay[]][];
$: {
pluginEntries = groupBy(filterList(list, ["PLUGIN_MAIN", "PLUGIN_DATA", "PLUGIN_ETC"]), "name");
}
let useSyncPluginEtc = plugin.settings.usePluginEtc;
</script>
<div class="buttonsWrap">
<div class="buttons">
<button on:click={() => scanAgain()}>Scan changes</button>
<button on:click={() => replicate()}>Sync once</button>
<button on:click={() => requestUpdate()}>Refresh</button>
{#if isMaintenanceMode}
<button on:click={() => requestReload()}>Reload</button>
{/if}
</div>
<div class="buttons">
<button on:click={() => selectAllNewest(true)}>Select All Shiny</button>
<button on:click={() => selectAllNewest(false)}>{ICON_EMOJI_FLAGGED} Select Flagged Shiny</button>
<button on:click={() => resetSelectNewest()}>Deselect all</button>
<button on:click={() => applyAll()} class="mod-cta">Apply All Selected</button>
</div>
</div>
<div class="loading">
{#if loading || $pluginV2Progress !== 0}
<span>Updating list...{$pluginV2Progress == 0 ? "" : ` (${$pluginV2Progress})`}</span>
{/if}
</div>
<div class="list">
{#if list.length == 0}
<div class="center">No Items.</div>
{:else}
{#each displayEntries as [key, label]}
<div>
<h3>{label}</h3>
{#each displayKeys[key] as name}
{@const bindKey = `${key}/${name}`}
{@const mode = automaticListDisp.get(bindKey) ?? MODE_SELECTIVE}
<div class="labelrow {hideEven ? 'hideeven' : ''}">
<div class="title">
<button class="status" on:click={(evt) => askMode(evt, `${key}/${name}`, bindKey)}>
{getIcon(mode)}
</button>
<span class="name">{(key == "THEME" && nameMap.get(`themes/${name}`)) || name}</span>
</div>
<div class="body">
{#if mode == MODE_SELECTIVE || mode == MODE_SHINY}
<PluginCombo {...options} isFlagged={mode == MODE_SHINY} list={list.filter((e) => e.category == key && e.name == name)} hidden={false} />
{:else}
<div class="statusnote">{TITLES[mode]}</div>
{/if}
</div>
</div>
{/each}
</div>
{/each}
<div>
<h3>Plugins</h3>
{#each pluginEntries as [name, listX]}
{@const bindKeyAll = `${PREFIX_PLUGIN_ALL}/${name}`}
{@const modeAll = automaticListDisp.get(bindKeyAll) ?? MODE_SELECTIVE}
{@const bindKeyMain = `${PREFIX_PLUGIN_MAIN}/${name}`}
{@const modeMain = automaticListDisp.get(bindKeyMain) ?? MODE_SELECTIVE}
{@const bindKeyData = `${PREFIX_PLUGIN_DATA}/${name}`}
{@const modeData = automaticListDisp.get(bindKeyData) ?? MODE_SELECTIVE}
{@const bindKeyETC = `${PREFIX_PLUGIN_ETC}/${name}`}
{@const modeEtc = automaticListDisp.get(bindKeyETC) ?? MODE_SELECTIVE}
<div class="labelrow {hideEven ? 'hideeven' : ''}">
<div class="title">
<button class="status" on:click={(evt) => askMode(evt, `${PREFIX_PLUGIN_ALL}/${name}`, bindKeyAll)}>
{getIcon(modeAll)}
</button>
<span class="name">{nameMap.get(`plugins/${name}`) || name}</span>
</div>
<div class="body">
{#if modeAll == MODE_SELECTIVE || modeAll == MODE_SHINY}
<PluginCombo {...options} isFlagged={modeAll == MODE_SHINY} list={listX} hidden={true} />
{/if}
</div>
</div>
{#if modeAll == MODE_SELECTIVE || modeAll == MODE_SHINY}
<div class="filerow {hideEven ? 'hideeven' : ''}">
<div class="filetitle">
<button class="status" on:click={(evt) => askMode(evt, `${PREFIX_PLUGIN_MAIN}/${name}/MAIN`, bindKeyMain)}>
{getIcon(modeMain)}
</button>
<span class="name">MAIN</span>
</div>
<div class="body">
{#if modeMain == MODE_SELECTIVE || modeMain == MODE_SHINY}
<PluginCombo {...options} isFlagged={modeMain == MODE_SHINY} list={filterList(listX, ["PLUGIN_MAIN"])} hidden={false} />
{:else}
<div class="statusnote">{TITLES[modeMain]}</div>
{/if}
</div>
</div>
<div class="filerow {hideEven ? 'hideeven' : ''}">
<div class="filetitle">
<button class="status" on:click={(evt) => askMode(evt, `${PREFIX_PLUGIN_DATA}/${name}`, bindKeyData)}>
{getIcon(modeData)}
</button>
<span class="name">DATA</span>
</div>
<div class="body">
{#if modeData == MODE_SELECTIVE || modeData == MODE_SHINY}
<PluginCombo {...options} isFlagged={modeData == MODE_SHINY} list={filterList(listX, ["PLUGIN_DATA"])} hidden={false} />
{:else}
<div class="statusnote">{TITLES[modeData]}</div>
{/if}
</div>
</div>
{#if useSyncPluginEtc}
<div class="filerow {hideEven ? 'hideeven' : ''}">
<div class="filetitle">
<button class="status" on:click={(evt) => askMode(evt, `${PREFIX_PLUGIN_ETC}/${name}`, bindKeyETC)}>
{getIcon(modeEtc)}
</button>
<span class="name">Other files</span>
</div>
<div class="body">
{#if modeEtc == MODE_SELECTIVE || modeEtc == MODE_SHINY}
<PluginCombo {...options} isFlagged={modeEtc == MODE_SHINY} list={filterList(listX, ["PLUGIN_ETC"])} hidden={false} />
{:else}
<div class="statusnote">{TITLES[modeEtc]}</div>
{/if}
</div>
</div>
{/if}
{:else}
<div class="noterow">
<div class="statusnote">{TITLES[modeAll]}</div>
</div>
{/if}
{/each}
</div>
{/if}
</div>
{#if isMaintenanceMode}
<div class="buttons">
<div>
<h3>Maintenance Commands</h3>
<div class="maintenancerow">
<label for="">Delete All of </label>
<select bind:value={deleteTerm}>
{#each allTerms as term}
<option value={term}>{term}</option>
{/each}
</select>
<button
class="status"
on:click={(evt) => {
deleteAllItems(deleteTerm);
}}
>
🗑️
</button>
</div>
</div>
</div>
{/if}
<div class="buttons">
<label><span>Hide not applicable items</span><input type="checkbox" bind:checked={hideEven} /></label>
</div>
<div class="buttons">
<label><span>Maintenance mode</span><input type="checkbox" bind:checked={isMaintenanceMode} /></label>
</div>
<style>
.buttonsWrap {
padding-bottom: 4px;
}
h3 {
position: sticky;
top: 0;
background-color: var(--modal-background);
}
.labelrow {
margin-left: 0.4em;
display: flex;
justify-content: flex-start;
align-items: center;
border-top: 1px solid var(--background-modifier-border);
padding: 4px;
flex-wrap: wrap;
}
.filerow {
margin-left: 1.25em;
display: flex;
justify-content: flex-start;
align-items: center;
padding-right: 4px;
flex-wrap: wrap;
}
.filerow.hideeven:has(.even),
.labelrow.hideeven:has(.even) {
display: none;
}
.noterow {
min-height: 2em;
display: flex;
}
button.status {
flex-grow: 0;
margin: 2px 4px;
min-width: 3em;
max-width: 4em;
}
.statusnote {
display: flex;
justify-content: flex-end;
padding-right: var(--size-4-12);
align-items: center;
min-width: 10em;
flex-grow: 1;
}
.list {
overflow-y: auto;
}
.title {
color: var(--text-normal);
font-size: var(--font-ui-medium);
line-height: var(--line-height-tight);
margin-right: auto;
}
.body {
/* margin-left: 0.4em; */
margin-left: auto;
display: flex;
justify-content: flex-start;
align-items: center;
/* flex-wrap: wrap; */
}
.filetitle {
color: var(--text-normal);
font-size: var(--font-ui-medium);
line-height: var(--line-height-tight);
margin-right: auto;
}
.buttons {
display: flex;
flex-direction: row;
justify-content: flex-end;
margin-top: 8px;
flex-wrap: wrap;
}
.buttons > button {
margin-left: 4px;
width: auto;
}
label {
display: flex;
justify-content: center;
align-items: center;
}
label > span {
margin-right: 0.25em;
}
:global(.is-mobile) .title,
:global(.is-mobile) .filetitle {
width: 100%;
}
.center {
display: flex;
justify-content: center;
align-items: center;
min-height: 3em;
}
.maintenancerow {
display: flex;
justify-content: flex-end;
align-items: center;
}
.maintenancerow label {
margin-right: 0.5em;
margin-left: 0.5em;
}
.loading {
transition: height 0.25s ease-in-out;
transition-delay: 4ms;
overflow-y: hidden;
flex-shrink: 0;
display: flex;
justify-content: flex-start;
align-items: center;
}
.loading:empty {
height: 0px;
transition: height 0.25s ease-in-out;
transition-delay: 1s;
}
.loading:not(:empty) {
height: 2em;
transition: height 0.25s ease-in-out;
transition-delay: 0;
}
</style>

79
src/ui/components/MultipleRegExpControl.svelte Normal file
View File

@@ -0,0 +1,79 @@
<script lang="ts">
export let patterns = [] as string[];
export let originals = [] as string[];
export let apply: (args: string[]) => Promise<void> = (_: string[]) => Promise.resolve();
function revert() {
patterns = [...originals];
}
const CHECK_OK = "✔";
const CHECK_NG = "⚠";
const MARK_MODIFIED = "✏ ";
function checkRegExp(pattern: string) {
if (pattern.trim() == "") return "";
try {
const _ = new RegExp(pattern);
return CHECK_OK;
} catch (ex) {
return CHECK_NG;
}
}
$: status = patterns.map((e) => checkRegExp(e));
$: modified = patterns.map((e, i) => (e != originals?.[i] ?? "" ? MARK_MODIFIED : ""));
function remove(idx: number) {
patterns[idx] = "";
}
function add() {
patterns = [...patterns, ""];
}
</script>
<ul>
{#each patterns as pattern, idx}
<!-- svelte-ignore a11y-label-has-associated-control -->
<li><label>{modified[idx]}{status[idx]}</label><input type="text" bind:value={pattern} class={modified[idx]} /><button class="iconbutton" on:click={() => remove(idx)}>🗑</button></li>
{/each}
<li>
<label><button on:click={() => add()}>Add</button></label>
</li>
<li class="buttons">
<button on:click={() => apply(patterns)} disabled={status.some((e) => e == CHECK_NG) || modified.every((e) => e == "")}>Apply</button>
<button on:click={() => revert()} disabled={status.some((e) => e == CHECK_NG) || modified.every((e) => e == "")}>Revert</button>
</li>
</ul>
<style>
label {
min-width: 4em;
width: 4em;
display: inline-flex;
flex-direction: row;
justify-content: flex-end;
}
ul {
flex-grow: 1;
display: inline-flex;
flex-direction: column;
list-style-type: none;
margin-block-start: 0;
margin-block-end: 0;
margin-inline-start: 0px;
margin-inline-end: 0px;
padding-inline-start: 0;
}
li {
padding: var(--size-2-1) var(--size-4-1);
display: inline-flex;
flex-grow: 1;
align-items: center;
justify-content: flex-end;
gap: var(--size-4-2);
}
li input {
min-width: 10em;
}
button.iconbutton {
max-width: 4em;
}
</style>

224
src/PluginCombo.svelte → src/ui/components/PluginCombo.svelte
View File

@@ -1,39 +1,42 @@
<script lang="ts">
import type { PluginDataExDisplay } from "./CmdConfigSync";
import { Logger } from "./lib/src/logger";
import { versionNumberString2Number } from "./lib/src/strbin";
import { type FilePath, LOG_LEVEL_NOTICE } from "./lib/src/types";
import { getDocData } from "./lib/src/utils";
import type ObsidianLiveSyncPlugin from "./main";
import { askString, scheduleTask } from "./utils";
import { PluginDataExDisplayV2, type IPluginDataExDisplay } from "../../features/CmdConfigSync";
import { Logger } from "../../lib/src/common/logger";
import { type FilePath, LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE } from "../../lib/src/common/types";
import { getDocData, timeDeltaToHumanReadable, unique } from "../../lib/src/common/utils";
import type ObsidianLiveSyncPlugin from "../../main";
import { askString } from "../../common/utils";
import { Menu } from "obsidian";
export let list: PluginDataExDisplay[] = [];
export let list: IPluginDataExDisplay[] = [];
export let thisTerm = "";
export let hideNotApplicable = false;
export let selectNewest = 0;
export let selectNewestStyle = 0;
export let applyAllPluse = 0;
export let applyData: (data: PluginDataExDisplay) => Promise<boolean>;
export let compareData: (dataA: PluginDataExDisplay, dataB: PluginDataExDisplay) => Promise<boolean>;
export let deleteData: (data: PluginDataExDisplay) => Promise<boolean>;
export let applyData: (data: IPluginDataExDisplay) => Promise<boolean>;
export let compareData: (dataA: IPluginDataExDisplay, dataB: IPluginDataExDisplay, compareEach?: boolean) => Promise<boolean>;
export let deleteData: (data: IPluginDataExDisplay) => Promise<boolean>;
export let hidden: boolean;
export let plugin: ObsidianLiveSyncPlugin;
export let isMaintenanceMode: boolean = false;
export let isFlagged: boolean = false;
const addOn = plugin.addOnConfigSync;
let selected = "";
export let selected = "";
let freshness = "";
let equivalency = "";
let version = "";
let canApply: boolean = false;
let canCompare: boolean = false;
let pickToCompare: boolean = false;
let currentSelectNewest = 0;
let currentApplyAll = 0;
// Selectable terminals
let terms = [] as string[];
async function comparePlugin(local: PluginDataExDisplay, remote: PluginDataExDisplay) {
async function comparePlugin(local: IPluginDataExDisplay | undefined, remote: IPluginDataExDisplay | undefined) {
let freshness = "";
let equivalency = "";
let version = "";
@@ -41,25 +44,28 @@
let canApply: boolean = false;
let canCompare = false;
if (!local && !remote) {
// NO OP. whats happened?
// NO OP. what's happened?
freshness = "";
} else if (local && !remote) {
freshness = "⚠ Local only";
freshness = "Local only";
} else if (remote && !local) {
freshness = "✓ Remote only";
freshness = "Remote only";
canApply = true;
} else {
const dtDiff = (local?.mtime ?? 0) - (remote?.mtime ?? 0);
const diff = timeDeltaToHumanReadable(Math.abs(dtDiff));
if (dtDiff / 1000 < -10) {
freshness = "✓ Newer";
// freshness = "✓ Newer";
freshness = `Newer (${diff})`;
canApply = true;
contentCheck = true;
} else if (dtDiff / 1000 > 10) {
freshness = "⚠ Older";
// freshness = "⚠ Older";
freshness = `Older (${diff})`;
canApply = true;
contentCheck = true;
} else {
freshness = "⚖️ Same old";
freshness = "Same";
canApply = false;
contentCheck = true;
}
@@ -67,25 +73,26 @@
const localVersionStr = local?.version || "0.0.0";
const remoteVersionStr = remote?.version || "0.0.0";
if (local?.version || remote?.version) {
const localVersion = versionNumberString2Number(localVersionStr);
const remoteVersion = versionNumberString2Number(remoteVersionStr);
if (localVersion == remoteVersion) {
version = "⚖️ Same ver.";
} else if (localVersion > remoteVersion) {
version = `⚠ Lower ${localVersionStr} > ${remoteVersionStr}`;
} else if (localVersion < remoteVersion) {
version = `✓ Higher ${localVersionStr} < ${remoteVersionStr}`;
const compare = `${localVersionStr}`.localeCompare(remoteVersionStr, undefined, { numeric: true });
if (compare == 0) {
version = "Same";
} else if (compare < 0) {
version = `Lower (${localVersionStr} < ${remoteVersionStr})`;
} else if (compare > 0) {
version = `Higher (${localVersionStr} > ${remoteVersionStr})`;
}
}
if (contentCheck) {
const { canApply, equivalency, canCompare } = await checkEquivalency(local, remote);
return { canApply, freshness, equivalency, version, canCompare };
if (local && remote) {
const { canApply, equivalency, canCompare } = await checkEquivalency(local, remote);
return { canApply, freshness, equivalency, version, canCompare };
}
}
return { canApply, freshness, equivalency, version, canCompare };
}
async function checkEquivalency(local: PluginDataExDisplay, remote: PluginDataExDisplay) {
async function checkEquivalency(local: IPluginDataExDisplay, remote: IPluginDataExDisplay) {
let equivalency = "";
let canApply = false;
let canCompare = false;
@@ -100,17 +107,21 @@
return 0b0000010; //"LOCAL_ONLY";
} else if (!localFile && remoteFile) {
return 0b0001000; //"REMOTE ONLY"
} else {
if (getDocData(localFile.data) == getDocData(remoteFile.data)) {
} else if (localFile && remoteFile) {
const localDoc = getDocData(localFile.data);
const remoteDoc = getDocData(remoteFile.data);
if (localDoc == remoteDoc) {
return 0b0000100; //"EVEN"
} else {
return 0b0010000; //"DIFFERENT";
}
} else {
return 0b0010000; //"DIFFERENT";
}
})
.reduce((p, c) => p | (c as number), 0 as number);
if (matchingStatus == 0b0000100) {
equivalency = "⚖️ Same";
equivalency = "Same";
canApply = false;
} else if (matchingStatus <= 0b0000100) {
equivalency = "Same or local only";
@@ -118,30 +129,37 @@
} else if (matchingStatus == 0b0010000) {
canApply = true;
canCompare = true;
equivalency = "≠ Different";
equivalency = "Different";
} else {
canApply = true;
canCompare = true;
equivalency = "≠ Different";
equivalency = "Mixed";
}
return { equivalency, canApply, canCompare };
}
async function performCompare(local: PluginDataExDisplay, remote: PluginDataExDisplay) {
async function performCompare(local: IPluginDataExDisplay | undefined, remote: IPluginDataExDisplay | undefined) {
const result = await comparePlugin(local, remote);
canApply = result.canApply;
freshness = result.freshness;
equivalency = result.equivalency;
version = result.version;
canCompare = result.canCompare;
if (local?.files.length != 1 || !local?.files?.first()?.filename?.endsWith(".json")) {
canCompare = false;
pickToCompare = false;
if (canCompare) {
if (local?.files.length == remote?.files.length && local?.files.length == 1 && local?.files[0].filename == remote?.files[0].filename) {
pickToCompare = false;
} else {
pickToCompare = true;
// pickToCompare = false;
// canCompare = false;
}
}
}
async function updateTerms(list: PluginDataExDisplay[], selectNewest: boolean, isMaintenanceMode: boolean) {
async function updateTerms(list: IPluginDataExDisplay[], selectNewest: boolean, isMaintenanceMode: boolean) {
const local = list.find((e) => e.term == thisTerm);
selected = "";
// selected = "";
if (isMaintenanceMode) {
terms = [...new Set(list.map((e) => e.term))];
} else if (hideNotApplicable) {
@@ -157,7 +175,7 @@
} else {
terms = [...new Set(list.map((e) => e.term))].filter((e) => e != thisTerm);
}
let newest: PluginDataExDisplay = local;
let newest: IPluginDataExDisplay | undefined = local;
if (selectNewest) {
for (const term of terms) {
const remote = list.find((e) => e.term == term);
@@ -170,12 +188,25 @@
}
// selectNewest = false;
}
if (terms.indexOf(selected) < 0) {
selected = "";
}
}
$: {
// React pulse and select
const doSelectNewest = selectNewest != currentSelectNewest;
currentSelectNewest = selectNewest;
let doSelectNewest = false;
if (selectNewest != currentSelectNewest) {
if (selectNewestStyle == 1) {
doSelectNewest = true;
} else if (selectNewestStyle == 2) {
doSelectNewest = isFlagged;
} else if (selectNewestStyle == 3) {
selected = "";
}
// currentSelectNewest = selectNewest;
}
updateTerms(list, doSelectNewest, isMaintenanceMode);
currentSelectNewest = selectNewest;
}
$: {
// React pulse and apply
@@ -207,25 +238,71 @@
const local = list.find((e) => e.term == thisTerm);
const selectedItem = list.find((e) => e.term == selected);
if (selectedItem && (await applyData(selectedItem))) {
scheduleTask("update-plugin-list", 250, () => addOn.updatePluginList(true, local.documentPath));
addOn.updatePluginList(true, local?.documentPath);
}
}
async function compareSelected() {
const local = list.find((e) => e.term == thisTerm);
const selectedItem = list.find((e) => e.term == selected);
if (local && selectedItem && (await compareData(local, selectedItem))) {
scheduleTask("update-plugin-list", 250, () => addOn.updatePluginList(true, local.documentPath));
await compareItems(local, selectedItem);
}
async function compareItems(local: IPluginDataExDisplay | undefined, remote: IPluginDataExDisplay | undefined, filename?: string) {
if (local && remote) {
if (!filename) {
if (await compareData(local, remote)) {
addOn.updatePluginList(true, local.documentPath);
}
return;
} else {
const localCopy = local instanceof PluginDataExDisplayV2 ? new PluginDataExDisplayV2(local) : { ...local };
const remoteCopy = remote instanceof PluginDataExDisplayV2 ? new PluginDataExDisplayV2(remote) : { ...remote };
localCopy.files = localCopy.files.filter((e) => e.filename == filename);
remoteCopy.files = remoteCopy.files.filter((e) => e.filename == filename);
if (await compareData(localCopy, remoteCopy, true)) {
addOn.updatePluginList(true, local.documentPath);
}
}
return;
} else {
if (!remote && !local) {
Logger(`Could not find both remote and local item`, LOG_LEVEL_INFO);
} else if (!remote) {
Logger(`Could not find remote item`, LOG_LEVEL_INFO);
} else if (!local) {
Logger(`Could not locally item`, LOG_LEVEL_INFO);
}
}
}
async function pickCompareItem(evt: MouseEvent) {
const local = list.find((e) => e.term == thisTerm);
const selectedItem = list.find((e) => e.term == selected);
if (!local) return;
if (!selectedItem) return;
const menu = new Menu();
menu.addItem((item) => item.setTitle("Compare file").setIsLabel(true));
menu.addSeparator();
const files = unique(local.files.map((e) => e.filename).concat(selectedItem.files.map((e) => e.filename)));
for (const filename of files) {
menu.addItem((item) => {
item.setTitle(filename).onClick((e) => compareItems(local, selectedItem, filename));
});
}
menu.showAtMouseEvent(evt);
}
async function deleteSelected() {
const selectedItem = list.find((e) => e.term == selected);
// const deletedPath = selectedItem.documentPath;
if (selectedItem && (await deleteData(selectedItem))) {
scheduleTask("update-plugin-list", 250, () => addOn.reloadPluginList(true));
addOn.reloadPluginList(true);
}
}
async function duplicateItem() {
const local = list.find((e) => e.term == thisTerm);
if (!local) {
Logger(`Could not find local item`, LOG_LEVEL_VERBOSE);
return;
}
const duplicateTermName = await askString(plugin.app, "Duplicate", "device name", "");
if (duplicateTermName) {
if (duplicateTermName.contains("/")) {
@@ -242,10 +319,10 @@
{#if terms.length > 0}
<span class="spacer" />
{#if !hidden}
<span class="messages">
<span class="message">{freshness}</span>
<span class="message">{equivalency}</span>
<span class="message">{version}</span>
<span class="chip-wrap">
<span class="chip modified">{freshness}</span>
<span class="chip content">{equivalency}</span>
<span class="chip version">{version}</span>
</span>
<select bind:value={selected}>
<option value={""}>-</option>
@@ -255,7 +332,12 @@
</select>
{#if canApply || (isMaintenanceMode && selected != "")}
{#if canCompare}
<button on:click={compareSelected}>🔍</button>
{#if pickToCompare}
<button on:click={pickCompareItem}>🗃️</button>
{:else}
<!--🔍 -->
<button on:click={compareSelected}>⮂</button>
{/if}
{:else}
<button disabled />
{/if}
@@ -307,12 +389,46 @@
padding: 0 1em;
line-height: var(--line-height-tight);
}
span.messages {
/* span.messages {
display: flex;
flex-direction: column;
align-items: center;
}
} */
:global(.is-mobile) .spacer {
margin-left: auto;
}
.chip-wrap {
display: flex;
gap: 2px;
flex-direction: column;
justify-content: center;
align-items: flex-start;
}
.chip {
display: inline-block;
border-radius: 2px;
font-size: 0.8em;
padding: 0 4px;
margin: 0 2px;
border-color: var(--tag-border-color);
background-color: var(--tag-background);
color: var(--tag-color);
}
.chip:empty {
display: none;
}
.chip:not(:empty)::before {
min-width: 1.8em;
display: inline-block;
}
.chip.content:not(:empty)::before {
content: "📄: ";
}
.chip.version:not(:empty)::before {
content: "🏷️: ";
}
.chip.modified:not(:empty)::before {
content: "📅: ";
}
</style>

359
src/ui/settingConstants.ts Normal file
View File

@@ -0,0 +1,359 @@
import { $t } from "src/lib/src/common/i18n";
import { DEFAULT_SETTINGS, configurationNames, type ConfigurationItem, type FilterBooleanKeys, type FilterNumberKeys, type FilterStringKeys, type ObsidianLiveSyncSettings } from "src/lib/src/common/types";
export type OnDialogSettings = {
configPassphrase: string,
preset: "" | "PERIODIC" | "LIVESYNC" | "DISABLE",
syncMode: "ONEVENTS" | "PERIODIC" | "LIVESYNC"
dummy: number,
deviceAndVaultName: string,
}
export const OnDialogSettingsDefault: OnDialogSettings = {
configPassphrase: "",
preset: "",
syncMode: "ONEVENTS",
dummy: 0,
deviceAndVaultName: "",
}
export const AllSettingDefault =
{ ...DEFAULT_SETTINGS, ...OnDialogSettingsDefault }
export type AllSettings = ObsidianLiveSyncSettings & OnDialogSettings;
export type AllStringItemKey = FilterStringKeys<AllSettings>;
export type AllNumericItemKey = FilterNumberKeys<AllSettings>;
export type AllBooleanItemKey = FilterBooleanKeys<AllSettings>;
export type AllSettingItemKey = AllStringItemKey | AllNumericItemKey | AllBooleanItemKey;
export type ValueOf<T extends AllSettingItemKey> =
T extends AllStringItemKey ? string :
T extends AllNumericItemKey ? number :
T extends AllBooleanItemKey ? boolean :
AllSettings[T];
export const SettingInformation: Partial<Record<keyof AllSettings, ConfigurationItem>> = {
"liveSync": {
"name": "Sync Mode"
},
"couchDB_URI": {
"name": "URI",
"placeHolder": "https://........"
},
"couchDB_USER": {
"name": "Username",
"desc": "username"
},
"couchDB_PASSWORD": {
"name": "Password",
"desc": "password"
},
"couchDB_DBNAME": {
"name": "Database name"
},
"passphrase": {
"name": "Passphrase",
"desc": "Encrypting passphrase. If you change the passphrase of an existing database, overwriting the remote database is strongly recommended."
},
"showStatusOnEditor": {
"name": "Show status inside the editor",
"desc": "Reflected after reboot"
},
"showOnlyIconsOnEditor": {
"name": "Show status as icons only"
},
"showStatusOnStatusbar": {
"name": "Show status on the status bar",
"desc": "Reflected after reboot."
},
"lessInformationInLog": {
"name": "Show only notifications",
"desc": "Prevent logging and show only notification. Please disable when you report the logs"
},
"showVerboseLog": {
"name": "Verbose Log",
"desc": "Show verbose log. Please enable when you report the logs"
},
"hashCacheMaxCount": {
"name": "Memory cache size (by total items)"
},
"hashCacheMaxAmount": {
"name": "Memory cache size (by total characters)",
"desc": "(Mega chars)"
},
"writeCredentialsForSettingSync": {
"name": "Write credentials in the file",
"desc": "(Not recommended) If set, credentials will be stored in the file."
},
"notifyAllSettingSyncFile": {
"name": "Notify all setting files"
},
"configPassphrase": {
"name": "Passphrase of sensitive configuration items",
"desc": "This passphrase will not be copied to another device. It will be set to `Default` until you configure it again."
},
"configPassphraseStore": {
"name": "Encrypting sensitive configuration items"
},
"syncOnSave": {
"name": "Sync on Save",
"desc": "When you save a file, sync automatically"
},
"syncOnEditorSave": {
"name": "Sync on Editor Save",
"desc": "When you save a file in the editor, sync automatically"
},
"syncOnFileOpen": {
"name": "Sync on File Open",
"desc": "When you open a file, sync automatically"
},
"syncOnStart": {
"name": "Sync on Start",
"desc": "Start synchronization after launching Obsidian."
},
"syncAfterMerge": {
"name": "Sync after merging file",
"desc": "Sync automatically after merging files"
},
"trashInsteadDelete": {
"name": "Use the trash bin",
"desc": "Do not delete files that are deleted in remote, just move to trash."
},
"doNotDeleteFolder": {
"name": "Keep empty folder",
"desc": "Normally, a folder is deleted when it becomes empty after a synchronization. Enabling this will prevent it from getting deleted"
},
"resolveConflictsByNewerFile": {
"name": "Always overwrite with a newer file (beta)",
"desc": "(Def off) Resolve conflicts by newer files automatically."
},
"checkConflictOnlyOnOpen": {
"name": "Postpone resolution of inactive files"
},
"showMergeDialogOnlyOnActive": {
"name": "Postpone manual resolution of inactive files"
},
"disableMarkdownAutoMerge": {
"name": "Always resolve conflicts manually",
"desc": "If this switch is turned on, a merge dialog will be displayed, even if the sensible-merge is possible automatically. (Turn on to previous behavior)"
},
"writeDocumentsIfConflicted": {
"name": "Always reflect synchronized changes even if the note has a conflict",
"desc": "Turn on to previous behavior"
},
"syncInternalFilesInterval": {
"name": "Scan hidden files periodically",
"desc": "Seconds, 0 to disable"
},
"batchSave": {
"name": "Batch database update",
"desc": "Reducing the frequency with which on-disk changes are reflected into the DB"
},
"readChunksOnline": {
"name": "Fetch chunks on demand",
"desc": "(ex. Read chunks online) If this option is enabled, LiveSync reads chunks online directly instead of replicating them locally. Increasing Custom chunk size is recommended."
},
"syncMaxSizeInMB": {
"name": "Maximum file size",
"desc": "(MB) If this is set, changes to local and remote files that are larger than this will be skipped. If the file becomes smaller again, a newer one will be used."
},
"useIgnoreFiles": {
"name": "(Beta) Use ignore files",
"desc": "If this is set, changes to local files which are matched by the ignore files will be skipped. Remote changes are determined using local ignore files."
},
"ignoreFiles": {
"name": "Ignore files",
"desc": "We can use multiple ignore files, e.g.) `.gitignore, .dockerignore`"
},
"batch_size": {
"name": "Batch size",
"desc": "Number of change feed items to process at a time. Defaults to 50. Minimum is 2."
},
"batches_limit": {
"name": "Batch limit",
"desc": "Number of batches to process at a time. Defaults to 40. Minimum is 2. This along with batch size controls how many docs are kept in memory at a time."
},
"useTimeouts": {
"name": "Use timeouts instead of heartbeats",
"desc": "If this option is enabled, PouchDB will hold the connection open for 60 seconds, and if no change arrives in that time, close and reopen the socket, instead of holding it open indefinitely. Useful when a proxy limits request duration but can increase resource usage."
},
"concurrencyOfReadChunksOnline": {
"name": "Batch size of on-demand fetching"
},
"minimumIntervalOfReadChunksOnline": {
"name": "The delay for consecutive on-demand fetches"
},
"suspendFileWatching": {
"name": "Suspend file watching",
"desc": "Stop watching for file change."
},
"suspendParseReplicationResult": {
"name": "Suspend database reflecting",
"desc": "Stop reflecting database changes to storage files."
},
"writeLogToTheFile": {
"name": "Write logs into the file",
"desc": "Warning! This will have a serious impact on performance. And the logs will not be synchronised under the default name. Please be careful with logs; they often contain your confidential information."
},
"deleteMetadataOfDeletedFiles": {
"name": "Do not keep metadata of deleted files."
},
"useIndexedDBAdapter": {
"name": "Use an old adapter for compatibility",
"desc": "Before v0.17.16, we used an old adapter for the local database. Now the new adapter is preferred. However, it needs local database rebuilding. Please disable this toggle when you have enough time. If leave it enabled, also while fetching from the remote database, you will be asked to disable this."
},
"watchInternalFileChanges": {
"name": "Scan changes on customization sync",
"desc": "Do not use internal API"
},
"doNotSuspendOnFetching": {
"name": "Fetch database with previous behaviour"
},
"disableCheckingConfigMismatch": {
"name": "Do not check configuration mismatch before replication"
},
"usePluginSync": {
"name": "Enable customization sync"
},
"autoSweepPlugins": {
"name": "Scan customization automatically",
"desc": "Scan customization before replicating."
},
"autoSweepPluginsPeriodic": {
"name": "Scan customization periodically",
"desc": "Scan customization every 1 minute."
},
"notifyPluginOrSettingUpdated": {
"name": "Notify customized",
"desc": "Notify when other device has newly customized."
},
"remoteType": {
"name": "Remote Type",
"desc": "Remote server type"
},
"endpoint": {
"name": "Endpoint URL",
"placeHolder": "https://........"
},
"accessKey": {
"name": "Access Key"
},
"secretKey": {
"name": "Secret Key"
},
"region": {
"name": "Region",
"placeHolder": "auto"
},
"bucket": {
"name": "Bucket Name"
},
"useCustomRequestHandler": {
"name": "Use Custom HTTP Handler",
"desc": "If your Object Storage could not configured accepting CORS, enable this."
},
"maxChunksInEden": {
"name": "Maximum Incubating Chunks",
"desc": "The maximum number of chunks that can be incubated within the document. Chunks exceeding this number will immediately graduate to independent chunks."
},
"maxTotalLengthInEden": {
"name": "Maximum Incubating Chunk Size",
"desc": "The maximum total size of chunks that can be incubated within the document. Chunks exceeding this size will immediately graduate to independent chunks."
},
"maxAgeInEden": {
"name": "Maximum Incubation Period",
"desc": "The maximum duration for which chunks can be incubated within the document. Chunks exceeding this period will graduate to independent chunks."
},
"settingSyncFile": {
"name": "Filename",
"desc": "If you set this, all settings are saved in a markdown file. You will be notified when new settings arrive. You can set different files by the platform."
},
"preset": {
"name": "Presets",
"desc": "Apply preset configuration"
},
"syncMode": {
name: "Sync Mode",
},
"periodicReplicationInterval": {
"name": "Periodic Sync interval",
"desc": "Interval (sec)"
},
"syncInternalFilesBeforeReplication": {
"name": "Scan for hidden files before replication"
},
"automaticallyDeleteMetadataOfDeletedFiles": {
"name": "Delete old metadata of deleted files on start-up",
"desc": "(Days passed, 0 to disable automatic-deletion)"
},
"additionalSuffixOfDatabaseName": {
"name": "Database suffix",
"desc": "LiveSync could not handle multiple vaults which have same name without different prefix, This should be automatically configured."
},
"hashAlg": {
"name": configurationNames["hashAlg"]?.name || "",
"desc": "xxhash64 is the current default."
},
"deviceAndVaultName": {
"name": "Device name",
"desc": "Unique name between all synchronized devices. To edit this setting, please disable customization sync once."
},
"displayLanguage": {
"name": "Display Language",
"desc": "Not all messages have been translated. And, please revert to \"Default\" when reporting errors."
},
enableChunkSplitterV2: {
name: "Use splitting-limit-capped chunk splitter",
desc: "If enabled, chunks will be split into no more than 100 items. However, dedupe is slightly weaker."
},
disableWorkerForGeneratingChunks: {
name: "Do not split chunks in the background",
desc: "If disabled(toggled), chunks will be split on the UI thread (Previous behaviour)."
},
processSmallFilesInUIThread: {
name: "Process small files in the foreground",
desc: "If enabled, the file under 1kb will be processed in the UI thread."
},
batchSaveMinimumDelay: {
name: "Minimum delay for batch database updating",
desc: "Seconds. Saving to the local database will be delayed until this value after we stop typing or saving."
},
batchSaveMaximumDelay: {
name: "Maximum delay for batch database updating",
desc: "Saving will be performed forcefully after this number of seconds."
},
"notifyThresholdOfRemoteStorageSize": {
name: "Notify when the estimated remote storage size exceeds on start up",
desc: "MB (0 to disable)."
},
"usePluginSyncV2": {
name: "Enable per-file-saved customization sync",
desc: "If enabled per-filed efficient customization sync will be used. We need a small migration when enabling this. And all devices should be updated to v0.23.18. Once we enabled this, we lost a compatibility with old versions."
}
}
function translateInfo(infoSrc: ConfigurationItem | undefined | false) {
if (!infoSrc) return false;
const info = { ...infoSrc };
info.name = $t(info.name);
if (info.desc) {
info.desc = $t(info.desc);
}
return info;
}
function _getConfig(key: AllSettingItemKey) {
if (key in configurationNames) {
return configurationNames[key as keyof ObsidianLiveSyncSettings];
}
if (key in SettingInformation) {
return SettingInformation[key as keyof ObsidianLiveSyncSettings];
}
return false;
}
export function getConfig(key: AllSettingItemKey) {
return translateInfo(_getConfig(key));
}
export function getConfName(key: AllSettingItemKey) {
const conf = getConfig(key);
if (!conf) return `${key} (No info)`;
return conf.name;
}

107
styles.css
View File

@@ -77,15 +77,6 @@
border-top: 1px solid var(--background-modifier-border);
}
/* .sls-table-head{
width:50%;
}
.sls-table-tail{
width:50%;
} */
.sls-header-button {
margin-left: 2em;
}
@@ -95,13 +86,26 @@
}
:root {
--slsmessage: "";
--sls-log-text: "";
}
.sls-troubleshoot-preview {
max-width: max-content;
}
.sls-troubleshoot-preview img {
max-width: 100%;
}
.CodeMirror-wrap::before,
.cm-s-obsidian>.cm-editor::before,
.canvas-wrapper::before {
content: attr(data-log);
.markdown-preview-view.cm-s-obsidian::before,
.markdown-source-view.cm-s-obsidian::before,
.canvas-wrapper::before,
.empty-state::before {
content: var(--sls-log-text, "");
font-variant-numeric: tabular-nums;
font-variant-emoji: emoji;
tab-size: 4;
text-align: right;
white-space: pre-wrap;
position: absolute;
@@ -116,6 +120,20 @@
filter: grayscale(100%);
}
.empty-state::before,
.markdown-preview-view.cm-s-obsidian::before,
.markdown-source-view.cm-s-obsidian::before {
top: var(--header-height);
right: 1em;
}
.is-mobile .empty-state::before,
.is-mobile .markdown-preview-view.cm-s-obsidian::before,
.is-mobile .markdown-source-view.cm-s-obsidian::before {
top: var(--view-header-height);
right: 1em;
}
.canvas-wrapper::before {
right: 48px;
}
@@ -124,7 +142,7 @@
right: 0px;
}
.cm-s-obsidian>.cm-editor::before {
.cm-s-obsidian > .cm-editor::before {
right: 16px;
}
@@ -153,8 +171,8 @@ div.sls-setting-menu-btn {
/* width: 100%; */
}
.sls-setting-tab:hover~div.sls-setting-menu-btn,
.sls-setting-label.selected .sls-setting-tab:checked~div.sls-setting-menu-btn {
.sls-setting-tab:hover ~ div.sls-setting-menu-btn,
.sls-setting-label.selected .sls-setting-tab:checked ~ div.sls-setting-menu-btn {
background-color: var(--interactive-accent);
color: var(--text-on-accent);
}
@@ -253,12 +271,28 @@ div.sls-setting-menu-btn {
content: "✏";
}
.sls-item-dirty-help::after {
content: " ❓";
}
.sls-item-invalid-value {
background-color: rgba(var(--background-modifier-error-rgb), 0.3) !important;
}
.sls-setting-disabled input[type=text],
.sls-setting-disabled input[type=number],
.sls-setting-disabled input[type=password] {
filter: brightness(80%);
color: var(--text-muted);
}
.sls-setting-hidden {
display: none;
}
.password-input > .setting-item-control >input {
-webkit-text-security: disc;
.password-input > .setting-item-control > input {
-webkit-text-security: disc;
}
span.ls-mark-cr::after {
@@ -270,4 +304,39 @@ span.ls-mark-cr::after {
.deleted span.ls-mark-cr::after {
color: var(--text-on-accent);
}
}
.ls-imgdiff-wrap {
display: flex;
justify-content: center;
align-items: center;
}
.ls-imgdiff-wrap .overlay {
position: relative;
}
.ls-imgdiff-wrap .overlay .img-base {
position: relative;
top: 0;
left: 0;
}
.ls-imgdiff-wrap .overlay .img-overlay {
-webkit-filter: invert(100%) opacity(50%);
filter: invert(100%) opacity(50%);
position: absolute;
top: 0;
left: 0;
animation: ls-blink-diff 0.5s cubic-bezier(0.4, 0, 1, 1) infinite alternate;
}
@keyframes ls-blink-diff {
0% {
opacity: 0;
}
50% {
opacity: 0;
}
100% {
opacity: 1;
}
}

1
tsconfig.json
View File

@@ -16,6 +16,7 @@
"importHelpers": false,
"alwaysStrict": true,
"allowImportingTsExtensions": true,
"noEmit": true,
"lib": [
"es2018",
"DOM",

77
updates.md
View File

@@ -1,43 +1,56 @@
### 0.20.0
### 0.23.0
Incredibly new features!
At 0.20.0, Self-hosted LiveSync has changed the binary file format and encrypting format, for efficient synchronisation.
The dialogue will be shown and asks us to decide whether to keep v1 or use v2. Once we have enabled v2, all subsequent edits will be saved in v2. Therefore, devices running 0.19 or below cannot understand this and they might say that decryption error. Please update all devices.
Then we will have an impressive performance.
Now, we can use object storage (MinIO, S3, R2 or anything you like) for synchronising! Moreover, despite that, we can use all the features as if we were using CouchDB.
Note: As this is a pretty experimental feature, hence we have some limitations.
- This is built on the append-only architecture. It will not shrink used storage if we do not perform a rebuild.
- A bit fragile. However, our version x.yy.0 is always so.
- When the first synchronisation, the entire history to date is transferred. For this reason, it is preferable to do this under the WiFi network.
- Do not worry, from the second synchronisation, we always transfer only differences.
Of course, these are very impactful changes. If you have any questions or troubled things, please feel free to open an issue and mention me.
I hope this feature empowers users to maintain independence and self-host their data, offering an alternative for those who prefer to manage their own storage solutions and avoid being stuck on the right side of a sudden change in business model.
Note: if you want to roll it back to v1, please enable `Use binary and encryption version 1` on the `Hatch` pane and perform the `rebuild everything` once.
Of course, I use Self-hosted MinIO for testing and recommend this. It is for the same reason as using CouchDB. -- open, controllable, auditable and indeed already audited by numerous eyes.
Extra but notable information:
Let me write one more acknowledgement.
This format change gives us the ability to detect some `marks` in the binary files as same as text files. Therefore, we can split binary files and some specific sort of them (i.e., PDF files) at the specific character. It means that editing the middle of files could be detected with marks.
Now only a few chunks are transferred, even if we add a comment to the PDF or put new files into the ZIP archives.
I have a lot of respect for that plugin, even though it is sometimes treated as if it is a competitor, remotely-save. I think it is a great architecture that embodies a different approach to my approach of recreating history. This time, with all due respect, I have used some of its code as a reference.
Hooray for open source, and generous licences, and the sharing of knowledge by experts.
#### Version history
- 0.20.2
- 0.23.19:
- Fixed:
- Customisation Sync now checks the difference while storing or applying the configuration.
- No longer storing the same configuration multiple times.
- Time difference in the dialogue has been fixed.
- 0.23.18:
- New feature:
- We can delete all data of customization sync from the `Delete all customization sync data` on the `Hatch` pane.
- Fixed:
- Prevent keep restarting on iOS by yielding microtasks.
- 0.20.1
- Fixed:
- No more UI freezing and keep restarting on iOS.
- Diff of Non-markdown documents are now shown correctly.
- Per-file-saved customization sync has been shipped.
- We can synchronise plug-igs etc., more smoothly.
- Default: disabled. We need a small migration when enabling this. And all devices should be updated to v0.23.18. Once we enabled this, we lost compatibility with old versions.
- Customisation sync has got beta3.
- We can set `Flag` to each item to select the newest, automatically.
- This configuration is per device.
- Improved:
- Performance has been a bit improved.
- Customization sync has gotten faster.
- However, We lost forward compatibility again (only for this feature). Please update all devices.
- Misc
- Terser configuration has been more aggressive.
- 0.20.0
- Improved:
- A New binary file handling implemented
- A new encrypted format has been implemented
- Now the chunk sizes will be adjusted for efficient sync
- Start-up speed has been improved.
- Fixed:
- levels of exception in some logs have been fixed
- Tidied:
- Some Lint warnings have been suppressed.
- On the customisation sync dialogue, buttons are kept within the screen.
- No more unnecessary entries on `data.json` for customisation sync.
- Selections are no longer lost while updating customisation items.
- Tidied on source codes:
- Many typos have been fixed.
- Some unnecessary type casting removed.
- 0.23.17:
- Improved:
- Overall performance has been improved by using PouchDB 9.0.0.
- Configuration mismatch detection is refined. We can resolve mismatches more smoothly and naturally.
More detail is on `troubleshooting.md` on the repository.
- Fixed:
- Customisation Sync will be disabled when a corrupted configuration is detected.
Therefore, the Device Name can be changed even in the event of a configuration mismatch.
- New feature:
- We can get a notification about the storage usage of the remote database.
- Default: We will be asked.
- If the remote storage usage approaches the configured value, we will be asked whether we want to Rebuild or increase the limit.
... To continue on to `updates_old.md`.
Older notes is in [updates_old.md](https://github.com/vrtmrz/obsidian-livesync/blob/main/updates_old.md).

476
updates_old.md
View File

@@ -1,3 +1,479 @@
### 0.23.0
Incredibly new features!
Now, we can use object storage (MinIO, S3, R2 or anything you like) for synchronising! Moreover, despite that, we can use all the features as if we were using CouchDB.
Note: As this is a pretty experimental feature, hence we have some limitations.
- This is built on the append-only architecture. It will not shrink used storage if we do not perform a rebuild.
- A bit fragile. However, our version x.yy.0 is always so.
- When the first synchronisation, the entire history to date is transferred. For this reason, it is preferable to do this under the WiFi network.
- Do not worry, from the second synchronisation, we always transfer only differences.
I hope this feature empowers users to maintain independence and self-host their data, offering an alternative for those who prefer to manage their own storage solutions and avoid being stuck on the right side of a sudden change in business model.
Of course, I use Self-hosted MinIO for testing and recommend this. It is for the same reason as using CouchDB. -- open, controllable, auditable and indeed already audited by numerous eyes.
Let me write one more acknowledgement.
I have a lot of respect for that plugin, even though it is sometimes treated as if it is a competitor, remotely-save. I think it is a great architecture that embodies a different approach to my approach of recreating history. This time, with all due respect, I have used some of its code as a reference.
Hooray for open source, and generous licences, and the sharing of knowledge by experts.
#### Version history
- 0.23.16:
- Maintenance Update:
- Library refining (Phase 1 - step 2). There are no significant changes on the user side.
- Including the following fixes of potentially problems:
- the problem which the path had been obfuscating twice has been resolved.
- Note: Potential problems of the library; which has not happened in Self-hosted LiveSync for some reasons.
- 0.23.15:
- Maintenance Update:
- Library refining (Phase 1). There are no significant changes on the user side.
- 0.23.14:
- Fixed:
- No longer batch-saving ignores editor inputs.
- The file-watching and serialisation processes have been changed to the one which is similar to previous implementations.
- We can configure the settings (Especially about text-boxes) even if we have configured the device name.
- Improved:
- We can configure the delay of batch-saving.
- Default: 5 seconds, the same as the previous hard-coded value. (Note: also, the previous behaviour was not correct).
- Also, we can configure the limit of delaying batch-saving.
- The performance of showing status indicators has been improved.
- 0.23.13:
- Fixed:
- No longer files have been trimmed even delimiters have been continuous.
- Fixed the toggle title to `Do not split chunks in the background` from `Do not split chunks in the foreground`.
- Non-configured item mismatches are no longer detected.
- 0.23.12:
- Improved:
- Now notes will be split into chunks in the background thread to improve smoothness.
- Default enabled, to disable, toggle `Do not split chunks in the foreground` on `Hatch` -> `Compatibility`.
- If you want to process very small notes in the foreground, please enable `Process small files in the foreground` on `Hatch` -> `Compatibility`.
- We can use a `splitting-limit-capped chunk splitter`; which performs more simple and make less amount of chunks.
- Default disabled, to enable, toggle `Use splitting-limit-capped chunk splitter` on `Sync settings` -> `Performance tweaks`
- Tidied
- Some files have been separated into multiple files to make them more explicit in what they are responsible for.
- 0.23.11:
- Fixed:
- Now we *surely* can set the device name and enable customised synchronisation.
- Unnecessary dialogue update processes have been eliminated.
- Customisation sync no longer stores half-collected files.
- No longer hangs up when removing or renaming files with the `Sync on Save` toggle enabled.
- Improved:
- Customisation sync now performs data deserialization more smoothly.
- New translations have been merged.
- 0.23.10
- Fixed:
- No longer configurations have been locked in the minimal setup.
- 0.23.9
- Fixed:
- No longer unexpected parallel replication is performed.
- Now we can set the device name and enable customised synchronisation again.
- 0.23.8
- New feature:
- Now we are ready for i18n.
- Patch or PR of `rosetta.ts` are welcome!
- The setting dialogue has been refined. Very controllable, clearly displayed disabled items, and ready to i18n.
- Fixed:
- Many memory leaks have been rescued.
- Chunk caches now work well.
- Many trivial but potential bugs are fixed.
- No longer error messages will be shown on retrieving checkpoint or server information.
- Now we can check and correct tweak mismatch during the setup
- Improved:
- Customisation synchronisation has got more smoother.
- Tidied
- Practically unused functions have been removed or are being prepared for removal.
- Many of the type-errors and lint errors have been corrected.
- Unused files have been removed.
- Note:
- From this version, some test files have been included. However, they are not enabled and released in the release build.
- To try them, please run Self-hosted LiveSync in the dev build.
- 0.23.7
- Fixed:
- No longer missing tasks which have queued as the same key (e.g., for the same operation to the same file).
- This occurs, for example, with hidden files that have been changed multiple times in a very short period of time, such as `appearance.json`. Thanks for the report!
- Some trivial issues have been fixed.
- New feature:
- Reloading Obsidian can be scheduled until that file and database operations are stable.
- 0.23.6:
- Fixed:
- Now the remote chunks could be decrypted even if we are using `Incubate chunks in Document`. (The note of 0.23.6 has been fixed).
- Chunk retrieving with `Incubate chunks in document` got more efficiently.
- No longer task processor misses the completed tasks.
- Replication is no longer started automatically during changes in window visibility (e.g., task switching on the desktop) when off-focused.
- 0.23.5:
- New feature:
- Now we can check configuration mismatching between clients before synchronisation.
- Default: enabled / Preferred: enabled / We can disable this by the `Do not check configuration mismatch before replication` toggle in the `Hatch` pane.
- It detects configuration mismatches and prevents synchronisation failures and wasted storage.
- Now we can perform remote database compaction from the `Maintenance` pane.
- Fixed:
- We can detect the bucket could not be reachable.
- Note:
- Known inexplicable behaviour: Recently, (Maybe while enabling `Incubate chunks in Document` and `Fetch chunks on demand` or some more toggles), our customisation sync data is sometimes corrupted. It will be addressed by the next release.
- 0.23.4
- Fixed:
- No longer experimental configuration is shown on the Minimal Setup.
- New feature:
- We can now use `Incubate Chunks in Document` to reduce non-well-formed chunks.
- Default: disabled / Preferred: enabled in all devices.
- When we enabled this toggle, newly created chunks are temporarily kept within the document, and graduated to become independent chunks once stabilised.
- The [design document](https://github.com/vrtmrz/obsidian-livesync/blob/3925052f9290b3579e45a4b716b3679c833d8ca0/docs/design_docs_of_keep_newborn_chunks.md) has been also available..
- 0.23.3
- Fixed: No longer unwanted `\f` in journal sync.
- 0.23.2
- Sorry for all the fixes to experimental features. (These things were also critical for dogfooding). The next release would be the main fixes! Thank you for your patience and understanding!
- Fixed:
- Journal Sync will not hang up during big replication, especially the initial one.
- All changes which have been replicated while rebuilding will not be postponed (Previous behaviour).
- Improved:
- Now Journal Sync works efficiently in download and parse, or pack and upload.
- Less server storage and faster packing/unpacking usage by the new chunk format.
- 0.23.1
- Fixed:
- Now journal synchronisation considers untransferred each from sent and received.
- Journal sync now handles retrying.
- Journal synchronisation no longer considers the synchronisation of chunks as revision updates (Simply ignored).
- Journal sync now splits the journal pack to prevent mobile device rebooting.
- Maintenance menus which had been on the command palette are now back in the maintain pane on the setting dialogue.
- Improved:
- Now all changes which have been replicated while rebuilding will be postponed.
- 0.23.0
- New feature:
- Now we can use Object Storage.
### 0.22.0
A few years passed since Self-hosted LiveSync was born, and our codebase had been very complicated. This could be patient now, but it should be a tremendous hurt.
Therefore at v0.22.0, for future maintainability, I refined task scheduling logic totally.
Of course, I think this would be our suffering in some cases. However, I would love to ask you for your cooperation and contribution.
Sorry for being absent so much long. And thank you for your patience!
Note: we got a very performance improvement.
Note at 0.22.2: **Now, to rescue mobile devices, Maximum file size is set to 50 by default**. Please configure the limit as you need. If you do not want to limit the sizes, set zero manually, please.
#### Version history
- 0.22.19
- Fixed:
- No longer data corrupting due to false BASE64 detections.
- Improved:
- A bit more efficient in Automatic data compression.
- 0.22.18
- New feature (Very Experimental):
- Now we can use `Automatic data compression` to reduce amount of traffic and the usage of remote database.
- Please make sure all devices are updated to v0.22.18 before trying this feature.
- If you are using some other utilities which connected to your vault, please make sure that they have compatibilities.
- Note: Setting `File Compression` on the remote database works for shrink the size of remote database. Please refer the [Doc](https://docs.couchdb.org/en/stable/config/couchdb.html#couchdb/file_compression).
- 0.22.17:
- Fixed:
- Error handling on booting now works fine.
- Replication is now started automatically in LiveSync mode.
- Batch database update is now disabled in LiveSync mode.
- No longer automatically reconnection while off-focused.
- Status saves are thinned out.
- Now Self-hosted LiveSync waits for all files between the local database and storage to be surely checked.
- Improved:
- The job scheduler is now more robust and stable.
- The status indicator no longer flickers and keeps zero for a while.
- No longer meaningless frequent updates of status indicators.
- Now we can configure regular expression filters in handy UI. Thank you so much, @eth-p!
- `Fetch` or `Rebuild everything` is now more safely performed.
- Minor things
- Some utility function has been added.
- Customisation sync now less wrong messages.
- Digging the weeds for eradication of type errors.
- 0.22.16:
- Fixed:
- Fixed the issue that binary files were sometimes corrupted.
- Fixed customisation sync data could be corrupted.
- Improved:
- Now the remote database costs lower memory.
- This release requires a brief wait on the first synchronisation, to track the latest changeset again.
- Description added for the `Device name`.
- Refactored:
- Many type-errors have been resolved.
- Obsolete file has been deleted.
- 0.22.15:
- Improved:
- Faster start-up by removing too many logs which indicates normality
- By streamlined scanning of customised synchronisation extra phases have been deleted.
... To continue on to `updates_old.md`.
- 0.22.14:
- New feature:
- We can disable the status bar in the setting dialogue.
- Improved:
- Now some files are handled as correct data type.
- Customisation sync now uses the digest of each file for better performance.
- The status in the Editor now works performant.
- Refactored:
- Common functions have been ready and the codebase has been organised.
- Stricter type checking following TypeScript updates.
- Remove old iOS workaround for simplicity and performance.
- 0.22.13:
- Improved:
- Now using HTTP for the remote database URI warns of an error (on mobile) or notice (on desktop).
- Refactored:
- Dependencies have been polished.
- 0.22.12:
- Changed:
- The default settings has been changed.
- Improved:
- Default and preferred settings are applied on completion of the wizard.
- Fixed:
- Now Initialisation `Fetch` will be performed smoothly and there will be fewer conflicts.
- No longer stuck while Handling transferred or initialised documents.
- 0.22.11:
- Fixed:
- `Verify and repair all files` is no longer broken.
- New feature:
- Now `Verify and repair all files` is able to...
- Restore if the file only in the local database.
- Show the history.
- Improved:
- Performance improved.
- 0.22.10
- Fixed:
- No longer unchanged hidden files and customisations are saved and transferred now.
- File integrity of vault history indicates the integrity correctly.
- Improved:
- In the report, the schema of the remote database URI is now printed.
- 0.22.9
- Fixed:
- Fixed a bug on `fetch chunks on demand` that could not fetch the chunks on demand.
- Improved:
- `fetch chunks on demand` works more smoothly.
- Initialisation `Fetch` is now more efficient.
- Tidied:
- Removed some meaningless codes.
- 0.22.8
- Fixed:
- Now fetch and unlock the locked remote database works well again.
- No longer crash on symbolic links inside hidden folders.
- Improved:
- Chunks are now created more efficiently.
- Splitting old notes into a larger chunk.
- Better performance in saving notes.
- Network activities are indicated as an icon.
- Less memory used for binary processing.
- Tidied:
- Cleaned unused functions up.
- Sorting out the codes that have become nonsense.
- Changed:
- Now no longer `fetch chunks on demand` needs `Pacing replication`
- The setting `Do not pace synchronization` has been deleted.
- 0.22.7
- Fixed:
- No longer deleted hidden files were ignored.
- The document history dialogue is now able to process the deleted revisions.
- Deletion of a hidden file is now surely performed even if the file is already conflicted.
- 0.22.6
- Fixed:
- Fixed a problem with synchronisation taking a long time to start in some cases.
- The first synchronisation after update might take a bit longer.
- Now we can disable E2EE encryption.
- Improved:
- `Setup Wizard` is now more clear.
- `Minimal Setup` is now more simple.
- Self-hosted LiveSync now be able to use even if there are vaults with the same name.
- Database suffix will automatically added.
- Now Self-hosted LiveSync waits until set-up is complete.
- Show reload prompts when possibly recommended while settings.
- New feature:
- A guidance dialogue prompting for settings will be shown after the installation.
- Changed
- `Open setup URI` is now `Use the copied setup URI`
- `Copy setup URI` is now `Copy current settings as a new setup URI`
- `Setup Wizard` is now `Minimal Setup`
- `Check database configuration` is now `Check and Fix database configuration`
- 0.22.5
- Fixed:
- Some description of settings have been refined
- New feature:
- TroubleShooting is now shown in the setting dialogue.
- 0.22.4
- Fixed:
- Now the result of conflict resolution could be surely written into the storage.
- Deleted files can be handled correctly again in the history dialogue and conflict dialogue.
- Some wrong log messages were fixed.
- Change handling now has become more stable.
- Some event handling became to be safer.
- Improved:
- Dumping document information shows conflicts and revisions.
- The timestamp-only differences can be surely cached.
- Timestamp difference detection can be rounded by two seconds.
- Refactored:
- A bit of organisation to write the test.
- 0.22.3
- Fixed:
- No longer detects storage changes which have been caused by Self-hosted LiveSync itself.
- Setting sync file will be detected only if it has been configured now.
- And its log will be shown only while the verbose log is enabled.
- Customisation file enumeration has got less blingy.
- Deletion of files is now reliably synchronised.
- Fixed and improved:
- In-editor-status is now shown in the following areas:
- Note editing pane (Source mode and live-preview mode).
- New tab pane.
- Canvas pane.
- 0.22.2
- Fixed:
- Now the results of resolving conflicts are surely synchronised.
- Modified:
- Some setting items got new clear names. (`Sync Settings` -> `Targets`).
- New feature:
- We can limit the synchronising files by their size. (`Sync Settings` -> `Targets` -> `Maximum file size`).
- It depends on the size of the newer one.
- At Obsidian 1.5.3 on mobile, we should set this to around 50MB to avoid restarting Obsidian.
- Now the settings could be stored in a specific markdown file to synchronise or switch it (`General Setting` -> `Share settings via markdown`).
- [Screwdriver](https://github.com/vrtmrz/obsidian-screwdriver) is quite good, but mostly we only need this.
- Customisation of the obsoleted device is now able to be deleted at once.
- We have to put the maintenance mode in at the Customisation sync dialogue.
- 0.22.1
- New feature:
- We can perform automatic conflict resolution for inactive files, and postpone only manual ones by `Postpone manual resolution of inactive files`.
- Now we can see the image in the document history dialogue.
- We can see the difference of the image, in the document history dialogue.
- And also we can highlight differences.
- Improved:
- Hidden file sync has been stabilised.
- Now automatically reloads the conflict-resolution dialogue when new conflicted revisions have arrived.
- Fixed:
- No longer periodic process runs after unloading the plug-in.
- Now the modification of binary files is surely stored in the storage.
- 0.22.0
- Refined:
- Task scheduling logics has been rewritten.
- Screen updates are also now efficient.
- Possibly many bugs and fragile behaviour has been fixed.
- Status updates and logging have been thinned out to display.
- Fixed:
- Remote-chunk-fetching now works with keeping request intervals
- New feature:
- We can show only the icons in the editor.
- Progress indicators have been more meaningful:
- 📥 Unprocessed transferred items
- 📄 Working database operation
- 💾 Working write storage processes
- ⏳ Working read storage processes
- 🛫 Pending read storage processes
- ⚙️ Working or pending storage processes of hidden files
- 🧩 Waiting chunks
- 🔌 Working Customisation items (Configuration, snippets and plug-ins)
... To continue on to `updates_old.md`.
### 0.21.0
The E2EE encryption V2 format has been reverted. That was probably the cause of the glitch.
Instead, to maintain efficiency, files are treated with Blob until just before saving. Along with this, the old-fashioned encryption format has also been discontinued.
There are both forward and backwards compatibilities, with recent versions. However, unfortunately, we lost compatibility with filesystem-livesync or some.
It will be addressed soon. Please be patient if you are using filesystem-livesync with E2EE.
- 0.21.5
- Improved:
- Now all revisions will be shown only its first a few letters.
- Now ID of the documents is shown in the log with the first 8 letters.
- Fixed:
- Check before modifying files has been implemented.
- Content change detection has been improved.
- 0.21.4
- This release had been skipped.
- 0.21.3
- Implemented:
- Now we can use SHA1 for hash function as fallback.
- 0.21.2
- IMPORTANT NOTICE: **0.21.1 CONTAINS A BUG WHILE REBUILDING THE DATABASE. IF YOU HAVE BEEN REBUILT, PLEASE MAKE SURE THAT ALL FILES ARE SANE.**
- This has been fixed in this version.
- Fixed:
- No longer files are broken while rebuilding.
- Now, Large binary files can be written correctly on a mobile platform.
- Any decoding errors now make zero-byte files.
- Modified:
- All files are processed sequentially for each.
- 0.21.1
- Fixed:
- No more infinity loops on larger files.
- Show message on decode error.
- Refactored:
- Fixed to avoid obsolete global variables.
- 0.21.0
- Changes and performance improvements:
- Now the saving files are processed by Blob.
- The V2-Format has been reverted.
- New encoding format has been enabled in default.
- WARNING: Since this version, the compatibilities with older Filesystem LiveSync have been lost.
## 0.20.0
At 0.20.0, Self-hosted LiveSync has changed the binary file format and encrypting format, for efficient synchronisation.
The dialogue will be shown and asks us to decide whether to keep v1 or use v2. Once we have enabled v2, all subsequent edits will be saved in v2. Therefore, devices running 0.19 or below cannot understand this and they might say that decryption error. Please update all devices.
Then we will have an impressive performance.
Of course, these are very impactful changes. If you have any questions or troubled things, please feel free to open an issue and mention me.
Note: if you want to roll it back to v1, please enable `Use binary and encryption version 1` on the `Hatch` pane and perform the `rebuild everything` once.
Extra but notable information:
This format change gives us the ability to detect some `marks` in the binary files as same as text files. Therefore, we can split binary files and some specific sort of them (i.e., PDF files) at the specific character. It means that editing the middle of files could be detected with marks.
Now only a few chunks are transferred, even if we add a comment to the PDF or put new files into the ZIP archives.
- 0.20.7
- Fixed
- To better replication, path obfuscation is now deterministic even if with E2EE.
Note: Compatible with previous database without any conversion. Only new files will be obfuscated in deterministic.
- 0.20.6
- Fixed
- Now empty file could be decoded.
- Local files are no longer pre-saved before fetching from a remote database.
- No longer deadlock while applying customisation sync.
- Configuration with multiple files is now able to be applied correctly.
- Deleting folder propagation now works without enabling the use of a trash bin.
- 0.20.5
- Fixed
- Now the files which having digit or character prefixes in the path will not be ignored.
- 0.20.4
- Fixed
- The text-input-dialogue is no longer broken.
- Finally, we can use the Setup URI again on mobile.
- 0.20.3
- New feature:
- We can launch Customization sync from the Ribbon if we enabled it.
- Fixed:
- Setup URI is now back to the previous spec; be encrypted by V1.
- It may avoid the trouble with iOS 17.
- The Settings dialogue is now registered at the beginning of the start-up process.
- We can change the configuration even though LiveSync could not be launched in normal.
- Improved:
- Enumerating documents has been faster.
- 0.20.2
- New feature:
- We can delete all data of customization sync from the `Delete all customization sync data` on the `Hatch` pane.
- Fixed:
- Prevent keep restarting on iOS by yielding microtasks.
- 0.20.1
- Fixed:
- No more UI freezing and keep restarting on iOS.
- Diff of Non-markdown documents are now shown correctly.
- Improved:
- Performance has been a bit improved.
- Customization sync has gotten faster.
- However, We lost forward compatibility again (only for this feature). Please update all devices.
- Misc
- Terser configuration has been more aggressive.
- 0.20.0
- Improved:
- A New binary file handling implemented
- A new encrypted format has been implemented
- Now the chunk sizes will be adjusted for efficient sync
- Fixed:
- levels of exception in some logs have been fixed
- Tidied:
- Some Lint warnings have been suppressed.
### 0.19.0
#### Customization sync

1
utils/.gitignore vendored Normal file
View File

@@ -0,0 +1 @@
fly.toml

29
utils/couchdb/couchdb-init.sh Executable file
View File

@@ -0,0 +1,29 @@
#!/bin/bash
if [[ -z "$hostname" ]]; then
echo "ERROR: Hostname missing"
exit 1
fi
if [[ -z "$username" ]]; then
echo "ERROR: Username missing"
exit 1
fi
if [[ -z "$password" ]]; then
echo "ERROR: Password missing"
exit 1
fi
echo "-- Configuring CouchDB by REST APIs... -->"
until (curl -X POST "${hostname}/_cluster_setup" -H "Content-Type: application/json" -d "{\"action\":\"enable_single_node\",\"username\":\"${username}\",\"password\":\"${password}\",\"bind_address\":\"0.0.0.0\",\"port\":5984,\"singlenode\":true}" --user "${username}:${password}"); do sleep 5; done
until (curl -X PUT "${hostname}/_node/nonode@nohost/_config/chttpd/require_valid_user" -H "Content-Type: application/json" -d '"true"' --user "${username}:${password}"); do sleep 5; done
until (curl -X PUT "${hostname}/_node/nonode@nohost/_config/chttpd_auth/require_valid_user" -H "Content-Type: application/json" -d '"true"' --user "${username}:${password}"); do sleep 5; done
until (curl -X PUT "${hostname}/_node/nonode@nohost/_config/httpd/WWW-Authenticate" -H "Content-Type: application/json" -d '"Basic realm=\"couchdb\""' --user "${username}:${password}"); do sleep 5; done
until (curl -X PUT "${hostname}/_node/nonode@nohost/_config/httpd/enable_cors" -H "Content-Type: application/json" -d '"true"' --user "${username}:${password}"); do sleep 5; done
until (curl -X PUT "${hostname}/_node/nonode@nohost/_config/chttpd/enable_cors" -H "Content-Type: application/json" -d '"true"' --user "${username}:${password}"); do sleep 5; done
until (curl -X PUT "${hostname}/_node/nonode@nohost/_config/chttpd/max_http_request_size" -H "Content-Type: application/json" -d '"4294967296"' --user "${username}:${password}"); do sleep 5; done
until (curl -X PUT "${hostname}/_node/nonode@nohost/_config/couchdb/max_document_size" -H "Content-Type: application/json" -d '"50000000"' --user "${username}:${password}"); do sleep 5; done
until (curl -X PUT "${hostname}/_node/nonode@nohost/_config/cors/credentials" -H "Content-Type: application/json" -d '"true"' --user "${username}:${password}"); do sleep 5; done
until (curl -X PUT "${hostname}/_node/nonode@nohost/_config/cors/origins" -H "Content-Type: application/json" -d '"app://obsidian.md,capacitor://localhost,http://localhost"' --user "${username}:${password}"); do sleep 5; done
echo "<-- Configuring CouchDB by REST APIs Done!"

4
utils/flyio/delete-server.sh Executable file
View File

@@ -0,0 +1,4 @@
#!/bin/bash
set -e
fly scale count 0 -y
fly apps destroy $(fly status -j | jq -r .Name) -y

43
utils/flyio/deploy-server.sh Executable file
View File

@@ -0,0 +1,43 @@
#!/bin/bash
## Script for deploy and automatic setup CouchDB onto fly.io.
## We need Deno for generating the Setup-URI.
source setenv.sh $@
export hostname="https://$appname.fly.dev"
echo "-- YOUR CONFIGURATION --"
echo "URL : $hostname"
echo "username: $username"
echo "password: $password"
echo "region : $region"
echo ""
echo "-- START DEPLOYING --> "
set -e
fly launch --name=$appname --env="COUCHDB_USER=$username" --copy-config=true --detach --no-deploy --region ${region} --yes
fly secrets set COUCHDB_PASSWORD=$password
fly deploy
set +e
../couchdb/couchdb-init.sh
# flyctl deploy
echo "OK!"
if command -v deno >/dev/null 2>&1; then
echo "Setup finished! Also, we can set up Self-hosted LiveSync instantly, by the following setup uri."
echo "Passphrase of setup-uri will be printed only one time. Keep it safe!"
echo "--- configured ---"
echo "database : ${database}"
echo "E2EE passphrase: ${passphrase}"
echo "--- setup uri ---"
deno run -A generate_setupuri.ts
else
echo "Setup finished! Here is the configured values (reprise)!"
echo "-- YOUR CONFIGURATION --"
echo "URL : $hostname"
echo "username: $username"
echo "password: $password"
echo "-- YOUR CONFIGURATION --"
echo "If we had Deno, we would got the setup uri directly!"
fi

40
utils/flyio/fly.template.toml Normal file
View File

@@ -0,0 +1,40 @@
## CouchDB for fly.io image
app = ''
primary_region = 'nrt'
swap_size_mb = 512
[build]
image = "couchdb:latest"
[mounts]
source = "couchdata"
destination = "/opt/couchdb/data"
initial_size = "1GB"
auto_extend_size_threshold = 90
auto_extend_size_increment = "1GB"
auto_extend_size_limit = "2GB"
[env]
COUCHDB_USER = ""
ERL_FLAGS = "-couch_ini /opt/couchdb/etc/default.ini /opt/couchdb/etc/default.d/ /opt/couchdb/etc/local.d /opt/couchdb/etc/local.ini /opt/couchdb/data/persistence.ini"
[http_service]
internal_port = 5984
force_https = true
auto_stop_machines = true
auto_start_machines = true
min_machines_running = 0
processes = ['app']
[[vm]]
cpu_kind = 'shared'
cpus = 1
memory_mb = 256
[[files]]
guest_path = "/docker-entrypoint2.sh"
raw_value = "#!/bin/bash\ntouch /opt/couchdb/data/persistence.ini\nchmod +w /opt/couchdb/data/persistence.ini\n/docker-entrypoint.sh $@"
[experimental]
entrypoint = ["tini", "--", "/docker-entrypoint2.sh"]

42
utils/flyio/generate_setupuri.ts Normal file
View File

@@ -0,0 +1,42 @@
import { encrypt } from "npm:octagonal-wheels@0.1.11/encryption/encryption.js";
const noun = ["waterfall", "river", "breeze", "moon", "rain", "wind", "sea", "morning", "snow", "lake", "sunset", "pine", "shadow", "leaf", "dawn", "glitter", "forest", "hill", "cloud", "meadow", "sun", "glade", "bird", "brook", "butterfly", "bush", "dew", "dust", "field", "fire", "flower", "firefly", "feather", "grass", "haze", "mountain", "night", "pond", "darkness", "snowflake", "silence", "sound", "sky", "shape", "surf", "thunder", "violet", "water", "wildflower", "wave", "water", "resonance", "sun", "log", "dream", "cherry", "tree", "fog", "frost", "voice", "paper", "frog", "smoke", "star"];
const adjectives = ["autumn", "hidden", "bitter", "misty", "silent", "empty", "dry", "dark", "summer", "icy", "delicate", "quiet", "white", "cool", "spring", "winter", "patient", "twilight", "dawn", "crimson", "wispy", "weathered", "blue", "billowing", "broken", "cold", "damp", "falling", "frosty", "green", "long", "late", "lingering", "bold", "little", "morning", "muddy", "old", "red", "rough", "still", "small", "sparkling", "thrumming", "shy", "wandering", "withered", "wild", "black", "young", "holy", "solitary", "fragrant", "aged", "snowy", "proud", "floral", "restless", "divine", "polished", "ancient", "purple", "lively", "nameless"];
function friendlyString() {
return `${adjectives[Math.floor(Math.random() * adjectives.length)]}-${noun[Math.floor(Math.random() * noun.length)]}`;
}
const uri_passphrase = `${Deno.env.get("uri_passphrase") ?? friendlyString()}`;
const URIBASE = "obsidian://setuplivesync?settings=";
async function main() {
const conf = {
"couchDB_URI": `${Deno.env.get("hostname")}`,
"couchDB_USER": `${Deno.env.get("username")}`,
"couchDB_PASSWORD": `${Deno.env.get("password")}`,
"couchDB_DBNAME": `${Deno.env.get("database")}`,
"syncOnStart": true,
"gcDelay": 0,
"periodicReplication": true,
"syncOnFileOpen": true,
"encrypt": true,
"passphrase": `${Deno.env.get("passphrase")}`,
"usePathObfuscation": true,
"batchSave": true,
"batch_size": 50,
"batches_limit": 50,
"useHistory": true,
"disableRequestURI": true,
"customChunkSize": 50,
"syncAfterMerge": false,
"concurrencyOfReadChunksOnline": 100,
"minimumIntervalOfReadChunksOnline": 100,
}
const encryptedConf = encodeURIComponent(await encrypt(JSON.stringify(conf), uri_passphrase, false));
const theURI = `${URIBASE}${encryptedConf}`;
console.log(theURI);
console.log("\nYour passphrase of Setup-URI is: ", uri_passphrase);
console.log("This passphrase is never shown again, so please note it in a safe place.")
}
await main();

30
utils/flyio/setenv.sh Executable file
View File

@@ -0,0 +1,30 @@
random_num() {
echo $RANDOM
}
random_noun() {
nouns=("waterfall" "river" "breeze" "moon" "rain" "wind" "sea" "morning" "snow" "lake" "sunset" "pine" "shadow" "leaf" "dawn" "glitter" "forest" "hill" "cloud" "meadow" "sun" "glade" "bird" "brook" "butterfly" "bush" "dew" "dust" "field" "fire" "flower" "firefly" "feather" "grass" "haze" "mountain" "night" "pond" "darkness" "snowflake" "silence" "sound" "sky" "shape" "surf" "thunder" "violet" "water" "wildflower" "wave" "water" "resonance" "sun" "log" "dream" "cherry" "tree" "fog" "frost" "voice" "paper" "frog" "smoke" "star")
echo ${nouns[$(($RANDOM % ${#nouns[*]}))]}
}
random_adjective() {
adjectives=("autumn" "hidden" "bitter" "misty" "silent" "empty" "dry" "dark" "summer" "icy" "delicate" "quiet" "white" "cool" "spring" "winter" "patient" "twilight" "dawn" "crimson" "wispy" "weathered" "blue" "billowing" "broken" "cold" "damp" "falling" "frosty" "green" "long" "late" "lingering" "bold" "little" "morning" "muddy" "old" "red" "rough" "still" "small" "sparkling" "thrumming" "shy" "wandering" "withered" "wild" "black" "young" "holy" "solitary" "fragrant" "aged" "snowy" "proud" "floral" "restless" "divine" "polished" "ancient" "purple" "lively" "nameless")
echo ${adjectives[$(($RANDOM % ${#adjectives[*]}))]}
}
cp ./fly.template.toml ./fly.toml
if [ "$1" = "renew" ]; then
unset appname
unset username
unset password
unset database
unset passphrase
unset region
fi
[ -z $appname ] && export appname=$(random_adjective)-$(random_noun)-$(random_num)
[ -z $username ] && export username=$(random_adjective)-$(random_noun)-$(random_num)
[ -z $password ] && export password=$(random_adjective)-$(random_noun)-$(random_num)
[ -z $database ] && export database="obsidiannotes"
[ -z $passphrase ] && export passphrase=$(random_adjective)-$(random_noun)-$(random_num)
[ -z $region ] && export region="nrt"

167
utils/readme.md Normal file
View File

@@ -0,0 +1,167 @@
<!-- For translation: 20240206r0 -->
# Utilities
Here are some useful things.
## couchdb
### couchdb-init.sh
This script can configure CouchDB with the necessary settings by REST APIs.
#### Materials
- Mandatory: curl
#### Usage
```sh
export hostname=http://localhost:5984/
export username=couchdb-admin-username
export password=couchdb-admin-password
./couchdb-init.sh
```
curl result will be shown, however, all of them can be ignored if the script has been run completely.
## fly.io
### deploy-server.sh
A fully automated CouchDB deployment script. We can deploy CouchDB onto fly.io. The only we need is an account of it.
All omitted configurations will be determined at random. (And, it is preferred). The region is configured to `nrt`.
If Japan is not close to you, please choose a region closer to you. However, the deployed database will work if you leave it at all.
#### Materials
- Mandatory: curl, flyctl
- Recommended: deno
#### Usage
```sh
#export appname=
#export username=
#export password=
#export database=
#export passphrase=
export region=nrt #pick your nearest location
./deploy-server.sh
```
The result of this command is as follows.
```
-- YOUR CONFIGURATION --
URL : https://young-darkness-25342.fly.dev
username: billowing-cherry-22580
password: misty-dew-13571
region : nrt
-- START DEPLOYING -->
An existing fly.toml file was found
Using build strategies '[the "couchdb:latest" docker image]'. Remove [build] from fly.toml to force a rescan
Creating app in /home/vorotamoroz/dev/obsidian-livesync/utils/flyio
We're about to launch your app on Fly.io. Here's what you're getting:
Organization: vorotamoroz (fly launch defaults to the personal org)
Name: young-darkness-25342 (specified on the command line)
Region: Tokyo, Japan (specified on the command line)
App Machines: shared-cpu-1x, 256MB RAM (specified on the command line)
Postgres: <none> (not requested)
Redis: <none> (not requested)
Created app 'young-darkness-25342' in organization 'personal'
Admin URL: https://fly.io/apps/young-darkness-25342
Hostname: young-darkness-25342.fly.dev
Wrote config file fly.toml
Validating /home/vorotamoroz/dev/obsidian-livesync/utils/flyio/fly.toml
Platform: machines
✓ Configuration is valid
Your app is ready! Deploy with `flyctl deploy`
Secrets are staged for the first deployment
==> Verifying app config
Validating /home/vorotamoroz/dev/obsidian-livesync/utils/flyio/fly.toml
Platform: machines
✓ Configuration is valid
--> Verified app config
==> Building image
Searching for image 'couchdb:latest' remotely...
image found: img_ox20prk63084j1zq
Watch your deployment at https://fly.io/apps/young-darkness-25342/monitoring
Provisioning ips for young-darkness-25342
Dedicated ipv6: 2a09:8280:1::37:fde9
Shared ipv4: 66.241.124.163
Add a dedicated ipv4 with: fly ips allocate-v4
Creating a 1 GB volume named 'couchdata' for process group 'app'. Use 'fly vol extend' to increase its size
This deployment will:
* create 1 "app" machine
No machines in group app, launching a new machine
WARNING The app is not listening on the expected address and will not be reachable by fly-proxy.
You can fix this by configuring your app to listen on the following addresses:
- 0.0.0.0:5984
Found these processes inside the machine with open listening sockets:
PROCESS | ADDRESSES
-----------------*---------------------------------------
/.fly/hallpass | [fdaa:0:73b9:a7b:22e:3851:7f28:2]:22
Finished launching new machines
NOTE: The machines for [app] have services with 'auto_stop_machines = true' that will be stopped when idling
-------
Checking DNS configuration for young-darkness-25342.fly.dev
Visit your newly deployed app at https://young-darkness-25342.fly.dev/
-- Configuring CouchDB by REST APIs... -->
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to young-darkness-25342.fly.dev:443
{"ok":true}
""
""
""
""
""
""
""
""
""
<-- Configuring CouchDB by REST APIs Done!
OK!
Setup finished! Also, we can set up Self-hosted LiveSync instantly, by the following setup uri.
Passphrase of setup-uri will be printed only one time. Keep it safe!
--- configured ---
database : obsidiannotes
E2EE passphrase: dark-wildflower-26467
--- setup uri ---
obsidian://setuplivesync?settings=%5B%22gZkBwjFbLqxbdSIbJymU%2FmTPBPAKUiHVGDRKYiNnKhW0auQeBgJOfvnxexZtMCn8sNiIUTAlxNaMGF2t%2BCEhpJoeCP%2FO%2BrwfN5LaNDQyky1Uf7E%2B64A5UWyjOYvZDOgq4iCKSdBAXp9oO%2BwKh4MQjUZ78vIVvJp8Mo6NWHfm5fkiWoAoddki1xBMvi%2BmmN%2FhZatQGcslVb9oyYWpZocduTl0a5Dv%2FQviGwlYQ%2F4NY0dVDIoOdvaYS%2FX4GhNAnLzyJKMXhPEJHo9FvR%2FEOBuwyfMdftV1SQUZ8YDCuiR3T7fh7Kn1c6OFgaFMpFm%2BWgIJ%2FZpmAyhZFpEcjpd7ty%2BN9kfd9gQsZM4%2BYyU9OwDd2DahVMBWkqoV12QIJ8OlJScHHdcUfMW5ex%2F4UZTWKNEHJsigITXBrtq11qGk3rBfHys8O0vY6sz%2FaYNM3iAOsR1aoZGyvwZm4O6VwtzK8edg0T15TL4O%2B7UajQgtCGxgKNYxb8EMOGeskv7NifYhjCWcveeTYOJzBhnIDyRbYaWbkAXQgHPBxzJRkkG%2FpBPfBBoJarj7wgjMvhLJ9xtL4FbP6sBNlr8jtAUCoq4L7LJcRNF4hlgvjJpL2BpFZMzkRNtUBcsRYR5J%2BM1X2buWi2BHncbSiRRDKEwNOQkc%2FmhMJjbAn%2F8eNKRuIICOLD5OvxD7FZNCJ0R%2BWzgrzcNV%22%2C%22ec7edc900516b4fcedb4c7cc01000000%22%2C%22fceb5fe54f6619ee266ed9a887634e07%22%5D
Your passphrase of Setup-URI is: patient-haze
This passphrase is never shown again, so please note it in a safe place.
```
All we have to do is copy the setup-URI (`obsidian`://...`) and open it from Self-hosted LiveSync on Obsidian.
If you did not install Deno, configurations will be printed again, instead of the setup-URI. In this case, we should configure it manually.
### delete-server.sh
The pair script of `deploy-server.sh`. We can delete the deployed server by this with fly.toml.
#### Materials
- Mandatory: flyctl, jq
- Recommended: none
#### Usage
```sh
./delete-server.sh
```
```
App 'young-darkness-25342 is going to be scaled according to this plan:
-1 machines for group 'app' on region 'nrt' of size 'shared-cpu-1x'
Executing scale plan
Destroyed e28667eec57158 group:app region:nrt size:shared-cpu-1x
Destroyed app young-darkness-25342
```