public/obsidian-livesync
1
0
Fork 0
mirror of https://github.com/vrtmrz/obsidian-livesync.git synced 2026-02-22 20:18:48 +00:00
Code Issues Packages Projects Releases Wiki Activity

Compare commits

base: public:0.20.4
Branches Tags
public:main public:beta public:snyk-upgrade-7fd31e6f474568014502e7205847f257 public:snyk-fix-90f3b71450f7543a1c61724d8afa0e71 public:refactor_services_2 public:refactor_services public:dev public:snyk-upgrade-fe9625385aa4be849c7157033fd2abb0 public:snyk-upgrade-99a31e70c822894595544b0c9813b11a public:svelteui public:disenchant public:snyk-fix-61f76ef0bce2a276df9552c2fc80a14f public:snyk-upgrade-2baa7938ba318464c3bad5c3fd5cbda5 public:snyk-upgrade-7af62a575eab20e8fb4e9565bf1adcfc public:snyk-upgrade-b32f281b31a81483f126b8fb6c54a957 public:snyk-upgrade-bd61b1103442d3faecc9fc726c34e9a4 public:snyk-upgrade-5b2d6584430c061b5187611a5d7fb64a public:snyk-fix-0e9fbbcb374fcb8d5e8cb027d8414fb7 public:snyk-upgrade-5242daf91e706d1ae500c9ea6545636e public:snyk-upgrade-70434a6822fbad83424135dfad9df909 public:snyk-upgrade-8fa01ccad0352125f1866d41e57d35e1 public:snyk-upgrade-39eb97b4655e70091f718a88fccd57c1 public:snyk-upgrade-2b99d54d19e207887e36c3071b796925 public:forprivacy public:snyk-upgrade-dad9088d0a333677e0157e3deef7fc38 public:snyk-upgrade-b9a76d74b3e5723759af5b3319aabb80 public:snyk-upgrade-7da6459d8ff5d293d80de9d80061d912 public:snyk-upgrade-6a3786fc4cbe394542e9d1b6058c46e7 public:snyk-upgrade-8fc9e62771dc41b0f23dc0efefb5c949
public:0.25.43-patched-9 public:0.25.43-patched-8 public:0.25.43-patched-7 public:0.25.43-patched-6 public:0.25.43-patched-5 public:0.25.43-patched-4 public:0.25.43-patched-3 public:0.25.43-patched-2 public:0.25.43-patched-1 public:0.25.43 public:0.25.42 public:0.25.41-patched-3 public:0.25.41-patched-2 public:0.25.41-patched-1 public:0.25.41 public:0.25.40 public:0.25.39 public:0.25.38 public:0.25.37 public:0.25.36 public:0.25.35 public:0.25.34 public:0.25.33 public:0.25.32 public:0.25.31 public:0.25.30 public:0.25.30-dev public:0.25.29 public:0.25.28 public:0.25.27 public:0.25.26 public:0.25.25 public:0.25.24 public:0.25.24.beta3 public:0.25.24.beta2 public:0.25.24.beta1 public:0.25.23 public:0.25.23.beta1 public:0.25.22 public:0.25.21 public:0.25.21.beta2 public:0.25.21.beta1 public:0.25.20 public:0.25.19 public:0.25.18 public:0.25.17 public:0.25.16 public:0.25.15 public:0.25.14 public:0.25.13 public:0.25.12 public:0.25.11 public:0.25.10 public:0.25.9 public:0.25.8 public:0.25.7 public:0.25.6 public:0.25.5 public:0.25.4 public:0.25.3 public:0.25.2 public:0.25.1 public:0.25.0 public:0.25.0-beta1 public:0.24.31 public:0.24.30 public:0.24.29 public:0.24.28 public:0.24.27 public:0.24.26 public:0.24.25 public:0.24.24 public:0.24.23 public:0.24.22 public:0.24.21 public:0.24.20 public:0.24.19 public:0.24.18 public:0.24.17 public:0.24.16 public:0.24.15 public:0.24.14 public:0.24.13 public:0.24.12 public:0.24.11 public:0.24.10 public:0.24.9 public:0.24.8 public:0.24.7 public:0.24.6 public:0.24.5 public:0.24.4 public:0.24.3 public:0.24.2 public:0.24.1 public:0.24.0 public:0.24.0.dev-rc8 public:0.24.0.dev-rc7 public:0.24.0.dev-rc6 public:0.24.0.dev-rc5 public:0.24.0.dev-rc4 public:0.24.0.dev-rc3 public:0.24.0.dev-rc2 public:0.23.23 public:0.23.22 public:0.23.21 public:0.23.20 public:0.23.19 public:0.23.18 public:0.23.17 public:0.23.16 public:0.23.15 public:0.23.14 public:0.23.13 public:0.23.12 public:0.23.11 public:0.23.10 public:0.23.9 public:0.23.8 public:0.23.7 public:0.23.6 public:0.23.5 public:0.23.4 public:0.23.3 public:0.23.2 public:0.23.1 public:0.23.0 public:0.22.19 public:0.22.18 public:0.22.17 public:0.22.16 public:0.22.15 public:0.22.14 public:0.22.13 public:0.22.12 public:0.22.11 public:0.22.10 public:0.22.9 public:0.22.8 public:0.22.7 public:0.22.6 public:0.22.5 public:0.22.4 public:0.22.3 public:0.22.2 public:0.22.1 public:0.22.0 public:0.21.5 public:0.21.4 public:0.21.3 public:0.21.2 public:0.21.1 public:0.21.0 public:0.20.7 public:0.20.6 public:0.20.5 public:0.20.4 public:0.20.3 public:0.20.2 public:0.20.1 public:0.20.0 public:0.19.23 public:0.19.22 public:0.19.21 public:0.19.20 public:0.19.19 public:0.19.18 public:0.19.17 public:0.19.16 public:0.19.15 public:0.19.14 public:0.19.13 public:0.19.12 public:0.19.11 public:0.19.10 public:0.19.9 public:0.19.8 public:0.19.7 public:0.19.6 public:0.19.5 public:0.19.4 public:0.19.3 public:0.19.2 public:0.19.1 public:0.19.0 public:0.18.6 public:0.18.5 public:0.18.4 public:0.18.3 public:0.18.2 public:0.18.1 public:0.18.0 public:0.17.34 public:0.17.33 public:0.17.32 public:0.17.31 public:0.17.30 public:0.17.29 public:0.17.28 public:0.17.27 public:0.17.26 public:0.17.25 public:0.17.24 public:0.17.23 public:0.17.22 public:0.17.21 public:0.17.20 public:0.17.19 public:0.17.18 public:0.17.17 public:0.17.16 public:0.17.15 public:0.17.14 public:0.17.13 public:0.17.12 public:0.17.11 public:0.17.10 public:0.17.9 public:0.17.8 public:0.17.7 public:0.17.6 public:0.17.5 public:0.17.4 public:0.17.3 public:0.17.2 public:0.17.1 public:0.17.0 public:0.16.8 public:0.16.7 public:0.16.6 public:0.16.5 public:0.16.4 public:0.16.3 public:0.16.2 public:0.16.1 public:0.16.0 public:0.15.10 public:0.15.9 public:0.15.8 public:0.15.7 public:0.15.6 public:0.15.5 public:0.15.4 public:0.15.3 public:0.15.2 public:0.15.1 public:0.15.0 public:0.14.8 public:0.14.7 public:0.14.6 public:0.14.5 public:0.14.4 public:0.14.3 public:0.14.2 public:0.14.1 public:0.14.0 public:0.13.4 public:0.13.3 public:0.13.2 public:0.13.1 public:0.13.0 public:0.12.5 public:0.12.4 public:0.12.3 public:0.12.2 public:0.12.1 public:0.12.0 public:0.11.10 public:0.11.9 public:0.11.8 public:0.11.7 public:0.11.6 public:0.11.5 public:0.11.4 public:0.11.3 public:0.11.2 public:0.11.1 public:0.11.0 public:0.10.1 public:0.10.0 public:0.9.0 public:0.8.8 public:0.8.7 public:0.8.6 public:0.8.5 public:0.8.3 public:0.8.2 public:0.8.1 public:0.8.0 public:0.7.2 public:0.7.1 public:0.7.0 public:0.6.1 public:0.6.0 public:0.5.0 public:0.4.1 public:0.4.0 public:0.3.1 public:0.3.0 public:0.2.1 public:0.2.0 public:0.1.26 public:0.1.25 public:0.1.24 public:0.1.23 public:0.1.22 public:0.1.21 public:0.1.20 public:0.1.19 public:0.1.18 public:0.1.17 public:0.1.16 public:0.1.15 public:0.1.14 public:0.1.13 public:0.1.12 public:0.1.11 public:0.1.10 public:0.1.9 public:0.1.8 public:0.1.7 public:0.1.6 public:0.1.5 public:0.1.4 public:0.1.3 public:0.1.2 public:0.1.1 public:0.1.0 public:0.0.9 public:0.0.8 public:0.0.7 public:0.0.6
...
compare: public:0.22.14
Branches Tags
public:beta public:main public:snyk-upgrade-7fd31e6f474568014502e7205847f257 public:snyk-fix-90f3b71450f7543a1c61724d8afa0e71 public:refactor_services_2 public:refactor_services public:dev public:snyk-upgrade-fe9625385aa4be849c7157033fd2abb0 public:snyk-upgrade-99a31e70c822894595544b0c9813b11a public:svelteui public:disenchant public:snyk-fix-61f76ef0bce2a276df9552c2fc80a14f public:snyk-upgrade-2baa7938ba318464c3bad5c3fd5cbda5 public:snyk-upgrade-7af62a575eab20e8fb4e9565bf1adcfc public:snyk-upgrade-b32f281b31a81483f126b8fb6c54a957 public:snyk-upgrade-bd61b1103442d3faecc9fc726c34e9a4 public:snyk-upgrade-5b2d6584430c061b5187611a5d7fb64a public:snyk-fix-0e9fbbcb374fcb8d5e8cb027d8414fb7 public:snyk-upgrade-5242daf91e706d1ae500c9ea6545636e public:snyk-upgrade-70434a6822fbad83424135dfad9df909 public:snyk-upgrade-8fa01ccad0352125f1866d41e57d35e1 public:snyk-upgrade-39eb97b4655e70091f718a88fccd57c1 public:snyk-upgrade-2b99d54d19e207887e36c3071b796925 public:forprivacy public:snyk-upgrade-dad9088d0a333677e0157e3deef7fc38 public:snyk-upgrade-b9a76d74b3e5723759af5b3319aabb80 public:snyk-upgrade-7da6459d8ff5d293d80de9d80061d912 public:snyk-upgrade-6a3786fc4cbe394542e9d1b6058c46e7 public:snyk-upgrade-8fc9e62771dc41b0f23dc0efefb5c949
public:0.25.43-patched-9 public:0.25.43-patched-8 public:0.25.43-patched-7 public:0.25.43-patched-6 public:0.25.43-patched-5 public:0.25.43-patched-4 public:0.25.43-patched-3 public:0.25.43-patched-2 public:0.25.43-patched-1 public:0.25.43 public:0.25.42 public:0.25.41-patched-3 public:0.25.41-patched-2 public:0.25.41-patched-1 public:0.25.41 public:0.25.40 public:0.25.39 public:0.25.38 public:0.25.37 public:0.25.36 public:0.25.35 public:0.25.34 public:0.25.33 public:0.25.32 public:0.25.31 public:0.25.30 public:0.25.30-dev public:0.25.29 public:0.25.28 public:0.25.27 public:0.25.26 public:0.25.25 public:0.25.24 public:0.25.24.beta3 public:0.25.24.beta2 public:0.25.24.beta1 public:0.25.23 public:0.25.23.beta1 public:0.25.22 public:0.25.21 public:0.25.21.beta2 public:0.25.21.beta1 public:0.25.20 public:0.25.19 public:0.25.18 public:0.25.17 public:0.25.16 public:0.25.15 public:0.25.14 public:0.25.13 public:0.25.12 public:0.25.11 public:0.25.10 public:0.25.9 public:0.25.8 public:0.25.7 public:0.25.6 public:0.25.5 public:0.25.4 public:0.25.3 public:0.25.2 public:0.25.1 public:0.25.0 public:0.25.0-beta1 public:0.24.31 public:0.24.30 public:0.24.29 public:0.24.28 public:0.24.27 public:0.24.26 public:0.24.25 public:0.24.24 public:0.24.23 public:0.24.22 public:0.24.21 public:0.24.20 public:0.24.19 public:0.24.18 public:0.24.17 public:0.24.16 public:0.24.15 public:0.24.14 public:0.24.13 public:0.24.12 public:0.24.11 public:0.24.10 public:0.24.9 public:0.24.8 public:0.24.7 public:0.24.6 public:0.24.5 public:0.24.4 public:0.24.3 public:0.24.2 public:0.24.1 public:0.24.0 public:0.24.0.dev-rc8 public:0.24.0.dev-rc7 public:0.24.0.dev-rc6 public:0.24.0.dev-rc5 public:0.24.0.dev-rc4 public:0.24.0.dev-rc3 public:0.24.0.dev-rc2 public:0.23.23 public:0.23.22 public:0.23.21 public:0.23.20 public:0.23.19 public:0.23.18 public:0.23.17 public:0.23.16 public:0.23.15 public:0.23.14 public:0.23.13 public:0.23.12 public:0.23.11 public:0.23.10 public:0.23.9 public:0.23.8 public:0.23.7 public:0.23.6 public:0.23.5 public:0.23.4 public:0.23.3 public:0.23.2 public:0.23.1 public:0.23.0 public:0.22.19 public:0.22.18 public:0.22.17 public:0.22.16 public:0.22.15 public:0.22.14 public:0.22.13 public:0.22.12 public:0.22.11 public:0.22.10 public:0.22.9 public:0.22.8 public:0.22.7 public:0.22.6 public:0.22.5 public:0.22.4 public:0.22.3 public:0.22.2 public:0.22.1 public:0.22.0 public:0.21.5 public:0.21.4 public:0.21.3 public:0.21.2 public:0.21.1 public:0.21.0 public:0.20.7 public:0.20.6 public:0.20.5 public:0.20.4 public:0.20.3 public:0.20.2 public:0.20.1 public:0.20.0 public:0.19.23 public:0.19.22 public:0.19.21 public:0.19.20 public:0.19.19 public:0.19.18 public:0.19.17 public:0.19.16 public:0.19.15 public:0.19.14 public:0.19.13 public:0.19.12 public:0.19.11 public:0.19.10 public:0.19.9 public:0.19.8 public:0.19.7 public:0.19.6 public:0.19.5 public:0.19.4 public:0.19.3 public:0.19.2 public:0.19.1 public:0.19.0 public:0.18.6 public:0.18.5 public:0.18.4 public:0.18.3 public:0.18.2 public:0.18.1 public:0.18.0 public:0.17.34 public:0.17.33 public:0.17.32 public:0.17.31 public:0.17.30 public:0.17.29 public:0.17.28 public:0.17.27 public:0.17.26 public:0.17.25 public:0.17.24 public:0.17.23 public:0.17.22 public:0.17.21 public:0.17.20 public:0.17.19 public:0.17.18 public:0.17.17 public:0.17.16 public:0.17.15 public:0.17.14 public:0.17.13 public:0.17.12 public:0.17.11 public:0.17.10 public:0.17.9 public:0.17.8 public:0.17.7 public:0.17.6 public:0.17.5 public:0.17.4 public:0.17.3 public:0.17.2 public:0.17.1 public:0.17.0 public:0.16.8 public:0.16.7 public:0.16.6 public:0.16.5 public:0.16.4 public:0.16.3 public:0.16.2 public:0.16.1 public:0.16.0 public:0.15.10 public:0.15.9 public:0.15.8 public:0.15.7 public:0.15.6 public:0.15.5 public:0.15.4 public:0.15.3 public:0.15.2 public:0.15.1 public:0.15.0 public:0.14.8 public:0.14.7 public:0.14.6 public:0.14.5 public:0.14.4 public:0.14.3 public:0.14.2 public:0.14.1 public:0.14.0 public:0.13.4 public:0.13.3 public:0.13.2 public:0.13.1 public:0.13.0 public:0.12.5 public:0.12.4 public:0.12.3 public:0.12.2 public:0.12.1 public:0.12.0 public:0.11.10 public:0.11.9 public:0.11.8 public:0.11.7 public:0.11.6 public:0.11.5 public:0.11.4 public:0.11.3 public:0.11.2 public:0.11.1 public:0.11.0 public:0.10.1 public:0.10.0 public:0.9.0 public:0.8.8 public:0.8.7 public:0.8.6 public:0.8.5 public:0.8.3 public:0.8.2 public:0.8.1 public:0.8.0 public:0.7.2 public:0.7.1 public:0.7.0 public:0.6.1 public:0.6.0 public:0.5.0 public:0.4.1 public:0.4.0 public:0.3.1 public:0.3.0 public:0.2.1 public:0.2.0 public:0.1.26 public:0.1.25 public:0.1.24 public:0.1.23 public:0.1.22 public:0.1.21 public:0.1.20 public:0.1.19 public:0.1.18 public:0.1.17 public:0.1.16 public:0.1.15 public:0.1.14 public:0.1.13 public:0.1.12 public:0.1.11 public:0.1.10 public:0.1.9 public:0.1.8 public:0.1.7 public:0.1.6 public:0.1.5 public:0.1.4 public:0.1.3 public:0.1.2 public:0.1.1 public:0.1.0 public:0.0.9 public:0.0.8 public:0.0.7 public:0.0.6

76 Commits
0.20.4 ... 0.22.14

Author SHA1 Message Date
vorotamoroz
e76e7ae8ea bump 2024-03-19 17:59:38 +01:00
vorotamoroz
f7fbe85d65 - New feature: 
- We can disable the status bar in the setting dialogue.
- Improved:
  - Now some files are handled as correct data type.
  - Customisation sync now uses the digest of each file for better performance.
  - The status in the Editor now works performant.
- Refactored:
  - Common functions have been ready and the codebase has been organised.
  - Stricter type checking following TypeScript updates.
  - Remove old iOS workaround for simplicity and performance.
 2024-03-19 17:58:55 +01:00
vorotamoroz
0313443b29 Merge pull request #389 from Seeker0472/fix-command 
Fixed docker-compose command in docs
 2024-03-19 14:06:23 +09:00
seeker0472
755c30f468 fix docker-compose command 2024-03-17 14:30:35 +08:00
vorotamoroz
b00b0cc5e5 bump 2024-03-15 10:37:15 +01:00
vorotamoroz
d7985a6b41 Improved: 
- Now using HTTP for the remote database URI warns of an error (on mobile) or notice (on desktop).
 2024-03-15 10:36:00 +01:00
vorotamoroz
486e816902 Update dependencies 2024-03-15 10:35:41 +01:00
vorotamoroz
ef9b19c24b bump 2024-03-04 04:07:51 +00:00
vorotamoroz
4ed9494176 Changed: 
- The default settings has been changed.
Improved:
- Default and preferred settings are applied on completion of the wizard.
Fixed:
- Now Initialisation `Fetch` will be performed smoothly and there will be fewer conflicts.
- No longer stuck while Handling transferred or initialised documents.
 2024-03-04 04:07:11 +00:00
vorotamoroz
fcd56d59d5 bump 2024-03-01 08:33:37 +00:00
vorotamoroz
1cabfcfd19 Fixed: 
- `Verify and repair all files` is no longer broken.
New feature::
- Now `Verify and repair all files` can restore or show history
Improved:
- Performance improved
 2024-03-01 08:32:48 +00:00
vorotamoroz
37a18dbfef bump 2024-03-01 03:28:46 +00:00
vorotamoroz
e7edf88713 Fixed 
- No longer unchanged hidden files and customisations are saved and transferred now.
- File integrity of vault history indicates the integrity correctly.
Improved
- In the report, the schema of the remote database URI is now printed.
 2024-03-01 03:28:06 +00:00
vorotamoroz
90ff75ab35 add notes. 2024-02-29 00:30:07 +00:00
vorotamoroz
bff1d661f5 Update troubleshooting.md 
Fix grammar
 2024-02-29 00:42:28 +09:00
vorotamoroz
6b59c14774 Update doc 2024-02-28 08:29:06 +00:00
vorotamoroz
8249274eac bump 2024-02-28 08:28:07 +00:00
vorotamoroz
3c6dae7814 - Fixed: 
- Fixed a bug on `fetch chunks on demand` that could not fetch the chunks on demand.
- Improved:
  - `fetch chunks on demand` works more smoothly.
  - Initialisation `Fetch` is now more efficient.
- Tidied:
  - Removed some meaningless codes.
 2024-02-28 08:27:17 +00:00
vorotamoroz
60cf8fe640 bump 2024-02-27 08:36:37 +00:00
vorotamoroz
3d89b3863f Fixed: 
- Now fetch and unlock the locked remote database works well again.
- No longer crash on symbolic links inside hidden folders.
Improved:
- Chunks are now created more efficiently.
- Better performance in saving notes.
- Network activities are indicated as an icon.
- Less memory used for binary processing.
Tidied:
- Cleaned unused functions up.
- Sorting out the codes that have become nonsense.
Changed:
- Now no longer `fetch chunks on demand` needs `Pacing replication`
 2024-02-27 08:35:46 +00:00
vorotamoroz
ee9364310d bump 2024-02-20 09:36:43 +00:00
vorotamoroz
86b9695bc2 Fixed: 
- No longer deleted hidden files were ignored.
- The document history dialogue is now able to process the deleted revisions.
- Deletion of a hidden file is now surely performed even if the file is already conflicted.
 2024-02-20 09:32:48 +00:00
vorotamoroz
e05f8771b9 bump 2024-02-20 05:15:48 +00:00
vorotamoroz
65619c2478 Fixed: 
- Fixed a problem with synchronisation taking a long time to start in some cases.
- Now we can disable E2EE encryption.
Improved:
- `Setup Wizard` is now more clear.
- `Minimal Setup` is now more simple.
- Self-hosted LiveSync now be able to use even if there are vaults with the same name.
- Now Self-hosted LiveSync waits until set-up is complete.
- Show reload prompts when possibly recommended while settings.
New feature:
- A guidance dialogue prompting for settings will be shown after the installation.
Changed
- Some setting names has been changed
 2024-02-20 05:13:53 +00:00
vorotamoroz
1552fa9d9e Merge branch 'main' of https://github.com/vrtmrz/obsidian-livesync 2024-02-15 08:36:49 +00:00
vorotamoroz
767f12b52f Rewrite 2024-02-15 08:35:12 +00:00
vorotamoroz
4071ba120e Update troubleshooting.md 2024-02-15 13:56:15 +09:00
vorotamoroz
2c0e3ba01c update faq 2024-02-15 04:38:31 +00:00
vorotamoroz
90adf06830 Merge branch 'main' of https://github.com/vrtmrz/obsidian-livesync 2024-02-15 04:04:06 +00:00
vorotamoroz
cf8e7ff6ca bump 2024-02-15 04:04:01 +00:00
vorotamoroz
95c3ff5043 Update common lib. 2024-02-15 04:02:28 +00:00
vorotamoroz
7ea3515801 Fixed: 
- Some description of settings have been refined
New feature:
- TroubleShooting is now shown in the setting dialogue.
 2024-02-15 04:02:20 +00:00
vorotamoroz
f866981a8a Update troubleshooting.md 2024-02-14 17:38:52 +09:00
vorotamoroz
8f36d6f893 fix minor layout 2024-02-09 03:20:31 +00:00
vorotamoroz
6dd86e9392 modify minor layouts. 2024-02-09 03:16:33 +00:00
vorotamoroz
d22716bef0 make more clear documents. 2024-02-09 03:13:03 +00:00
vorotamoroz
5d9baec5e4 update colab note 2024-02-07 10:17:45 +00:00
vorotamoroz
27d71ca2fb New utilities. 2024-02-06 11:03:51 +00:00
vorotamoroz
c024ed13d3 Refining the content. 2024-02-06 11:02:48 +00:00
vorotamoroz
b9527ccab0 bump 2024-01-30 17:31:52 +00:00
vorotamoroz
fa3aa2702c Fixed: 
- Now the result of conflict resolution could be surely written into the storage.
- Deleted files can be handled correctly again in the history dialogue and conflict dialogue.
- Some wrong log messages were fixed.
- Change handling now has become more stable.
- Some event handling became to be safer.
Improved:
- Dumping document information shows conflicts and revisions.
- The timestamp-only differences can be surely cached.
- Timestamp difference detection can be rounded by two seconds.
Refactored:
- A bit of organisation to write the test.
 2024-01-30 17:31:02 +00:00
vorotamoroz
93e7cbb133 bump. 2024-01-29 08:41:03 +00:00
vorotamoroz
716ae32e02 Fixed: 
- Deletion of files is now reliably synchronised.
 2024-01-29 08:40:41 +00:00
vorotamoroz
d6d8cbcf5a bump 2024-01-29 07:57:02 +00:00
vorotamoroz
efd348b266 Fixed: 
- No longer detects storage changes which have been caused by Self-hosted LiveSync itself.
- Setting sync file will be detected only if it has been configured now.
  - And its log will be shown only while the verbose log is enabled.
- Customisation file enumeration has got less blingy.
Fixed and improved:
- In-editor-status is now shown in the following areas:
  - Note editing pane (Source mode and live-preview mode).
  - New tab pane.
  - Canvas pane.
 2024-01-29 07:56:02 +00:00
vorotamoroz
8969b1800a bump 2024-01-24 08:53:00 +00:00
vorotamoroz
2c8e026e29 Fixed: 
- Now the results of resolving conflicts are surely synchronised.
Modified:
- Some setting items got new clear names.
New feature:
- We can limit the synchronising files by their size.
- Now the settings could be stored in a specific markdown file to synchronise or switch it
- Customisation of the obsoleted device is now able to be deleted at once.
 2024-01-24 08:52:47 +00:00
vorotamoroz
a6c27eab3d Merge pull request #367 from calvinbui/patch-1 
Skip workspace-mobile.json for cross-platform sync
 2024-01-24 15:55:50 +09:00
vorotamoroz
9b5c57d540 Merge pull request #336 from toon159/patch-1 
Lower payload size limit and batch limit from 10 to 2
 2024-01-22 13:03:29 +09:00
Calvin Bui
c251c596e8 Skip workspace-mobile.json for cross-platform sync 2024-01-18 13:23:48 +11:00
vorotamoroz
61188cfaef bump 2024-01-16 08:36:37 +00:00
vorotamoroz
97d944fd75 New feature: 
- We can perform automatic conflict resolution for inactive files, and postpone only manual ones by `Postpone manual resolution of inactive files`.
- Now we can see the image in the document history dialogue.
  - We can see the difference of the image, in the document history dialogue.
	- And also we can highlight differences.

Improved:
- Hidden file sync has been stabilised.
- Now automatically reloads the conflict-resolution dialogue when new conflicted revisions have arrived.

Fixed:
- No longer periodic process runs after unloading the plug-in.
- Now the modification of binary files is surely stored in the storage.
 2024-01-16 08:32:43 +00:00
vorotamoroz
d3dc1e7328 Minor fix and refine the readme 2024-01-12 10:29:18 +00:00
vorotamoroz
45304af369 bump 2024-01-12 09:38:57 +00:00
vorotamoroz
7f422d58f2 - Refined: 
- Task scheduling logics has been rewritten.
  - Possibly many bugs and fragile behaviour has been fixed
- Fixed:
  - Remote-chunk-fetching now works with keeping request intervals
- New feature:
  - We can show only the icons in the editor.
 2024-01-12 09:36:49 +00:00
vorotamoroz
c2491fdfad bump 2023-12-11 12:55:01 +09:00
vorotamoroz
06a6e391e8 Fixed for change detection bug. 2023-12-11 12:53:50 +09:00
vorotamoroz
f99475f6b7 bump 2023-12-11 12:46:23 +09:00
vorotamoroz
109fc00b9d Fixed 
- Now ID of the documents is shown in the log with the first 8 letters.
 2023-12-11 12:45:40 +09:00
vorotamoroz
c071d822e1 - Improved: 
- Now all revisions will be shown only its first a few letters.
- Fixed:
  - Check before modifying files has been implemented.
  - Content change detection has been improved.
 2023-12-11 12:22:17 +09:00
vorotamoroz
d2de5b4710 bump 2023-12-04 19:39:47 +09:00
vorotamoroz
cf5ecd8922 Implemented: 
- Now we can use SHA1 for hash function as fallback.
 2023-12-04 19:39:04 +09:00
vorotamoroz
b337a05b5a bump 2023-11-27 07:13:15 +00:00
vorotamoroz
9ea6bee9d1 - Fixed: 
- No longer files are broken while rebuilding.
    - Now, Large binary files can be written correctly on a mobile platform.
    - Any decoding errors now make zero-byte files.
  - Modified:
    - All files are processed sequentially for each.
 2023-11-27 06:55:55 +00:00
vorotamoroz
9747c26d50 bump 2023-11-25 02:22:26 +09:00
vorotamoroz
bb4b764586 - Fixed: 
- No more infinity loops on larger files.
    - Show message on decode error.
  - Refactored:
    - Fixed to avoid obsolete global variables.
 2023-11-25 02:21:44 +09:00
vorotamoroz
279b4b41e5 bump 2023-11-24 10:32:46 +00:00
vorotamoroz
b644fb791d - Changes and performance improvements: 
- Now the saving files are processed by Blob.
    - The V2-Format has been reverted.
    - New encoding format has been enabled in default.
    - WARNING: Since this version, the compatibilities with older Filesystem LiveSync have been lost.
 2023-11-24 10:31:58 +00:00
Vichaya Raksakunpanich
5802ed31be Update ObsidianLiveSyncSettingTab.ts 
Lower payload size limit and batch limit to 2 due to IBM Cloudant read/write limitation. Hope it will fix the "Replication error".
 2023-11-24 16:25:16 +07:00
vorotamoroz
ac9428e96b Fixed 
- To better replication, path obfuscation is now deterministic even if with E2EE.
 2023-11-15 08:44:03 +00:00
vorotamoroz
280d9e1dd9 Fixed: Fixed the issue of TOML editing. 2023-11-07 01:07:58 +00:00
vorotamoroz
f7209e566c bump 2023-10-24 10:07:29 +01:00
vorotamoroz
4a9ab2d1de Fixed: 
- No longer enumerating file names is broken.
 2023-10-24 10:07:17 +01:00
vorotamoroz
cb74b5ee93 - Fixed 
- Now empty file could be decoded.
    - Local files are no longer pre-saved before fetching from a remote database.
    - No longer deadlock while applying customisation sync.
    - Configuration with multiple files is now able to be applied correctly.
    - Deleting folder propagation now works without enabling the use of a trash bin.
 2023-10-24 09:54:56 +01:00
vorotamoroz
60eecd7001 bump 2023-10-17 12:00:59 +09:00
vorotamoroz
4bd7b54bcd Fixed: 
- Now the files which having digit or character prefixes in the path will not be ignored.
 2023-10-17 12:00:19 +09:00
51 changed files with 5410 additions and 3715 deletions
Expand all files Collapse all files

115
README.md
View File

@@ -1,95 +1,84 @@
<!-- For translation: 20240227r0 -->
# Self-hosted LiveSync
[Japanese docs](./README_ja.md) - [Chinese docs](./README_cn.md).
Self-hosted LiveSync is a community-implemented synchronization plugin.
A self-hosted or purchased CouchDB acts as the intermediate server. Available on every obsidian-compatible platform.
Note: It has no compatibility with the official "Obsidian Sync".
Self-hosted LiveSync is a community-implemented synchronization plugin, available on every obsidian-compatible platform and using CouchDB as the server.
![obsidian_live_sync_demo](https://user-images.githubusercontent.com/45774780/137355323-f57a8b09-abf2-4501-836c-8cb7d2ff24a3.gif)
Before installing or upgrading LiveSync, please back your vault up.
Note: This plugin cannot synchronise with the official "Obsidian Sync".
## Features
- Visual conflict resolver included.
- Bidirectional synchronization between devices nearly in real-time
- You can use CouchDB or its compatibles like IBM Cloudant.
- End-to-End encryption is supported.
- Plugin synchronization(Beta)
- Receive WebClip from [obsidian-livesync-webclip](https://chrome.google.com/webstore/detail/obsidian-livesync-webclip/jfpaflmpckblieefkegjncjoceapakdf) (End-to-End encryption will not be applicable.)
- Synchronize vaults very efficiently with less traffic.
- Good at conflicted modification.
- Automatic merging for simple conflicts.
- Using OSS solution for the server.
- Compatible solutions can be used.
- Supporting End-to-end encryption.
- Synchronisation of settings, snippets, themes, and plug-ins, via [Customization sync(Beta)](#customization-sync) or [Hidden File Sync](#hiddenfilesync)
- WebClip from [obsidian-livesync-webclip](https://chrome.google.com/webstore/detail/obsidian-livesync-webclip/jfpaflmpckblieefkegjncjoceapakdf)
Useful for researchers, engineers and developers with a need to keep their notes fully self-hosted for security reasons. Or just anyone who would like the peace of mind of knowing that their notes are fully private.
This plug-in might be useful for researchers, engineers, and developers with a need to keep their notes fully self-hosted for security reasons. Or just anyone who would like the peace of mind of knowing that their notes are fully private.
## IMPORTANT NOTICE
- Do not enable this plugin with another synchronization solution at the same time (including iCloud and Obsidian Sync). Before enabling this plugin, make sure to disable all the other synchronization methods to avoid content corruption or duplication. If you want to synchronize to two or more services, do them one by one and never enable two synchronization methods at the same time.
This includes not putting your vault inside a cloud-synchronized folder (eg. an iCloud folder or Dropbox folder)
- This is a synchronization plugin. Not a backup solution. Do not rely on this for backup.
- If the device's storage runs out, database corruption may happen.
- Hidden files or any other invisible files wouldn't be kept in the database, and thus won't be synchronized. (**and may also get deleted**)
>[!IMPORTANT]
> - Before installing or upgrading this plug-in, please back your vault up.
> - Do not enable this plugin with another synchronization solution at the same time (including iCloud and Obsidian Sync).
> - This is a synchronization plugin. Not a backup solution. Do not rely on this for backup.
## How to use
### Get your database ready.
### 3-minute setup - CouchDB on fly.io
First, get your database ready. fly.io is preferred for testing. Or you can use your own server with CouchDB. For more information, refer below:
1. [Setup fly.io](docs/setup_flyio.md)
2. [Setup IBM Cloudant](docs/setup_cloudant.md)
3. [Setup your CouchDB](docs/setup_own_server.md)
**Recommended for beginners**
### Configure the plugin
[![LiveSync Setup onto Fly.io SpeedRun 2024 using Google Colab](https://img.youtube.com/vi/7sa_I1832Xc/0.jpg)](https://www.youtube.com/watch?v=7sa_I1832Xc)
See [Quick setup guide](doccs/../docs/quick_setup.md)
1. [Setup CouchDB on fly.io](docs/setup_flyio.md)
2. Configure plug-in in [Quick Setup](docs/quick_setup.md)
## Something looks corrupted...
### Manually Setup
Please open the configuration link again and Answer below:
- If your local database looks corrupted (in other words, when your Obsidian getting weird even standalone.)
- Answer `No` to `Keep local DB?`
- If your remote database looks corrupted (in other words, when something happens while replicating)
- Answer `No` to `Keep remote DB?`
1. Setup the server
1. [Setup CouchDB on fly.io](docs/setup_flyio.md)
2. [Setup your CouchDB](docs/setup_own_server.md)
2. Configure plug-in in [Quick Setup](docs/quick_setup.md)
> [!TIP]
> We are still able to use IBM Cloudant. However, it is not recommended for several reasons nowadays. Here is [Setup IBM Cloudant](docs/setup_cloudant.md)
If you answered `No` to both, your databases will be rebuilt by the content on your device. And the remote database will lock out other devices. You have to synchronize all your devices again. (When this time, almost all your files should be synchronized with a timestamp. So you can use an existing vault).
## Information in StatusBar
Synchronization status is shown in statusbar.
Synchronization status is shown in the status bar with the following icons.
- Activity Indicator
- 📲 Network request
- Status
- ⏹️ Stopped
- 💤 LiveSync enabled. Waiting for changes.
- ⚡️ Synchronization in progress.
- ⚠ An error occurred.
- ↑ Uploaded chunks and metadata
- ↓ Downloaded chunks and metadata
- ⏳ Number of pending processes
- 🧩 Number of files waiting for their chunks.
If you have deleted or renamed files, please wait until ⏳ icon disappears.
- 💤 LiveSync enabled. Waiting for changes
- ⚡️ Synchronization in progress
- ⚠ An error occurred
- Statistical indicator
- ↑ Uploaded chunks and metadata
- ↓ Downloaded chunks and metadata
- Progress indicator
- 📥 Unprocessed transferred items
- 📄 Working database operation
- 💾 Working write storage processes
- ⏳ Working read storage processes
- 🛫 Pending read storage processes
- ⚙️ Working or pending storage processes of hidden files
- 🧩 Waiting chunks
- 🔌 Working Customisation items (Configuration, snippets, and plug-ins)
To prevent file and database corruption, please wait to stop Obsidian until all progress indicators have disappeared as possible (The plugin will also try to resume, though). Especially in case of if you have deleted or renamed files.
## Hints
- If a folder becomes empty after a replication, it will be deleted by default. But you can toggle this behaviour. Check the [Settings](docs/settings.md).
- LiveSync mode drains more batteries in mobile devices. Periodic sync with some automatic sync is recommended.
- Mobile Obsidian can not connect to non-secure (HTTP) or locally-signed servers, even if the root certificate is installed on the device.
- There are no 'exclude_folders' like configurations.
- While synchronizing, files are compared by their modification time and the older ones will be overwritten by the newer ones. Then plugin checks for conflicts and if a merge is needed, a dialog will open.
- Rarely, a file in the database could be corrupted. The plugin will not write to local storage when a file looks corrupted. If a local version of the file is on your device, the corruption could be fixed by editing the local file and synchronizing it. But if the file does not exist on any of your devices, then it can not be rescued. In this case, you can delete these items from the settings dialog.
- To stop the boot-up sequence (eg. for fixing problems on databases), you can put a `redflag.md` file (or directory) at the root of your vault.
Tip for iOS: a redflag directory can be created at the root of the vault using the File application.
- Also, with `redflag2.md` placed, we can automatically rebuild both the local and the remote databases during the boot-up sequence. With `redflag3.md`, we can discard only the local database and fetch from the remote again.
- Q: The database is growing, how can I shrink it down?
A: each of the docs is saved with their past 100 revisions for detecting and resolving conflicts. Picturing that one device has been offline for a while, and comes online again. The device has to compare its notes with the remotely saved ones. If there exists a historic revision in which the note used to be identical, it could be updated safely (like git fast-forward). Even if that is not in revision histories, we only have to check the differences after the revision that both devices commonly have. This is like git's conflict-resolving method. So, We have to make the database again like an enlarged git repo if you want to solve the root of the problem.
- And more technical Information is in the [Technical Information](docs/tech_info.md)
- If you want to synchronize files without obsidian, you can use [filesystem-livesync](https://github.com/vrtmrz/filesystem-livesync).
- WebClipper is also available on Chrome Web Store:[obsidian-livesync-webclip](https://chrome.google.com/webstore/detail/obsidian-livesync-webclip/jfpaflmpckblieefkegjncjoceapakdf)
Repo is here: [obsidian-livesync-webclip](https://github.com/vrtmrz/obsidian-livesync-webclip). (Docs are a work in progress.)
## Troubleshooting
If you are having problems getting the plugin working see: [Troubleshooting](docs/troubleshooting.md)
## Tips and Troubleshooting
If you are having problems getting the plugin working see: [Tips and Troubleshooting](docs/troubleshooting.md)
## License
The source code is licensed under the MIT License.
Licensed under the MIT License.

153
README_ja.md
View File

@@ -1,84 +1,85 @@
<!-- For translation: 20240227r0 -->
# Self-hosted LiveSync
[英語版ドキュメント](./README.md) - [中国語版ドキュメント](./README_cn.md).
**旧): obsidian-livesync**
Obsidianで利用可能なすべてのプラットフォームで使える、CouchDBをサーバに使用する、コミュニティ版の同期プラグイン
セルフホストしたデータベースを使って、双方向のライブシンクするObsidianのプラグイン。
**公式のSyncとは互換性はありません**
![obsidian_live_sync_demo](https://user-images.githubusercontent.com/45774780/137355323-f57a8b09-abf2-4501-836c-8cb7d2ff24a3.gif)
**インストールする前に、Vaultのバックアップを確実に取得してください**
[英語版](./README.md)
## こんなことができるプラグインです。
- Windows, Mac, iPad, iPhone, Android, Chromebookで動く
- セルフホストしたデータベースに同期して
- 複数端末で同時にその変更をほぼリアルタイムで配信し
- さらに、他の端末での変更も別の端末に配信する、双方向リアルタイムなLiveSyncを実現でき、
- 発生した変更の衝突はその場で解決できます。
- 同期先のホストにはCouchDBまたはその互換DBaaSのIBM Cloudantをサーバーに使用できます。あなたのデータは、あなたのものです。
- もちろんLiveではない同期もできます。
- 万が一のために、サーバーに送る内容を暗号化できます(betaです)。
- [Webクリッパー](https://chrome.google.com/webstore/detail/obsidian-livesync-webclip/jfpaflmpckblieefkegjncjoceapakdf) もあります(End-to-End暗号化対象外です)
NDAや類似の契約や義務、倫理を守る必要のある、研究者、設計者、開発者のような方に特にオススメです。
特にエンタープライズでは、たとえEnd to Endの暗号化が行われていても、管理下にあるサーバーにのみデータを格納することが求められる場合があります。
# 重要なお知らせ
- ❌ファイルの重複や破損を避けるため、複数の同期手段を同時に使用しないでください。
これは、Vaultをクラウド管理下のフォルダに置くことも含みます。(例えば、iCloudの管理フォルダ内に入れたり)。
- ⚠️このプラグインは、端末間でのノートの反映を目的として作成されました。バックアップ等が目的ではありません。そのため、バックアップは必ず別のソリューションで行うようにしてください。
- ストレージの空き容量が枯渇した場合、データベースが破損することがあります。
# このプラグインの使い方
1. Community Pluginsから、Self-holsted LiveSyncと検索しインストールするか、このリポジトリのReleasesから`main.js`, `manifest.json`, `style.css` をダウンロードしvaultの中の`.obsidian/plugins/obsidian-livesync`に入れて、Obsidianを再起動してください。
2. サーバーをセットアップします。IBM Cloudantがお手軽かつ堅牢で便利です。完全にセルフホストする際にはお持ちのサーバーにCouchDBをインストールする必要があります。詳しくは下記を参照してください
1. [IBM Cloudantのセットアップ](docs/setup_cloudant_ja.md)
2. [独自のCouchDBのセットアップ](docs/setup_own_server_ja.md)
備考： IBM Cloudantのアカウント登録が出来ないケースがあるようです。代替を探していて、今 [using fly.io](https://github.com/vrtmrz/obsidian-livesync/discussions/85)を検討しています。
1. [Quick setup](docs/quick_setup_ja.md)から、セットアップウィザード使ってセットアップしてください。
# テストサーバー
もし、CouchDBをインストールしたり、Cloudantのインスタンスをセットアップしたりするのに気が引ける場合、[Self-hosted LiveSyncのテストサーバー](https://olstaste.vrtmrz.net/)を作りましたので、使ってみてください。
備考: 制限事項をよく確認して使用してください。くれぐれも、本当に使用している自分のVaultを同期しないようにしてください。
# WebClipperあります
Self-hosted LiveSync用にWebClipperも作りました。Chrome Web Storeからダウンロードできます。
[obsidian-livesync-webclip](https://chrome.google.com/webstore/detail/obsidian-livesync-webclip/jfpaflmpckblieefkegjncjoceapakdf)
リポジトリはこちらです: [obsidian-livesync-webclip](https://github.com/vrtmrz/obsidian-livesync-webclip)。
相変わらずドキュメントは間に合っていません。
# ステータスバーの情報
右下のステータスバーに、同期の状態が表示されます
- 同期状態
- ⏹️ 同期は停止しています
- 💤 同期はLiveSync中で、なにか起こるのを待っています
- ⚡️ 同期中です
- ⚠ エラーが発生しています
- ↑ 送信したデータ数
- ↓ 受信したデータ数
- ⏳ 保留している処理の数です
ファイルを削除したりリネームした場合、この表示が消えるまでお待ちください。
# さらなる補足
- ファイルは同期された後、タイムスタンプを比較して新しければいったん新しい方で上書きされます。その後、衝突が発生したかによって、マージが行われます。
- まれにファイルが破損することがあります。破損したファイルに関してはディスクへの反映を試みないため、実際には使用しているデバイスには少し古いファイルが残っていることが多いです。そのファイルを再度更新してもらうと、データベースが更新されて問題なくなるケースがあります。ファイルがどの端末にも存在しない場合は、設定画面から、削除できます。
- データベースの復旧中に再起動した場合など、うまくローカルデータベースを修正できない際には、Vaultのトップに`redflag.md`というファイルを置いてください。起動時のシーケンスがスキップされます。
- データベースが大きくなってきてるんだけど、小さくできる？→各ノートは、それぞれの古い100リビジョンとともに保存されています。例えば、しばらくオフラインだったあるデバイスが、久しぶりに同期したと想定してみてください。そのとき、そのデバイスは最新とは少し異なるリビジョンを持ってるはずです。その場合でも、リモートのリビジョン履歴にリモートのものが存在した場合、安全にマージできます。もしリビジョン履歴に存在しなかった場合、確認しなければいけない差分も、対象を存在して持っている共通のリビジョン以降のみに絞れます。ちょうどGitのような方法で、衝突を解決している形になるのです。そのため、肥大化したリポジトリの解消と同様に、本質的にデータベースを小さくしたい場合は、データベースの作り直しが必要です。
- その他の技術的なお話は、[技術的な内容](docs/tech_info_ja.md)に書いてあります。
※公式のSyncと同期することはできません。
# ライセンス
## 機能
- 高効率・低トラフィックでVault同士を同期
- 競合解決がいい感じ
- 単純な競合なら自動マージします
- OSSソリューションを同期サーバに使用
- 互換ソリューションも使用可能です
- End-to-End暗号化実装済み
- 設定・スニペット・テーマ、プラグインの同期が可能
- [Webクリッパー](https://chrome.google.com/webstore/detail/obsidian-livesync-webclip/jfpaflmpckblieefkegjncjoceapakdf) もあります
The source code is licensed MIT.
NDAや類似の契約や義務、倫理を守る必要のある、研究者、設計者、開発者のような方に特にオススメです。
>[!IMPORTANT]
> - インストール・アップデート前には必ずVaultをバックアップしてください
> - 複数の同期ソリューションを同時に有効にしないでください（これはiCloudや公式のSyncも含みます）
> - このプラグインは同期プラグインです。バックアップとして使用しないでください
## このプラグインの使い方
### 3分セットアップ - CouchDB on fly.io
**はじめての方におすすめ**
[![LiveSync Setup onto Fly.io SpeedRun 2024 using Google Colab](https://img.youtube.com/vi/7sa_I1832Xc/0.jpg)](https://www.youtube.com/watch?v=7sa_I1832Xc)
1. [Fly.ioにCouchDBをセットアップする](docs/setup_flyio.md)
2. [Quick Setup](docs/quick_setup_ja.md)でプラグインを設定する
### Manually Setup
1. サーバのセットアップ
1. [Fly.ioにCouchDBをセットアップする](docs/setup_flyio.md)
2. [CouchDBをセットアップする](docs/setup_own_server_ja.md)
2. [Quick Setup](docs/quick_setup_ja.md)でプラグインを設定する
> [!TIP]
> IBM Cloudantもまだ使用できますが、いくつかの理由で現在はおすすめしていません。[IBM Cloudantのセットアップ](docs/setup_cloudant_ja.md)はまだあります。
## ステータスバーの説明
同期ステータスはステータスバーに、下記のアイコンとともに表示されます
- アクティビティー
- 📲 ネットワーク接続中
- 同期ステータス
- ⏹️ 停止中
- 💤 変更待ち（LiveSync中）
- ⚡️ 同期の進行中
- ⚠ エラー
- 統計情報
- ↑ アップロードしたチャンクとメタデータ数
- ↓ ダウンロードしたチャンクとメタデータ数
- 進捗情報
- 📥 転送後、未処理の項目数
- 📄 稼働中データベース操作数
- 💾 稼働中のストレージ書き込み数操作数
- ⏳ 稼働中のストレージ読み込み数操作数
- 🛫 待機中のストレージ読み込み数操作数
- ⚙️ 隠しファイルの操作数（待機・稼働中合計）
- 🧩 取得待ちを行っているチャンク数
- 🔌 設定同期関連の操作数
データベースやファイルの破損を避けるため、Obsidianの終了は進捗情報が表示されなくなるまで待ってください（プラグインも復帰を試みますが）。特にファイルを削除やリネームした場合は気をつけてください。
## Tips and Troubleshooting
何かこまったら、[Tips and Troubleshooting](docs/troubleshooting.md)をご参照ください。
## License
Licensed under the MIT License.

322
deploy_couchdb_to_flyio_v2_with_swap.ipynb
View File

@@ -1,322 +0,0 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/gist/vrtmrz/37c3efd7842e49947aaaa7f665e5020a/deploy_couchdb_to_flyio_v2_with_swap.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "markdown",
"source": [
"History:\n",
"- 18, May, 2023: Initial.\n",
"- 19, Jun., 2023: Patched for enabling swap.\n",
"- 22, Aug., 2023: Generating Setup-URI implemented."
],
"metadata": {
"id": "HiRV7G8Gk1Rs"
}
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "2Vh0mEQEZuAK"
},
"outputs": [],
"source": [
"# Configurations\n",
"import os\n",
"os.environ['region']=\"nrt\"\n",
"os.environ['couchUser']=\"alkcsa93\"\n",
"os.environ['couchPwd']=\"c349usdfnv48fsasd\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "SPmbB0jZauQ1"
},
"outputs": [],
"source": [
"# Delete once\n",
"!rm ./fly.toml"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Nze7QoxLZ7Yx"
},
"outputs": [],
"source": [
"# Installation\n",
"# You have to set up your account in here.\n",
"!curl -L https://fly.io/install.sh | sh\n",
"!/root/.fly/bin/flyctl auth signup"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "MVJwsIYrbgtx"
},
"outputs": [],
"source": [
"# Generate server\n",
"!/root/.fly/bin/flyctl launch --auto-confirm --generate-name --detach --no-deploy --region ${region}\n",
"!/root/.fly/bin/fly volumes create --region ${region} couchdata --size 2 --yes"
]
},
{
"cell_type": "code",
"source": [
"# Check the toml once.\n",
"!cat fly.toml"
],
"metadata": {
"id": "2RSoO9o-i2TT"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "zUtPZLVnbvdQ"
},
"outputs": [],
"source": [
"# Modify fly.toml\n",
"## Port modification\n",
"!sed -i 's/8080/5984/g' fly.toml\n",
"## Add user into.\n",
"!echo -e \"\\n[env]\\n COUCHDB_USER = \\\"${couchUser}\\\"\" >> ./fly.toml\n",
"## Set the location of an ini file which to save configurations persistently via erlang flags.\n",
"!echo -e \"\\nERL_FLAGS=\\\"-couch_ini /opt/couchdb/etc/default.ini /opt/couchdb/etc/default.d/ /opt/couchdb/etc/local.d /opt/couchdb/etc/local.ini /opt/couchdb/data/persistence.ini\\\"\" >> ./fly.toml\n",
"## Mounting volumes to store data and ini file.\n",
"!echo -e \"\\n[mounts]\\n source=\\\"couchdata\\\"\\n destination=\\\"/opt/couchdb/data\\\"\" >> ./fly.toml\n",
"!cat fly.toml"
]
},
{
"cell_type": "code",
"source": [
"# Make the Dockerfile to modify the permission of the ini file. If you want to use a specific version, you should change `latest` here.\n",
"!echo -e \"\\n[build]\\n dockerfile = \\\"./Dockerfile\\\"\" >> ./fly.toml"
],
"metadata": {
"id": "LQPsZ_dYxkTu"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"!echo -e \"FROM couchdb:latest\\nRUN sed -i '2itouch /opt/couchdb/data/persistence.ini && chmod +w /opt/couchdb/data/persistence.ini && fallocate -l 512M /swapfile && chmod 0600 /swapfile && mkswap /swapfile && echo 10 > /proc/sys/vm/swappiness && swapon /swapfile && echo 1 > /proc/sys/vm/overcommit_memory' /docker-entrypoint.sh\" > ./Dockerfile"
],
"metadata": {
"id": "44cBeGJ9on5i"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Check dockerfile\n",
"!cat ./Dockerfile"
],
"metadata": {
"id": "ai2R3BbpxRSe"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "xWdsTCI6bzk2"
},
"outputs": [],
"source": [
"# Configure password\n",
"!/root/.fly/bin/flyctl secrets set COUCHDB_PASSWORD=${couchPwd}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "k0WIQlShcXGa"
},
"outputs": [],
"source": [
"# Deploy server\n",
"# Be sure to shutdown after the test.\n",
"!/root/.fly/bin/flyctl deploy --detach --remote-only\n",
"!/root/.fly/bin/flyctl status"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "0ySggkdlfq7M"
},
"outputs": [],
"source": [
"import subprocess, json\n",
"result = subprocess.run([\"/root/.fly/bin/flyctl\",\"status\",\"-j\"], capture_output=True, text=True)\n",
"if result.returncode==0:\n",
" hostname = json.loads(result.stdout)[\"Hostname\"]\n",
" os.environ['couchHost']=\"https://%s\" % (hostname)\n",
" print(\"Your couchDB server is https://%s/\" % (hostname))\n",
"else:\n",
" print(\"Something occured.\")\n"
]
},
{
"cell_type": "code",
"source": [
"# Finish setting up the CouchDB\n",
"# Please repeat until the request is completed without error messages\n",
"# i.e., You have to redo this block while \"curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to xxxx\" is showing.\n",
"!curl -X POST \"${couchHost}/_cluster_setup\" -H \"Content-Type: application/json\" -d \"{\\\"action\\\":\\\"enable_single_node\\\",\\\"username\\\":\\\"${couchUser}\\\",\\\"password\\\":\\\"${couchPwd}\\\",\\\"bind_address\\\":\\\"0.0.0.0\\\",\\\"port\\\":5984,\\\"singlenode\\\":true}\" --user \"${couchUser}:${couchPwd}\""
],
"metadata": {
"id": "cGlSzVqlQG_z"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Please repeat until all lines are completed without error messages\n",
"!curl -X PUT \"${couchHost}/_node/nonode@nohost/_config/chttpd/require_valid_user\" -H \"Content-Type: application/json\" -d '\"true\"' --user \"${couchUser}:${couchPwd}\"\n",
"!curl -X PUT \"${couchHost}/_node/nonode@nohost/_config/chttpd_auth/require_valid_user\" -H \"Content-Type: application/json\" -d '\"true\"' --user \"${couchUser}:${couchPwd}\"\n",
"!curl -X PUT \"${couchHost}/_node/nonode@nohost/_config/httpd/WWW-Authenticate\" -H \"Content-Type: application/json\" -d '\"Basic realm=\\\"couchdb\\\"\"' --user \"${couchUser}:${couchPwd}\"\n",
"!curl -X PUT \"${couchHost}/_node/nonode@nohost/_config/httpd/enable_cors\" -H \"Content-Type: application/json\" -d '\"true\"' --user \"${couchUser}:${couchPwd}\"\n",
"!curl -X PUT \"${couchHost}/_node/nonode@nohost/_config/chttpd/enable_cors\" -H \"Content-Type: application/json\" -d '\"true\"' --user \"${couchUser}:${couchPwd}\"\n",
"!curl -X PUT \"${couchHost}/_node/nonode@nohost/_config/chttpd/max_http_request_size\" -H \"Content-Type: application/json\" -d '\"4294967296\"' --user \"${couchUser}:${couchPwd}\"\n",
"!curl -X PUT \"${couchHost}/_node/nonode@nohost/_config/couchdb/max_document_size\" -H \"Content-Type: application/json\" -d '\"50000000\"' --user \"${couchUser}:${couchPwd}\"\n",
"!curl -X PUT \"${couchHost}/_node/nonode@nohost/_config/cors/credentials\" -H \"Content-Type: application/json\" -d '\"true\"' --user \"${couchUser}:${couchPwd}\"\n",
"!curl -X PUT \"${couchHost}/_node/nonode@nohost/_config/cors/origins\" -H \"Content-Type: application/json\" -d '\"app://obsidian.md,capacitor://localhost,http://localhost\"' --user \"${couchUser}:${couchPwd}\""
],
"metadata": {
"id": "JePzrsHypY18"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"Now, our CouchDB has been surely installed and configured. Cheers!\n",
"\n",
"In the steps that follow, create a setup-URI.\n",
"\n",
"This URI could be imported directly into Self-hosted LiveSync, to configure the use of the CouchDB which we configured now."
],
"metadata": {
"id": "YfSOomsoXbGS"
}
},
{
"cell_type": "code",
"source": [
"# Database config\n",
"import random, string\n",
"\n",
"def randomname(n):\n",
" return ''.join(random.choices(string.ascii_letters + string.digits, k=n))\n",
"\n",
"# The database name\n",
"os.environ['database']=\"obsidiannote\"\n",
"# The passphrase to E2EE\n",
"os.environ['passphrase']=randomname(20)\n",
"\n",
"print(\"Your database:\"+os.environ['database'])\n",
"print(\"Your passphrase:\"+os.environ['passphrase'])"
],
"metadata": {
"id": "416YncOqXdNn"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Install deno for make setup uri\n",
"!curl -fsSL https://deno.land/x/install/install.sh | sh"
],
"metadata": {
"id": "C4d7C0HAXgsr"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Fetch module for encrypting a Setup URI\n",
"!curl -o encrypt.ts https://gist.githubusercontent.com/vrtmrz/f9d1d95ee2ca3afa1a924a2c6759b854/raw/d7a070d864a6f61403d8dc74208238d5741aeb5a/encrypt.ts"
],
"metadata": {
"id": "hQL_Dx-PXise"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# Make buttons!\n",
"from IPython.display import HTML\n",
"result = subprocess.run([\"/root/.deno/bin/deno\",\"run\",\"-A\",\"encrypt.ts\"], capture_output=True, text=True)\n",
"text=\"\"\n",
"if result.returncode==0:\n",
" text = result.stdout.strip()\n",
" result = HTML(f\"<button onclick=navigator.clipboard.writeText('{text}')>Copy setup uri</button><br>Importing passphrase is `welcome`. <br>If you want to synchronise in live mode, please apply a preset after setup.)\")\n",
"else:\n",
" result = \"Failed to encrypt the setup URI\"\n",
"result"
],
"metadata": {
"id": "o0gX_thFXlIZ"
},
"execution_count": null,
"outputs": []
}
],
"metadata": {
"colab": {
"provenance": [],
"private_outputs": true,
"include_colab_link": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": "python"
},
"gpuClass": "standard"
},
"nbformat": 4,
"nbformat_minor": 0
}

102
docs/quick_setup.md
View File

@@ -2,34 +2,51 @@
[Japanese docs](./quick_setup_ja.md) - [Chinese docs](./quick_setup_cn.md).
The plugin has so many configuration options to deal with different circumstances. However, there are not so many settings that are actually used. Therefore, `The Setup wizard` has been implemented to simplify the initial setup.
Note: Subsequent devices are recommended to be set up using the `Copy setup URI` and `Open setup URI`.
## The Setup wizard
Open the `🧙‍♂️ Setup wizard` in the settings dialogue. If the plugin has not been configured before, it should already be open.
The plugin has so many configuration options to deal with different circumstances. However, only a few settings are required in the normal cases. Therefore, `The Setup wizard` has been implemented to simplify the setup.
![](../images/quick_setup_1.png)
- Discard the existing configuration and set up
If you have changed any settings, this button allows you to discard all changes before setting up.
There are three methods to set up Self-hosted LiveSync.
- Do not discard the existing configuration and set up
Simply reconfigure. Be careful. In wizard mode, you cannot see all configuration items, even if they have been configured.
1. [Using setup URIs](#1-using-setup-uris) *(Recommended)*
2. [Minimal setup](#2-minimal-setup)
3. [Full manually setup the and Enable on this dialogue](#3-manually-setup)
Pressing `Next` on one of the above options will put the configuration dialog into wizard mode.
## At the first device
### Wizard mode
### 1. Using setup URIs
> [!TIP] What is the setup URI? Why is it required?
> The setup URI is the encrypted representation of Self-hosted LiveSync configuration as a URI. This starts `obsidian://setuplivesync?settings=`. This is encrypted with a passphrase, so that it can be shared relatively securely between devices. It is a bit long, but it is one line. This allows a series of settings to be set at once without any inconsistencies.
>
> If you have configured the remote database by [Automated setup on Fly.io](./setup_flyio.md#a-very-automated-setup) or [set up your server with the tool](./setup_own_server.md#1-generate-the-setup-uri-on-a-desktop-device-or-server), **you should have one of them**
In this procedure, [this video](https://youtu.be/7sa_I1832Xc?t=146) may help us.
1. Click `Use` button (Or launch `Use the copied setup URI` from Command palette).
2. Paste the Setup URI into the dialogue
3. Type the passphrase of the Setup URI
4. Answer `yes` for `Importing LiveSync's conf, OK?`.
5. Answer `Set it up as secondary or subsequent device` for `How would you like to set it up?`.
6. Initialisation will begin, please hold a while.
7. You will asked about the hidden file synchronisation, answer as you like.
1. If you are new to Self-hosted LiveSync, we can configure it later so leave it once.
8. Synchronisation has been started! `Reload app without saving` is recommended after the indicators of Self-hosted LiveSync disappear.
OK, we can proceed the [next step](#).
### 2. Minimal setup
If you do not have any setup URI, Press the `start` button. The setting dialogue turns into the wizard mode and will display only minimal items.
>[!TIP]
> We can generate the setup URI with the tool in any time. Please use [this tool](./setup_own_server.md#1-generate-the-setup-uri-on-a-desktop-device-or-server).
![](../images/quick_setup_2.png)
Let's see how to use it step-by-step.
#### Remote database configuration
## Remote Database configuration
### Remote database configuration
Enter the information for the database we have set up.
1. Enter the information for the database we have set up.
![](../images/quick_setup_3.png)
@@ -37,13 +54,9 @@ Enter the information for the database we have set up.
#### Test database connection and Check database configuration
We can check the connectivity to the database, and the database settings.
![](../images/quick_setup_5.png)
#### Test Database Connection
Check whether we can connect to the database. If it fails, there are several possible reasons, but first attempt the `Check database configuration` check to see if it fails there too.
#### Check database configuration
#### Check and Fix database configuration
Check the database settings and fix any problems on the spot.
@@ -52,40 +65,43 @@ Check the database settings and fix any problems on the spot.
This item may vary depending on the connection. In the above case, press all three Fix buttons.
If the Fix buttons disappear and all become check marks, we are done.
### Confidentiality configuration
#### Confidentiality configuration (Optional but very preferred)
![](../images/quick_setup_4.png)
Encrypt your database in case of unintended database exposure; enable End to End encryption and the contents of your notes will be encrypted at the moment it leaves the device. We strongly recommend enabling it. And `Path Obfuscation` also obfuscates filenames. Now stable and recommended.
Encryption is based on 256-bit AES-GCM.
Enable End-to-end encryption and the contents of your notes will be encrypted at the moment it leaves the device. We strongly recommend enabling it. And `Path Obfuscation` also obfuscates filenames. Now stable and recommended.
These setting can be disabled if you are inside a closed network and it is clear that you will not be accessed by third parties.
![](../images/quick_setup_7.png)
> [!TIP]
> Encryption is based on 256-bit AES-GCM.
#### Next
Go to the Sync Settings.
We should proceed to the Next step.
#### Discard existing database and proceed
Discard the contents of the Remote database and go to the Sync Settings.
### Sync Settings
#### Sync Settings
Finally, finish the wizard by selecting a preset for synchronisation.
![](../images/quick_setup_9_1.png)
Select any synchronisation methods we want to use and `Apply` to initialise and build the local and remote databases as required. If `All done!` is displayed, we are done. Automatically, `Copy setup URI` will open and we will be asked for a passphrase to encrypt the `Setup URI`.
Select any synchronisation methods we want to use and `Apply`. If database initialisation is required, it will be performed at this time. When `All done!` is displayed, we are ready to synchronise.
The dialogue of `Copy settings as a new setup URI` will be open automatically. Please input a passphrase to encrypt the new `Setup URI`. (This passphrase is to encrypt the setup URI, not the vault).
![](../images/quick_setup_10.png)
Set the passphrase as you like.
The Setup URI will be copied to the clipboard, which you can then transfer to the second and subsequent devices in some way.
The Setup URI will be copied to the clipboard, please make a note(Not in Obsidian) of this.
# How to set up the second and subsequent units
After installing Self-hosted LiveSync on the first device, select `Open setup URI` from the command palette and enter the setup URI you transferred. Afterwards, enter your passphrase and a setup wizard will open.
Answer the following.
>[!TIP]
We can copy this in any time by `Copy current settings as a new setup URI`.
- `Yes` to `Importing LiveSync's conf, OK?`
- `Set it up as secondary or subsequent device` to `How would you like to set it up?`.
### 3. Manually setup
Then, The configuration will take effect and replication will start. Your files will be synchronised soon! You may need to close the settings dialog and reopen it to see the settings fields populated properly, but they will be set.
It is strongly recommended to perform a "minimal set-up" first and set up the other contents after making sure has been synchronised.
However, if you have some specific reasons to configure it manually, please click the `Enable` button of `Enable LiveSync on this device as the set-up was completed manually`.
And, please copy the setup URI by `Copy current settings as a new setup URI` and make a note(Not in Obsidian) of this.
## At the subsequent device
After installing Self-hosted LiveSync on the first device, we should have a setup URI. **The first choice is to use it**. Please share it with the device you want to setup.
It is completely same as [Using setup URIs on the first device](#1-using-setup-uris). Please refer it.

237
docs/setup_flyio.md
View File

@@ -1,24 +1,53 @@
<!-- For translation: 20240209r0 -->
# Setup CouchDB on fly.io
In some cases, the use of IBM Cloudant was found to be hard. We looked for alternatives, but there were no services available. Therefore, we have to build our own servers, which is quite a challenge. In this situation, with fly.io, most of the processes are simplified and only CouchDB needs to be configured.
This is how to configure fly.io and CouchDB on it for Self-hosted LiveSync.
It generally falls within the `Free Allowances` range in most cases.
> [!WARNING]
> It is **your** instance. In Obsidian, we have files locally. Hence, do not hesitate to destroy the remote database if you feel something have got weird. We can launch and switch to the new CouchDB instance anytime[^1].
>
[^1]: Actually, I am always building the database for reproduction of the issue like so.
**[Automatic setup using Colaboratory](#automatic-setup-using-colaboratory) is recommended, after reading this document through. It is reproducible and hard to fail.**
> [!NOTE]
> **What and why is the Fly.io?**
> At some point, we started to experience problems related to our IBM Cloudant account. At the same time, Self-hosted LiveSync started to improve its functionality, requiring CouchDB in a more natural state to use all its features.
>
> Then we found Fly.io. Fly.io is the PaaS Platform, which can be useable for a very reasonable price. It generally falls within the `Free Allowances` range in most cases.
## Required materials
- A valid credit or debit card.
## Warning
## Setup CouchDB instance
- It will be `your` instance. Check the log regularly.
### A. Very automated setup
## Prerequisites
[![LiveSync Setup onto Fly.io SpeedRun 2024 using Google Colab](https://img.youtube.com/vi/7sa_I1832Xc/0.jpg)](https://www.youtube.com/watch?v=7sa_I1832Xc)
For simplicity, the following description assumes that the settings are as shown in the table below. Please read it in accordance with the actual settings you wish to make.
1. Open [setup-flyio-on-the-fly-v2.ipynb](../setup-flyio-on-the-fly-v2.ipynb).
2. Press the `Open in Colab` button.
3. Choose a region and run all blocks (Refer to video).
1. If you do not have the account yet, the sign-up page will be shown, please follow the instructions. The [Official document is here](https://fly.io/docs/hands-on/sign-up/).
4. Copy the Setup-URI and Use it in the Obsidian.
5. You have been synchronised. Use the Setup-URI in subsequent devices.
Steps 4 and 5 are detailed in the [Quick Setup](./quick_setup.md#1-using-setup-uris).
> [!NOTE]
> Your automatically configured configurations will be shown on the result in the Colab note like below, and **it will not be saved**. Please make a note of it somewhere.
> ```
> -- YOUR CONFIGURATION --
> URL : https://billowing-dawn-6619.fly.dev
> username: billowing-cherry-22580
> password: misty-dew-13571
> region : nrt
> ```
### B. Scripted Setup
Please refer to the document of [deploy-server.sh](../utils/readme.md#deploy-serversh).
### C. Manual Setup
| Used in the text | Meaning and where to use | Memo |
| ---------------- | --------------------------- | ------------------------------------------------------------------------ |
@@ -26,11 +55,7 @@ For simplicity, the following description assumes that the settings are as shown
| dfusiuada9suy | Password | |
| nrt | Region to make the instance | We can use any [region](https://fly.io/docs/reference/regions/) near us. |
## Steps with your computer
If you want to avoid installing anything, please skip to [Automatic setup using Colaboratory](#automatic-setup-using-colaboratory).
### 1. Install flyctl
#### 1. Install flyctl
- Mac or Linux
@@ -44,7 +69,7 @@ $ curl -L https://fly.io/install.sh | sh
$ iwr https://fly.io/install.ps1 -useb | iex
```
### 2. Sign up or Sign in to fly.io
#### 2. Sign up or Sign in to fly.io
- Sign up
@@ -58,148 +83,90 @@ $ fly auth signup
$ fly auth login
```
For more information, please refer [Sign up](https://fly.io/docs/hands-on/sign-up/) and [Sign in](https://fly.io/docs/hands-on/sign-in/).
For more information, please refer to [Sign up](https://fly.io/docs/hands-on/sign-up/) and [Sign in](https://fly.io/docs/hands-on/sign-in/).
### 3. Make configuration files
#### 3. Make a configuration file
Please be careful, `nrt` is the region where near to Japan. Please use your preferred region.
1. Make `fly.toml` from template `fly.template.toml`.
We can simply copy and rename the file. The template is on [utils/flyio/fly.template.toml](../utils/flyio/fly.template.toml)
2. Decide the instance name, initialize the App, and set credentials.
1. Make fly.toml
>[!TIP]
> - The name `billowing-dawn-6619` is randomly decided name, and it will be a part of the CouchDB URL. It should be globally unique. Therefore, it is recommended to use something random for this name.
> - Explicit naming is very good for humans. However, we do not often get the chance to actually enter this manually (have designed so). This database may contain important information for you. The needle should be hidden in the haystack.
```bash
$ fly launch --name=billowing-dawn-6619 --env="COUCHDB_USER=campanella" --copy-config=true --detach --no-deploy --region nrt --yes
$ fly secrets set COUCHDB_PASSWORD=dfusiuada9suy
```
#### 4. Deploy
```
$ flyctl launch --generate-name --detach --no-deploy --region nrt
Creating app in /home/vrtmrz/dev/fly/demo
Scanning source code
Could not find a Dockerfile, nor detect a runtime or framework from source code. Continuing with a blank app.
automatically selected personal organization: vorotamoroz
App will use 'nrt' region as primary
$ flyctl deploy
An existing fly.toml file was found
Using build strategies '[the "couchdb:latest" docker image]'. Remove [build] from fly.toml to force a rescan
Creating app in /home/vorotamoroz/dev/obsidian-livesync/utils/flyio
We're about to launch your app on Fly.io. Here's what you're getting:
Organization: vorotamoroz (fly launch defaults to the personal org)
Name: billowing-dawn-6619 (specified on the command line)
Region: Tokyo, Japan (specified on the command line)
App Machines: shared-cpu-1x, 256MB RAM (specified on the command line)
Postgres: <none> (not requested)
Redis: <none> (not requested)
Created app 'billowing-dawn-6619' in organization 'personal'
Admin URL: https://fly.io/apps/billowing-dawn-6619
Hostname: billowing-dawn-6619.fly.dev
Wrote config file fly.toml
```
`billowing-dawn-6619` is an automatically generated name. It is used as the hostname. Please note it in something.
Note: we can specify this without `--generate-name`, but does not recommend in the trial phases.
1. Make volume
```
$ flyctl volumes create --region nrt couchdata --size 2 --yes
ID: vol_g67340kxgmmvydxw
Name: couchdata
App: billowing-dawn-6619
Region: nrt
Zone: 35b7
Size GB: 2
Encrypted: true
Created at: 02 Jun 23 01:19 UTC
```
3. Edit fly.toml
Changes:
- Change exposing port from `8080` to `5984`
- Mounting the volume `couchdata` created in step 2 under `/opt/couchdb/data`
- Set `campanella` for the administrator of CouchDB
- Customise CouchDB to use persistent ini-file; which is located under the data directory.
- To use Dockerfile
```diff
# fly.toml app configuration file generated for billowing-dawn-6619 on 2023-06-02T10:18:59+09:00
#
# See https://fly.io/docs/reference/configuration/ for information about how to use this file.
#
app = "billowing-dawn-6619"
primary_region = "nrt"
[http_service]
- internal_port = 8080
+ internal_port = 5984
force_https = true
auto_stop_machines = true
auto_start_machines = true
min_machines_running = 0
+[mounts]
+ source="couchdata"
+ destination="/opt/couchdb/data"
+
+[env]
+ COUCHDB_USER = "campanella"
+ ERL_FLAGS="-couch_ini /opt/couchdb/etc/default.ini /opt/couchdb/etc/default.d/ /opt/couchdb/etc/local.d /opt/couchdb/etc/local.ini /opt/couchdb/data/persistence.ini"
+
+[build]
+ dockerfile = "./Dockerfile"
```
4. Make `Dockerfile`
Create a Dockerfile that patches the start-up script to fix ini file permissions.
```dockerfile
FROM couchdb:latest
RUN sed -i '2itouch /opt/couchdb/data/persistence.ini && chmod +w /opt/couchdb/data/persistence.ini' /docker-entrypoint.sh
```
5. Set credential
```
flyctl secrets set COUCHDB_PASSWORD=dfusiuada9suy
```
### 4. Deploy
```
$ flyctl deploy --detach --remote-only
Validating /home/vorotamoroz/dev/obsidian-livesync/utils/flyio/fly.toml
Platform: machines
✓ Configuration is valid
Your app is ready! Deploy with `flyctl deploy`
Secrets are staged for the first deployment
==> Verifying app config
Validating /home/vrtmrz/dev/fly/demo/fly.toml
Validating /home/vorotamoroz/dev/obsidian-livesync/utils/flyio/fly.toml
Platform: machines
✓ Configuration is valid
--> Verified app config
==> Building image
Remote builder fly-builder-bold-sky-4515 ready
==> Creating build context
--> Creating build context done
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
Searching for image 'couchdb:latest' remotely...
image found: img_ox20prk63084j1zq
-------------:SNIPPED:-------------
Watch your app at https://fly.io/apps/billowing-dawn-6619/monitoring
Watch your deployment at https://fly.io/apps/billowing-dawn-6619/monitoring
Provisioning ips for billowing-dawn-6619
Dedicated ipv6: 2a09:8280:1::2d:240f
Shared ipv4: 66.241.125.213
Dedicated ipv6: 2a09:8280:1::37:fde9
Shared ipv4: 66.241.124.163
Add a dedicated ipv4 with: fly ips allocate-v4
Creating a 1 GB volume named 'couchdata' for process group 'app'. Use 'fly vol extend' to increase its size
This deployment will:
* create 1 "app" machine
No machines in group app, launching a new machine
Machine e7845d1f297183 [app] has state: started
WARNING The app is not listening on the expected address and will not be reachable by fly-proxy.
You can fix this by configuring your app to listen on the following addresses:
- 0.0.0.0:5984
Found these processes inside the machine with open listening sockets:
PROCESS | ADDRESSES
-----------------*---------------------------------------
/.fly/hallpass | [fdaa:0:73b9:a7b:22e:3851:7f28:2]:22
Finished launching new machines
NOTE: The machines for [app] have services with 'auto_stop_machines = true' that will be stopped when idling
-------
Checking DNS configuration for billowing-dawn-6619.fly.dev
Visit your newly deployed app at https://billowing-dawn-6619.fly.dev/
```
Now your CouchDB has been launched. (Do not forget to delete it if no longer need).
If failed, please check by `flyctl doctor`. Failure of remote build may be resolved by `flyctl` wireguard reset` or something.
```
$ flyctl status
App
Name = billowing-dawn-6619
Owner = personal
Hostname = billowing-dawn-6619.fly.dev
Image = billowing-dawn-6619:deployment-01H1WWB3CK5Z9ZX71KHBSDGHF1
Platform = machines
Machines
PROCESS ID VERSION REGION STATE CHECKS LAST UPDATED
app e7845d1f297183 1 nrt started 2023-06-02T01:43:34Z
```
### 5. Apply CouchDB configuration
#### 5. Apply CouchDB configuration
After the initial setup, CouchDB needs some more customisations to be used from Self-hosted LiveSync. It can be configured in browsers or by HTTP-REST APIs.
@@ -273,15 +240,13 @@ iwr -UseBasicParsing -Method 'PUT' -ContentType 'application/json; charset=utf-8
Note: Each of these should also be repeated until finished in 200.
### 6. Use it from Self-hosted LiveSync
#### 6. Use it from Self-hosted LiveSync
Now the CouchDB is ready to use from Self-hosted LiveSync. We can use `https://billowing-dawn-6619.fly.dev` in URI, `campanella` in `Username` and `dfusiuada9suy` in `Password` on Self-hosted LiveSync. `Database name` could be anything you want.
`Enhance chunk size` could be up to around `100`.
Now the CouchDB is ready to use from Self-hosted LiveSync. We can use `https://billowing-dawn-6619.fly.dev` in URI, `campanella` in `Username` and `dfusiuada9suy` in `Password` on Self-hosted LiveSync. The `Database name` could be anything you want.
Please refer to the [Minimal Setup of the Quick Setup](./quick_setup.md#2-minimal-setup).
## Automatic setup using Colaboratory
## Delete the Instance
We can perform all these steps by using [this Colaboratory notebook](/deploy_couchdb_to_flyio_v2_with_swap.ipynb) without installing anything.
If you want to delete the CouchDB instance, you can do that in [fly.io Dashboard](https://fly.io/dashboard/personal)
## After testing / before creating a new instance
**Be sure to delete the instance. We can check instances on the [Dashboard](https://fly.io/dashboard/personal)**
If you have done with [B. Scripted Setup](#b-scripted-setup), we can use [delete-server.sh](../utils/readme.md#delete-serversh).

208
docs/setup_own_server.md
View File

@@ -1,138 +1,126 @@
# Setup a CouchDB server
## Table of Contents
- [Configure](#configure)
- [Run](#run)
- [Docker CLI](#docker-cli)
- [Docker Compose](#docker-compose)
- [Access from a mobile device](#access-from-a-mobile-device)
- [Testing from a mobile](#testing-from-a-mobile)
- [Setting up your domain](#setting-up-your-domain)
- [Reverse Proxies](#reverse-proxies)
- [Traefik](#traefik)
- [Setup a CouchDB server](#setup-a-couchdb-server)
- [Table of Contents](#table-of-contents)
- [1. Prepare CouchDB](#1-prepare-couchdb)
- [A. Using Docker container](#a-using-docker-container)
- [1. Prepare](#1-prepare)
- [2. Run docker container](#2-run-docker-container)
- [B. Install CouchDB directly](#b-install-couchdb-directly)
- [2. Run couchdb-init.sh for initialise](#2-run-couchdb-initsh-for-initialise)
- [3. Expose CouchDB to the Internet](#3-expose-couchdb-to-the-internet)
- [4. Client Setup](#4-client-setup)
- [1. Generate the setup URI on a desktop device or server](#1-generate-the-setup-uri-on-a-desktop-device-or-server)
- [2. Setup Self-hosted LiveSync to Obsidian](#2-setup-self-hosted-livesync-to-obsidian)
- [Manual setup information](#manual-setup-information)
- [Setting up your domain](#setting-up-your-domain)
- [Reverse Proxies](#reverse-proxies)
- [Traefik](#traefik)
---
## Configure
## 1. Prepare CouchDB
### A. Using Docker container
The easiest way to set up a CouchDB instance is using the official [docker image](https://hub.docker.com/_/couchdb).
#### 1. Prepare
```bash
Some initial configuration is required. Create a `local.ini` to use Self-hosted LiveSync as follows ([CouchDB has to be version 3.2 or higher](https://docs.couchdb.org/en/latest/config/http.html#chttpd/enable_cors), if lower `enable_cors = true` has to be under section `[httpd]` ):
# Prepare environment variables.
export hostname=localhost:5984
export username=goojdasjdas #Please change as you like.
export password=kpkdasdosakpdsa #Please change as you like
```ini
[couchdb]
single_node=true
max_document_size = 50000000
[chttpd]
require_valid_user = true
max_http_request_size = 4294967296
enable_cors = true
[chttpd_auth]
require_valid_user = true
authentication_redirect = /_utils/session.html
[httpd]
WWW-Authenticate = Basic realm="couchdb"
bind_address = 0.0.0.0
[cors]
origins = app://obsidian.md, capacitor://localhost, http://localhost
credentials = true
headers = accept, authorization, content-type, origin, referer
methods = GET,PUT,POST,HEAD,DELETE
max_age = 3600
# Prepare directories which saving data and configurations.
mkdir couchdb-data
mkdir couchdb-etc
```
## Run
#### 2. Run docker container
### Docker CLI
1. Boot Check.
```
$ docker run --name couchdb-for-ols --rm -it -e COUCHDB_USER=${username} -e COUCHDB_PASSWORD=${password} -v ${PWD}/couchdb-data:/opt/couchdb/data -v ${PWD}/couchdb-etc:/opt/couchdb/etc/local.d -p 5984:5984 couchdb
```
If your container has been exited, please check the permission of couchdb-data, and couchdb-etc.
Once CouchDB run, these directories will be owned by uid:`5984`. Please chown it for you again.
You can launch CouchDB using your `local.ini` like this:
2. Enable it in background
```
$ docker run --rm -it -e COUCHDB_USER=admin -e COUCHDB_PASSWORD=password -v /path/to/local.ini:/opt/couchdb/etc/local.ini -p 5984:5984 couchdb
$ docker run --name couchdb-for-ols -d --restart always -e COUCHDB_USER=${username} -e COUCHDB_PASSWORD=${password} -v ${PWD}/couchdb-data:/opt/couchdb/data -v ${PWD}/couchdb-etc:/opt/couchdb/etc/local.d -p 5984:5984 couchdb
```
*Remember to replace the path with the path to your local.ini*
### B. Install CouchDB directly
Please refer the [official document](https://docs.couchdb.org/en/stable/install/index.html). However, we do not have to configure it fully. Just administrator needs to be configured.
Run in detached mode:
## 2. Run couchdb-init.sh for initialise
```
$ docker run -d --restart always -e COUCHDB_USER=admin -e COUCHDB_PASSWORD=password -v /path/to/local.ini:/opt/couchdb/etc/local.ini -p 5984:5984 couchdb
```
*Remember to replace the path with the path to your local.ini*
### Docker Compose
Create a directory, place your `local.ini` within it, and create a `docker-compose.yml` alongside it. Make sure to have write permissions for `local.ini` and the about to be created `data` folder after the container start. The directory structure should look similar to this:
```
obsidian-livesync
├── docker-compose.yml
└── local.ini
$ curl -s https://raw.githubusercontent.com/vrtmrz/obsidian-livesync/main/utils/couchdb/couchdb-init.sh | bash
```
A good place to start for `docker-compose.yml`:
```yaml
version: "2.1"
services:
couchdb:
image: couchdb
container_name: obsidian-livesync
user: 1000:1000
environment:
- COUCHDB_USER=admin
- COUCHDB_PASSWORD=password
volumes:
- ./data:/opt/couchdb/data
- ./local.ini:/opt/couchdb/etc/local.ini
ports:
- 5984:5984
restart: unless-stopped
If it results like following:
```
-- Configuring CouchDB by REST APIs... -->
{"ok":true}
""
""
""
""
""
""
""
""
""
<-- Configuring CouchDB by REST APIs Done!
```
And finally launch the container
Your CouchDB has been initialised successfully. If you want this manually, please read the script.
## 3. Expose CouchDB to the Internet
- You can skip this instruction if you using only in intranet and only with desktop devices.
- For mobile devices, Obsidian requires a valid SSL certificate. Usually, it needs exposing the internet.
Whatever solutions we can use. For the simplicity, following sample uses Cloudflare Zero Trust for testing.
```
# -d will launch detached so the container runs in background
docker compose up -d
$ cloudflared tunnel --url http://localhost:5984
2024-02-14T10:35:25Z INF Thank you for trying Cloudflare Tunnel. Doing so, without a Cloudflare account, is a quick way to experiment and try it out. However, be aware that these account-less Tunnels have no uptime guarantee. If you intend to use Tunnels in production you should use a pre-created named tunnel by following: https://developers.cloudflare.com/cloudflare-one/connections/connect-apps
2024-02-14T10:35:25Z INF Requesting new quick Tunnel on trycloudflare.com...
2024-02-14T10:35:26Z INF +--------------------------------------------------------------------------------------------+
2024-02-14T10:35:26Z INF | Your quick Tunnel has been created! Visit it at (it may take some time to be reachable): |
2024-02-14T10:35:26Z INF | https://tiles-photograph-routine-groundwater.trycloudflare.com |
2024-02-14T10:35:26Z INF +--------------------------------------------------------------------------------------------+
:
:
:
```
Now `https://tiles-photograph-routine-groundwater.trycloudflare.com` is our server. Make it into background once please.
## 4. Client Setup
> [!TIP]
> Now manually configuration is not recommended for some reasons. However, if you want to do so, please use `Setup wizard`. The recommended extra configurations will be also set.
### 1. Generate the setup URI on a desktop device or server
```bash
$ export hostname=https://tiles-photograph-routine-groundwater.trycloudflare.com #Point to your vault
$ export database=obsidiannotes #Please change as you like
$ export passphrase=dfsapkdjaskdjasdas #Please change as you like
$ deno run -A https://raw.githubusercontent.com/vrtmrz/obsidian-livesync/main/utils/flyio/generate_setupuri.ts
obsidian://setuplivesync?settings=%5B%22tm2DpsOE74nJAryprZO2M93wF%2Fvg.......4b26ed33230729%22%5D
```
## Access from a mobile device
If you want to access Self-hosted LiveSync from mobile devices, you need a valid SSL certificate.
### 2. Setup Self-hosted LiveSync to Obsidian
[This video](https://youtu.be/7sa_I1832Xc?t=146) may help us.
1. Install Self-hosted LiveSync
2. Choose `Use the copied setup URI` from the command palette and paste the setup URI. (obsidian://setuplivesync?settings=.....).
3. Type `welcome` for setup-uri passphrase.
4. Answer `yes` and `Set it up...`, and finish the first dialogue with `Keep them disabled`.
5. `Reload app without save` once.
### Testing from a mobile
In the testing phase, [localhost.run](https://localhost.run/) or something like services is very useful.
---
example using localhost.run:
```
$ ssh -R 80:localhost:5984 nokey@localhost.run
Warning: Permanently added the RSA host key for IP address '35.171.254.69' to the list of known hosts.
===============================================================================
Welcome to localhost.run!
Follow your favourite reverse tunnel at [https://twitter.com/localhost_run].
**You need a SSH key to access this service.**
If you get a permission denied follow Gitlab's most excellent howto:
https://docs.gitlab.com/ee/ssh/
*Only rsa and ed25519 keys are supported*
To set up and manage custom domains go to https://admin.localhost.run/
More details on custom domains (and how to enable subdomains of your custom
domain) at https://localhost.run/docs/custom-domains
To explore using localhost.run visit the documentation site:
https://localhost.run/docs/
===============================================================================
** your connection id is xxxxxxxxxxxxxxxxxxxxxxxxxxxx, please mention it if you send me a message about an issue. **
xxxxxxxx.localhost.run tunneled with tls termination, https://xxxxxxxx.localhost.run
Connection to localhost.run closed by remote host.
Connection to localhost.run closed.
```
https://xxxxxxxx.localhost.run is the temporary server address.
## Manual setup information
### Setting up your domain

2
docs/setup_own_server_cn.md
View File

@@ -91,7 +91,7 @@ services:
最后, 创建并启动容器:
```
# -d will launch detached so the container runs in background
docker compose up -d
docker-compose up -d
```
## 创建数据库

125
docs/troubleshooting.md
View File

@@ -1,12 +1,117 @@
# Info
In this document some tips for solving issues will be given.
<!-- 2024-02-15 -->
# Tips and Troubleshooting
# Issue with syncing with mobile device
- If you are getting problem like could not sync files when doing initial sync. Try lower batch size
- Open setting
- Open Live Sync settings
- Go to sync settings
- Go down to Advanced settings.
- Lower "Batch size" and "Batch limit" and try to sync again.
- If you keep getting error keep lowering until you find the sweet spot.
- [Tips and Troubleshooting](#tips-and-troubleshooting)
- [Notable bugs and fixes](#notable-bugs-and-fixes)
- [Binary files get bigger on iOS](#binary-files-get-bigger-on-ios)
- [Some setting name has been changed](#some-setting-name-has-been-changed)
- [FAQ](#faq)
- [Why `Use an old adapter for compatibility` is somehow enabled in my vault?](#why-use-an-old-adapter-for-compatibility-is-somehow-enabled-in-my-vault)
- [ZIP (or any extensions) files were not synchronised. Why?](#zip-or-any-extensions-files-were-not-synchronised-why)
- [I hope to report the issue, but you said you needs `Report`. How to make it?](#i-hope-to-report-the-issue-but-you-said-you-needs-report-how-to-make-it)
- [Where can I check the log?](#where-can-i-check-the-log)
- [Why are the logs volatile and ephemeral?](#why-are-the-logs-volatile-and-ephemeral)
- [Some network logs are not written into the file.](#some-network-logs-are-not-written-into-the-file)
- [If a file were deleted or trimmed, the capacity of the database should be reduced, right?](#if-a-file-were-deleted-or-trimmed-the-capacity-of-the-database-should-be-reduced-right)
- [Troubleshooting](#troubleshooting)
- [On the mobile device, cannot synchronise on the local network!](#on-the-mobile-device-cannot-synchronise-on-the-local-network)
- [I think that something bad happening on the vault...](#i-think-that-something-bad-happening-on-the-vault)
- [Tips](#tips)
- [Old tips](#old-tips)
<!-- - -->
## Notable bugs and fixes
### Binary files get bigger on iOS
- Reported at: v0.20.x
- Fixed at: v0.21.2 (Fixed but not reviewed)
- Required action: larger files will not be fixed automatically, please perform `Verify and repair all files`. If our local database and storage are not matched, we will be asked to apply which one.
### Some setting name has been changed
- Fixed at: v0.22.6
| Previous name | New name |
| ---------------------------- | ---------------------------------------- |
| Open setup URI | Use the copied setup URI |
| Copy setup URI | Copy current settings as a new setup URI |
| Setup Wizard | Minimal Setup |
| Check database configuration | Check and Fix database configuration |
## FAQ
### Why `Use an old adapter for compatibility` is somehow enabled in my vault?
Because you are a compassionate and experienced user. Before v0.17.16, we used an old adapter for the local database. At that time, current default adapter has not been stable.
The new adapter has better performance and has a new feature like purging. Therefore, we should use new adapters and current default is so.
However, when switching from an old adapter to a new adapter, some converting or local database rebuilding is required, and it takes a few time. It was a long time ago now, but we once inconvenienced everyone in a hurry when we changed the format of our database.
For these reasons, this toggle is automatically on if we have upgraded from vault which using an old adapter.
When you rebuild everything or fetch from the remote again, you will be asked to switch this.
Therefore, experienced users (especially those stable enough not to have to rebuild the database) may have this toggle enabled in their Vault.
Please disable it when you have enough time.
### ZIP (or any extensions) files were not synchronised. Why?
It depends on Obsidian detects. May toggling `Detect all extensions` of `File and links` (setting of Obsidian) will help us.
### I hope to report the issue, but you said you needs `Report`. How to make it?
We can copy the report to the clipboard, by pressing the `Make report` button on the `Hatch` pane.
![Screenshot](../images/hatch.png)
### Where can I check the log?
We can launch the log pane by `Show log` on the command palette.
And if you have troubled something, please enable the `Verbose Log` on the `General Setting` pane.
However, the logs would not be kept so long and cleared when restarted. If you want to check the logs, please enable `Write logs into the file` temporarily.
![ScreenShot](../images/write_logs_into_the_file.png)
> [!IMPORTANT]
> - Writing logs into the file will impact the performance.
> - Please make sure that you have erased all your confidential information before reporting issue.
### Why are the logs volatile and ephemeral?
To avoid unexpected exposure to our confidential things.
### Some network logs are not written into the file.
Especially the CORS error will be reported as a general error to the plug-in for security reasons. So we cannot detect and log it.
### If a file were deleted or trimmed, the capacity of the database should be reduced, right?
No, even though if files were deleted, chunks were not deleted.
Self-hosted LiveSync splits the files into multiple chunks and transfers only newly created. This behaviour enables us to less traffic. And, the chunks will be shared between the files to reduce the total usage of the database.
And one more thing, we can handle the conflicts on any device even though it has happened on other devices. This means that conflicts will happen in the past, after the time we have synchronised. Hence we cannot collect and delete the unused chunks even though if we are not currently referenced.
To shrink the database size, `Rebuild everything` only reliably and effectively. But do not worry, if we have synchronised well. We have the actual and real files. Only it takes a bit of time and traffics.
<!-- Add here -->
## Troubleshooting
<!-- Add here -->
### On the mobile device, cannot synchronise on the local network!
Obsidian mobile is not able to connect to the non-secure end-point, such as starting with `http://`. Make sure your URI of CouchDB. Also not able to use a self-signed certificate.
### I think that something bad happening on the vault...
Place `redflag.md` on top of the vault, and restart Obsidian. The most simple way is to create a new note and rename it to `redflag`. Of course, we can put it without Obsidian.
If there is `redflag.md`, Self-hosted LiveSync suspends all database and storage processes.
## Tips
<!-- Add here -->
### Old tips
- Rarely, a file in the database could be corrupted. The plugin will not write to local storage when a file looks corrupted. If a local version of the file is on your device, the corruption could be fixed by editing the local file and synchronizing it. But if the file does not exist on any of your devices, then it can not be rescued. In this case, you can delete these items from the settings dialog.
- To stop the boot-up sequence (eg. for fixing problems on databases), you can put a `redflag.md` file (or directory) at the root of your vault.
Tip for iOS: a redflag directory can be created at the root of the vault using the File application.
- Also, with `redflag2.md` placed, we can automatically rebuild both the local and the remote databases during the boot-up sequence. With `redflag3.md`, we can discard only the local database and fetch from the remote again.
- Q: The database is growing, how can I shrink it down?
A: each of the docs is saved with their past 100 revisions for detecting and resolving conflicts. Picturing that one device has been offline for a while, and comes online again. The device has to compare its notes with the remotely saved ones. If there exists a historic revision in which the note used to be identical, it could be updated safely (like git fast-forward). Even if that is not in revision histories, we only have to check the differences after the revision that both devices commonly have. This is like git's conflict-resolving method. So, We have to make the database again like an enlarged git repo if you want to solve the root of the problem.
- And more technical Information is in the [Technical Information](tech_info.md)
- If you want to synchronize files without obsidian, you can use [filesystem-livesync](https://github.com/vrtmrz/filesystem-livesync).
- WebClipper is also available on Chrome Web Store:[obsidian-livesync-webclip](https://chrome.google.com/webstore/detail/obsidian-livesync-webclip/jfpaflmpckblieefkegjncjoceapakdf)
Repo is here: [obsidian-livesync-webclip](https://github.com/vrtmrz/obsidian-livesync-webclip). (Docs are a work in progress.)

BIN
images/hatch.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 13 KiB

BIN
images/quick_setup_1.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 25 KiB

After

Width:  |  Height:  |  Size: 20 KiB

BIN
images/quick_setup_5.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 36 KiB

After

Width:  |  Height:  |  Size: 5.7 KiB

BIN
images/quick_setup_7.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 10 KiB

BIN
images/write_logs_into_the_file.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 17 KiB

4
manifest.json
View File

@@ -1,10 +1,10 @@
{
"id": "obsidian-livesync",
"name": "Self-hosted LiveSync",
"version": "0.20.4",
"version": "0.22.14",
"minAppVersion": "0.9.12",
"description": "Community implementation of self-hosted livesync. Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.",
"author": "vorotamoroz",
"authorUrl": "https://github.com/vrtmrz",
"isDesktopOnly": false
}
}

2674
package-lock.json generated
View File

File diff suppressed because it is too large Load Diff

54
package.json
View File

@@ -1,6 +1,6 @@
{
"name": "obsidian-livesync",
"version": "0.20.4",
"version": "0.22.14",
"description": "Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.",
"main": "main.js",
"type": "module",
@@ -13,29 +13,29 @@
"author": "vorotamoroz",
"license": "MIT",
"devDependencies": {
"@tsconfig/svelte": "^5.0.0",
"@types/diff-match-patch": "^1.0.32",
"@types/node": "^20.2.5",
"@types/pouchdb": "^6.4.0",
"@types/pouchdb-adapter-http": "^6.1.3",
"@types/pouchdb-adapter-idb": "^6.1.4",
"@types/pouchdb-browser": "^6.1.3",
"@types/pouchdb-core": "^7.0.11",
"@types/pouchdb-mapreduce": "^6.1.7",
"@types/pouchdb-replication": "^6.4.4",
"@types/transform-pouch": "^1.0.2",
"@typescript-eslint/eslint-plugin": "^6.2.1",
"@typescript-eslint/parser": "^6.2.1",
"@tsconfig/svelte": "^5.0.2",
"@types/diff-match-patch": "^1.0.36",
"@types/node": "^20.11.28",
"@types/pouchdb": "^6.4.2",
"@types/pouchdb-adapter-http": "^6.1.6",
"@types/pouchdb-adapter-idb": "^6.1.7",
"@types/pouchdb-browser": "^6.1.5",
"@types/pouchdb-core": "^7.0.14",
"@types/pouchdb-mapreduce": "^6.1.10",
"@types/pouchdb-replication": "^6.4.7",
"@types/transform-pouch": "^1.0.6",
"@typescript-eslint/eslint-plugin": "^7.2.0",
"@typescript-eslint/parser": "^7.2.0",
"builtin-modules": "^3.3.0",
"esbuild": "0.18.17",
"esbuild-svelte": "^0.7.4",
"eslint": "^8.46.0",
"esbuild": "0.20.2",
"esbuild-svelte": "^0.8.0",
"eslint": "^8.57.0",
"eslint-config-airbnb-base": "^15.0.0",
"eslint-plugin-import": "^2.28.0",
"eslint-plugin-import": "^2.29.1",
"events": "^3.3.0",
"obsidian": "^1.4.11",
"postcss": "^8.4.27",
"postcss-load-config": "^4.0.1",
"obsidian": "^1.5.7",
"postcss": "^8.4.35",
"postcss-load-config": "^5.0.3",
"pouchdb-adapter-http": "^8.0.1",
"pouchdb-adapter-idb": "^8.0.1",
"pouchdb-adapter-indexeddb": "^8.0.1",
@@ -46,16 +46,16 @@
"pouchdb-merge": "^8.0.1",
"pouchdb-replication": "^8.0.1",
"pouchdb-utils": "^8.0.1",
"svelte": "^4.1.2",
"svelte-preprocess": "^5.0.4",
"terser": "^5.19.2",
"svelte": "^4.2.12",
"svelte-preprocess": "^5.1.3",
"terser": "^5.29.2",
"transform-pouch": "^2.0.0",
"tslib": "^2.6.1",
"typescript": "^5.1.6"
"tslib": "^2.6.2",
"typescript": "^5.4.2"
},
"dependencies": {
"diff-match-patch": "^1.0.5",
"idb": "^7.1.1",
"idb": "^8.0.0",
"minimatch": "^9.0.3",
"xxhash-wasm": "0.4.2",
"xxhash-wasm-102": "npm:xxhash-wasm@^1.0.2"

151
setup-flyio-on-the-fly-v2.ipynb Normal file
View File

@@ -0,0 +1,151 @@
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"provenance": [],
"private_outputs": true,
"authorship_tag": "ABX9TyMexQ5pErH5LBG2tENtEVWf",
"include_colab_link": true
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"language_info": {
"name": "python"
}
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/gist/vrtmrz/9402b101746e08e969b1a4f5f0deb465/setup-flyio-on-the-fly-v2.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "markdown",
"source": [
"- Initial version 7th Feb. 2024"
],
"metadata": {
"id": "AzLlAcLFRO5A"
}
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "z1x8DQpa9opC"
},
"outputs": [],
"source": [
"# Install prerequesties\n",
"!curl -L https://fly.io/install.sh | sh\n",
"!curl -fsSL https://deno.land/x/install/install.sh | sh\n",
"!apt update && apt -y install jq\n",
"import os\n",
"%env PATH=/root/.fly/bin:/root/.deno/bin/:{os.environ[\"PATH\"]}\n",
"!git clone --recursive https://github.com/vrtmrz/obsidian-livesync"
]
},
{
"cell_type": "code",
"source": [
"# Login up sign up\n",
"!flyctl auth signup"
],
"metadata": {
"id": "mGN08BaFDviy"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"Select a region and execute the block."
],
"metadata": {
"id": "BBFTFOP6vA8m"
}
},
{
"cell_type": "code",
"source": [
"# see https://fly.io/docs/reference/regions/\n",
"region = \"nrt/Tokyo, Japan\" #@param [\"ams/Amsterdam, Netherlands\",\"arn/Stockholm, Sweden\",\"atl/Atlanta, Georgia (US)\",\"bog/Bogotá, Colombia\",\"bos/Boston, Massachusetts (US)\",\"cdg/Paris, France\",\"den/Denver, Colorado (US)\",\"dfw/Dallas, Texas (US)\",\"ewr/Secaucus, NJ (US)\",\"eze/Ezeiza, Argentina\",\"gdl/Guadalajara, Mexico\",\"gig/Rio de Janeiro, Brazil\",\"gru/Sao Paulo, Brazil\",\"hkg/Hong Kong, Hong Kong\",\"iad/Ashburn, Virginia (US)\",\"jnb/Johannesburg, South Africa\",\"lax/Los Angeles, California (US)\",\"lhr/London, United Kingdom\",\"mad/Madrid, Spain\",\"mia/Miami, Florida (US)\",\"nrt/Tokyo, Japan\",\"ord/Chicago, Illinois (US)\",\"otp/Bucharest, Romania\",\"phx/Phoenix, Arizona (US)\",\"qro/Querétaro, Mexico\",\"scl/Santiago, Chile\",\"sea/Seattle, Washington (US)\",\"sin/Singapore, Singapore\",\"sjc/San Jose, California (US)\",\"syd/Sydney, Australia\",\"waw/Warsaw, Poland\",\"yul/Montreal, Canada\",\"yyz/Toronto, Canada\" ] {allow-input: true}\n",
"%env region={region.split(\"/\")[0]}\n",
"#%env appame=\n",
"#%env username=\n",
"#%env password=\n",
"#%env database=\n",
"#%env passphrase=\n",
"\n",
"# automatic setup leave it -->\n",
"%cd obsidian-livesync/utils/flyio\n",
"!./deploy-server.sh | tee deploy-result.txt\n",
"\n",
"## Show result button\n",
"from IPython.display import HTML\n",
"last_line=\"\"\n",
"with open('deploy-result.txt', 'r') as f:\n",
" last_line = f.readlines()[-1]\n",
" last_line = str.strip(last_line)\n",
"\n",
"if last_line.startswith(\"obsidian://\"):\n",
" result = HTML(f\"Copy your setup-URI with this button! -&gt; <button onclick=\\\"navigator.clipboard.writeText('{last_line}')\\\">Copy setup uri</button><br>Importing passphrase is `welcome`. <br>If you want to synchronise in live mode, please apply a preset after ensuring the imported configuration works.\")\n",
"else:\n",
" result = \"Failed to encrypt the setup URI\"\n",
"result"
],
"metadata": {
"id": "TNl0A603EF9E"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"If you see the `Copy setup URI` button, Congratulations! Your CouchDB is ready to use! Please click the button. And open this on Obsidian.\n",
"\n",
"And, you should keep the output to your secret memo.\n",
"\n"
],
"metadata": {
"id": "oeIzExnEKhFp"
}
},
{
"cell_type": "markdown",
"source": [
"\n",
"\n",
"---\n",
"\n",
"\n",
"If you want to delete this CouchDB instance, you can do it by executing next cell. \n",
"If your fly.toml has been gone, access https://fly.io/dashboard and check the existing app."
],
"metadata": {
"id": "sdQrqOjERN3K"
}
},
{
"cell_type": "code",
"source": [
"!./delete-server.sh"
],
"metadata": {
"id": "7JMSkNvVIIfg"
},
"execution_count": null,
"outputs": []
}
]
}

233
src/CmdConfigSync.ts
View File

@@ -1,20 +1,21 @@
import { writable } from 'svelte/store';
import { Notice, type PluginManifest, parseYaml, normalizePath } from "./deps";
import { Notice, type PluginManifest, parseYaml, normalizePath, type ListedFiles } from "./deps";
import type { EntryDoc, LoadedEntry, InternalFileEntry, FilePathWithPrefix, FilePath, DocumentID, AnyEntry } from "./lib/src/types";
import type { EntryDoc, LoadedEntry, InternalFileEntry, FilePathWithPrefix, FilePath, DocumentID, AnyEntry, SavingEntry } from "./lib/src/types";
import { LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE, MODE_SELECTIVE } from "./lib/src/types";
import { ICXHeader, PERIODIC_PLUGIN_SWEEP, } from "./types";
import { delay, getDocData } from "./lib/src/utils";
import { createSavingEntryFromLoadedEntry, createTextBlob, delay, fireAndForget, getDocData, isDocContentSame, sendSignal, waitForSignal } from "./lib/src/utils";
import { Logger } from "./lib/src/logger";
import { WrappedNotice } from "./lib/src/wrapper";
import { readString, crc32CKHash, decodeBinary, encodeBinary } from "./lib/src/strbin";
import { readString, decodeBinary, arrayBufferToBase64, digestHash } from "./lib/src/strbin";
import { serialized } from "./lib/src/lock";
import { LiveSyncCommands } from "./LiveSyncCommands";
import { stripAllPrefixes } from "./lib/src/path";
import { PeriodicProcessor, askYesNo, disposeMemoObject, memoIfNotExist, memoObject, retrieveMemoObject, scheduleTask } from "./utils";
import { PluginDialogModal } from "./dialogs";
import { JsonResolveModal } from "./JsonResolveModal";
import { pipeGeneratorToGenerator, processAllGeneratorTasksWithConcurrencyLimit } from './lib/src/task';
import { QueueProcessor } from './lib/src/processor';
import { pluginScanningCount } from './lib/src/stores';
import type ObsidianLiveSyncPlugin from './main';
const d = "\u200b";
const d2 = "\n";
@@ -29,7 +30,8 @@ function serialize(data: PluginDataEx): string {
ret += data.mtime + d2;
for (const file of data.files) {
ret += file.filename + d + (file.displayName ?? "") + d + (file.version ?? "") + d2;
ret += file.mtime + d + file.size + d2;
const hash = digestHash((file.data ?? []).join());
ret += file.mtime + d + file.size + d + hash + d2;
for (const data of file.data ?? []) {
ret += data + d
}
@@ -93,6 +95,7 @@ function deserialize2(str: string): PluginDataEx {
tokens.nextLine();
const mtime = Number(tokens.next());
const size = Number(tokens.next());
const hash = tokens.next();
tokens.nextLine();
const data = [] as string[];
let piece = "";
@@ -108,9 +111,11 @@ function deserialize2(str: string): PluginDataEx {
version,
mtime,
size,
data
data,
hash
}
)
tokens.nextLine();
} while (filename);
return result;
}
@@ -134,10 +139,11 @@ export const pluginIsEnumerating = writable(false);
export type PluginDataExFile = {
filename: string,
data?: string[],
data: string[],
mtime: number,
size: number,
version?: string,
hash?: string,
displayName?: string,
}
export type PluginDataExDisplay = {
@@ -161,14 +167,21 @@ export type PluginDataEx = {
mtime: number,
};
export class ConfigSync extends LiveSyncCommands {
confirmPopup: WrappedNotice = null;
constructor(plugin: ObsidianLiveSyncPlugin) {
super(plugin);
pluginScanningCount.onChanged((e) => {
const total = e.value;
pluginIsEnumerating.set(total != 0);
// if (total == 0) {
// Logger(`Processing configurations done`, LOG_LEVEL_INFO, "get-plugins");
// }
})
}
get kvDB() {
return this.plugin.kvDB;
}
ensureDirectoryEx(fullPath: string) {
return this.plugin.ensureDirectoryEx(fullPath);
}
pluginDialog: PluginDialogModal = null;
pluginDialog?: PluginDialogModal = undefined;
periodicPluginSweepProcessor = new PeriodicProcessor(this.plugin, async () => await this.scanAllConfigFiles(false));
pluginList: PluginDataExDisplay[] = [];
@@ -176,7 +189,7 @@ export class ConfigSync extends LiveSyncCommands {
if (!this.settings.usePluginSync) {
return;
}
if (this.pluginDialog != null) {
if (this.pluginDialog) {
this.pluginDialog.open();
} else {
this.pluginDialog = new PluginDialogModal(this.app, this.plugin);
@@ -187,7 +200,7 @@ export class ConfigSync extends LiveSyncCommands {
hidePluginSyncModal() {
if (this.pluginDialog != null) {
this.pluginDialog.close();
this.pluginDialog = null;
this.pluginDialog = undefined;
}
}
onunload() {
@@ -262,16 +275,28 @@ export class ConfigSync extends LiveSyncCommands {
await this.updatePluginList(showMessage);
}
async loadPluginData(path: FilePathWithPrefix): Promise<PluginDataExDisplay | false> {
const wx = await this.localDatabase.getDBEntry(path, null, false, false);
const wx = await this.localDatabase.getDBEntry(path, undefined, false, false);
if (wx) {
const data = deserialize(getDocData(wx.data), {}) as PluginDataEx;
const xFiles = [] as PluginDataExFile[];
let missingHash = false;
for (const file of data.files) {
const work = { ...file };
const tempStr = getDocData(work.data);
work.data = [crc32CKHash(tempStr)];
const work = { ...file, data: [] as string[] };
if (!file.hash) {
// debugger;
const tempStr = getDocData(work.data);
const hash = digestHash(tempStr);
file.hash = hash;
missingHash = true;
}
work.data = [file.hash];
xFiles.push(work);
}
if (missingHash) {
Logger(`Digest created for ${path} to improve checking`, LOG_LEVEL_VERBOSE);
wx.data = serialize(data);
fireAndForget(() => this.localDatabase.putDBEntry(createSavingEntryFromLoadedEntry(wx)));
}
return ({
...data,
documentPath: this.getPath(wx),
@@ -301,63 +326,71 @@ export class ConfigSync extends LiveSyncCommands {
this.plugin.saveSettingData();
}
}
pluginScanProcessor = new QueueProcessor(async (v: AnyEntry[]) => {
const plugin = v[0];
const path = plugin.path || this.getPath(plugin);
const oldEntry = (this.pluginList.find(e => e.documentPath == path));
if (oldEntry && oldEntry.mtime == plugin.mtime) return [];
try {
const pluginData = await this.loadPluginData(path);
if (pluginData) {
return [pluginData];
}
// Failed to load
return [];
} catch (ex) {
Logger(`Something happened at enumerating customization :${path}`, LOG_LEVEL_NOTICE);
Logger(ex, LOG_LEVEL_VERBOSE);
}
return [];
}, { suspended: false, batchSize: 1, concurrentLimit: 5, delay: 100, yieldThreshold: 10, maintainDelay: false }).pipeTo(
new QueueProcessor(
async (pluginDataList) => {
// Concurrency is two, therefore, we can unlock the previous awaiting.
sendSignal("plugin-next-load");
let newList = [...this.pluginList];
for (const item of pluginDataList) {
newList = newList.filter(x => x.documentPath != item.documentPath);
newList.push(item)
}
this.pluginList = newList;
pluginList.set(newList);
if (pluginDataList.length != 10) {
// If the queue is going to be empty, await subsequent for a while.
await waitForSignal("plugin-next-load", 1000);
}
return;
}
, { suspended: false, batchSize: 10, concurrentLimit: 2, delay: 100, yieldThreshold: 25, totalRemainingReactiveSource: pluginScanningCount, maintainDelay: false })).startPipeline().root.onIdle(() => {
// Logger(`All files enumerated`, LOG_LEVEL_INFO, "get-plugins");
this.createMissingConfigurationEntry();
});
async updatePluginList(showMessage: boolean, updatedDocumentPath?: FilePathWithPrefix): Promise<void> {
const logLevel = showMessage ? LOG_LEVEL_NOTICE : LOG_LEVEL_INFO;
// pluginList.set([]);
if (!this.settings.usePluginSync) {
this.pluginScanProcessor.clearQueue();
this.pluginList = [];
pluginList.set(this.pluginList)
return;
}
await Promise.resolve(); // Just to prevent warning.
scheduleTask("update-plugin-list-task", 200, async () => {
await serialized("update-plugin-list", async () => {
try {
const updatedDocumentId = updatedDocumentPath ? await this.path2id(updatedDocumentPath) : "";
const plugins = updatedDocumentPath ?
this.localDatabase.findEntries(updatedDocumentId, updatedDocumentId + "\u{10ffff}", { include_docs: true, key: updatedDocumentId, limit: 1 }) :
this.localDatabase.findEntries(ICXHeader + "", `${ICXHeader}\u{10ffff}`, { include_docs: true });
let count = 0;
pluginIsEnumerating.set(true);
for await (const v of processAllGeneratorTasksWithConcurrencyLimit(20, pipeGeneratorToGenerator(plugins, async plugin => {
const path = plugin.path || this.getPath(plugin);
if (updatedDocumentPath && updatedDocumentPath != path) {
return false;
}
const oldEntry = (this.pluginList.find(e => e.documentPath == path));
if (oldEntry && oldEntry.mtime == plugin.mtime) return false;
try {
count++;
if (count % 10 == 0) Logger(`Enumerating files... ${count}`, logLevel, "get-plugins");
Logger(`plugin-${path}`, LOG_LEVEL_VERBOSE);
return this.loadPluginData(path);
// return entries;
} catch (ex) {
//TODO
Logger(`Something happened at enumerating customization :${path}`, LOG_LEVEL_NOTICE);
console.warn(ex);
}
return false;
}))) {
if ("ok" in v) {
if (v.ok !== false) {
let newList = [...this.pluginList];
const item = v.ok;
newList = newList.filter(x => x.documentPath != item.documentPath);
newList.push(item)
if (updatedDocumentPath != "") newList = newList.filter(e => e.documentPath != updatedDocumentPath);
this.pluginList = newList;
pluginList.set(newList);
}
}
}
Logger(`All files enumerated`, logLevel, "get-plugins");
this.createMissingConfigurationEntry();
} finally {
pluginIsEnumerating.set(false);
}
});
});
try {
const updatedDocumentId = updatedDocumentPath ? await this.path2id(updatedDocumentPath) : "";
const plugins = updatedDocumentPath ?
this.localDatabase.findEntries(updatedDocumentId, updatedDocumentId + "\u{10ffff}", { include_docs: true, key: updatedDocumentId, limit: 1 }) :
this.localDatabase.findEntries(ICXHeader + "", `${ICXHeader}\u{10ffff}`, { include_docs: true });
for await (const v of plugins) {
const path = v.path || this.getPath(v);
if (updatedDocumentPath && updatedDocumentPath != path) continue;
this.pluginScanProcessor.enqueue(v);
}
} finally {
pluginIsEnumerating.set(false);
}
pluginIsEnumerating.set(false);
// return entries;
}
async compareUsingDisplayData(dataA: PluginDataExDisplay, dataB: PluginDataExDisplay) {
@@ -412,12 +445,12 @@ export class ConfigSync extends LiveSyncCommands {
try {
// console.dir(f);
const path = `${baseDir}/${f.filename}`;
await this.ensureDirectoryEx(path);
await this.vaultAccess.ensureDirectory(path);
if (!content) {
const dt = decodeBinary(f.data);
await this.app.vault.adapter.writeBinary(path, dt);
await this.vaultAccess.adapterWrite(path, dt);
} else {
await this.app.vault.adapter.write(path, content);
await this.vaultAccess.adapterWrite(path, content);
}
Logger(`Applying ${f.filename} of ${data.displayName || data.name}.. Done`);
@@ -483,9 +516,9 @@ export class ConfigSync extends LiveSyncCommands {
if (this.plugin.settings.usePluginSync && this.plugin.settings.notifyPluginOrSettingUpdated) {
if (!this.pluginDialog || (this.pluginDialog && !this.pluginDialog.isOpened())) {
const fragment = createFragment((doc) => {
doc.createEl("span", null, (a) => {
doc.createEl("span", undefined, (a) => {
a.appendText(`Some configuration has been arrived, Press `);
a.appendChild(a.createEl("a", null, (anchor) => {
a.appendChild(a.createEl("a", undefined, (anchor) => {
anchor.text = "HERE";
anchor.addEventListener("click", () => {
this.showPluginSyncModal();
@@ -537,16 +570,16 @@ export class ConfigSync extends LiveSyncCommands {
recentProcessedInternalFiles = [] as string[];
async makeEntryFromFile(path: FilePath): Promise<false | PluginDataExFile> {
const stat = await this.app.vault.adapter.stat(path);
const stat = await this.vaultAccess.adapterStat(path);
let version: string | undefined;
let displayName: string | undefined;
if (!stat) {
return false;
}
const contentBin = await this.app.vault.adapter.readBinary(path);
const contentBin = await this.vaultAccess.adapterReadBinary(path);
let content: string[];
try {
content = await encodeBinary(contentBin, this.settings.useV1);
content = await arrayBufferToBase64(contentBin);
if (path.toLowerCase().endsWith("/manifest.json")) {
const v = readString(new Uint8Array(contentBin));
try {
@@ -649,10 +682,10 @@ export class ConfigSync extends LiveSyncCommands {
return
}
const content = serialize(dt);
const content = createTextBlob(serialize(dt));
try {
const old = await this.localDatabase.getDBEntryMeta(prefixedFileName, null, false);
let saveData: LoadedEntry;
const old = await this.localDatabase.getDBEntryMeta(prefixedFileName, undefined, false);
let saveData: SavingEntry;
if (old === false) {
saveData = {
_id: id,
@@ -661,22 +694,34 @@ export class ConfigSync extends LiveSyncCommands {
mtime,
ctime: mtime,
datatype: "newnote",
size: content.length,
size: content.size,
children: [],
deleted: false,
type: "newnote",
};
} else {
if (old.mtime == mtime) {
// Logger(`STORAGE --> DB:${file.path}: (hidden) Not changed`, LOG_LEVEL_VERBOSE);
// Logger(`STORAGE --> DB:${prefixedFileName}: (config) Skipped (Same time)`, LOG_LEVEL_VERBOSE);
return true;
}
const oldC = await this.localDatabase.getDBEntryFromMeta(old, {}, false, false);
if (oldC) {
const d = await deserialize(getDocData(oldC.data), {}) as PluginDataEx;
const diffs = (d.files.map(previous => ({ prev: previous, curr: dt.files.find(e => e.filename == previous.filename) })).map(async e => {
try { return await isDocContentSame(e.curr?.data ?? [], e.prev.data) } catch (_) { return false }
}))
const isSame = (await Promise.all(diffs)).every(e => e == true);
if (isSame) {
Logger(`STORAGE --> DB:${prefixedFileName}: (config) Skipped (Same content)`, LOG_LEVEL_VERBOSE);
return true;
}
}
saveData =
{
...old,
data: content,
mtime,
size: content.length,
size: content.size,
datatype: "newnote",
children: [],
deleted: false,
@@ -698,7 +743,7 @@ export class ConfigSync extends LiveSyncCommands {
async watchVaultRawEventsAsync(path: FilePath) {
if (!this.settings.usePluginSync) return false;
if (!this.isTargetPath(path)) return false;
const stat = await this.app.vault.adapter.stat(path);
const stat = await this.vaultAccess.adapterStat(path);
// Make sure that target is a file.
if (stat && stat.type != "file")
return false;
@@ -736,7 +781,11 @@ export class ConfigSync extends LiveSyncCommands {
const filesOnDB = ((await this.localDatabase.allDocsRaw({ startkey: ICXHeader + "", endkey: `${ICXHeader}\u{10ffff}`, include_docs: true })).rows.map(e => e.doc) as InternalFileEntry[]).filter(e => !e.deleted);
let deleteCandidate = filesOnDB.map(e => this.getPath(e)).filter(e => e.startsWith(`${ICXHeader}${term}/`));
for (const vp of virtualPathsOfLocalFiles) {
const p = files.find(e => e.key == vp).file;
const p = files.find(e => e.key == vp)?.file;
if (!p) {
Logger(`scanAllConfigFiles - File not found: ${vp}`, LOG_LEVEL_VERBOSE);
continue;
}
await this.storeCustomizationFiles(p);
deleteCandidate = deleteCandidate.filter(e => e != vp);
}
@@ -751,10 +800,11 @@ export class ConfigSync extends LiveSyncCommands {
const mtime = new Date().getTime();
await serialized("file-x-" + prefixedFileName, async () => {
try {
const old = await this.localDatabase.getDBEntryMeta(prefixedFileName, null, false) as InternalFileEntry | false;
const old = await this.localDatabase.getDBEntryMeta(prefixedFileName, undefined, false) as InternalFileEntry | false;
let saveData: InternalFileEntry;
if (old === false) {
Logger(`STORAGE -x> DB:${prefixedFileName}: (config) already deleted (Not found on database)`);
return;
} else {
if (old.deleted) {
Logger(`STORAGE -x> DB:${prefixedFileName}: (config) already deleted`);
@@ -793,7 +843,14 @@ export class ConfigSync extends LiveSyncCommands {
lastDepth: number
) {
if (lastDepth == -1) return [];
const w = await this.app.vault.adapter.list(path);
let w: ListedFiles;
try {
w = await this.app.vault.adapter.list(path);
} catch (ex) {
Logger(`Could not traverse(ConfigSync):${path}`, LOG_LEVEL_INFO);
Logger(ex, LOG_LEVEL_VERBOSE);
return [];
}
let files = [
...w.files
];

319
src/CmdHiddenFileSync.ts
View File

@@ -1,26 +1,23 @@
import { normalizePath, type PluginManifest } from "./deps";
import { type EntryDoc, type LoadedEntry, type InternalFileEntry, type FilePathWithPrefix, type FilePath, LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE, MODE_SELECTIVE, MODE_PAUSED } from "./lib/src/types";
import { normalizePath, type PluginManifest, type ListedFiles } from "./deps";
import { type EntryDoc, type LoadedEntry, type InternalFileEntry, type FilePathWithPrefix, type FilePath, LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE, MODE_SELECTIVE, MODE_PAUSED, type SavingEntry, type DocumentID } from "./lib/src/types";
import { type InternalFileInfo, ICHeader, ICHeaderEnd } from "./types";
import { Parallels, delay, isDocContentSame } from "./lib/src/utils";
import { readAsBlob, isDocContentSame, sendSignal, readContent, createBlob } from "./lib/src/utils";
import { Logger } from "./lib/src/logger";
import { PouchDB } from "./lib/src/pouchdb-browser.js";
import { scheduleTask, isInternalMetadata, PeriodicProcessor } from "./utils";
import { WrappedNotice } from "./lib/src/wrapper";
import { decodeBinary, encodeBinary } from "./lib/src/strbin";
import { isInternalMetadata, PeriodicProcessor } from "./utils";
import { serialized } from "./lib/src/lock";
import { JsonResolveModal } from "./JsonResolveModal";
import { LiveSyncCommands } from "./LiveSyncCommands";
import { addPrefix, stripAllPrefixes } from "./lib/src/path";
import { KeyedQueueProcessor, QueueProcessor } from "./lib/src/processor";
import { hiddenFilesEventCount, hiddenFilesProcessingCount } from "./lib/src/stores";
export class HiddenFileSync extends LiveSyncCommands {
periodicInternalFileScanProcessor: PeriodicProcessor = new PeriodicProcessor(this.plugin, async () => this.settings.syncInternalFiles && this.localDatabase.isReady && await this.syncInternalFilesAndDatabase("push", false));
confirmPopup: WrappedNotice = null;
get kvDB() {
return this.plugin.kvDB;
}
ensureDirectoryEx(fullPath: string) {
return this.plugin.ensureDirectoryEx(fullPath);
}
getConflictedDoc(path: FilePathWithPrefix, rev: string) {
return this.plugin.getConflictedDoc(path, rev);
}
@@ -68,29 +65,24 @@ export class HiddenFileSync extends LiveSyncCommands {
realizeSettingSyncMode(): Promise<void> {
this.periodicInternalFileScanProcessor?.disable();
if (this.plugin.suspended)
return;
return Promise.resolve();
if (!this.plugin.isReady)
return;
return Promise.resolve();
this.periodicInternalFileScanProcessor.enable(this.settings.syncInternalFiles && this.settings.syncInternalFilesInterval ? (this.settings.syncInternalFilesInterval * 1000) : 0);
return;
return Promise.resolve();
}
procInternalFiles: string[] = [];
async execInternalFile() {
await serialized("execInternal", async () => {
const w = [...this.procInternalFiles];
this.procInternalFiles = [];
Logger(`Applying hidden ${w.length} files change...`);
await this.syncInternalFilesAndDatabase("pull", false, false, w);
Logger(`Applying hidden ${w.length} files changed`);
});
}
procInternalFile(filename: string) {
this.procInternalFiles.push(filename);
scheduleTask("procInternal", 500, async () => {
await this.execInternalFile();
});
this.internalFileProcessor.enqueueWithKey(filename, filename);
}
internalFileProcessor = new KeyedQueueProcessor<string, any>(
async (filenames) => {
Logger(`START :Applying hidden ${filenames.length} files change`, LOG_LEVEL_VERBOSE);
await this.syncInternalFilesAndDatabase("pull", false, false, filenames);
Logger(`DONE :Applying hidden ${filenames.length} files change`, LOG_LEVEL_VERBOSE);
return;
}, { batchSize: 100, concurrentLimit: 1, delay: 10, yieldThreshold: 10, suspended: false, totalRemainingReactiveSource: hiddenFilesEventCount }
);
recentProcessedInternalFiles = [] as string[];
async watchVaultRawEventsAsync(path: FilePath) {
@@ -103,13 +95,15 @@ export class HiddenFileSync extends LiveSyncCommands {
Logger(`Hidden file skipped: ${path} is synchronized in customization sync.`, LOG_LEVEL_VERBOSE);
return;
}
const stat = await this.app.vault.adapter.stat(path);
const stat = await this.vaultAccess.adapterStat(path);
// sometimes folder is coming.
if (stat && stat.type != "file")
if (stat != null && stat.type != "file") {
return;
const storageMTime = ~~((stat && stat.mtime || 0) / 1000);
}
const mtime = stat == null ? 0 : stat?.mtime ?? 0;
const storageMTime = ~~((mtime) / 1000);
const key = `${path}-${storageMTime}`;
if (this.recentProcessedInternalFiles.contains(key)) {
if (mtime != 0 && this.recentProcessedInternalFiles.contains(key)) {
//If recently processed, it may caused by self.
return;
}
@@ -129,7 +123,7 @@ export class HiddenFileSync extends LiveSyncCommands {
if (storageMTime == 0) {
await this.deleteInternalFileOnDatabase(path);
} else {
await this.storeInternalFileToDatabase({ path: path, ...stat });
await this.storeInternalFileToDatabase({ path: path, mtime, ctime: stat?.ctime ?? mtime, size: stat?.size ?? 0 });
}
}
@@ -137,25 +131,56 @@ export class HiddenFileSync extends LiveSyncCommands {
async resolveConflictOnInternalFiles() {
// Scan all conflicted internal files
const conflicted = this.localDatabase.findEntries(ICHeader, ICHeaderEnd, { conflicts: true });
for await (const doc of conflicted) {
if (!("_conflicts" in doc))
continue;
if (isInternalMetadata(doc._id)) {
await this.resolveConflictOnInternalFile(doc.path);
this.conflictResolutionProcessor.suspend();
try {
for await (const doc of conflicted) {
if (!("_conflicts" in doc))
continue;
if (isInternalMetadata(doc._id)) {
this.conflictResolutionProcessor.enqueue(doc.path);
}
}
} catch (ex) {
Logger("something went wrong on resolving all conflicted internal files");
Logger(ex, LOG_LEVEL_VERBOSE);
}
await this.conflictResolutionProcessor.startPipeline().waitForPipeline();
}
async resolveConflictOnInternalFile(path: FilePathWithPrefix): Promise<boolean> {
async resolveByNewerEntry(id: DocumentID, path: FilePathWithPrefix, currentDoc: EntryDoc, currentRev: string, conflictedRev: string) {
const conflictedDoc = await this.localDatabase.getRaw(id, { rev: conflictedRev });
// determine which revision should been deleted.
// simply check modified time
const mtimeCurrent = ("mtime" in currentDoc && currentDoc.mtime) || 0;
const mtimeConflicted = ("mtime" in conflictedDoc && conflictedDoc.mtime) || 0;
// Logger(`Revisions:${new Date(mtimeA).toLocaleString} and ${new Date(mtimeB).toLocaleString}`);
// console.log(`mtime:${mtimeA} - ${mtimeB}`);
const delRev = mtimeCurrent < mtimeConflicted ? currentRev : conflictedRev;
// delete older one.
await this.localDatabase.removeRevision(id, delRev);
Logger(`Older one has been deleted:${path}`);
const cc = await this.localDatabase.getRaw(id, { conflicts: true });
if (cc._conflicts?.length === 0) {
await this.extractInternalFileFromDatabase(stripAllPrefixes(path))
} else {
this.conflictResolutionProcessor.enqueue(path);
}
// check the file again
}
conflictResolutionProcessor = new QueueProcessor(async (paths: FilePathWithPrefix[]) => {
const path = paths[0];
sendSignal(`cancel-internal-conflict:${path}`);
try {
// Retrieve data
const id = await this.path2id(path, ICHeader);
const doc = await this.localDatabase.getRaw(id, { conflicts: true });
// If there is no conflict, return with false.
if (!("_conflicts" in doc))
return false;
// if (!("_conflicts" in doc)){
// return [];
// }
if (doc._conflicts === undefined) return [];
if (doc._conflicts.length == 0)
return false;
return [];
Logger(`Hidden file conflicted:${path}`);
const conflicts = doc._conflicts.sort((a, b) => Number(a.split("-")[0]) - Number(b.split("-")[0]));
const revA = doc._rev;
@@ -166,54 +191,59 @@ export class HiddenFileSync extends LiveSyncCommands {
const conflictedRevNo = Number(conflictedRev.split("-")[0]);
//Search
const revFrom = (await this.localDatabase.getRaw<EntryDoc>(id, { revs_info: true }));
const commonBase = revFrom._revs_info.filter(e => e.status == "available" && Number(e.rev.split("-")[0]) < conflictedRevNo).first()?.rev ?? "";
const commonBase = revFrom._revs_info?.filter(e => e.status == "available" && Number(e.rev.split("-")[0]) < conflictedRevNo).first()?.rev ?? "";
const result = await this.plugin.mergeObject(path, commonBase, doc._rev, conflictedRev);
if (result) {
Logger(`Object merge:${path}`, LOG_LEVEL_INFO);
const filename = stripAllPrefixes(path);
const isExists = await this.app.vault.adapter.exists(filename);
const isExists = await this.plugin.vaultAccess.adapterExists(filename);
if (!isExists) {
await this.ensureDirectoryEx(filename);
await this.vaultAccess.ensureDirectory(filename);
}
await this.plugin.vaultAccess.adapterWrite(filename, result);
const stat = await this.vaultAccess.adapterStat(filename);
if (!stat) {
throw new Error(`conflictResolutionProcessor: Failed to stat file ${filename}`);
}
await this.app.vault.adapter.write(filename, result);
const stat = await this.app.vault.adapter.stat(filename);
await this.storeInternalFileToDatabase({ path: filename, ...stat });
await this.extractInternalFileFromDatabase(filename);
await this.localDatabase.removeRaw(id, revB);
return this.resolveConflictOnInternalFile(path);
await this.localDatabase.removeRevision(id, revB);
this.conflictResolutionProcessor.enqueue(path);
return [];
} else {
Logger(`Object merge is not applicable.`, LOG_LEVEL_VERBOSE);
}
const docAMerge = await this.localDatabase.getDBEntry(path, { rev: revA });
const docBMerge = await this.localDatabase.getDBEntry(path, { rev: revB });
if (docAMerge != false && docBMerge != false) {
if (await this.showJSONMergeDialogAndMerge(docAMerge, docBMerge)) {
await delay(200);
// Again for other conflicted revisions.
return this.resolveConflictOnInternalFile(path);
}
return false;
}
return [{ path, revA, revB, id, doc }];
}
const revBDoc = await this.localDatabase.getRaw(id, { rev: revB });
// determine which revision should been deleted.
// simply check modified time
const mtimeA = ("mtime" in doc && doc.mtime) || 0;
const mtimeB = ("mtime" in revBDoc && revBDoc.mtime) || 0;
// Logger(`Revisions:${new Date(mtimeA).toLocaleString} and ${new Date(mtimeB).toLocaleString}`);
// console.log(`mtime:${mtimeA} - ${mtimeB}`);
const delRev = mtimeA < mtimeB ? revA : revB;
// delete older one.
await this.localDatabase.removeRaw(id, delRev);
Logger(`Older one has been deleted:${path}`);
// check the file again
return this.resolveConflictOnInternalFile(path);
// When not JSON file, resolve conflicts by choosing a newer one.
await this.resolveByNewerEntry(id, path, doc, revA, revB);
return [];
} catch (ex) {
Logger(`Failed to resolve conflict (Hidden): ${path}`);
Logger(ex, LOG_LEVEL_VERBOSE);
return false;
return [];
}
}, {
suspended: false, batchSize: 1, concurrentLimit: 5, delay: 10, keepResultUntilDownstreamConnected: true, yieldThreshold: 10,
pipeTo: new QueueProcessor(async (results) => {
const { id, doc, path, revA, revB } = results[0];
const docAMerge = await this.localDatabase.getDBEntry(path, { rev: revA });
const docBMerge = await this.localDatabase.getDBEntry(path, { rev: revB });
if (docAMerge != false && docBMerge != false) {
if (await this.showJSONMergeDialogAndMerge(docAMerge, docBMerge)) {
// Again for other conflicted revisions.
this.conflictResolutionProcessor.enqueue(path);
}
return;
} else {
// If either revision could not read, force resolving by the newer one.
await this.resolveByNewerEntry(id, path, doc, revA, revB);
}
}, { suspended: false, batchSize: 1, concurrentLimit: 1, delay: 10, keepResultUntilDownstreamConnected: false, yieldThreshold: 10 })
})
queueConflictCheck(path: FilePathWithPrefix) {
this.conflictResolutionProcessor.enqueue(path);
}
//TODO: Tidy up. Even though it is experimental feature, So dirty...
@@ -278,28 +308,38 @@ export class HiddenFileSync extends LiveSyncCommands {
acc[stripAllPrefixes(this.getPath(cur))] = cur;
return acc;
}, {} as { [key: string]: InternalFileEntry; });
const para = Parallels();
for (const filename of allFileNames) {
await new QueueProcessor(async (filenames: FilePath[]) => {
const filename = filenames[0];
processed++;
if (processed % 100 == 0) {
Logger(`Hidden file: ${processed}/${fileCount}`, logLevel, "sync_internal");
}
if (!filename) continue;
if (!filename) return [];
if (ignorePatterns.some(e => filename.match(e)))
continue;
return [];
if (await this.plugin.isIgnoredByIgnoreFiles(filename)) {
continue;
return [];
}
const fileOnStorage = filename in filesMap ? filesMap[filename] : undefined;
const fileOnDatabase = filename in filesOnDBMap ? filesOnDBMap[filename] : undefined;
const cache = filename in caches ? caches[filename] : { storageMtime: 0, docMtime: 0 };
await para.wait(5);
const proc = (async (xFileOnStorage: InternalFileInfo, xFileOnDatabase: InternalFileEntry) => {
return [{
filename,
fileOnStorage,
fileOnDatabase,
}]
}, { suspended: true, batchSize: 1, concurrentLimit: 10, delay: 0, totalRemainingReactiveSource: hiddenFilesProcessingCount })
.pipeTo(new QueueProcessor(async (params) => {
const
{
filename,
fileOnStorage: xFileOnStorage,
fileOnDatabase: xFileOnDatabase
} = params[0];
if (xFileOnStorage && xFileOnDatabase) {
const cache = filename in caches ? caches[filename] : { storageMtime: 0, docMtime: 0 };
// Both => Synchronize
if ((direction != "pullForce" && direction != "pushForce") && xFileOnDatabase.mtime == cache.docMtime && xFileOnStorage.mtime == cache.storageMtime) {
return;
@@ -335,16 +375,21 @@ export class HiddenFileSync extends LiveSyncCommands {
}
}
} else if (xFileOnStorage && !xFileOnDatabase) {
await this.storeInternalFileToDatabase(xFileOnStorage);
if (direction == "push" || direction == "pushForce" || direction == "safe") {
await this.storeInternalFileToDatabase(xFileOnStorage);
} else {
await this.extractInternalFileFromDatabase(xFileOnStorage.path);
}
} else {
throw new Error("Invalid state on hidden file sync");
// Something corrupted?
}
return;
}, { suspended: true, batchSize: 1, concurrentLimit: 5, delay: 0 }))
.root
.enqueueAll(allFileNames)
.startPipeline().waitForPipeline();
});
para.add(proc(fileOnStorage, fileOnDatabase))
}
await para.all();
await this.kvDB.set("diff-caches-internal", caches);
// When files has been retrieved from the database. they must be reloaded.
@@ -360,7 +405,7 @@ export class HiddenFileSync extends LiveSyncCommands {
const enabledPlugins = this.app.plugins.enabledPlugins as Set<string>;
const enabledPluginManifests = manifests.filter(e => enabledPlugins.has(e.id));
for (const manifest of enabledPluginManifests) {
if (manifest.dir in updatedFolders) {
if (manifest.dir && manifest.dir in updatedFolders) {
// If notified about plug-ins, reloading Obsidian may not be necessary.
updatedCount -= updatedFolders[manifest.dir];
const updatePluginId = manifest.id;
@@ -408,20 +453,12 @@ export class HiddenFileSync extends LiveSyncCommands {
const id = await this.path2id(file.path, ICHeader);
const prefixedFileName = addPrefix(file.path, ICHeader);
const contentBin = await this.app.vault.adapter.readBinary(file.path);
let content: string[];
try {
content = await encodeBinary(contentBin, this.settings.useV1);
} catch (ex) {
Logger(`The file ${file.path} could not be encoded`);
Logger(ex, LOG_LEVEL_VERBOSE);
return false;
}
const content = createBlob(await this.plugin.vaultAccess.adapterReadAuto(file.path));
const mtime = file.mtime;
return await serialized("file-" + prefixedFileName, async () => {
try {
const old = await this.localDatabase.getDBEntry(prefixedFileName, null, false, false);
let saveData: LoadedEntry;
const old = await this.localDatabase.getDBEntry(prefixedFileName, undefined, false, false);
let saveData: SavingEntry;
if (old === false) {
saveData = {
_id: id,
@@ -436,7 +473,7 @@ export class HiddenFileSync extends LiveSyncCommands {
type: "newnote",
};
} else {
if (isDocContentSame(old.data, content) && !forceWrite) {
if (await isDocContentSame(readAsBlob(old), content) && !forceWrite) {
// Logger(`STORAGE --> DB:${file.path}: (hidden) Not changed`, LOG_LEVEL_VERBOSE);
return;
}
@@ -446,13 +483,13 @@ export class HiddenFileSync extends LiveSyncCommands {
data: content,
mtime,
size: file.size,
datatype: "newnote",
datatype: old.datatype,
children: [],
deleted: false,
type: "newnote",
type: old.datatype,
};
}
const ret = await this.localDatabase.putDBEntry(saveData, true);
const ret = await this.localDatabase.putDBEntry(saveData);
Logger(`STORAGE --> DB:${file.path}: (hidden) Done`);
return ret;
} catch (ex) {
@@ -472,7 +509,7 @@ export class HiddenFileSync extends LiveSyncCommands {
}
await serialized("file-" + prefixedFileName, async () => {
try {
const old = await this.localDatabase.getDBEntryMeta(prefixedFileName, null, true) as InternalFileEntry | false;
const old = await this.localDatabase.getDBEntryMeta(prefixedFileName, undefined, true) as InternalFileEntry | false;
let saveData: InternalFileEntry;
if (old === false) {
saveData = {
@@ -486,6 +523,14 @@ export class HiddenFileSync extends LiveSyncCommands {
type: "newnote",
};
} else {
// Remove all conflicted before deleting.
const conflicts = await this.localDatabase.getRaw(old._id, { conflicts: true });
if (conflicts._conflicts !== undefined) {
for (const conflictRev of conflicts._conflicts) {
await this.localDatabase.removeRevision(old._id, conflictRev);
Logger(`STORAGE -x> DB:${filename}: (hidden) conflict removed ${old._rev} => ${conflictRev}`, LOG_LEVEL_VERBOSE);
}
}
if (old.deleted) {
Logger(`STORAGE -x> DB:${filename}: (hidden) already deleted`);
return;
@@ -511,7 +556,7 @@ export class HiddenFileSync extends LiveSyncCommands {
}
async extractInternalFileFromDatabase(filename: FilePath, force = false) {
const isExists = await this.app.vault.adapter.exists(filename);
const isExists = await this.plugin.vaultAccess.adapterExists(filename);
const prefixedFileName = addPrefix(filename, ICHeader);
if (await this.plugin.isIgnoredByIgnoreFiles(filename)) {
return;
@@ -519,8 +564,7 @@ export class HiddenFileSync extends LiveSyncCommands {
return await serialized("file-" + prefixedFileName, async () => {
try {
// Check conflicted status
//TODO option
const fileOnDB = await this.localDatabase.getDBEntry(prefixedFileName, { conflicts: true }, false, true);
const fileOnDB = await this.localDatabase.getDBEntry(prefixedFileName, { conflicts: true }, false, true, true);
if (fileOnDB === false)
throw new Error(`File not found on database.:${filename}`);
// Prevent overwrite for Prevent overwriting while some conflicted revision exists.
@@ -528,13 +572,13 @@ export class HiddenFileSync extends LiveSyncCommands {
Logger(`Hidden file ${filename} has conflicted revisions, to keep in safe, writing to storage has been prevented`, LOG_LEVEL_INFO);
return;
}
const deleted = "deleted" in fileOnDB ? fileOnDB.deleted : false;
const deleted = fileOnDB.deleted || fileOnDB._deleted || false;
if (deleted) {
if (!isExists) {
Logger(`STORAGE <x- DB:${filename}: deleted (hidden) Deleted on DB, but the file is already not found on storage.`);
Logger(`STORAGE <x- DB:${filename}: deleted (hidden) Deleted on DB, but the file is already not found on storage.`);
} else {
Logger(`STORAGE <x- DB:${filename}: deleted (hidden).`);
await this.app.vault.adapter.remove(filename);
await this.plugin.vaultAccess.adapterRemove(filename);
try {
//@ts-ignore internalAPI
await this.app.vault.adapter.reconcileInternalFile(filename);
@@ -546,8 +590,8 @@ export class HiddenFileSync extends LiveSyncCommands {
return true;
}
if (!isExists) {
await this.ensureDirectoryEx(filename);
await this.app.vault.adapter.writeBinary(filename, decodeBinary(fileOnDB.data), { mtime: fileOnDB.mtime, ctime: fileOnDB.ctime });
await this.vaultAccess.ensureDirectory(filename);
await this.plugin.vaultAccess.adapterWrite(filename, readContent(fileOnDB), { mtime: fileOnDB.mtime, ctime: fileOnDB.ctime });
try {
//@ts-ignore internalAPI
await this.app.vault.adapter.reconcileInternalFile(filename);
@@ -558,13 +602,13 @@ export class HiddenFileSync extends LiveSyncCommands {
Logger(`STORAGE <-- DB:${filename}: written (hidden,new${force ? ", force" : ""})`);
return true;
} else {
const contentBin = await this.app.vault.adapter.readBinary(filename);
const content = await encodeBinary(contentBin, this.settings.useV1);
if (isDocContentSame(content, fileOnDB.data) && !force) {
const content = await this.plugin.vaultAccess.adapterReadAuto(filename);
const docContent = readContent(fileOnDB);
if (await isDocContentSame(content, docContent) && !force) {
// Logger(`STORAGE <-- DB:${filename}: skipped (hidden) Not changed`, LOG_LEVEL_VERBOSE);
return true;
}
await this.app.vault.adapter.writeBinary(filename, decodeBinary(fileOnDB.data), { mtime: fileOnDB.mtime, ctime: fileOnDB.ctime });
await this.plugin.vaultAccess.adapterWrite(filename, docContent, { mtime: fileOnDB.mtime, ctime: fileOnDB.ctime });
try {
//@ts-ignore internalAPI
await this.app.vault.adapter.reconcileInternalFile(filename);
@@ -587,7 +631,7 @@ export class HiddenFileSync extends LiveSyncCommands {
showJSONMergeDialogAndMerge(docA: LoadedEntry, docB: LoadedEntry): Promise<boolean> {
return serialized("conflict:merge-data", () => new Promise((res) => {
return new Promise((res) => {
Logger("Opening data-merging dialog", LOG_LEVEL_VERBOSE);
const docs = [docA, docB];
const path = stripAllPrefixes(docA.path);
@@ -613,13 +657,17 @@ export class HiddenFileSync extends LiveSyncCommands {
}
}
if (!keep && result) {
const isExists = await this.app.vault.adapter.exists(filename);
const isExists = await this.plugin.vaultAccess.adapterExists(filename);
if (!isExists) {
await this.ensureDirectoryEx(filename);
await this.vaultAccess.ensureDirectory(filename);
}
await this.app.vault.adapter.write(filename, result);
const stat = await this.app.vault.adapter.stat(filename);
await this.storeInternalFileToDatabase({ path: filename, ...stat }, true);
await this.plugin.vaultAccess.adapterWrite(filename, result);
const stat = await this.plugin.vaultAccess.adapterStat(filename);
if (!stat) {
throw new Error("Stat failed");
}
const mtime = stat?.mtime ?? 0;
await this.storeInternalFileToDatabase({ path: filename, mtime, ctime: stat?.ctime ?? mtime, size: stat?.size ?? 0 }, true);
try {
//@ts-ignore internalAPI
await this.app.vault.adapter.reconcileInternalFile(filename);
@@ -641,7 +689,7 @@ export class HiddenFileSync extends LiveSyncCommands {
}
});
modal.open();
}));
});
}
async scanInternalFiles(): Promise<InternalFileInfo[]> {
@@ -653,11 +701,11 @@ export class HiddenFileSync extends LiveSyncCommands {
const root = this.app.vault.getRoot();
const findRoot = root.path;
const filenames = (await this.getFiles(findRoot, [], null, ignoreFilter)).filter(e => e.startsWith(".")).filter(e => !e.startsWith(".trash"));
const filenames = (await this.getFiles(findRoot, [], undefined, ignoreFilter)).filter(e => e.startsWith(".")).filter(e => !e.startsWith(".trash"));
const files = filenames.filter(path => synchronisedInConfigSync.every(filterFile => !path.toLowerCase().startsWith(filterFile))).map(async (e) => {
return {
path: e as FilePath,
stat: await this.app.vault.adapter.stat(e)
stat: await this.plugin.vaultAccess.adapterStat(e)
};
});
const result: InternalFileInfo[] = [];
@@ -666,9 +714,12 @@ export class HiddenFileSync extends LiveSyncCommands {
if (await this.plugin.isIgnoredByIgnoreFiles(w.path)) {
continue
}
const mtime = w.stat?.mtime ?? 0
const ctime = w.stat?.ctime ?? mtime;
const size = w.stat?.size ?? 0;
result.push({
...w,
...w.stat
mtime, ctime, size
});
}
return result;
@@ -679,11 +730,17 @@ export class HiddenFileSync extends LiveSyncCommands {
async getFiles(
path: string,
ignoreList: string[],
filter: RegExp[],
ignoreFilter: RegExp[]
filter?: RegExp[],
ignoreFilter?: RegExp[]
) {
const w = await this.app.vault.adapter.list(path);
let w: ListedFiles;
try {
w = await this.app.vault.adapter.list(path);
} catch (ex) {
Logger(`Could not traverse(HiddenSync):${path}`, LOG_LEVEL_INFO);
Logger(ex, LOG_LEVEL_VERBOSE);
return [];
}
const filesSrc = [
...w.files
.filter((e) => !ignoreList.some((ee) => e.endsWith(ee)))

36
src/CmdPluginAndTheirSettings.ts
View File

@@ -1,8 +1,8 @@
import { normalizePath, type PluginManifest } from "./deps";
import type { DocumentID, EntryDoc, FilePathWithPrefix, LoadedEntry } from "./lib/src/types";
import type { DocumentID, EntryDoc, FilePathWithPrefix, LoadedEntry, SavingEntry } from "./lib/src/types";
import { LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE } from "./lib/src/types";
import { type PluginDataEntry, PERIODIC_PLUGIN_SWEEP, type PluginList, type DevicePluginList, PSCHeader, PSCHeaderEnd } from "./types";
import { getDocData, isDocContentSame } from "./lib/src/utils";
import { createTextBlob, getDocData, isDocContentSame } from "./lib/src/utils";
import { Logger } from "./lib/src/logger";
import { PouchDB } from "./lib/src/pouchdb-browser.js";
import { isPluginMetadata, PeriodicProcessor } from "./utils";
@@ -186,18 +186,17 @@ export class PluginAndTheirSettings extends LiveSyncCommands {
}
Logger(`Reading plugin:${m.name}(${m.id})`, LOG_LEVEL_VERBOSE);
const path = normalizePath(m.dir) + "/";
const adapter = this.app.vault.adapter;
const files = ["manifest.json", "main.js", "styles.css", "data.json"];
const pluginData: { [key: string]: string; } = {};
for (const file of files) {
const thePath = path + file;
if (await adapter.exists(thePath)) {
pluginData[file] = await adapter.read(thePath);
if (await this.plugin.vaultAccess.adapterExists(thePath)) {
pluginData[file] = await this.plugin.vaultAccess.adapterRead(thePath);
}
}
let mtime = 0;
if (await adapter.exists(path + "/data.json")) {
mtime = (await adapter.stat(path + "/data.json")).mtime;
if (await this.plugin.vaultAccess.adapterExists(path + "/data.json")) {
mtime = (await this.plugin.vaultAccess.adapterStat(path + "/data.json")).mtime;
}
const p: PluginDataEntry = {
@@ -211,13 +210,14 @@ export class PluginAndTheirSettings extends LiveSyncCommands {
mtime: mtime,
type: "plugin",
};
const d: LoadedEntry = {
const blob = createTextBlob(JSON.stringify(p));
const d: SavingEntry = {
_id: p._id,
path: p._id as string as FilePathWithPrefix,
data: JSON.stringify(p),
data: blob,
ctime: mtime,
mtime: mtime,
size: 0,
size: blob.size,
children: [],
datatype: "plain",
type: "plain"
@@ -228,7 +228,7 @@ export class PluginAndTheirSettings extends LiveSyncCommands {
if (old !== false) {
const oldData = { data: old.data, deleted: old._deleted };
const newData = { data: d.data, deleted: d._deleted };
if (isDocContentSame(oldData.data, newData.data) && oldData.deleted == newData.deleted) {
if (await isDocContentSame(oldData.data, newData.data) && oldData.deleted == newData.deleted) {
Logger(`Nothing changed:${m.name}`);
return;
}
@@ -268,7 +268,6 @@ export class PluginAndTheirSettings extends LiveSyncCommands {
async applyPluginData(plugin: PluginDataEntry) {
await serialized("plugin-" + plugin.manifest.id, async () => {
const pluginTargetFolderPath = normalizePath(plugin.manifest.dir) + "/";
const adapter = this.app.vault.adapter;
// @ts-ignore
const stat = this.app.plugins.enabledPlugins.has(plugin.manifest.id) == true;
if (stat) {
@@ -277,7 +276,7 @@ export class PluginAndTheirSettings extends LiveSyncCommands {
Logger(`Unload plugin:${plugin.manifest.id}`, LOG_LEVEL_NOTICE);
}
if (plugin.dataJson)
await adapter.write(pluginTargetFolderPath + "data.json", plugin.dataJson);
await this.plugin.vaultAccess.adapterWrite(pluginTargetFolderPath + "data.json", plugin.dataJson);
Logger("wrote:" + pluginTargetFolderPath + "data.json", LOG_LEVEL_NOTICE);
if (stat) {
// @ts-ignore
@@ -298,14 +297,13 @@ export class PluginAndTheirSettings extends LiveSyncCommands {
}
const pluginTargetFolderPath = normalizePath(plugin.manifest.dir) + "/";
const adapter = this.app.vault.adapter;
if ((await adapter.exists(pluginTargetFolderPath)) === false) {
await adapter.mkdir(pluginTargetFolderPath);
if ((await this.plugin.vaultAccess.adapterExists(pluginTargetFolderPath)) === false) {
await this.app.vault.adapter.mkdir(pluginTargetFolderPath);
}
await adapter.write(pluginTargetFolderPath + "main.js", plugin.mainJs);
await adapter.write(pluginTargetFolderPath + "manifest.json", plugin.manifestJson);
await this.plugin.vaultAccess.adapterWrite(pluginTargetFolderPath + "main.js", plugin.mainJs);
await this.plugin.vaultAccess.adapterWrite(pluginTargetFolderPath + "manifest.json", plugin.manifestJson);
if (plugin.styleCss)
await adapter.write(pluginTargetFolderPath + "styles.css", plugin.styleCss);
await this.plugin.vaultAccess.adapterWrite(pluginTargetFolderPath + "styles.css", plugin.styleCss);
if (stat) {
// @ts-ignore
await this.app.plugins.loadPlugin(plugin.manifest.id);

81
src/CmdSetupLiveSync.ts
View File

@@ -5,7 +5,7 @@ import { PouchDB } from "./lib/src/pouchdb-browser.js";
import { askSelectString, askYesNo, askString } from "./utils";
import { decrypt, encrypt } from "./lib/src/e2ee_v2";
import { LiveSyncCommands } from "./LiveSyncCommands";
import { delay } from "./lib/src/utils";
import { delay, fireAndForget } from "./lib/src/utils";
import { confirmWithMessage } from "./dialogs";
import { Platform } from "./deps";
import { fetchAllUsedChunks } from "./lib/src/utils_couchdb";
@@ -17,25 +17,25 @@ export class SetupLiveSync extends LiveSyncCommands {
this.plugin.addCommand({
id: "livesync-copysetupuri",
name: "Copy the setup URI",
callback: this.command_copySetupURI.bind(this),
name: "Copy settings as a new setup URI",
callback: () => fireAndForget(this.command_copySetupURI()),
});
this.plugin.addCommand({
id: "livesync-copysetupuri-short",
name: "Copy the setup URI (With customization sync)",
callback: this.command_copySetupURIWithSync.bind(this),
name: "Copy settings as a new setup URI (With customization sync)",
callback: () => fireAndForget(this.command_copySetupURIWithSync()),
});
this.plugin.addCommand({
id: "livesync-copysetupurifull",
name: "Copy the setup URI (Full)",
callback: this.command_copySetupURIFull.bind(this),
name: "Copy settings as a new setup URI (Full)",
callback: () => fireAndForget(this.command_copySetupURIFull()),
});
this.plugin.addCommand({
id: "livesync-opensetupuri",
name: "Open the setup URI",
callback: this.command_openSetupURI.bind(this),
name: "Use the copied setup URI (Formerly Open setup URI)",
callback: () => fireAndForget(this.command_openSetupURI()),
});
}
onInitializeDatabase(showNotice: boolean) { }
@@ -60,7 +60,7 @@ export class SetupLiveSync extends LiveSyncCommands {
delete setting[k];
}
}
const encryptedSetting = encodeURIComponent(await encrypt(JSON.stringify(setting), encryptingPassphrase, false, true));
const encryptedSetting = encodeURIComponent(await encrypt(JSON.stringify(setting), encryptingPassphrase, false));
const uri = `${configURIBase}${encryptedSetting}`;
await navigator.clipboard.writeText(uri);
Logger("Setup URI copied to clipboard", LOG_LEVEL_NOTICE);
@@ -70,13 +70,13 @@ export class SetupLiveSync extends LiveSyncCommands {
if (encryptingPassphrase === false)
return;
const setting = { ...this.settings, configPassphraseStore: "", encryptedCouchDBConnection: "", encryptedPassphrase: "" };
const encryptedSetting = encodeURIComponent(await encrypt(JSON.stringify(setting), encryptingPassphrase, false, true));
const encryptedSetting = encodeURIComponent(await encrypt(JSON.stringify(setting), encryptingPassphrase, false));
const uri = `${configURIBase}${encryptedSetting}`;
await navigator.clipboard.writeText(uri);
Logger("Setup URI copied to clipboard", LOG_LEVEL_NOTICE);
}
async command_copySetupURIWithSync() {
this.command_copySetupURI(false);
await this.command_copySetupURI(false);
}
async command_openSetupURI() {
const setupURI = await askString(this.app, "Easy setup", "Set up URI", `${configURIBase}aaaaa`);
@@ -108,6 +108,7 @@ export class SetupLiveSync extends LiveSyncCommands {
newSettingW.configPassphraseStore = "";
newSettingW.encryptedPassphrase = "";
newSettingW.encryptedCouchDBConnection = "";
newSettingW.additionalSuffixOfDatabaseName = `${("appId" in this.app ? this.app.appId : "")}`
const setupJustImport = "Just import setting";
const setupAsNew = "Set it up as secondary or subsequent device";
const setupAsMerge = "Secondary device but try keeping local changes";
@@ -115,6 +116,7 @@ export class SetupLiveSync extends LiveSyncCommands {
const setupManually = "Leave everything to me";
newSettingW.syncInternalFiles = false;
newSettingW.usePluginSync = false;
newSettingW.isConfigured = true;
// Migrate completely obsoleted configuration.
if (!newSettingW.useIndexedDBAdapter) {
newSettingW.useIndexedDBAdapter = true;
@@ -132,7 +134,7 @@ export class SetupLiveSync extends LiveSyncCommands {
} else if (setupType == setupAsMerge) {
this.plugin.settings = newSettingW;
this.plugin.usedPassphrase = "";
await this.fetchLocalWithKeepLocal();
await this.fetchLocalWithRebuild();
} else if (setupType == setupAgain) {
const confirm = "I know this operation will rebuild all my databases with files on this device, and files that are on the remote database and I didn't synchronize to any other devices will be lost and want to proceed indeed.";
if (await askSelectString(this.app, "Do you really want to do this?", ["Cancel", confirm]) != confirm) {
@@ -321,7 +323,6 @@ Of course, we are able to disable these features.`
this.plugin.settings.suspendFileWatching = false;
await this.plugin.syncAllFiles(true);
await this.plugin.loadQueuedFiles();
this.plugin.procQueuedFiles();
await this.plugin.saveSettings();
}
@@ -334,10 +335,18 @@ Of course, we are able to disable these features.`
const ret = await confirmWithMessage(this.plugin, "Database adapter", message, choices, CHOICE_YES, 10);
if (ret == CHOICE_YES) {
this.plugin.settings.useIndexedDBAdapter = false;
this.plugin.settings.useIndexedDBAdapter = true;
}
}
}
async resetLocalDatabase() {
if (this.plugin.settings.isConfigured && this.plugin.settings.additionalSuffixOfDatabaseName == "") {
// Discard the non-suffixed database
await this.plugin.resetLocalDatabase();
}
this.plugin.settings.additionalSuffixOfDatabaseName = `${("appId" in this.app ? this.app.appId : "")}`
await this.plugin.resetLocalDatabase();
}
async fetchRemoteChunks() {
if (!this.plugin.settings.doNotSuspendOnFetching && this.plugin.settings.readChunksOnline) {
Logger(`Fetching chunks`, LOG_LEVEL_NOTICE);
@@ -350,45 +359,36 @@ Of course, we are able to disable these features.`
Logger(`Fetching chunks done`, LOG_LEVEL_NOTICE);
}
}
async fetchLocal() {
async fetchLocal(makeLocalChunkBeforeSync?: boolean) {
this.suspendExtraSync();
this.askUseNewAdapter();
await this.askUseNewAdapter();
this.plugin.settings.isConfigured = true;
await this.suspendReflectingDatabase();
await this.plugin.realizeSettingSyncMode();
await this.plugin.resetLocalDatabase();
await this.resetLocalDatabase();
await delay(1000);
await this.plugin.markRemoteResolved();
await this.plugin.openDatabase();
this.plugin.isReady = true;
if (makeLocalChunkBeforeSync) {
await this.plugin.initializeDatabase(true);
}
await this.plugin.markRemoteResolved();
await delay(500);
await this.plugin.replicateAllFromServer(true);
await delay(1000);
await this.plugin.replicateAllFromServer(true);
await this.fetchRemoteChunks();
// if (!tryLessFetching) {
// await this.fetchRemoteChunks();
// }
await this.resumeReflectingDatabase();
await this.askHiddenFileConfiguration({ enableFetch: true });
}
async fetchLocalWithKeepLocal() {
this.suspendExtraSync();
this.askUseNewAdapter();
await this.suspendReflectingDatabase();
await this.plugin.realizeSettingSyncMode();
await this.plugin.resetLocalDatabase();
await delay(1000);
await this.plugin.initializeDatabase(true);
await this.plugin.markRemoteResolved();
await this.plugin.openDatabase();
this.plugin.isReady = true;
await delay(500);
await this.plugin.replicateAllFromServer(true);
await delay(1000);
await this.plugin.replicateAllFromServer(true);
await this.fetchRemoteChunks();
await this.resumeReflectingDatabase();
await this.askHiddenFileConfiguration({ enableFetch: true });
async fetchLocalWithRebuild() {
return await this.fetchLocal(true);
}
async rebuildRemote() {
this.suspendExtraSync();
this.plugin.settings.isConfigured = true;
await this.plugin.realizeSettingSyncMode();
await this.plugin.markRemoteLocked();
await this.plugin.tryResetRemoteDatabase();
@@ -402,9 +402,10 @@ Of course, we are able to disable these features.`
}
async rebuildEverything() {
this.suspendExtraSync();
this.askUseNewAdapter();
await this.askUseNewAdapter();
this.plugin.settings.isConfigured = true;
await this.plugin.realizeSettingSyncMode();
await this.plugin.resetLocalDatabase();
await this.resetLocalDatabase();
await delay(1000);
await this.plugin.initializeDatabase(true);
await this.plugin.markRemoteLocked();

82
src/ConflictResolveModal.ts
View File

@@ -1,23 +1,41 @@
import { App, Modal } from "./deps";
import { DIFF_DELETE, DIFF_EQUAL, DIFF_INSERT } from "diff-match-patch";
import { type diff_result } from "./lib/src/types";
import { CANCELLED, LEAVE_TO_SUBSEQUENT, RESULT_TIMED_OUT, type diff_result } from "./lib/src/types";
import { escapeStringToHTML } from "./lib/src/strbin";
import { delay, sendValue, waitForValue } from "./lib/src/utils";
export type MergeDialogResult = typeof LEAVE_TO_SUBSEQUENT | typeof CANCELLED | string;
export class ConflictResolveModal extends Modal {
// result: Array<[number, string]>;
result: diff_result;
filename: string;
callback: (remove_rev: string) => Promise<void>;
constructor(app: App, filename: string, diff: diff_result, callback: (remove_rev: string) => Promise<void>) {
response: MergeDialogResult = CANCELLED;
isClosed = false;
consumed = false;
constructor(app: App, filename: string, diff: diff_result) {
super(app);
this.result = diff;
this.callback = callback;
this.filename = filename;
// Send cancel signal for the previous merge dialogue
// if not there, simply be ignored.
// sendValue("close-resolve-conflict:" + this.filename, false);
sendValue("cancel-resolve-conflict:" + this.filename, true);
}
onOpen() {
const { contentEl } = this;
// Send cancel signal for the previous merge dialogue
// if not there, simply be ignored.
sendValue("cancel-resolve-conflict:" + this.filename, true);
setTimeout(async () => {
const forceClose = await waitForValue("cancel-resolve-conflict:" + this.filename);
// debugger;
if (forceClose) {
this.sendResponse(CANCELLED);
}
}, 10)
// sendValue("close-resolve-conflict:" + this.filename, false);
this.titleEl.setText("Conflicting changes");
contentEl.empty();
contentEl.createEl("span", { text: this.filename });
@@ -44,42 +62,32 @@ export class ConflictResolveModal extends Modal {
div2.innerHTML = `
<span class='deleted'>A:${date1}</span><br /><span class='added'>B:${date2}</span><br>
`;
contentEl.createEl("button", { text: "Keep A" }, (e) => {
e.addEventListener("click", async () => {
const callback = this.callback;
this.callback = null;
this.close();
await callback(this.result.right.rev);
});
});
contentEl.createEl("button", { text: "Keep B" }, (e) => {
e.addEventListener("click", async () => {
const callback = this.callback;
this.callback = null;
this.close();
await callback(this.result.left.rev);
});
});
contentEl.createEl("button", { text: "Concat both" }, (e) => {
e.addEventListener("click", async () => {
const callback = this.callback;
this.callback = null;
this.close();
await callback("");
});
});
contentEl.createEl("button", { text: "Not now" }, (e) => {
e.addEventListener("click", () => {
this.close();
});
});
contentEl.createEl("button", { text: "Keep A" }, (e) => e.addEventListener("click", () => this.sendResponse(this.result.right.rev)));
contentEl.createEl("button", { text: "Keep B" }, (e) => e.addEventListener("click", () => this.sendResponse(this.result.left.rev)));
contentEl.createEl("button", { text: "Concat both" }, (e) => e.addEventListener("click", () => this.sendResponse(LEAVE_TO_SUBSEQUENT)));
contentEl.createEl("button", { text: "Not now" }, (e) => e.addEventListener("click", () => this.sendResponse(CANCELLED)));
}
sendResponse(result: MergeDialogResult) {
this.response = result;
this.close();
}
onClose() {
const { contentEl } = this;
contentEl.empty();
if (this.callback != null) {
this.callback(null);
if (this.consumed) {
return;
}
this.consumed = true;
sendValue("close-resolve-conflict:" + this.filename, this.response);
sendValue("cancel-resolve-conflict:" + this.filename, false);
}
}
async waitForResult(): Promise<MergeDialogResult> {
await delay(100);
const r = await waitForValue<MergeDialogResult>("close-resolve-conflict:" + this.filename);
if (r === RESULT_TIMED_OUT) return CANCELLED;
return r;
}
}

150
src/DocumentHistoryModal.ts
View File

@@ -5,9 +5,36 @@ import ObsidianLiveSyncPlugin from "./main";
import { type DocumentID, type FilePathWithPrefix, type LoadedEntry, LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE } from "./lib/src/types";
import { Logger } from "./lib/src/logger";
import { isErrorOfMissingDoc } from "./lib/src/utils_couchdb";
import { getDocData } from "./lib/src/utils";
import { stripPrefix } from "./lib/src/path";
import { getDocData, readContent } from "./lib/src/utils";
import { isPlainText, stripPrefix } from "./lib/src/path";
function isImage(path: string) {
const ext = path.split(".").splice(-1)[0].toLowerCase();
return ["png", "jpg", "jpeg", "gif", "bmp", "webp"].includes(ext);
}
function isComparableText(path: string) {
const ext = path.split(".").splice(-1)[0].toLowerCase();
return isPlainText(path) || ["md", "mdx", "txt", "json"].includes(ext);
}
function isComparableTextDecode(path: string) {
const ext = path.split(".").splice(-1)[0].toLowerCase();
return ["json"].includes(ext)
}
function readDocument(w: LoadedEntry) {
if (w.data.length == 0) return "";
if (isImage(w.path)) {
return new Uint8Array(decodeBinary(w.data));
}
if (w.data == "plain") return getDocData(w.data);
if (isComparableTextDecode(w.path)) return readString(new Uint8Array(decodeBinary(w.data)));
if (isComparableText(w.path)) return getDocData(w.data);
try {
return readString(new Uint8Array(decodeBinary(w.data)));
} catch (ex) {
// NO OP.
}
return getDocData(w.data);
}
export class DocumentHistoryModal extends Modal {
plugin: ObsidianLiveSyncPlugin;
range!: HTMLInputElement;
@@ -39,13 +66,13 @@ export class DocumentHistoryModal extends Modal {
}
}
async loadFile(initialRev: string) {
async loadFile(initialRev?: string) {
if (!this.id) {
this.id = await this.plugin.path2id(this.file);
}
const db = this.plugin.localDatabase;
try {
const w = await db.localDatabase.get(this.id, { revs_info: true });
const w = await db.getRaw(this.id, { revs_info: true });
this.revs_info = w._revs_info?.filter((e) => e?.status == "available") ?? [];
this.range.max = `${Math.max(this.revs_info.length - 1, 0)}`;
this.range.value = this.range.max;
@@ -56,7 +83,6 @@ export class DocumentHistoryModal extends Modal {
this.range.max = "0";
this.range.value = "";
this.range.disabled = true;
this.showDiff
this.contentView.setText(`History of this file was not recorded.`);
} else {
this.contentView.setText(`Error occurred.`);
@@ -76,6 +102,22 @@ export class DocumentHistoryModal extends Modal {
const rev = this.revs_info[index];
await this.showExactRev(rev.rev);
}
BlobURLs = new Map<string, string>();
revokeURL(key: string) {
const v = this.BlobURLs.get(key);
if (v) {
URL.revokeObjectURL(v);
}
this.BlobURLs.delete(key);
}
generateBlobURL(key: string, data: Uint8Array) {
this.revokeURL(key);
const v = URL.createObjectURL(new Blob([data], { endings: "transparent", type: "application/octet-stream" }));
this.BlobURLs.set(key, v);
return v;
}
async showExactRev(rev: string) {
const db = this.plugin.localDatabase;
const w = await db.getDBEntry(this.file, { rev: rev }, false, false, true);
@@ -88,42 +130,67 @@ export class DocumentHistoryModal extends Modal {
} else {
this.currentDoc = w;
this.info.innerHTML = `Modified:${new Date(w.mtime).toLocaleString()}`;
let result = "";
const w1data = w.datatype == "plain" ? getDocData(w.data) : readString(new Uint8Array(decodeBinary(w.data)));
let result = undefined;
const w1data = readDocument(w);
this.currentDeleted = !!w.deleted;
this.currentText = w1data;
// this.currentText = w1data;
if (this.showDiff) {
const prevRevIdx = this.revs_info.length - 1 - ((this.range.value as any) / 1 - 1);
if (prevRevIdx >= 0 && prevRevIdx < this.revs_info.length) {
const oldRev = this.revs_info[prevRevIdx].rev;
const w2 = await db.getDBEntry(this.file, { rev: oldRev }, false, false, true);
if (w2 != false) {
const dmp = new diff_match_patch();
const w2data = w2.datatype == "plain" ? getDocData(w2.data) : readString(new Uint8Array(decodeBinary(w2.data)));
const diff = dmp.diff_main(w2data, w1data);
dmp.diff_cleanupSemantic(diff);
for (const v of diff) {
const x1 = v[0];
const x2 = v[1];
if (x1 == DIFF_DELETE) {
result += "<span class='history-deleted'>" + escapeStringToHTML(x2) + "</span>";
} else if (x1 == DIFF_EQUAL) {
result += "<span class='history-normal'>" + escapeStringToHTML(x2) + "</span>";
} else if (x1 == DIFF_INSERT) {
result += "<span class='history-added'>" + escapeStringToHTML(x2) + "</span>";
if (typeof w1data == "string") {
result = "";
const dmp = new diff_match_patch();
const w2data = readDocument(w2) as string;
const diff = dmp.diff_main(w2data, w1data);
dmp.diff_cleanupSemantic(diff);
for (const v of diff) {
const x1 = v[0];
const x2 = v[1];
if (x1 == DIFF_DELETE) {
result += "<span class='history-deleted'>" + escapeStringToHTML(x2) + "</span>";
} else if (x1 == DIFF_EQUAL) {
result += "<span class='history-normal'>" + escapeStringToHTML(x2) + "</span>";
} else if (x1 == DIFF_INSERT) {
result += "<span class='history-added'>" + escapeStringToHTML(x2) + "</span>";
}
}
result = result.replace(/\n/g, "<br>");
} else if (isImage(this.file)) {
const src = this.generateBlobURL("base", w1data);
const overlay = this.generateBlobURL("overlay", readDocument(w2) as Uint8Array);
result =
`<div class='ls-imgdiff-wrap'>
<div class='overlay'>
<img class='img-base' src="${src}">
<img class='img-overlay' src='${overlay}'>
</div>
</div>`;
this.contentView.removeClass("op-pre");
}
}
}
result = result.replace(/\n/g, "<br>");
} else {
result = escapeStringToHTML(w1data);
}
if (result == undefined) {
if (typeof w1data != "string") {
if (isImage(this.file)) {
const src = this.generateBlobURL("base", w1data);
result =
`<div class='ls-imgdiff-wrap'>
<div class='overlay'>
<img class='img-base' src="${src}">
</div>
</div>`;
this.contentView.removeClass("op-pre");
}
} else {
result = escapeStringToHTML(w1data);
}
} else {
result = escapeStringToHTML(w1data);
}
if (result == undefined) result = typeof w1data == "string" ? escapeStringToHTML(w1data) : "Binary file";
this.contentView.innerHTML = (this.currentDeleted ? "(At this revision, the file has been deleted)\n" : "") + result;
}
}
@@ -180,12 +247,10 @@ export class DocumentHistoryModal extends Modal {
Logger(`Old content copied to clipboard`, LOG_LEVEL_NOTICE);
});
});
async function focusFile(path: string) {
const targetFile = app.vault
.getFiles()
.find((f) => f.path === path);
const focusFile = async (path: string) => {
const targetFile = this.plugin.app.vault.getFileByPath(path);
if (targetFile) {
const leaf = app.workspace.getLeaf(false);
const leaf = this.plugin.app.workspace.getLeaf(false);
await leaf.openFile(targetFile);
} else {
Logger("The file could not view on the editor", LOG_LEVEL_NOTICE)
@@ -198,24 +263,25 @@ export class DocumentHistoryModal extends Modal {
const pathToWrite = stripPrefix(this.file);
if (!isValidPath(pathToWrite)) {
Logger("Path is not valid to write content.", LOG_LEVEL_INFO);
return;
}
if (this.currentDoc?.datatype == "plain") {
await this.app.vault.adapter.write(pathToWrite, getDocData(this.currentDoc.data));
await focusFile(pathToWrite);
this.close();
} else if (this.currentDoc?.datatype == "newnote") {
await this.app.vault.adapter.writeBinary(pathToWrite, decodeBinary(this.currentDoc.data));
await focusFile(pathToWrite);
this.close();
} else {
Logger(`Could not parse entry`, LOG_LEVEL_NOTICE);
if (!this.currentDoc) {
Logger("No active file loaded.", LOG_LEVEL_INFO);
return;
}
const d = readContent(this.currentDoc);
await this.plugin.vaultAccess.adapterWrite(pathToWrite, d);
await focusFile(pathToWrite);
this.close();
});
});
}
onClose() {
const { contentEl } = this;
contentEl.empty();
this.BlobURLs.forEach(value => {
console.log(value);
if (value) URL.revokeObjectURL(value);
})
}
}

22
src/GlobalHistory.svelte
View File

@@ -2,12 +2,11 @@
import ObsidianLiveSyncPlugin from "./main";
import { onDestroy, onMount } from "svelte";
import type { AnyEntry, FilePathWithPrefix } from "./lib/src/types";
import { getDocData, isDocContentSame } from "./lib/src/utils";
import { getDocData, isDocContentSame, readAsBlob } from "./lib/src/utils";
import { diff_match_patch } from "./deps";
import { DocumentHistoryModal } from "./DocumentHistoryModal";
import { isPlainText, stripAllPrefixes } from "./lib/src/path";
import { TFile } from "./deps";
import { encodeBinary } from "./lib/src/strbin";
export let plugin: ObsidianLiveSyncPlugin;
let showDiffInfo = false;
@@ -67,10 +66,7 @@
for (const revInfo of reversedRevs) {
if (revInfo.status == "available") {
const doc =
(!isPlain && showDiffInfo) || (checkStorageDiff && revInfo.rev == docA._rev)
? await db.getDBEntry(path, { rev: revInfo.rev }, false, false, true)
: await db.getDBEntryMeta(path, { rev: revInfo.rev }, true);
const doc = (!isPlain && showDiffInfo) || (checkStorageDiff && revInfo.rev == docA._rev) ? await db.getDBEntry(path, { rev: revInfo.rev }, false, false, true) : await db.getDBEntryMeta(path, { rev: revInfo.rev }, true);
if (doc === false) continue;
const rev = revInfo.rev;
@@ -108,17 +104,11 @@
}
if (rev == docA._rev) {
if (checkStorageDiff) {
const abs = plugin.app.vault.getAbstractFileByPath(stripAllPrefixes(plugin.getPath(docA)));
const abs = plugin.vaultAccess.getAbstractFileByPath(stripAllPrefixes(plugin.getPath(docA)));
if (abs instanceof TFile) {
let result = false;
if (isPlainText(docA.path)) {
const data = await plugin.app.vault.read(abs);
result = isDocContentSame(data, doc.data);
} else {
const data = await plugin.app.vault.readBinary(abs);
const dataEEncoded = await encodeBinary(data, plugin.settings.useV1);
result = isDocContentSame(dataEEncoded, doc.data);
}
const data = await plugin.vaultAccess.adapterReadAuto(abs);
const d = readAsBlob(doc);
const result = await isDocContentSame(data, d);
if (result) {
diffDetail += " ⚖️";
} else {

2
src/JsonResolveModal.ts
View File

@@ -1,6 +1,7 @@
import { App, Modal } from "./deps";
import { type FilePath, type LoadedEntry } from "./lib/src/types";
import JsonResolvePane from "./JsonResolvePane.svelte";
import { waitForSignal } from "./lib/src/utils";
export class JsonResolveModal extends Modal {
// result: Array<[number, string]>;
@@ -20,6 +21,7 @@ export class JsonResolveModal extends Modal {
this.nameA = nameA;
this.nameB = nameB;
this.defaultSelect = defaultSelect;
waitForSignal(`cancel-internal-conflict:${filename}`).then(() => this.close());
}
async UICallback(keepRev: string, mergedStr?: string) {
this.close();

3
src/LiveSyncCommands.ts
View File

@@ -14,6 +14,9 @@ export abstract class LiveSyncCommands {
get localDatabase() {
return this.plugin.localDatabase;
}
get vaultAccess() {
return this.plugin.vaultAccess;
}
id2path(id: DocumentID, entry?: EntryHasPath, stripPrefix?: boolean): FilePathWithPrefix {
return this.plugin.id2path(id, entry, stripPrefix);
}

11
src/LogDisplayModal.ts
View File

@@ -1,5 +1,6 @@
import { App, Modal } from "./deps";
import { logMessageStore } from "./lib/src/stores";
import type { ReactiveInstance, } from "./lib/src/reactive";
import { logMessages } from "./lib/src/stores";
import { escapeStringToHTML } from "./lib/src/strbin";
import ObsidianLiveSyncPlugin from "./main";
@@ -21,14 +22,16 @@ export class LogDisplayModal extends Modal {
div.addClass("op-scrollable");
div.addClass("op-pre");
this.logEl = div;
this.unsubscribe = logMessageStore.observe((e) => {
function updateLog(logs: ReactiveInstance<string[]>) {
const e = logs.value;
let msg = "";
for (const v of e) {
msg += escapeStringToHTML(v) + "<br>";
}
this.logEl.innerHTML = msg;
})
logMessageStore.invalidate();
}
logMessages.onChanged(updateLog);
this.unsubscribe = () => logMessages.offChanged(updateLog);
}
onClose() {
const { contentEl } = this;

29
src/LogPane.svelte
View File

@@ -1,26 +1,27 @@
<script lang="ts">
import { onDestroy, onMount } from "svelte";
import { logMessageStore } from "./lib/src/stores";
import { logMessages } from "./lib/src/stores";
import type { ReactiveInstance } from "./lib/src/reactive";
import { Logger } from "./lib/src/logger";
let unsubscribe: () => void;
let messages = [] as string[];
let wrapRight = false;
let autoScroll = true;
let suspended = false;
function updateLog(logs: ReactiveInstance<string[]>) {
const e = logs.value;
if (!suspended) {
messages = [...e];
setTimeout(() => {
if (scroll) scroll.scrollTop = scroll.scrollHeight;
}, 10);
}
}
onMount(async () => {
unsubscribe = logMessageStore.observe((e) => {
if (!suspended) {
messages = [...e];
if (autoScroll) {
if (scroll) scroll.scrollTop = scroll.scrollHeight;
}
}
});
logMessageStore.invalidate();
setTimeout(() => {
if (scroll) scroll.scrollTop = scroll.scrollHeight;
}, 100);
logMessages.onChanged(updateLog);
Logger("Log window opened");
unsubscribe = () => logMessages.offChanged(updateLog);
});
onDestroy(() => {
if (unsubscribe) unsubscribe();

625
src/ObsidianLiveSyncSettingTab.ts
View File

@@ -1,13 +1,14 @@
import { App, PluginSettingTab, Setting, sanitizeHTMLToDom, TextAreaComponent, MarkdownRenderer, stringifyYaml } from "./deps";
import { DEFAULT_SETTINGS, type ObsidianLiveSyncSettings, type ConfigPassphraseStore, type RemoteDBSettings, type FilePathWithPrefix, type HashAlgorithm, type DocumentID, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE, LOG_LEVEL_INFO } from "./lib/src/types";
import { delay } from "./lib/src/utils";
import { Semaphore } from "./lib/src/semaphore";
import { DEFAULT_SETTINGS, type ObsidianLiveSyncSettings, type ConfigPassphraseStore, type RemoteDBSettings, type FilePathWithPrefix, type HashAlgorithm, type DocumentID, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE, LOG_LEVEL_INFO, type LoadedEntry, PREFERRED_SETTING_CLOUDANT, PREFERRED_SETTING_SELF_HOSTED } from "./lib/src/types";
import { createBlob, delay, isDocContentSame, readAsBlob } from "./lib/src/utils";
import { versionNumberString2Number } from "./lib/src/strbin";
import { Logger } from "./lib/src/logger";
import { checkSyncInfo, isCloudantURI } from "./lib/src/utils_couchdb.js";
import { checkSyncInfo, isCloudantURI } from "./lib/src/utils_couchdb";
import { testCrypt } from "./lib/src/e2ee_v2";
import ObsidianLiveSyncPlugin from "./main";
import { askYesNo, performRebuildDB, requestToCouchDB, scheduleTask } from "./utils";
import { request, type ButtonComponent, TFile } from "obsidian";
import { shouldBeIgnored } from "./lib/src/path";
export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
@@ -26,6 +27,18 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
}
this.plugin.addLog(`Connected to ${db.info.db_name}`, LOG_LEVEL_NOTICE);
}
askReload() {
scheduleTask("configReload", 250, async () => {
if (await askYesNo(this.app, "Do you want to restart and reload Obsidian now?") == "yes") {
// @ts-ignore
this.app.commands.executeCommandById("app:reload")
}
})
}
closeSetting() {
// @ts-ignore
this.plugin.app.setting.close()
}
display(): void {
const { containerEl } = this;
let encrypt = this.plugin.settings.encrypt;
@@ -34,6 +47,10 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
containerEl.empty();
// const preferred_setting = isCloudantURI(this.plugin.settings.couchDB_URI) ? PREFERRED_SETTING_CLOUDANT : PREFERRED_SETTING_SELF_HOSTED;
// const default_setting = { ...DEFAULT_SETTINGS };
containerEl.createEl("h2", { text: "Settings for Self-hosted LiveSync." });
containerEl.addClass("sls-setting");
containerEl.removeClass("isWizard");
@@ -69,11 +86,11 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
}
w.querySelectorAll(`.sls-setting-label`).forEach((element) => {
element.removeClass("selected");
(element.querySelector<HTMLInputElement>("input[type=radio]")).checked = false;
(element.querySelector<HTMLInputElement>("input[type=radio]"))!.checked = false;
});
w.querySelectorAll(`.sls-setting-label.c-${screen}`).forEach((element) => {
element.addClass("selected");
(element.querySelector<HTMLInputElement>("input[type=radio]")).checked = true;
(element.querySelector<HTMLInputElement>("input[type=radio]"))!.checked = true;
});
this.selectedScreen = screen;
};
@@ -103,7 +120,7 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
tmpDiv.innerHTML = `<button> OK, I read all. </button>`;
if (lastVersion > this.plugin.settings.lastReadUpdates) {
const informationButtonDiv = h3El.appendChild(tmpDiv);
informationButtonDiv.querySelector("button").addEventListener("click", async () => {
informationButtonDiv.querySelector("button")?.addEventListener("click", async () => {
this.plugin.settings.lastReadUpdates = lastVersion;
await this.plugin.saveSettings();
informationButtonDiv.remove();
@@ -132,29 +149,27 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
const setupWizardEl = containerEl.createDiv();
setupWizardEl.createEl("h3", { text: "Setup wizard" });
new Setting(setupWizardEl)
.setName("Discard the existing configuration and set up")
.setName("Use the copied setup URI")
.setDesc("To setup Self-hosted LiveSync, this method is the most preferred one.")
.addButton((text) => {
// eslint-disable-next-line require-await
text.setButtonText("Next").onClick(async () => {
if (JSON.stringify(this.plugin.settings) != JSON.stringify(DEFAULT_SETTINGS)) {
this.plugin.replicator.closeReplication();
this.plugin.settings = { ...DEFAULT_SETTINGS };
this.plugin.saveSettings();
Logger("Configuration has been flushed, please open it again", LOG_LEVEL_NOTICE)
// @ts-ignore
this.plugin.app.setting.close()
} else {
containerEl.addClass("isWizard");
applyDisplayEnabled();
inWizard = true;
changeDisplay("0")
}
text.setButtonText("Use").onClick(async () => {
this.closeSetting();
await this.plugin.addOnSetup.command_openSetupURI();
})
})
if (this.plugin.settings.isConfigured) {
new Setting(setupWizardEl)
.setName("Copy current settings as a new setup URI")
.addButton((text) => {
text.setButtonText("Copy").onClick(async () => {
await this.plugin.addOnSetup.command_copySetupURI();
})
})
}
new Setting(setupWizardEl)
.setName("Do not discard the existing configuration and set up again")
.setName("Minimal setup")
.addButton((text) => {
text.setButtonText("Next").onClick(async () => {
text.setButtonText("Start").onClick(async () => {
this.plugin.settings.liveSync = false;
this.plugin.settings.periodicReplication = false;
this.plugin.settings.syncOnSave = false;
@@ -168,33 +183,100 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
applyDisplayEnabled();
inWizard = true;
changeDisplay("0")
})
})
const infoWarnForSubsequent = setupWizardEl.createEl("div", { text: `To set up second or subsequent device, please use 'Copy setup URI' and 'Open setup URI'` });
infoWarnForSubsequent.addClass("op-warn-info");
new Setting(setupWizardEl)
.setName("Copy setup URI")
.addButton((text) => {
text.setButtonText("Copy setup URI").onClick(() => {
// @ts-ignore
this.plugin.app.commands.executeCommandById("obsidian-livesync:livesync-copysetupuri")
if (!this.plugin.settings.isConfigured) {
new Setting(setupWizardEl)
.setName("Enable LiveSync on this device as the set-up was completed manually")
.addButton((text) => {
text.setButtonText("Enable").onClick(async () => {
this.plugin.settings.isConfigured = true;
await this.plugin.saveSettings();
this.askReload();
})
})
}
if (this.plugin.settings.isConfigured) {
new Setting(setupWizardEl)
.setName("Discard exist settings and databases")
.addButton((text) => {
text.setButtonText("Discard").onClick(async () => {
if (await askYesNo(this.plugin.app, "Do you really want to discard exist settings and databases?") == "yes") {
this.plugin.settings = { ...DEFAULT_SETTINGS };
await this.plugin.saveSettingData();
await this.plugin.resetLocalDatabase();
// await this.plugin.initializeDatabase();
this.askReload();
}
}).setWarning()
})
}
setupWizardEl.createEl("h3", { text: "Online Tips" });
const repo = "vrtmrz/obsidian-livesync";
const topPath = "/docs/troubleshooting.md";
const rawRepoURI = `https://raw.githubusercontent.com/${repo}/main`;
setupWizardEl.createEl("div", "", el => el.innerHTML = `<a href='https://github.com/${repo}/blob/main${topPath}' target="_blank">Open in browser</a>`);
const troubleShootEl = setupWizardEl.createEl("div", { text: "", cls: "sls-troubleshoot-preview" });
const loadMarkdownPage = async (pathAll: string, basePathParam: string = "") => {
troubleShootEl.style.minHeight = troubleShootEl.clientHeight + "px";
troubleShootEl.empty();
const fullPath = pathAll.startsWith("/") ? pathAll : `${basePathParam}/${pathAll}`;
const directoryArr = fullPath.split("/");
const filename = directoryArr.pop();
const directly = directoryArr.join("/");
const basePath = directly;
let remoteTroubleShootMDSrc = "";
try {
remoteTroubleShootMDSrc = await request(`${rawRepoURI}${basePath}/${filename}`);
} catch (ex: any) {
remoteTroubleShootMDSrc = "Error Occurred!!\n" + ex.toString();
}
const remoteTroubleShootMD = remoteTroubleShootMDSrc.replace(/\((.*?(.png)|(.jpg))\)/g, `(${rawRepoURI}${basePath}/$1)`)
// Render markdown
await MarkdownRenderer.render(this.plugin.app, `<a class='sls-troubleshoot-anchor'></a> [Tips and Troubleshooting](${topPath}) [PageTop](${filename})\n\n${remoteTroubleShootMD}`, troubleShootEl, `${rawRepoURI}`, this.plugin);
// Menu
troubleShootEl.querySelector<HTMLAnchorElement>(".sls-troubleshoot-anchor")?.parentElement?.setCssStyles({ position: "sticky", top: "-1em", backgroundColor: "var(--modal-background)" });
// Trap internal links.
troubleShootEl.querySelectorAll<HTMLAnchorElement>("a.internal-link").forEach((anchorEl) => {
anchorEl.addEventListener("click", async (evt) => {
const uri = anchorEl.getAttr("data-href");
if (!uri) return;
if (uri.startsWith("#")) {
evt.preventDefault();
const elements = Array.from(troubleShootEl.querySelectorAll<HTMLHeadingElement>("[data-heading]"))
const p = elements.find(e => e.getAttr("data-heading")?.toLowerCase().split(" ").join("-") == uri.substring(1).toLowerCase());
if (p) {
p.setCssStyles({ scrollMargin: "3em" });
p.scrollIntoView({ behavior: "instant", block: "start" });
}
} else {
evt.preventDefault();
await loadMarkdownPage(uri, basePath);
troubleShootEl.setCssStyles({ scrollMargin: "1em" });
troubleShootEl.scrollIntoView({ behavior: "instant", block: "start" });
}
})
})
.addButton((text) => {
text.setButtonText("Open setup URI").onClick(() => {
// @ts-ignore
this.plugin.app.commands.executeCommandById("obsidian-livesync:livesync-opensetupuri")
})
})
troubleShootEl.style.minHeight = "";
}
loadMarkdownPage(topPath);
addScreenElement("110", setupWizardEl);
const containerRemoteDatabaseEl = containerEl.createDiv();
containerRemoteDatabaseEl.createEl("h3", { text: "Remote Database configuration" });
const syncWarn = containerRemoteDatabaseEl.createEl("div", { text: `These settings are kept locked while any synchronization options are enabled. Disable these options in the "Sync Settings" tab to unlock.` });
if (this.plugin.settings.couchDB_URI.startsWith("http://")) {
if (this.plugin.isMobile) {
containerRemoteDatabaseEl.createEl("div", { text: `Configured as using plain HTTP. We cannot connect to the remote. Please set up the credentials and use HTTPS for the remote URI.` })
.addClass("op-warn");
} else {
containerRemoteDatabaseEl.createEl("div", { text: `Configured as using plain HTTP. We might fail on mobile devices.` })
.addClass("op-warn-info");
}
}
syncWarn.addClass("op-warn-info");
syncWarn.addClass("sls-hidden");
@@ -284,6 +366,7 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
new Setting(containerRemoteDatabaseEl)
.setName("Test Database Connection")
.setClass("wizardHidden")
.setDesc("Open database connection. If the remote database is not found and you have the privilege to create a database, the database will be created.")
.addButton((button) =>
button
@@ -295,8 +378,8 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
);
new Setting(containerRemoteDatabaseEl)
.setName("Check database configuration")
// .setDesc("Open database connection. If the remote database is not found and you have the privilege to create a database, the database will be created.")
.setName("Check and Fix database configuration")
.setDesc("Check the database configuration, and fix if there are any problems.")
.addButton((button) =>
button
.setButtonText("Check")
@@ -331,7 +414,7 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
tmpDiv.addClass("ob-btn-config-fix");
tmpDiv.innerHTML = `<label>${title}</label><button>Fix</button>`;
const x = checkResultDiv.appendChild(tmpDiv);
x.querySelector("button").addEventListener("click", async () => {
x.querySelector("button")?.addEventListener("click", async () => {
Logger(`CouchDB Configuration: ${title} -> Set ${key} to ${value}`)
const res = await requestToCouchDB(this.plugin.settings.couchDB_URI, this.plugin.settings.couchDB_USER, this.plugin.settings.couchDB_PASSWORD, undefined, key, value);
if (res.status == 200) {
@@ -427,15 +510,11 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
const origins = ["app://obsidian.md", "capacitor://localhost", "http://localhost"];
for (const org of origins) {
const rr = await requestToCouchDB(this.plugin.settings.couchDB_URI, this.plugin.settings.couchDB_USER, this.plugin.settings.couchDB_PASSWORD, org);
const responseHeaders = Object.entries(rr.headers)
const responseHeaders = Object.fromEntries(Object.entries(rr.headers)
.map((e) => {
e[0] = (e[0] + "").toLowerCase();
e[0] = `${e[0]}`.toLowerCase();
return e;
})
.reduce((obj, [key, val]) => {
obj[key] = val;
return obj;
}, {} as { [key: string]: string });
}));
addResult(`Origin check:${org}`);
if (responseHeaders["access-control-allow-credentials"] != "true") {
addResult("❗ CORS is not allowing credential");
@@ -451,7 +530,7 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
addResult("--Done--", ["ob-btn-config-head"]);
addResult("If you have some trouble with Connection-check even though all Config-check has been passed, Please check your reverse proxy's configuration.", ["ob-btn-config-info"]);
Logger(`Checking configuration done`, LOG_LEVEL_INFO);
} catch (ex) {
} catch (ex: any) {
if (ex?.status == 401) {
addResult(`❗ Access forbidden.`);
addResult(`We could not continue the test.`);
@@ -503,18 +582,18 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
passphraseSetting?.controlEl.toggleClass("sls-item-dirty", passphrase != this.plugin.settings.passphrase);
dynamicIteration?.controlEl.toggleClass("sls-item-dirty", useDynamicIterationCount != this.plugin.settings.useDynamicIterationCount);
usePathObfuscationEl?.controlEl.toggleClass("sls-item-dirty", usePathObfuscation != this.plugin.settings.usePathObfuscation);
if (encrypt != this.plugin.settings.encrypt ||
passphrase != this.plugin.settings.passphrase ||
useDynamicIterationCount != this.plugin.settings.useDynamicIterationCount ||
usePathObfuscation != this.plugin.settings.usePathObfuscation) {
applyE2EButtons.settingEl.removeClass("sls-setting-hidden");
} else {
applyE2EButtons.settingEl.addClass("sls-setting-hidden");
}
} else {
passphraseSetting.settingEl.addClass("sls-setting-hidden");
dynamicIteration.settingEl.addClass("sls-setting-hidden");
usePathObfuscationEl.settingEl.addClass("sls-setting-hidden");
}
if (encrypt != this.plugin.settings.encrypt ||
passphrase != this.plugin.settings.passphrase ||
useDynamicIterationCount != this.plugin.settings.useDynamicIterationCount ||
usePathObfuscation != this.plugin.settings.usePathObfuscation) {
applyE2EButtons.settingEl.removeClass("sls-setting-hidden");
} else {
applyE2EButtons.settingEl.addClass("sls-setting-hidden");
}
}
@@ -649,6 +728,7 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
this.plugin.settings.passphrase = passphrase;
this.plugin.settings.useDynamicIterationCount = useDynamicIterationCount;
this.plugin.settings.usePathObfuscation = usePathObfuscation;
this.plugin.settings.isConfigured = true;
await this.plugin.saveSettings();
updateE2EControls();
if (sendToServer) {
@@ -659,7 +739,7 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
}
};
const rebuildDB = async (method: "localOnly" | "remoteOnly" | "rebuildBothByThisDevice") => {
const rebuildDB = async (method: "localOnly" | "remoteOnly" | "rebuildBothByThisDevice" | "localOnlyWithChunks") => {
if (encrypt && passphrase == "") {
Logger("If you enable encryption, you have to set the passphrase", LOG_LEVEL_NOTICE);
return;
@@ -677,61 +757,39 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
this.plugin.settings.passphrase = passphrase;
this.plugin.settings.useDynamicIterationCount = useDynamicIterationCount;
this.plugin.settings.usePathObfuscation = usePathObfuscation;
this.plugin.settings.isConfigured = true;
Logger("All synchronization have been temporarily disabled. Please enable them after the fetching, if you need them.", LOG_LEVEL_NOTICE)
await this.plugin.saveSettings();
updateE2EControls();
applyDisplayEnabled();
// @ts-ignore
this.plugin.app.setting.close();
this.closeSetting();
await delay(2000);
await performRebuildDB(this.plugin, method);
}
let rebuildRemote = false;
new Setting(containerRemoteDatabaseEl)
.setName("")
.setClass("wizardOnly")
.addButton((button) =>
button
.setButtonText("Next")
.setClass("mod-cta")
.setCta()
.setDisabled(false)
.onClick(() => {
if (!this.plugin.settings.encrypt) {
this.plugin.settings.passphrase = "";
}
if (isCloudantURI(this.plugin.settings.couchDB_URI)) {
this.plugin.settings.customChunkSize = 0;
// this.plugin.settings.customChunkSize = 0;
this.plugin.settings = { ...this.plugin.settings, ...PREFERRED_SETTING_CLOUDANT };
} else {
this.plugin.settings.customChunkSize = 100;
this.plugin.settings = { ...this.plugin.settings, ...PREFERRED_SETTING_SELF_HOSTED };
}
rebuildRemote = false;
changeDisplay("30")
})
);
new Setting(containerRemoteDatabaseEl)
.setName("")
.setClass("wizardOnly")
.addButton((button) =>
button
.setButtonText("Discard existing database and proceed")
.setDisabled(false)
.setWarning()
.onClick(() => {
if (!this.plugin.settings.encrypt) {
this.plugin.settings.passphrase = "";
}
if (isCloudantURI(this.plugin.settings.couchDB_URI)) {
this.plugin.settings.customChunkSize = 0;
} else {
this.plugin.settings.customChunkSize = 100;
}
rebuildRemote = true;
changeDisplay("30")
})
);
)
;
addScreenElement("0", containerRemoteDatabaseEl);
const containerGeneralSettingsEl = containerEl.createDiv();
@@ -741,14 +799,35 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
new Setting(containerGeneralSettingsEl)
.setName("Show status inside the editor")
.setDesc("")
.setDesc("Reflected after reboot")
.addToggle((toggle) =>
toggle.setValue(this.plugin.settings.showStatusOnEditor).onChange(async (value) => {
this.plugin.settings.showStatusOnEditor = value;
await this.plugin.saveSettings();
this.display();
})
);
if (this.plugin.settings.showStatusOnEditor) {
new Setting(containerGeneralSettingsEl)
.setName("Show status as icons only")
.setDesc("")
.addToggle((toggle) =>
toggle.setValue(this.plugin.settings.showOnlyIconsOnEditor).onChange(async (value) => {
this.plugin.settings.showOnlyIconsOnEditor = value;
await this.plugin.saveSettings();
})
);
}
new Setting(containerGeneralSettingsEl)
.setName("Show status on the status bar")
.setDesc("Reflected after reboot.")
.addToggle((toggle) =>
toggle.setValue(this.plugin.settings.showStatusOnStatusbar).onChange(async (value) => {
this.plugin.settings.showStatusOnStatusbar = value;
await this.plugin.saveSettings();
this.display();
})
);
containerGeneralSettingsEl.createEl("h4", { text: "Logging" });
new Setting(containerGeneralSettingsEl)
.setName("Show only notifications")
@@ -807,6 +886,53 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
text.inputEl.setAttribute("type", "number");
});
containerGeneralSettingsEl.createEl("h4", { text: "Share settings via markdown" });
let settingSyncFile = this.plugin.settings.settingSyncFile;
let buttonApplyFilename: ButtonComponent;
new Setting(containerGeneralSettingsEl)
.setName("Filename")
.setDesc("If you set this, all settings are saved in a markdown file. You will also be notified when new settings were arrived. You can set different files by the platform.")
.addText((text) => {
text.setPlaceholder("livesync/setting.md")
.setValue(settingSyncFile)
.onChange((value) => {
settingSyncFile = value;
if (settingSyncFile == this.plugin.settings.settingSyncFile) {
buttonApplyFilename.removeCta()
buttonApplyFilename.setDisabled(true);
} else {
buttonApplyFilename.setCta()
buttonApplyFilename.setDisabled(false);
}
})
}).addButton(button => {
button.setButtonText("Apply")
.onClick(async () => {
this.plugin.settings.settingSyncFile = settingSyncFile;
await this.plugin.saveSettings();
this.display();
})
buttonApplyFilename = button;
})
new Setting(containerGeneralSettingsEl)
.setName("Write credentials in the file")
.setDesc("(Not recommended) If set, credentials will be stored in the file.")
.addToggle(toggle => {
toggle.setValue(this.plugin.settings.writeCredentialsForSettingSync)
.onChange(async (value) => {
this.plugin.settings.writeCredentialsForSettingSync = value;
await this.plugin.saveSettings();
})
});
new Setting(containerGeneralSettingsEl)
.setName("Notify all setting files")
.addToggle(toggle => {
toggle.setValue(this.plugin.settings.notifyAllSettingSyncFile)
.onChange(async (value) => {
this.plugin.settings.notifyAllSettingSyncFile = value;
await this.plugin.saveSettings();
})
});
containerGeneralSettingsEl.createEl("h4", { text: "Advanced Confidentiality" });
@@ -936,25 +1062,20 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
...presetAllDisabled
}
}
this.plugin.saveSettings();
await this.plugin.saveSettings();
this.display();
await this.plugin.realizeSettingSyncMode();
if (inWizard) {
// @ts-ignore
this.plugin.app.setting.close()
await this.plugin.resetLocalDatabase();
await this.plugin.initializeDatabase(true);
if (rebuildRemote) {
await this.plugin.markRemoteLocked();
await this.plugin.tryResetRemoteDatabase();
await this.plugin.markRemoteLocked();
await this.plugin.markRemoteResolved();
this.closeSetting();
if (!this.plugin.settings.isConfigured) {
this.plugin.settings.isConfigured = true;
await this.plugin.saveSettings();
await rebuildDB("localOnly");
Logger("All done! Please set up subsequent devices with 'Copy current settings as a new setup URI' and 'Use the copied setup URI'.", LOG_LEVEL_NOTICE);
await this.plugin.addOnSetup.command_copySetupURI();
} else {
this.askReload();
}
await this.plugin.replicate(true);
Logger("All done! Please set up subsequent devices with 'Copy setup URI' and 'Open setup URI'.", LOG_LEVEL_NOTICE);
// @ts-ignore
this.plugin.app.commands.executeCommandById("obsidian-livesync:livesync-copysetupuri")
}
})
);
@@ -1104,7 +1225,7 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
);
new Setting(containerSyncSettingEl)
.setName("Postpone resolution of unopened files")
.setName("Postpone resolution of inactive files")
.setClass("wizardHidden")
.addToggle((toggle) =>
toggle.setValue(this.plugin.settings.checkConflictOnlyOnOpen).onChange(async (value) => {
@@ -1112,6 +1233,15 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
await this.plugin.saveSettings();
})
);
new Setting(containerSyncSettingEl)
.setName("Postpone manual resolution of inactive files")
.setClass("wizardHidden")
.addToggle((toggle) =>
toggle.setValue(this.plugin.settings.showMergeDialogOnlyOnActive).onChange(async (value) => {
this.plugin.settings.showMergeDialogOnlyOnActive = value;
await this.plugin.saveSettings();
})
);
containerSyncSettingEl.createEl("h4", { text: "Compatibility" }).addClass("wizardHidden");
new Setting(containerSyncSettingEl)
.setName("Always resolve conflict manually")
@@ -1164,24 +1294,21 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
.addButton((button) => {
button.setButtonText("Merge")
.onClick(async () => {
// @ts-ignore
this.plugin.app.setting.close()
this.closeSetting()
await this.plugin.addOnSetup.configureHiddenFileSync("MERGE");
})
})
.addButton((button) => {
button.setButtonText("Fetch")
.onClick(async () => {
// @ts-ignore
this.plugin.app.setting.close()
this.closeSetting()
await this.plugin.addOnSetup.configureHiddenFileSync("FETCH");
})
})
.addButton((button) => {
button.setButtonText("Overwrite")
.onClick(async () => {
// @ts-ignore
this.plugin.app.setting.close()
this.closeSetting()
await this.plugin.addOnSetup.configureHiddenFileSync("OVERWRITE");
})
});
@@ -1215,9 +1342,9 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
});
text.inputEl.setAttribute("type", "number");
});
let skipPatternTextArea: TextAreaComponent = null;
let skipPatternTextArea: TextAreaComponent;
const defaultSkipPattern = "\\/node_modules\\/, \\/\\.git\\/, \\/obsidian-livesync\\/";
const defaultSkipPatternXPlat = defaultSkipPattern + ",\\/workspace$ ,\\/workspace.json$";
const defaultSkipPatternXPlat = defaultSkipPattern + ",\\/workspace$ ,\\/workspace.json$,\\/workspace-mobile.json$";
new Setting(containerSyncSettingEl)
.setName("Folders and files to ignore")
.setDesc(
@@ -1301,11 +1428,35 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
});
containerSyncSettingEl.createEl("h4", {
text: sanitizeHTMLToDom(`Synchronization target filters`),
text: sanitizeHTMLToDom(`Targets`),
}).addClass("wizardHidden");
new Setting(containerSyncSettingEl)
.setName("Regular expression to ignore files")
.setDesc("If this is set, any changes to local and remote files that match this will be skipped.")
.setName("Synchronising files")
.setDesc("(RegExp) Empty to sync all files. set filter as a regular expression to limit synchronising files.")
.setClass("wizardHidden")
.addTextArea((text) => {
text
.setValue(this.plugin.settings.syncOnlyRegEx)
.setPlaceholder("\\.md$|\\.txt")
.onChange(async (value) => {
let isValidRegExp = false;
try {
new RegExp(value);
isValidRegExp = true;
} catch (_) {
// NO OP.
}
if (isValidRegExp || value.trim() == "") {
this.plugin.settings.syncOnlyRegEx = value;
await this.plugin.saveSettings();
}
})
return text;
}
);
new Setting(containerSyncSettingEl)
.setName("Non-Synchronising files")
.setDesc("(RegExp) If this is set, any changes to local and remote files that match this will be skipped.")
.setClass("wizardHidden")
.addTextArea((text) => {
text
@@ -1328,29 +1479,22 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
}
);
new Setting(containerSyncSettingEl)
.setName("Regular expression for restricting synchronization targets")
.setDesc("If this is set, changes to local and remote files that only match this will be processed.")
.setName("Maximum file size")
.setDesc("(MB) If this is set, changes to local and remote files that are larger than this will be skipped. If the file becomes smaller again, a newer one will be used.")
.setClass("wizardHidden")
.addTextArea((text) => {
text
.setValue(this.plugin.settings.syncOnlyRegEx)
.setPlaceholder("\\.md$|\\.txt")
.addText((text) => {
text.setPlaceholder("")
.setValue(this.plugin.settings.syncMaxSizeInMB + "")
.onChange(async (value) => {
let isValidRegExp = false;
try {
new RegExp(value);
isValidRegExp = true;
} catch (_) {
// NO OP.
let v = Number(value);
if (isNaN(v) || v < 1) {
v = 0;
}
if (isValidRegExp || value.trim() == "") {
this.plugin.settings.syncOnlyRegEx = value;
await this.plugin.saveSettings();
}
})
return text;
}
);
this.plugin.settings.syncMaxSizeInMB = v;
await this.plugin.saveSettings();
});
text.inputEl.setAttribute("type", "number");
});
new Setting(containerSyncSettingEl)
.setName("(Beta) Use ignore files")
.setDesc("If this is set, changes to local files which are matched by the ignore files will be skipped. Remote changes are determined using local ignore files.")
@@ -1391,15 +1535,15 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
}).addClass("wizardHidden");
new Setting(containerSyncSettingEl)
.setName("Batch size")
.setDesc("Number of change feed items to process at a time. Defaults to 50.")
.setDesc("Number of change feed items to process at a time. Defaults to 50. Minimum is 2.")
.setClass("wizardHidden")
.addText((text) => {
text.setPlaceholder("")
.setValue(this.plugin.settings.batch_size + "")
.onChange(async (value) => {
let v = Number(value);
if (isNaN(v) || v < 10) {
v = 10;
if (isNaN(v) || v < 2) {
v = 2;
}
this.plugin.settings.batch_size = v;
await this.plugin.saveSettings();
@@ -1409,15 +1553,15 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
new Setting(containerSyncSettingEl)
.setName("Batch limit")
.setDesc("Number of batches to process at a time. Defaults to 40. This along with batch size controls how many docs are kept in memory at a time.")
.setDesc("Number of batches to process at a time. Defaults to 40. Minimum is 2. This along with batch size controls how many docs are kept in memory at a time.")
.setClass("wizardHidden")
.addText((text) => {
text.setPlaceholder("")
.setValue(this.plugin.settings.batches_limit + "")
.onChange(async (value) => {
let v = Number(value);
if (isNaN(v) || v < 10) {
v = 10;
if (isNaN(v) || v < 2) {
v = 2;
}
this.plugin.settings.batches_limit = v;
await this.plugin.saveSettings();
@@ -1508,14 +1652,17 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
const pluginConfig = JSON.parse(JSON.stringify(this.plugin.settings)) as ObsidianLiveSyncSettings;
pluginConfig.couchDB_DBNAME = REDACTED;
pluginConfig.couchDB_PASSWORD = REDACTED;
pluginConfig.couchDB_URI = isCloudantURI(pluginConfig.couchDB_URI) ? "cloudant" : "self-hosted";
const scheme = pluginConfig.couchDB_URI.startsWith("http:") ? "(HTTP)" : (pluginConfig.couchDB_URI.startsWith("https:")) ? "(HTTPS)" : ""
pluginConfig.couchDB_URI = isCloudantURI(pluginConfig.couchDB_URI) ? "cloudant" : `self-hosted${scheme}`;
pluginConfig.couchDB_USER = REDACTED;
pluginConfig.passphrase = REDACTED;
pluginConfig.encryptedPassphrase = REDACTED;
pluginConfig.encryptedCouchDBConnection = REDACTED;
pluginConfig.pluginSyncExtendedSetting = {};
const msgConfig = `----remote config----
const obsidianInfo = navigator.userAgent;
const msgConfig = `---- Obsidian info ----
${obsidianInfo}
---- remote config ----
${stringifyYaml(responseConfig)}
---- Plug-in config ---
version:${manifestVersion}
@@ -1553,43 +1700,113 @@ ${stringifyYaml(pluginConfig)}`;
c.addClass("op-warn");
}
}
new Setting(containerHatchEl)
.setName("Back to non-configured")
.addButton((button) =>
button
.setButtonText("Back")
.setDisabled(false)
.onClick(async () => {
this.plugin.settings.isConfigured = false;
await this.plugin.saveSettings();
this.askReload();
}));
const hatchWarn = containerHatchEl.createEl("div", { text: `To stop the boot up sequence for fixing problems on databases, you can put redflag.md on top of your vault (Rebooting obsidian is required).` });
hatchWarn.addClass("op-warn-info");
const addResult = (path: string, file: TFile | false, fileOnDB: LoadedEntry | false) => {
resultArea.appendChild(resultArea.createEl("div", {}, el => {
el.appendChild(el.createEl("h6", { text: path }));
el.appendChild(el.createEl("div", {}, infoGroupEl => {
infoGroupEl.appendChild(infoGroupEl.createEl("div", { text: `Storage : Modified: ${!file ? `Missing:` : `${new Date(file.stat.mtime).toLocaleString()}, Size:${file.stat.size}`}` }))
infoGroupEl.appendChild(infoGroupEl.createEl("div", { text: `Database: Modified: ${!fileOnDB ? `Missing:` : `${new Date(fileOnDB.mtime).toLocaleString()}, Size:${fileOnDB.size}`}` }))
}));
if (fileOnDB && file) {
el.appendChild(el.createEl("button", { text: "Show history" }, buttonEl => {
buttonEl.onClickEvent(() => {
this.plugin.showHistory(file, fileOnDB._id);
})
}))
}
if (file) {
el.appendChild(el.createEl("button", { text: "Storage -> Database" }, buttonEl => {
buttonEl.onClickEvent(() => {
this.plugin.updateIntoDB(file, undefined, true);
el.remove();
})
}))
}
if (fileOnDB) {
el.appendChild(el.createEl("button", { text: "Database -> Storage" }, buttonEl => {
buttonEl.onClickEvent(() => {
this.plugin.pullFile(this.plugin.getPath(fileOnDB), [], true, undefined, false);
el.remove();
})
}))
}
return el;
}))
}
const checkBetweenStorageAndDatabase = async (file: TFile, fileOnDB: LoadedEntry) => {
const dataContent = readAsBlob(fileOnDB);
const content = createBlob(await this.plugin.vaultAccess.vaultReadAuto(file))
if (await isDocContentSame(content, dataContent)) {
Logger(`Compare: SAME: ${file.path}`)
} else {
Logger(`Compare: CONTENT IS NOT MATCHED! ${file.path}`, LOG_LEVEL_NOTICE);
addResult(file.path, file, fileOnDB)
}
}
new Setting(containerHatchEl)
.setName("Verify and repair all files")
.setDesc("Verify and repair all files and update database without restoring")
.setDesc("Compare the content of files between on local database and storage. If not matched, you will asked which one want to keep.")
.addButton((button) =>
button
.setButtonText("Verify and repair")
.setButtonText("Verify all")
.setDisabled(false)
.setWarning()
.onClick(async () => {
const semaphore = Semaphore(10);
const files = this.app.vault.getFiles();
const documents = [] as FilePathWithPrefix[];
const adn = this.plugin.localDatabase.findAllNormalDocs()
for await (const i of adn) documents.push(this.plugin.getPath(i));
const allPaths = [...new Set([...documents, ...files.map(e => e.path as FilePathWithPrefix)])];
let i = 0;
const processes = files.map(e => (async (file) => {
const releaser = await semaphore.acquire(1, "verifyAndRepair");
for (const path of allPaths) {
i++;
Logger(`${i}/${files.length}\n${path}`, LOG_LEVEL_NOTICE, "verify");
if (shouldBeIgnored(path)) continue;
const abstractFile = this.plugin.vaultAccess.getAbstractFileByPath(path);
const fileOnStorage = abstractFile instanceof TFile ? abstractFile : false;
if (!await this.plugin.isTargetFile(path)) continue;
try {
Logger(`UPDATE DATABASE ${file.path}`);
await this.plugin.updateIntoDB(file, false, null, true);
i++;
Logger(`${i}/${files.length}\n${file.path}`, LOG_LEVEL_NOTICE, "verify");
if (fileOnStorage && this.plugin.isFileSizeExceeded(fileOnStorage.stat.size)) continue;
const fileOnDB = await this.plugin.localDatabase.getDBEntry(path);
if (fileOnDB && this.plugin.isFileSizeExceeded(fileOnDB.size)) continue;
} catch (ex) {
i++;
Logger(`Error while verifyAndRepair`, LOG_LEVEL_NOTICE);
Logger(ex);
} finally {
releaser();
if (!fileOnDB && fileOnStorage) {
Logger(`Compare: Not found on the local database: ${path}`, LOG_LEVEL_NOTICE);
addResult(path, fileOnStorage, false)
continue;
}
if (fileOnDB && !fileOnStorage) {
Logger(`Compare: Not found on the storage: ${path}`, LOG_LEVEL_NOTICE);
addResult(path, false, fileOnDB)
continue;
}
if (fileOnStorage && fileOnDB) {
await checkBetweenStorageAndDatabase(fileOnStorage, fileOnDB)
}
}
)(e));
await Promise.all(processes);
Logger("done", LOG_LEVEL_NOTICE, "verify");
})
);
const resultArea = containerHatchEl.createDiv({ text: "" });
new Setting(containerHatchEl)
.setName("Check and convert non-path-obfuscated files")
.setDesc("")
@@ -1612,6 +1829,7 @@ ${stringifyYaml(pluginConfig)}`;
//Prepare converted data
newDoc._id = idEncoded;
newDoc.path = docName as FilePathWithPrefix;
// @ts-ignore
delete newDoc._rev;
try {
const obfuscatedDoc = await this.plugin.localDatabase.getRaw(idEncoded, { revs_info: true });
@@ -1632,12 +1850,12 @@ ${stringifyYaml(pluginConfig)}`;
if ((await this.plugin.localDatabase.putRaw(doc)).ok) {
Logger(`Old ${docName} has been deleted`, LOG_LEVEL_NOTICE);
}
await this.plugin.showIfConflicted(docName as FilePathWithPrefix);
await this.plugin.queueConflictCheck(docName as FilePathWithPrefix);
} else {
Logger(`Converting ${docName} Failed!`, LOG_LEVEL_NOTICE);
Logger(ret, LOG_LEVEL_VERBOSE);
}
} catch (ex) {
} catch (ex: any) {
if (ex?.status == 404) {
// We can perform this safely
if ((await this.plugin.localDatabase.putRaw(newDoc)).ok) {
@@ -1680,12 +1898,7 @@ ${stringifyYaml(pluginConfig)}`;
toggle.setValue(this.plugin.settings.suspendFileWatching).onChange(async (value) => {
this.plugin.settings.suspendFileWatching = value;
await this.plugin.saveSettings();
scheduleTask("configReload", 250, async () => {
if (await askYesNo(this.app, "Do you want to restart and reload Obsidian now?") == "yes") {
// @ts-ignore
this.app.commands.executeCommandById("app:reload")
}
})
this.askReload();
})
);
new Setting(containerHatchEl)
@@ -1695,12 +1908,7 @@ ${stringifyYaml(pluginConfig)}`;
toggle.setValue(this.plugin.settings.suspendParseReplicationResult).onChange(async (value) => {
this.plugin.settings.suspendParseReplicationResult = value;
await this.plugin.saveSettings();
scheduleTask("configReload", 250, async () => {
if (await askYesNo(this.app, "Do you want to restart and reload Obsidian now?") == "yes") {
// @ts-ignore
this.app.commands.executeCommandById("app:reload")
}
})
this.askReload();
})
);
new Setting(containerHatchEl)
@@ -1713,15 +1921,15 @@ ${stringifyYaml(pluginConfig)}`;
})
);
new Setting(containerHatchEl)
.setName("Do not pace synchronization")
.setDesc("If this toggle enabled, synchronisation will not be paced by queued entries. If synchronisation has been deadlocked, please make this enabled once.")
.addToggle((toggle) =>
toggle.setValue(this.plugin.settings.doNotPaceReplication).onChange(async (value) => {
this.plugin.settings.doNotPaceReplication = value;
await this.plugin.saveSettings();
})
);
// new Setting(containerHatchEl)
// .setName("Do not pace synchronization")
// .setDesc("If this toggle enabled, synchronisation will not be paced by queued entries. If synchronisation has been deadlocked, please make this enabled once.")
// .addToggle((toggle) =>
// toggle.setValue(this.plugin.settings.doNotPaceReplication).onChange(async (value) => {
// this.plugin.settings.doNotPaceReplication = value;
// await this.plugin.saveSettings();
// })
// );
containerHatchEl.createEl("h4", {
text: sanitizeHTMLToDom(`Compatibility`),
cls: "wizardHidden"
@@ -1762,7 +1970,7 @@ ${stringifyYaml(pluginConfig)}`;
new Setting(containerHatchEl)
.setName("Use an old adapter for compatibility")
.setDesc("This option is not compatible with a database made by older versions. Changing this configuration will fetch the remote database again.")
.setDesc("Before v0.17.16, we used an old adapter for the local database. Now the new adapter is preferred. However, it needs local database rebuilding. Please disable this toggle when you have enough time. If leave it enabled, also while fetching from the remote database, you will be asked to disable this.")
.setClass("wizardHidden")
.addToggle((toggle) =>
toggle.setValue(!this.plugin.settings.useIndexedDBAdapter).onChange(async (value) => {
@@ -1785,7 +1993,7 @@ ${stringifyYaml(pluginConfig)}`;
let newDatabaseName = this.plugin.settings.additionalSuffixOfDatabaseName + "";
new Setting(containerHatchEl)
.setName("Database suffix")
.setDesc("LiveSync could not treat multiple vaults which have same name, please add some suffix from here.")
.setDesc("LiveSync could not handle multiple vaults which have same name without different prefix, This should be automatically configured.")
.addText((text) => {
text.setPlaceholder("")
.setValue(newDatabaseName)
@@ -1813,7 +2021,7 @@ ${stringifyYaml(pluginConfig)}`;
.setClass("wizardHidden")
.addDropdown((dropdown) =>
dropdown
.addOptions({ "": "Old Algorithm", "xxhash32": "xxhash32 (Fast)", "xxhash64": "xxhash64 (Fastest)" } as Record<HashAlgorithm, string>)
.addOptions({ "": "Old Algorithm", "xxhash32": "xxhash32 (Fast)", "xxhash64": "xxhash64 (Fastest)", "sha1": "Fallback (Without WebAssembly)" } as Record<HashAlgorithm, string>)
.setValue(this.plugin.settings.hashAlg)
.onChange(async (value) => {
this.plugin.settings.hashAlg = value as HashAlgorithm;
@@ -1832,15 +2040,6 @@ ${stringifyYaml(pluginConfig)}`;
await this.plugin.saveSettings();
})
);
new Setting(containerHatchEl)
.setName("Use binary and encryption version 1")
.setDesc("Use the previous data format for other products which shares the remote database.")
.addToggle((toggle) =>
toggle.setValue(this.plugin.settings.useV1).onChange(async (value) => {
this.plugin.settings.useV1 = value;
await this.plugin.saveSettings();
})
);
addScreenElement("50", containerHatchEl);
@@ -1979,6 +2178,19 @@ ${stringifyYaml(pluginConfig)}`;
})
)
new Setting(containerMaintenanceEl)
.setName("Fetch rebuilt DB (Save local documents before)")
.setDesc("Restore or reconstruct local database from remote database but use local chunks.")
.addButton((button) =>
button
.setButtonText("Save and Fetch")
.setWarning()
.setDisabled(false)
.onClick(async () => {
await rebuildDB("localOnlyWithChunks");
})
)
new Setting(containerMaintenanceEl)
.setName("Discard local database to reset or uninstall Self-hosted LiveSync")
.addButton((button) =>
@@ -2008,8 +2220,7 @@ ${stringifyYaml(pluginConfig)}`;
.setDisabled(false)
.setWarning()
.onClick(async () => {
// @ts-ignore
this.plugin.app.setting.close()
this.closeSetting()
await this.plugin.dbGC();
})
);
@@ -2031,7 +2242,7 @@ ${stringifyYaml(pluginConfig)}`;
applyDisplayEnabled();
if (this.selectedScreen == "") {
if (lastVersion != this.plugin.settings.lastReadUpdates) {
if (JSON.stringify(this.plugin.settings) != JSON.stringify(DEFAULT_SETTINGS)) {
if (this.plugin.settings.isConfigured) {
changeDisplay("100");
} else {
changeDisplay("110")

6
src/PluginCombo.svelte
View File

@@ -207,21 +207,21 @@
const local = list.find((e) => e.term == thisTerm);
const selectedItem = list.find((e) => e.term == selected);
if (selectedItem && (await applyData(selectedItem))) {
scheduleTask("update-plugin-list", 250, () => addOn.updatePluginList(true, local.documentPath));
addOn.updatePluginList(true, local?.documentPath);
}
}
async function compareSelected() {
const local = list.find((e) => e.term == thisTerm);
const selectedItem = list.find((e) => e.term == selected);
if (local && selectedItem && (await compareData(local, selectedItem))) {
scheduleTask("update-plugin-list", 250, () => addOn.updatePluginList(true, local.documentPath));
addOn.updatePluginList(true, local.documentPath);
}
}
async function deleteSelected() {
const selectedItem = list.find((e) => e.term == selected);
// const deletedPath = selectedItem.documentPath;
if (selectedItem && (await deleteData(selectedItem))) {
scheduleTask("update-plugin-list", 250, () => addOn.reloadPluginList(true));
addOn.reloadPluginList(true);
}
}
async function duplicateItem() {

46
src/PluginPane.svelte
View File

@@ -27,8 +27,10 @@
async function requestReload() {
await addOn.reloadPluginList(true);
}
let allTerms = [] as string[];
pluginList.subscribe((e) => {
list = e;
allTerms = unique(list.map((e) => e.term));
});
pluginIsEnumerating.subscribe((e) => {
loading = e;
@@ -172,7 +174,7 @@
.filter((e) => `${e.category}/${e.name}` == key)
.map((e) => e.files)
.flat()
.map((e) => e.filename)
.map((e) => e.filename),
);
automaticList.set(key, mode);
automaticListDisp = automaticList;
@@ -218,6 +220,16 @@
.sort((a, b) => (a.displayName ?? a.name).localeCompare(b.displayName ?? b.name))
.reduce((p, c) => ({ ...p, [c.category]: unique(c.category in p ? [...p[c.category], c.displayName ?? c.name] : [c.displayName ?? c.name]) }), {} as Record<string, string[]>);
}
let deleteTerm = "";
async function deleteAllItems(term: string) {
const deleteItems = list.filter((e) => e.term == term);
for (const item of deleteItems) {
await deleteData(item);
}
addOn.reloadPluginList(true);
}
</script>
<div>
@@ -322,6 +334,29 @@
</div>
{/if}
</div>
{#if isMaintenanceMode}
<div class="list">
<div>
<h3>Maintenance Commands</h3>
<div class="maintenancerow">
<label for="">Delete All of </label>
<select bind:value={deleteTerm}>
{#each allTerms as term}
<option value={term}>{term}</option>
{/each}
</select>
<button
class="status"
on:click={(evt) => {
deleteAllItems(deleteTerm);
}}
>
🗑️
</button>
</div>
</div>
</div>
{/if}
<div class="buttons">
<label><span>Hide not applicable items</span><input type="checkbox" bind:checked={hideEven} /></label>
</div>
@@ -423,4 +458,13 @@
align-items: center;
min-height: 3em;
}
.maintenancerow {
display: flex;
justify-content: flex-end;
align-items: center;
}
.maintenancerow label {
margin-right: 0.5em;
margin-left: 0.5em;
}
</style>

196
src/SerializedFileAccess.ts Normal file
View File

@@ -0,0 +1,196 @@
import { type App, TFile, type DataWriteOptions, TFolder, TAbstractFile } from "./deps";
import { serialized } from "./lib/src/lock";
import { Logger } from "./lib/src/logger";
import { isPlainText } from "./lib/src/path";
import type { FilePath } from "./lib/src/types";
import { createBinaryBlob, isDocContentSame } from "./lib/src/utils";
import type { InternalFileInfo } from "./types";
import { markChangesAreSame } from "./utils";
function getFileLockKey(file: TFile | TFolder | string) {
return `fl:${typeof (file) == "string" ? file : file.path}`;
}
function toArrayBuffer(arr: Uint8Array | ArrayBuffer | DataView): ArrayBufferLike {
if (arr instanceof Uint8Array) {
return arr.buffer;
}
if (arr instanceof DataView) {
return arr.buffer;
}
return arr;
}
async function processReadFile<T>(file: TFile | TFolder | string, proc: () => Promise<T>) {
const ret = await serialized(getFileLockKey(file), () => proc());
return ret;
}
async function processWriteFile<T>(file: TFile | TFolder | string, proc: () => Promise<T>) {
const ret = await serialized(getFileLockKey(file), () => proc());
return ret;
}
export class SerializedFileAccess {
app: App
constructor(app: App) {
this.app = app;
}
async adapterStat(file: TFile | string) {
const path = file instanceof TFile ? file.path : file;
return await processReadFile(file, () => this.app.vault.adapter.stat(path));
}
async adapterExists(file: TFile | string) {
const path = file instanceof TFile ? file.path : file;
return await processReadFile(file, () => this.app.vault.adapter.exists(path));
}
async adapterRemove(file: TFile | string) {
const path = file instanceof TFile ? file.path : file;
return await processReadFile(file, () => this.app.vault.adapter.remove(path));
}
async adapterRead(file: TFile | string) {
const path = file instanceof TFile ? file.path : file;
return await processReadFile(file, () => this.app.vault.adapter.read(path));
}
async adapterReadBinary(file: TFile | string) {
const path = file instanceof TFile ? file.path : file;
return await processReadFile(file, () => this.app.vault.adapter.readBinary(path));
}
async adapterReadAuto(file: TFile | string) {
const path = file instanceof TFile ? file.path : file;
if (isPlainText(path)) return await processReadFile(file, () => this.app.vault.adapter.read(path));
return await processReadFile(file, () => this.app.vault.adapter.readBinary(path));
}
async adapterWrite(file: TFile | string, data: string | ArrayBuffer | Uint8Array, options?: DataWriteOptions) {
const path = file instanceof TFile ? file.path : file;
if (typeof (data) === "string") {
return await processWriteFile(file, () => this.app.vault.adapter.write(path, data, options));
} else {
return await processWriteFile(file, () => this.app.vault.adapter.writeBinary(path, toArrayBuffer(data), options));
}
}
async vaultCacheRead(file: TFile) {
return await processReadFile(file, () => this.app.vault.cachedRead(file));
}
async vaultRead(file: TFile) {
return await processReadFile(file, () => this.app.vault.read(file));
}
async vaultReadBinary(file: TFile) {
return await processReadFile(file, () => this.app.vault.readBinary(file));
}
async vaultReadAuto(file: TFile) {
const path = file.path;
if (isPlainText(path)) return await processReadFile(file, () => this.app.vault.read(file));
return await processReadFile(file, () => this.app.vault.readBinary(file));
}
async vaultModify(file: TFile, data: string | ArrayBuffer | Uint8Array, options?: DataWriteOptions) {
if (typeof (data) === "string") {
return await processWriteFile(file, async () => {
const oldData = await this.app.vault.read(file);
if (data === oldData) {
if (options && options.mtime) markChangesAreSame(file, file.stat.mtime, options.mtime);
return false
}
await this.app.vault.modify(file, data, options)
return true;
}
);
} else {
return await processWriteFile(file, async () => {
const oldData = await this.app.vault.readBinary(file);
if (await isDocContentSame(createBinaryBlob(oldData), createBinaryBlob(data))) {
if (options && options.mtime) markChangesAreSame(file, file.stat.mtime, options.mtime);
return false;
}
await this.app.vault.modifyBinary(file, toArrayBuffer(data), options)
return true;
});
}
}
async vaultCreate(path: string, data: string | ArrayBuffer | Uint8Array, options?: DataWriteOptions): Promise<TFile> {
if (typeof (data) === "string") {
return await processWriteFile(path, () => this.app.vault.create(path, data, options));
} else {
return await processWriteFile(path, () => this.app.vault.createBinary(path, toArrayBuffer(data), options));
}
}
trigger(name: string, ...data: any[]) {
return this.app.vault.trigger(name, ...data);
}
async adapterAppend(normalizedPath: string, data: string, options?: DataWriteOptions) {
return await this.app.vault.adapter.append(normalizedPath, data, options)
}
async delete(file: TFile | TFolder, force = false) {
return await processWriteFile(file, () => this.app.vault.delete(file, force));
}
async trash(file: TFile | TFolder, force = false) {
return await processWriteFile(file, () => this.app.vault.trash(file, force));
}
getAbstractFileByPath(path: FilePath | string): TAbstractFile | null {
// Disabled temporary.
return this.app.vault.getAbstractFileByPath(path);
// // Hidden API but so useful.
// // @ts-ignore
// if ("getAbstractFileByPathInsensitive" in app.vault && (app.vault.adapter?.insensitive ?? false)) {
// // @ts-ignore
// return app.vault.getAbstractFileByPathInsensitive(path);
// } else {
// return app.vault.getAbstractFileByPath(path);
// }
}
getFiles() {
return this.app.vault.getFiles();
}
async ensureDirectory(fullPath: string) {
const pathElements = fullPath.split("/");
pathElements.pop();
let c = "";
for (const v of pathElements) {
c += v;
try {
await this.app.vault.adapter.mkdir(c);
} catch (ex: any) {
if (ex?.message == "Folder already exists.") {
// Skip if already exists.
} else {
Logger("Folder Create Error");
Logger(ex);
}
}
c += "/";
}
}
touchedFiles: string[] = [];
touch(file: TFile | FilePath) {
const f = file instanceof TFile ? file : this.getAbstractFileByPath(file) as TFile;
const key = `${f.path}-${f.stat.mtime}-${f.stat.size}`;
this.touchedFiles.unshift(key);
this.touchedFiles = this.touchedFiles.slice(0, 100);
}
recentlyTouched(file: TFile | InternalFileInfo) {
const key = file instanceof TFile ? `${file.path}-${file.stat.mtime}-${file.stat.size}` : `${file.path}-${file.mtime}-${file.size}`;
if (this.touchedFiles.indexOf(key) == -1) return false;
return true;
}
clearTouched() {
this.touchedFiles = [];
}
}

106
src/StorageEventManager.ts
View File

@@ -1,36 +1,37 @@
import type { SerializedFileAccess } from "./SerializedFileAccess";
import { Plugin, TAbstractFile, TFile, TFolder } from "./deps";
import { isPlainText, shouldBeIgnored } from "./lib/src/path";
import { getGlobalStore } from "./lib/src/store";
import { type FilePath, type ObsidianLiveSyncSettings } from "./lib/src/types";
import { type FileEventItem, type FileEventType, type FileInfo, type InternalFileInfo, type queueItem } from "./types";
import { recentlyTouched } from "./utils";
import { Logger } from "./lib/src/logger";
import { shouldBeIgnored } from "./lib/src/path";
import type { KeyedQueueProcessor } from "./lib/src/processor";
import { LOG_LEVEL_NOTICE, type FilePath, type ObsidianLiveSyncSettings } from "./lib/src/types";
import { delay } from "./lib/src/utils";
import { type FileEventItem, type FileEventType, type FileInfo, type InternalFileInfo } from "./types";
export abstract class StorageEventManager {
abstract fetchEvent(): FileEventItem | false;
abstract cancelRelativeEvent(item: FileEventItem): void;
abstract getQueueLength(): number;
abstract beginWatch(): void;
}
type LiveSyncForStorageEventManager = Plugin &
{
settings: ObsidianLiveSyncSettings
ignoreFiles: string[],
vaultAccess: SerializedFileAccess
} & {
isTargetFile: (file: string | TAbstractFile) => Promise<boolean>,
procFileEvent: (applyBatch?: boolean) => Promise<any>,
fileEventQueue: KeyedQueueProcessor<FileEventItem, any>,
isFileSizeExceeded: (size: number) => boolean;
};
export class StorageEventManagerObsidian extends StorageEventManager {
plugin: LiveSyncForStorageEventManager;
queuedFilesStore = getGlobalStore("queuedFiles", { queuedItems: [] as queueItem[], fileEventItems: [] as FileEventItem[] });
watchedFileEventQueue = [] as FileEventItem[];
constructor(plugin: LiveSyncForStorageEventManager) {
super();
this.plugin = plugin;
}
beginWatch() {
const plugin = this.plugin;
this.watchVaultChange = this.watchVaultChange.bind(this);
this.watchVaultCreate = this.watchVaultCreate.bind(this);
this.watchVaultDelete = this.watchVaultDelete.bind(this);
@@ -42,6 +43,7 @@ export class StorageEventManagerObsidian extends StorageEventManager {
plugin.registerEvent(plugin.app.vault.on("create", this.watchVaultCreate));
//@ts-ignore : Internal API
plugin.registerEvent(plugin.app.vault.on("raw", this.watchVaultRawEvents));
plugin.fileEventQueue.startPipeline();
}
watchVaultCreate(file: TAbstractFile, ctx?: any) {
@@ -89,7 +91,8 @@ export class StorageEventManagerObsidian extends StorageEventManager {
}
// Cache file and waiting to can be proceed.
async appendWatchEvent(params: { type: FileEventType, file: TAbstractFile | InternalFileInfo, oldPath?: string }[], ctx?: any) {
let forcePerform = false;
if (!this.plugin.settings.isConfigured) return;
if (this.plugin.settings.suspendFileWatching) return;
for (const param of params) {
if (shouldBeIgnored(param.file.path)) {
continue;
@@ -98,51 +101,31 @@ export class StorageEventManagerObsidian extends StorageEventManager {
const type = param.type;
const file = param.file;
const oldPath = param.oldPath;
const size = file instanceof TFile ? file.stat.size : (file as InternalFileInfo)?.size ?? 0;
if (this.plugin.isFileSizeExceeded(size) && (type == "CREATE" || type == "CHANGED")) {
Logger(`The storage file has been changed but exceeds the maximum size. Skipping: ${param.file.path}`, LOG_LEVEL_NOTICE);
continue;
}
if (file instanceof TFolder) continue;
if (!await this.plugin.isTargetFile(file.path)) continue;
if (this.plugin.settings.suspendFileWatching) continue;
let cache: null | string | ArrayBuffer;
// Stop cache using to prevent the corruption;
// let cache: null | string | ArrayBuffer;
// new file or something changed, cache the changes.
if (file instanceof TFile && (type == "CREATE" || type == "CHANGED")) {
if (recentlyTouched(file)) {
// Wait for a bit while to let the writer has marked `touched` at the file.
await delay(10);
if (this.plugin.vaultAccess.recentlyTouched(file)) {
continue;
}
if (!isPlainText(file.name)) {
cache = await this.plugin.app.vault.readBinary(file);
} else {
// cache = await this.app.vault.read(file);
cache = await this.plugin.app.vault.cachedRead(file);
if (!cache) cache = await this.plugin.app.vault.read(file);
}
// cache = await this.plugin.vaultAccess.vaultReadAuto(file);
// if (!isPlainText(file.name)) {
// cache = await this.plugin.vaultAccess.vaultReadBinary(file);
// } else {
// cache = await this.plugin.vaultAccess.vaultCacheRead(file);
// if (!cache) cache = await this.plugin.vaultAccess.vaultRead(file);
// }
}
if (type == "DELETE" || type == "RENAME") {
forcePerform = true;
}
if (this.plugin.settings.batchSave && !this.plugin.settings.liveSync) {
// if the latest event is the same type, omit that
// a.md MODIFY <- this should be cancelled when a.md MODIFIED
// b.md MODIFY <- this should be cancelled when b.md MODIFIED
// a.md MODIFY
// a.md CREATE
// :
let i = this.watchedFileEventQueue.length;
L1:
while (i >= 0) {
i--;
if (i < 0) break L1;
if (this.watchedFileEventQueue[i].args.file.path != file.path) {
continue L1;
}
if (this.watchedFileEventQueue[i].type != type) break L1;
this.watchedFileEventQueue.remove(this.watchedFileEventQueue[i]);
//this.queuedFilesStore.set({ queuedItems: this.queuedFiles, fileEventItems: this.watchedFileEventQueue });
this.queuedFilesStore.apply((value) => ({ ...value, fileEventItems: this.watchedFileEventQueue }));
}
}
const fileInfo = file instanceof TFile ? {
ctime: file.stat.ctime,
mtime: file.stat.mtime,
@@ -150,32 +133,17 @@ export class StorageEventManagerObsidian extends StorageEventManager {
path: file.path,
size: file.stat.size
} as FileInfo : file as InternalFileInfo;
this.watchedFileEventQueue.push({
this.plugin.fileEventQueue.enqueueWithKey(`file-${fileInfo.path}`, {
type,
args: {
file: fileInfo,
oldPath,
cache,
// cache,
ctx
},
key: atomicKey
})
}
// this.queuedFilesStore.set({ queuedItems: this.queuedFiles, fileEventItems: this.watchedFileEventQueue });
this.queuedFilesStore.apply((value) => ({ ...value, fileEventItems: this.watchedFileEventQueue }));
this.plugin.procFileEvent(forcePerform);
}
fetchEvent(): FileEventItem | false {
if (this.watchedFileEventQueue.length == 0) return false;
const item = this.watchedFileEventQueue.shift();
this.queuedFilesStore.apply((value) => ({ ...value, fileEventItems: this.watchedFileEventQueue }));
return item;
}
cancelRelativeEvent(item: FileEventItem) {
this.watchedFileEventQueue = [...this.watchedFileEventQueue].filter(e => e.key != item.key);
this.queuedFilesStore.apply((value) => ({ ...value, fileEventItems: this.watchedFileEventQueue }));
}
getQueueLength() {
return this.watchedFileEventQueue.length;
}
}

2
src/deps.ts
View File

@@ -4,7 +4,7 @@ export {
addIcon, App, debounce, Editor, FuzzySuggestModal, MarkdownRenderer, MarkdownView, Modal, Notice, Platform, Plugin, PluginSettingTab, requestUrl, sanitizeHTMLToDom, Setting, stringifyYaml, TAbstractFile, TextAreaComponent, TFile, TFolder,
parseYaml, ItemView, WorkspaceLeaf
} from "obsidian";
export type { DataWriteOptions, PluginManifest, RequestUrlParam, RequestUrlResponse, MarkdownFileInfo } from "obsidian";
export type { DataWriteOptions, PluginManifest, RequestUrlParam, RequestUrlResponse, MarkdownFileInfo, ListedFiles } from "obsidian";
import {
normalizePath as normalizePath_
} from "obsidian";

33
src/dialogs.ts
View File

@@ -7,10 +7,9 @@ import PluginPane from "./PluginPane.svelte";
export class PluginDialogModal extends Modal {
plugin: ObsidianLiveSyncPlugin;
logEl: HTMLDivElement;
component: PluginPane = null;
component: PluginPane | undefined;
isOpened() {
return this.component != null;
return this.component != undefined;
}
constructor(app: App, plugin: ObsidianLiveSyncPlugin) {
@@ -21,7 +20,7 @@ export class PluginDialogModal extends Modal {
onOpen() {
const { contentEl } = this;
this.titleEl.setText("Customization Sync (Beta2)")
if (this.component == null) {
if (!this.component) {
this.component = new PluginPane({
target: contentEl,
props: { plugin: this.plugin },
@@ -30,9 +29,9 @@ export class PluginDialogModal extends Modal {
}
onClose() {
if (this.component != null) {
if (this.component) {
this.component.$destroy();
this.component = null;
this.component = undefined;
}
}
}
@@ -94,13 +93,13 @@ export class InputStringDialog extends Modal {
}
export class PopoverSelectString extends FuzzySuggestModal<string> {
app: App;
callback: (e: string) => void = () => { };
callback: ((e: string) => void) | undefined = () => { };
getItemsFun: () => string[] = () => {
return ["yes", "no"];
}
constructor(app: App, note: string, placeholder: string | null, getItemsFun: () => string[], callback: (e: string) => void) {
constructor(app: App, note: string, placeholder: string | undefined, getItemsFun: (() => string[]) | undefined, callback: (e: string) => void) {
super(app);
this.app = app;
this.setPlaceholder((placeholder ?? "y/n) ") + note);
@@ -118,13 +117,14 @@ export class PopoverSelectString extends FuzzySuggestModal<string> {
onChooseItem(item: string, evt: MouseEvent | KeyboardEvent): void {
// debugger;
this.callback(item);
this.callback = null;
this.callback?.(item);
this.callback = undefined;
}
onClose(): void {
setTimeout(() => {
if (this.callback != null) {
if (this.callback) {
this.callback("");
this.callback = undefined;
}
}, 100);
}
@@ -136,16 +136,16 @@ export class MessageBox extends Modal {
title: string;
contentMd: string;
buttons: string[];
result: string;
result: string | false = false;
isManuallyClosed = false;
defaultAction: string | undefined;
timeout: number | undefined;
timer: ReturnType<typeof setInterval> = undefined;
timer: ReturnType<typeof setInterval> | undefined = undefined;
defaultButtonComponent: ButtonComponent | undefined;
onSubmit: (result: string | false) => void;
constructor(plugin: Plugin, title: string, contentMd: string, buttons: string[], defaultAction: (typeof buttons)[number], timeout: number, onSubmit: (result: (typeof buttons)[number] | false) => void) {
constructor(plugin: Plugin, title: string, contentMd: string, buttons: string[], defaultAction: (typeof buttons)[number], timeout: number | undefined, onSubmit: (result: (typeof buttons)[number] | false) => void) {
super(plugin.app);
this.plugin = plugin;
this.title = title;
@@ -156,6 +156,7 @@ export class MessageBox extends Modal {
this.timeout = timeout;
if (this.timeout) {
this.timer = setInterval(() => {
if (this.timeout === undefined) return;
this.timeout--;
if (this.timeout < 0) {
if (this.timer) {
@@ -166,7 +167,7 @@ export class MessageBox extends Modal {
this.isManuallyClosed = true;
this.close();
} else {
this.defaultButtonComponent.setButtonText(`( ${this.timeout} ) ${defaultAction}`);
this.defaultButtonComponent?.setButtonText(`( ${this.timeout} ) ${defaultAction}`);
}
}, 1000);
}
@@ -223,7 +224,7 @@ export class MessageBox extends Modal {
}
export function confirmWithMessage(plugin: Plugin, title: string, contentMd: string, buttons: string[], defaultAction?: (typeof buttons)[number], timeout?: number): Promise<(typeof buttons)[number] | false> {
export function confirmWithMessage(plugin: Plugin, title: string, contentMd: string, buttons: string[], defaultAction: (typeof buttons)[number], timeout?: number): Promise<(typeof buttons)[number] | false> {
return new Promise((res) => {
const dialog = new MessageBox(plugin, title, contentMd, buttons, defaultAction, timeout, (result) => res(result));
dialog.open();

2
src/lib

Submodule src/lib updated: b2788a8d98...29e23f5763

1893
src/main.ts
View File

File diff suppressed because it is too large Load Diff

7
src/stores.ts Normal file
View File

@@ -0,0 +1,7 @@
import { PersistentMap } from "./lib/src/PersistentMap";
export let sameChangePairs: PersistentMap<number[]>;
export function initializeStores(vaultName: string) {
sameChangePairs = new PersistentMap<number[]>(`ls-persist-same-changes-${vaultName}`);
}

177
src/utils.ts
View File

@@ -1,12 +1,16 @@
import { type DataWriteOptions, normalizePath, TFile, Platform, TAbstractFile, App, Plugin, type RequestUrlParam, requestUrl } from "./deps";
import { normalizePath, Platform, TAbstractFile, App, type RequestUrlParam, requestUrl, TFile } from "./deps";
import { path2id_base, id2path_base, isValidFilenameInLinux, isValidFilenameInDarwin, isValidFilenameInWidows, isValidFilenameInAndroid, stripAllPrefixes } from "./lib/src/path";
import { Logger } from "./lib/src/logger";
import { LOG_LEVEL_VERBOSE, type AnyEntry, type DocumentID, type EntryHasPath, type FilePath, type FilePathWithPrefix } from "./lib/src/types";
import { CHeader, ICHeader, ICHeaderLength, ICXHeader, PSCHeader } from "./types";
import { InputStringDialog, PopoverSelectString } from "./dialogs";
import ObsidianLiveSyncPlugin from "./main";
import type ObsidianLiveSyncPlugin from "./main";
import { writeString } from "./lib/src/strbin";
import { fireAndForget } from "./lib/src/utils";
import { sameChangePairs } from "./stores";
export { scheduleTask, setPeriodicTask, cancelTask, cancelAllTasks, cancelPeriodicTask, cancelAllPeriodicTask, } from "./lib/src/task";
// For backward compatibility, using the path for determining id.
// Only CouchDB unacceptable ID (that starts with an underscore) has been prefixed with "/".
@@ -43,49 +47,6 @@ export function getPathFromTFile(file: TAbstractFile) {
return file.path as FilePath;
}
const tasks: { [key: string]: ReturnType<typeof setTimeout> } = {};
export function scheduleTask(key: string, timeout: number, proc: (() => Promise<any> | void), skipIfTaskExist?: boolean) {
if (skipIfTaskExist && key in tasks) {
return;
}
cancelTask(key);
tasks[key] = setTimeout(async () => {
delete tasks[key];
await proc();
}, timeout);
}
export function cancelTask(key: string) {
if (key in tasks) {
clearTimeout(tasks[key]);
delete tasks[key];
}
}
export function cancelAllTasks() {
for (const v in tasks) {
clearTimeout(tasks[v]);
delete tasks[v];
}
}
const intervals: { [key: string]: ReturnType<typeof setInterval> } = {};
export function setPeriodicTask(key: string, timeout: number, proc: (() => Promise<any> | void)) {
cancelPeriodicTask(key);
intervals[key] = setInterval(async () => {
delete intervals[key];
await proc();
}, timeout);
}
export function cancelPeriodicTask(key: string) {
if (key in intervals) {
clearInterval(intervals[key]);
delete intervals[key];
}
}
export function cancelAllPeriodicTask() {
for (const v in intervals) {
clearInterval(intervals[v]);
delete intervals[v];
}
}
const memos: { [key: string]: any } = {};
export function memoObject<T>(key: string, obj: T): T {
@@ -281,14 +242,11 @@ export function mergeObject(
ret[key] = v;
}
}
const retSorted = Object.fromEntries(Object.entries(ret).sort((a, b) => a[0] < b[0] ? -1 : a[0] > b[0] ? 1 : 0));
if (Array.isArray(objA) && Array.isArray(objB)) {
return Object.values(Object.entries(ret)
.sort()
.reduce((p, [key, value]) => ({ ...p, [key]: value }), {}));
return Object.values(retSorted);
}
return Object.entries(ret)
.sort()
.reduce((p, [key, value]) => ({ ...p, [key]: value }), {});
return retSorted;
}
export function flattenObject(obj: Record<string | number | symbol, any>, path: string[] = []): [string, any][] {
@@ -303,20 +261,6 @@ export function flattenObject(obj: Record<string | number | symbol, any>, path:
return ret;
}
export function modifyFile(file: TFile, data: string | ArrayBuffer, options?: DataWriteOptions) {
if (typeof (data) === "string") {
return app.vault.modify(file, data, options);
} else {
return app.vault.modifyBinary(file, data, options);
}
}
export function createFile(path: string, data: string | ArrayBuffer, options?: DataWriteOptions): Promise<TFile> {
if (typeof (data) === "string") {
return app.vault.create(path, data, options);
} else {
return app.vault.createBinary(path, data, options);
}
}
export function isValidPath(filename: string) {
if (Platform.isDesktop) {
@@ -332,38 +276,10 @@ export function isValidPath(filename: string) {
return isValidFilenameInWidows(filename);
}
let touchedFiles: string[] = [];
export function getAbstractFileByPath(path: FilePath): TAbstractFile | null {
// Disabled temporary.
return app.vault.getAbstractFileByPath(path);
// // Hidden API but so useful.
// // @ts-ignore
// if ("getAbstractFileByPathInsensitive" in app.vault && (app.vault.adapter?.insensitive ?? false)) {
// // @ts-ignore
// return app.vault.getAbstractFileByPathInsensitive(path);
// } else {
// return app.vault.getAbstractFileByPath(path);
// }
}
export function trimPrefix(target: string, prefix: string) {
return target.startsWith(prefix) ? target.substring(prefix.length) : target;
}
export function touch(file: TFile | FilePath) {
const f = file instanceof TFile ? file : getAbstractFileByPath(file) as TFile;
const key = `${f.path}-${f.stat.mtime}-${f.stat.size}`;
touchedFiles.unshift(key);
touchedFiles = touchedFiles.slice(0, 100);
}
export function recentlyTouched(file: TFile) {
const key = `${file.path}-${file.stat.mtime}-${file.stat.size}`;
if (touchedFiles.indexOf(key) == -1) return false;
return true;
}
export function clearTouched() {
touchedFiles = [];
}
/**
* returns is internal chunk of file
@@ -394,7 +310,7 @@ export function isCustomisationSyncMetadata(str: string): boolean {
export const askYesNo = (app: App, message: string): Promise<"yes" | "no"> => {
return new Promise((res) => {
const popover = new PopoverSelectString(app, message, null, null, (result) => res(result as "yes" | "no"));
const popover = new PopoverSelectString(app, message, undefined, undefined, (result) => res(result as "yes" | "no"));
popover.open();
});
};
@@ -408,7 +324,7 @@ export const askSelectString = (app: App, message: string, items: string[]): Pro
};
export const askString = (app: App, title: string, key: string, placeholder: string, isPassword?: boolean): Promise<string | false> => {
export const askString = (app: App, title: string, key: string, placeholder: string, isPassword: boolean = false): Promise<string | false> => {
return new Promise((res) => {
const dialog = new InputStringDialog(app, title, key, placeholder, isPassword, (result) => res(result));
dialog.open();
@@ -419,8 +335,8 @@ export const askString = (app: App, title: string, key: string, placeholder: str
export class PeriodicProcessor {
_process: () => Promise<any>;
_timer?: number;
_plugin: Plugin;
constructor(plugin: Plugin, process: () => Promise<any>) {
_plugin: ObsidianLiveSyncPlugin;
constructor(plugin: ObsidianLiveSyncPlugin, process: () => Promise<any>) {
this._plugin = plugin;
this._process = process;
}
@@ -434,12 +350,19 @@ export class PeriodicProcessor {
enable(interval: number) {
this.disable();
if (interval == 0) return;
this._timer = window.setInterval(() => this.process().then(() => { }), interval);
this._timer = window.setInterval(() => fireAndForget(async () => {
await this.process();
if (this._plugin._unloaded) {
this.disable();
}
}), interval);
this._plugin.registerInterval(this._timer);
}
disable() {
if (this._timer !== undefined) window.clearInterval(this._timer);
this._timer = undefined;
if (this._timer !== undefined) {
window.clearInterval(this._timer);
this._timer = undefined;
}
}
}
@@ -474,14 +397,17 @@ export const _requestToCouchDB = async (baseUri: string, username: string, passw
};
return await requestUrl(requestParam);
}
export const requestToCouchDB = async (baseUri: string, username: string, password: string, origin: string, key?: string, body?: string, method?: string) => {
export const requestToCouchDB = async (baseUri: string, username: string, password: string, origin: string = "", key?: string, body?: string, method?: string) => {
const uri = `_node/_local/_config${key ? "/" + key : ""}`;
return await _requestToCouchDB(baseUri, username, password, origin, uri, body, method);
};
export async function performRebuildDB(plugin: ObsidianLiveSyncPlugin, method: "localOnly" | "remoteOnly" | "rebuildBothByThisDevice") {
export async function performRebuildDB(plugin: ObsidianLiveSyncPlugin, method: "localOnly" | "remoteOnly" | "rebuildBothByThisDevice" | "localOnlyWithChunks") {
if (method == "localOnly") {
await plugin.addOnSetup.fetchLocalWithKeepLocal();
await plugin.addOnSetup.fetchLocal();
}
if (method == "localOnlyWithChunks") {
await plugin.addOnSetup.fetchLocal(true);
}
if (method == "remoteOnly") {
await plugin.addOnSetup.rebuildRemote();
@@ -490,3 +416,48 @@ export async function performRebuildDB(plugin: ObsidianLiveSyncPlugin, method: "
await plugin.addOnSetup.rebuildEverything();
}
}
export const BASE_IS_NEW = Symbol("base");
export const TARGET_IS_NEW = Symbol("target");
export const EVEN = Symbol("even");
// Why 2000? : ZIP FILE Does not have enough resolution.
const resolution = 2000;
export function compareMTime(baseMTime: number, targetMTime: number): typeof BASE_IS_NEW | typeof TARGET_IS_NEW | typeof EVEN {
const truncatedBaseMTime = (~~(baseMTime / resolution)) * resolution;
const truncatedTargetMTime = (~~(targetMTime / resolution)) * resolution;
// Logger(`Resolution MTime ${truncatedBaseMTime} and ${truncatedTargetMTime} `, LOG_LEVEL_VERBOSE);
if (truncatedBaseMTime == truncatedTargetMTime) return EVEN;
if (truncatedBaseMTime > truncatedTargetMTime) return BASE_IS_NEW;
if (truncatedBaseMTime < truncatedTargetMTime) return TARGET_IS_NEW;
throw new Error("Unexpected error");
}
export function markChangesAreSame(file: TFile | AnyEntry | string, mtime1: number, mtime2: number) {
if (mtime1 === mtime2) return true;
const key = typeof file == "string" ? file : file instanceof TFile ? file.path : file.path ?? file._id;
const pairs = sameChangePairs.get(key, []) || [];
if (pairs.some(e => e == mtime1 || e == mtime2)) {
sameChangePairs.set(key, [...new Set([...pairs, mtime1, mtime2])]);
} else {
sameChangePairs.set(key, [mtime1, mtime2]);
}
}
export function isMarkedAsSameChanges(file: TFile | AnyEntry | string, mtimes: number[]) {
const key = typeof file == "string" ? file : file instanceof TFile ? file.path : file.path ?? file._id;
const pairs = sameChangePairs.get(key, []) || [];
if (mtimes.every(e => pairs.indexOf(e) !== -1)) {
return EVEN;
}
}
export function compareFileFreshness(baseFile: TFile | AnyEntry, checkTarget: TFile | AnyEntry): typeof BASE_IS_NEW | typeof TARGET_IS_NEW | typeof EVEN {
const modifiedBase = baseFile instanceof TFile ? baseFile?.stat?.mtime ?? 0 : baseFile?.mtime ?? 0;
const modifiedTarget = checkTarget instanceof TFile ? checkTarget?.stat?.mtime ?? 0 : checkTarget?.mtime ?? 0;
if (modifiedBase && modifiedTarget && isMarkedAsSameChanges(baseFile, [modifiedBase, modifiedTarget])) {
return EVEN;
}
return compareMTime(modifiedBase, modifiedTarget);
}

87
styles.css
View File

@@ -77,15 +77,6 @@
border-top: 1px solid var(--background-modifier-border);
}
/* .sls-table-head{
width:50%;
}
.sls-table-tail{
width:50%;
} */
.sls-header-button {
margin-left: 2em;
}
@@ -95,13 +86,23 @@
}
:root {
--slsmessage: "";
--sls-log-text: "";
}
.sls-troubleshoot-preview {
max-width: max-content;
}
.sls-troubleshoot-preview img {
max-width: 100%;
}
.CodeMirror-wrap::before,
.cm-s-obsidian>.cm-editor::before,
.canvas-wrapper::before {
content: attr(data-log);
.markdown-preview-view.cm-s-obsidian::before,
.markdown-source-view.cm-s-obsidian::before,
.canvas-wrapper::before,
.empty-state::before {
content: var(--sls-log-text, "");
text-align: right;
white-space: pre-wrap;
position: absolute;
@@ -116,6 +117,19 @@
filter: grayscale(100%);
}
.empty-state::before,
.markdown-preview-view.cm-s-obsidian::before,
.markdown-source-view.cm-s-obsidian::before {
top: var(--header-height);
right: 1em;
}
.is-mobile .empty-state::before,
.is-mobile .markdown-preview-view.cm-s-obsidian::before,
.is-mobile .markdown-source-view.cm-s-obsidian::before {
top: var(--view-header-height);
right: 1em;
}
.canvas-wrapper::before {
right: 48px;
}
@@ -124,7 +138,7 @@
right: 0px;
}
.cm-s-obsidian>.cm-editor::before {
.cm-s-obsidian > .cm-editor::before {
right: 16px;
}
@@ -153,8 +167,8 @@ div.sls-setting-menu-btn {
/* width: 100%; */
}
.sls-setting-tab:hover~div.sls-setting-menu-btn,
.sls-setting-label.selected .sls-setting-tab:checked~div.sls-setting-menu-btn {
.sls-setting-tab:hover ~ div.sls-setting-menu-btn,
.sls-setting-label.selected .sls-setting-tab:checked ~ div.sls-setting-menu-btn {
background-color: var(--interactive-accent);
color: var(--text-on-accent);
}
@@ -257,8 +271,8 @@ div.sls-setting-menu-btn {
display: none;
}
.password-input > .setting-item-control >input {
-webkit-text-security: disc;
.password-input > .setting-item-control > input {
-webkit-text-security: disc;
}
span.ls-mark-cr::after {
@@ -270,4 +284,39 @@ span.ls-mark-cr::after {
.deleted span.ls-mark-cr::after {
color: var(--text-on-accent);
}
}
.ls-imgdiff-wrap {
display: flex;
justify-content: center;
align-items: center;
}
.ls-imgdiff-wrap .overlay {
position: relative;
}
.ls-imgdiff-wrap .overlay .img-base {
position: relative;
top: 0;
left: 0;
}
.ls-imgdiff-wrap .overlay .img-overlay {
-webkit-filter: invert(100%) opacity(50%);
filter: invert(100%) opacity(50%);
position: absolute;
top: 0;
left: 0;
animation: ls-blink-diff 0.5s cubic-bezier(0.4, 0, 1, 1) infinite alternate;
}
@keyframes ls-blink-diff {
0% {
opacity: 0;
}
50% {
opacity: 0;
}
100% {
opacity: 1;
}
}

1
tsconfig.json
View File

@@ -16,6 +16,7 @@
"importHelpers": false,
"alwaysStrict": true,
"allowImportingTsExtensions": true,
"noEmit": true,
"lib": [
"es2018",
"DOM",

91
updates.md
View File

@@ -1,57 +1,52 @@
### 0.20.0
### 0.22.0
A few years passed since Self-hosted LiveSync was born, and our codebase had been very complicated. This could be patient now, but it should be a tremendous hurt.
Therefore at v0.22.0, for future maintainability, I refined task scheduling logic totally.
At 0.20.0, Self-hosted LiveSync has changed the binary file format and encrypting format, for efficient synchronisation.
The dialogue will be shown and asks us to decide whether to keep v1 or use v2. Once we have enabled v2, all subsequent edits will be saved in v2. Therefore, devices running 0.19 or below cannot understand this and they might say that decryption error. Please update all devices.
Then we will have an impressive performance.
Of course, I think this would be our suffering in some cases. However, I would love to ask you for your cooperation and contribution.
Of course, these are very impactful changes. If you have any questions or troubled things, please feel free to open an issue and mention me.
Sorry for being absent so much long. And thank you for your patience!
Note: if you want to roll it back to v1, please enable `Use binary and encryption version 1` on the `Hatch` pane and perform the `rebuild everything` once.
Extra but notable information:
This format change gives us the ability to detect some `marks` in the binary files as same as text files. Therefore, we can split binary files and some specific sort of them (i.e., PDF files) at the specific character. It means that editing the middle of files could be detected with marks.
Now only a few chunks are transferred, even if we add a comment to the PDF or put new files into the ZIP archives.
Note: we got a very performance improvement.
Note at 0.22.2: **Now, to rescue mobile devices, Maximum file size is set to 50 by default**. Please configure the limit as you need. If you do not want to limit the sizes, set zero manually, please.
#### Version history
- 0.20.4
- Fixed
- The text-input-dialogue is no longer broken.
- Finally, we can use the Setup URI again on mobile.
- 0.20.3
- 0.22.14:
- New feature:
- We can disable the status bar in the setting dialogue.
- Improved:
- Now some files are handled as correct data type.
- Customisation sync now uses the digest of each file for better performance.
- The status in the Editor now works performant.
- Refactored:
- Common functions have been ready and the codebase has been organised.
- Stricter type checking following TypeScript updates.
- Remove old iOS workaround for simplicity and performance.
- 0.22.13:
- Improved:
- Now using HTTP for the remote database URI warns of an error (on mobile) or notice (on desktop).
- Refactored:
- Dependencies have been polished.
- 0.22.12:
- Changed:
- The default settings has been changed.
- Improved:
- Default and preferred settings are applied on completion of the wizard.
- Fixed:
- Now Initialisation `Fetch` will be performed smoothly and there will be fewer conflicts.
- No longer stuck while Handling transferred or initialised documents.
- 0.22.11:
- Fixed:
- `Verify and repair all files` is no longer broken.
- New feature:
- We can launch Customization sync from the Ribbon if we enabled it.
- Fixed:
- Setup URI is now back to the previous spec; be encrypted by V1.
- It may avoid the trouble with iOS 17.
- The Settings dialogue is now registered at the beginning of the start-up process.
- We can change the configuration even though LiveSync could not be launched in normal.
- Now `Verify and repair all files` is able to...
- Restore if the file only in the local database.
- Show the history.
- Improved:
- Enumerating documents has been faster.
- 0.20.2
- New feature:
- We can delete all data of customization sync from the `Delete all customization sync data` on the `Hatch` pane.
- Performance improved.
- 0.22.10
- Fixed:
- Prevent keep restarting on iOS by yielding microtasks.
- 0.20.1
- Fixed:
- No more UI freezing and keep restarting on iOS.
- Diff of Non-markdown documents are now shown correctly.
- No longer unchanged hidden files and customisations are saved and transferred now.
- File integrity of vault history indicates the integrity correctly.
- Improved:
- Performance has been a bit improved.
- Customization sync has gotten faster.
- However, We lost forward compatibility again (only for this feature). Please update all devices.
- Misc
- Terser configuration has been more aggressive.
- 0.20.0
- Improved:
- A New binary file handling implemented
- A new encrypted format has been implemented
- Now the chunk sizes will be adjusted for efficient sync
- Fixed:
- levels of exception in some logs have been fixed
- Tidied:
- Some Lint warnings have been suppressed.
... To continue on to `updates_old.md`.
- In the report, the schema of the remote database URI is now printed.
... To continue on to `updates_old.md`.

247
updates_old.md
View File

@@ -1,3 +1,250 @@
### 0.22.0
A few years passed since Self-hosted LiveSync was born, and our codebase had been very complicated. This could be patient now, but it should be a tremendous hurt.
Therefore at v0.22.0, for future maintainability, I refined task scheduling logic totally.
Of course, I think this would be our suffering in some cases. However, I would love to ask you for your cooperation and contribution.
Sorry for being absent so much long. And thank you for your patience!
Note: we got a very performance improvement.
Note at 0.22.2: **Now, to rescue mobile devices, Maximum file size is set to 50 by default**. Please configure the limit as you need. If you do not want to limit the sizes, set zero manually, please.
#### Version history
- 0.22.9
- Fixed:
- Fixed a bug on `fetch chunks on demand` that could not fetch the chunks on demand.
- Improved:
- `fetch chunks on demand` works more smoothly.
- Initialisation `Fetch` is now more efficient.
- Tidied:
- Removed some meaningless codes.
- 0.22.8
- Fixed:
- Now fetch and unlock the locked remote database works well again.
- No longer crash on symbolic links inside hidden folders.
- Improved:
- Chunks are now created more efficiently.
- Splitting old notes into a larger chunk.
- Better performance in saving notes.
- Network activities are indicated as an icon.
- Less memory used for binary processing.
- Tidied:
- Cleaned unused functions up.
- Sorting out the codes that have become nonsense.
- Changed:
- Now no longer `fetch chunks on demand` needs `Pacing replication`
- The setting `Do not pace synchronization` has been deleted.
- 0.22.7
- Fixed:
- No longer deleted hidden files were ignored.
- The document history dialogue is now able to process the deleted revisions.
- Deletion of a hidden file is now surely performed even if the file is already conflicted.
- 0.22.6
- Fixed:
- Fixed a problem with synchronisation taking a long time to start in some cases.
- The first synchronisation after update might take a bit longer.
- Now we can disable E2EE encryption.
- Improved:
- `Setup Wizard` is now more clear.
- `Minimal Setup` is now more simple.
- Self-hosted LiveSync now be able to use even if there are vaults with the same name.
- Database suffix will automatically added.
- Now Self-hosted LiveSync waits until set-up is complete.
- Show reload prompts when possibly recommended while settings.
- New feature:
- A guidance dialogue prompting for settings will be shown after the installation.
- Changed
- `Open setup URI` is now `Use the copied setup URI`
- `Copy setup URI` is now `Copy current settings as a new setup URI`
- `Setup Wizard` is now `Minimal Setup`
- `Check database configuration` is now `Check and Fix database configuration`
- 0.22.5
- Fixed:
- Some description of settings have been refined
- New feature:
- TroubleShooting is now shown in the setting dialogue.
- 0.22.4
- Fixed:
- Now the result of conflict resolution could be surely written into the storage.
- Deleted files can be handled correctly again in the history dialogue and conflict dialogue.
- Some wrong log messages were fixed.
- Change handling now has become more stable.
- Some event handling became to be safer.
- Improved:
- Dumping document information shows conflicts and revisions.
- The timestamp-only differences can be surely cached.
- Timestamp difference detection can be rounded by two seconds.
- Refactored:
- A bit of organisation to write the test.
- 0.22.3
- Fixed:
- No longer detects storage changes which have been caused by Self-hosted LiveSync itself.
- Setting sync file will be detected only if it has been configured now.
- And its log will be shown only while the verbose log is enabled.
- Customisation file enumeration has got less blingy.
- Deletion of files is now reliably synchronised.
- Fixed and improved:
- In-editor-status is now shown in the following areas:
- Note editing pane (Source mode and live-preview mode).
- New tab pane.
- Canvas pane.
- 0.22.2
- Fixed:
- Now the results of resolving conflicts are surely synchronised.
- Modified:
- Some setting items got new clear names. (`Sync Settings` -> `Targets`).
- New feature:
- We can limit the synchronising files by their size. (`Sync Settings` -> `Targets` -> `Maximum file size`).
- It depends on the size of the newer one.
- At Obsidian 1.5.3 on mobile, we should set this to around 50MB to avoid restarting Obsidian.
- Now the settings could be stored in a specific markdown file to synchronise or switch it (`General Setting` -> `Share settings via markdown`).
- [Screwdriver](https://github.com/vrtmrz/obsidian-screwdriver) is quite good, but mostly we only need this.
- Customisation of the obsoleted device is now able to be deleted at once.
- We have to put the maintenance mode in at the Customisation sync dialogue.
- 0.22.1
- New feature:
- We can perform automatic conflict resolution for inactive files, and postpone only manual ones by `Postpone manual resolution of inactive files`.
- Now we can see the image in the document history dialogue.
- We can see the difference of the image, in the document history dialogue.
- And also we can highlight differences.
- Improved:
- Hidden file sync has been stabilised.
- Now automatically reloads the conflict-resolution dialogue when new conflicted revisions have arrived.
- Fixed:
- No longer periodic process runs after unloading the plug-in.
- Now the modification of binary files is surely stored in the storage.
- 0.22.0
- Refined:
- Task scheduling logics has been rewritten.
- Screen updates are also now efficient.
- Possibly many bugs and fragile behaviour has been fixed.
- Status updates and logging have been thinned out to display.
- Fixed:
- Remote-chunk-fetching now works with keeping request intervals
- New feature:
- We can show only the icons in the editor.
- Progress indicators have been more meaningful:
- 📥 Unprocessed transferred items
- 📄 Working database operation
- 💾 Working write storage processes
- ⏳ Working read storage processes
- 🛫 Pending read storage processes
- ⚙️ Working or pending storage processes of hidden files
- 🧩 Waiting chunks
- 🔌 Working Customisation items (Configuration, snippets and plug-ins)
... To continue on to `updates_old.md`.
### 0.21.0
The E2EE encryption V2 format has been reverted. That was probably the cause of the glitch.
Instead, to maintain efficiency, files are treated with Blob until just before saving. Along with this, the old-fashioned encryption format has also been discontinued.
There are both forward and backwards compatibilities, with recent versions. However, unfortunately, we lost compatibility with filesystem-livesync or some.
It will be addressed soon. Please be patient if you are using filesystem-livesync with E2EE.
- 0.21.5
- Improved:
- Now all revisions will be shown only its first a few letters.
- Now ID of the documents is shown in the log with the first 8 letters.
- Fixed:
- Check before modifying files has been implemented.
- Content change detection has been improved.
- 0.21.4
- This release had been skipped.
- 0.21.3
- Implemented:
- Now we can use SHA1 for hash function as fallback.
- 0.21.2
- IMPORTANT NOTICE: **0.21.1 CONTAINS A BUG WHILE REBUILDING THE DATABASE. IF YOU HAVE BEEN REBUILT, PLEASE MAKE SURE THAT ALL FILES ARE SANE.**
- This has been fixed in this version.
- Fixed:
- No longer files are broken while rebuilding.
- Now, Large binary files can be written correctly on a mobile platform.
- Any decoding errors now make zero-byte files.
- Modified:
- All files are processed sequentially for each.
- 0.21.1
- Fixed:
- No more infinity loops on larger files.
- Show message on decode error.
- Refactored:
- Fixed to avoid obsolete global variables.
- 0.21.0
- Changes and performance improvements:
- Now the saving files are processed by Blob.
- The V2-Format has been reverted.
- New encoding format has been enabled in default.
- WARNING: Since this version, the compatibilities with older Filesystem LiveSync have been lost.
## 0.20.0
At 0.20.0, Self-hosted LiveSync has changed the binary file format and encrypting format, for efficient synchronisation.
The dialogue will be shown and asks us to decide whether to keep v1 or use v2. Once we have enabled v2, all subsequent edits will be saved in v2. Therefore, devices running 0.19 or below cannot understand this and they might say that decryption error. Please update all devices.
Then we will have an impressive performance.
Of course, these are very impactful changes. If you have any questions or troubled things, please feel free to open an issue and mention me.
Note: if you want to roll it back to v1, please enable `Use binary and encryption version 1` on the `Hatch` pane and perform the `rebuild everything` once.
Extra but notable information:
This format change gives us the ability to detect some `marks` in the binary files as same as text files. Therefore, we can split binary files and some specific sort of them (i.e., PDF files) at the specific character. It means that editing the middle of files could be detected with marks.
Now only a few chunks are transferred, even if we add a comment to the PDF or put new files into the ZIP archives.
- 0.20.7
- Fixed
- To better replication, path obfuscation is now deterministic even if with E2EE.
Note: Compatible with previous database without any conversion. Only new files will be obfuscated in deterministic.
- 0.20.6
- Fixed
- Now empty file could be decoded.
- Local files are no longer pre-saved before fetching from a remote database.
- No longer deadlock while applying customisation sync.
- Configuration with multiple files is now able to be applied correctly.
- Deleting folder propagation now works without enabling the use of a trash bin.
- 0.20.5
- Fixed
- Now the files which having digit or character prefixes in the path will not be ignored.
- 0.20.4
- Fixed
- The text-input-dialogue is no longer broken.
- Finally, we can use the Setup URI again on mobile.
- 0.20.3
- New feature:
- We can launch Customization sync from the Ribbon if we enabled it.
- Fixed:
- Setup URI is now back to the previous spec; be encrypted by V1.
- It may avoid the trouble with iOS 17.
- The Settings dialogue is now registered at the beginning of the start-up process.
- We can change the configuration even though LiveSync could not be launched in normal.
- Improved:
- Enumerating documents has been faster.
- 0.20.2
- New feature:
- We can delete all data of customization sync from the `Delete all customization sync data` on the `Hatch` pane.
- Fixed:
- Prevent keep restarting on iOS by yielding microtasks.
- 0.20.1
- Fixed:
- No more UI freezing and keep restarting on iOS.
- Diff of Non-markdown documents are now shown correctly.
- Improved:
- Performance has been a bit improved.
- Customization sync has gotten faster.
- However, We lost forward compatibility again (only for this feature). Please update all devices.
- Misc
- Terser configuration has been more aggressive.
- 0.20.0
- Improved:
- A New binary file handling implemented
- A new encrypted format has been implemented
- Now the chunk sizes will be adjusted for efficient sync
- Fixed:
- levels of exception in some logs have been fixed
- Tidied:
- Some Lint warnings have been suppressed.
### 0.19.0
#### Customization sync

1
utils/.gitignore vendored Normal file
View File

@@ -0,0 +1 @@
fly.toml

29
utils/couchdb/couchdb-init.sh Executable file
View File

@@ -0,0 +1,29 @@
#!/bin/bash
if [[ -z "$hostname" ]]; then
echo "ERROR: Hostname missing"
exit 1
fi
if [[ -z "$username" ]]; then
echo "ERROR: Username missing"
exit 1
fi
if [[ -z "$password" ]]; then
echo "ERROR: Password missing"
exit 1
fi
echo "-- Configuring CouchDB by REST APIs... -->"
until (curl -X POST "${hostname}/_cluster_setup" -H "Content-Type: application/json" -d "{\"action\":\"enable_single_node\",\"username\":\"${username}\",\"password\":\"${password}\",\"bind_address\":\"0.0.0.0\",\"port\":5984,\"singlenode\":true}" --user "${username}:${password}"); do sleep 5; done
until (curl -X PUT "${hostname}/_node/nonode@nohost/_config/chttpd/require_valid_user" -H "Content-Type: application/json" -d '"true"' --user "${username}:${password}"); do sleep 5; done
until (curl -X PUT "${hostname}/_node/nonode@nohost/_config/chttpd_auth/require_valid_user" -H "Content-Type: application/json" -d '"true"' --user "${username}:${password}"); do sleep 5; done
until (curl -X PUT "${hostname}/_node/nonode@nohost/_config/httpd/WWW-Authenticate" -H "Content-Type: application/json" -d '"Basic realm=\"couchdb\""' --user "${username}:${password}"); do sleep 5; done
until (curl -X PUT "${hostname}/_node/nonode@nohost/_config/httpd/enable_cors" -H "Content-Type: application/json" -d '"true"' --user "${username}:${password}"); do sleep 5; done
until (curl -X PUT "${hostname}/_node/nonode@nohost/_config/chttpd/enable_cors" -H "Content-Type: application/json" -d '"true"' --user "${username}:${password}"); do sleep 5; done
until (curl -X PUT "${hostname}/_node/nonode@nohost/_config/chttpd/max_http_request_size" -H "Content-Type: application/json" -d '"4294967296"' --user "${username}:${password}"); do sleep 5; done
until (curl -X PUT "${hostname}/_node/nonode@nohost/_config/couchdb/max_document_size" -H "Content-Type: application/json" -d '"50000000"' --user "${username}:${password}"); do sleep 5; done
until (curl -X PUT "${hostname}/_node/nonode@nohost/_config/cors/credentials" -H "Content-Type: application/json" -d '"true"' --user "${username}:${password}"); do sleep 5; done
until (curl -X PUT "${hostname}/_node/nonode@nohost/_config/cors/origins" -H "Content-Type: application/json" -d '"app://obsidian.md,capacitor://localhost,http://localhost"' --user "${username}:${password}"); do sleep 5; done
echo "<-- Configuring CouchDB by REST APIs Done!"

4
utils/flyio/delete-server.sh Executable file
View File

@@ -0,0 +1,4 @@
#!/bin/bash
set -e
fly scale count 0 -y
fly apps destroy $(fly status -j | jq -r .Name) -y

43
utils/flyio/deploy-server.sh Executable file
View File

@@ -0,0 +1,43 @@
#!/bin/bash
## Script for deploy and automatic setup CouchDB onto fly.io.
## We need Deno for generating the Setup-URI.
source setenv.sh $@
export hostname="https://$appname.fly.dev"
echo "-- YOUR CONFIGURATION --"
echo "URL : $hostname"
echo "username: $username"
echo "password: $password"
echo "region : $region"
echo ""
echo "-- START DEPLOYING --> "
set -e
fly launch --name=$appname --env="COUCHDB_USER=$username" --copy-config=true --detach --no-deploy --region ${region} --yes
fly secrets set COUCHDB_PASSWORD=$password
fly deploy
set +e
../couchdb/couchdb-init.sh
# flyctl deploy
echo "OK!"
if command -v deno >/dev/null 2>&1; then
echo "Setup finished! Also, we can set up Self-hosted LiveSync instantly, by the following setup uri."
echo "Passphrase of setup-uri is \`welcome\`".
echo "--- configured ---"
echo "database : ${database}"
echo "E2EE passphrase: ${passphrase}"
echo "--- setup uri ---"
deno run -A generate_setupuri.ts
else
echo "Setup finished! Here is the configured values (reprise)!"
echo "-- YOUR CONFIGURATION --"
echo "URL : $hostname"
echo "username: $username"
echo "password: $password"
echo "-- YOUR CONFIGURATION --"
echo "If we had Deno, we would got the setup uri directly!"
fi

40
utils/flyio/fly.template.toml Normal file
View File

@@ -0,0 +1,40 @@
## CouchDB for fly.io image
app = ''
primary_region = 'nrt'
swap_size_mb = 512
[build]
image = "couchdb:latest"
[mounts]
source = "couchdata"
destination = "/opt/couchdb/data"
initial_size = "1GB"
auto_extend_size_threshold = 90
auto_extend_size_increment = "1GB"
auto_extend_size_limit = "2GB"
[env]
COUCHDB_USER = ""
ERL_FLAGS = "-couch_ini /opt/couchdb/etc/default.ini /opt/couchdb/etc/default.d/ /opt/couchdb/etc/local.d /opt/couchdb/etc/local.ini /opt/couchdb/data/persistence.ini"
[http_service]
internal_port = 5984
force_https = true
auto_stop_machines = true
auto_start_machines = true
min_machines_running = 0
processes = ['app']
[[vm]]
cpu_kind = 'shared'
cpus = 1
memory_mb = 256
[[files]]
guest_path = "/docker-entrypoint2.sh"
raw_value = "#!/bin/bash\ntouch /opt/couchdb/data/persistence.ini\nchmod +w /opt/couchdb/data/persistence.ini\n/docker-entrypoint.sh $@"
[experimental]
entrypoint = ["tini", "--", "/docker-entrypoint2.sh"]

180
utils/flyio/generate_setupuri.ts Normal file
View File

@@ -0,0 +1,180 @@
import { webcrypto } from "node:crypto";
const KEY_RECYCLE_COUNT = 100;
type KeyBuffer = {
key: CryptoKey;
salt: Uint8Array;
count: number;
};
let semiStaticFieldBuffer: Uint8Array;
const nonceBuffer: Uint32Array = new Uint32Array(1);
const writeString = (string: string) => {
// Prepare enough buffer.
const buffer = new Uint8Array(string.length * 4);
const length = string.length;
let index = 0;
let chr = 0;
let idx = 0;
while (idx < length) {
chr = string.charCodeAt(idx++);
if (chr < 128) {
buffer[index++] = chr;
} else if (chr < 0x800) {
// 2 bytes
buffer[index++] = 0xC0 | (chr >>> 6);
buffer[index++] = 0x80 | (chr & 0x3F);
} else if (chr < 0xD800 || chr > 0xDFFF) {
// 3 bytes
buffer[index++] = 0xE0 | (chr >>> 12);
buffer[index++] = 0x80 | ((chr >>> 6) & 0x3F);
buffer[index++] = 0x80 | (chr & 0x3F);
} else {
// 4 bytes - surrogate pair
chr = (((chr - 0xD800) << 10) | (string.charCodeAt(idx++) - 0xDC00)) + 0x10000;
buffer[index++] = 0xF0 | (chr >>> 18);
buffer[index++] = 0x80 | ((chr >>> 12) & 0x3F);
buffer[index++] = 0x80 | ((chr >>> 6) & 0x3F);
buffer[index++] = 0x80 | (chr & 0x3F);
}
}
return buffer.slice(0, index);
};
const KeyBuffs = new Map<string, KeyBuffer>();
async function getKeyForEncrypt(passphrase: string, autoCalculateIterations: boolean): Promise<[CryptoKey, Uint8Array]> {
// For performance, the plugin reuses the key KEY_RECYCLE_COUNT times.
const buffKey = `${passphrase}-${autoCalculateIterations}`;
const f = KeyBuffs.get(buffKey);
if (f) {
f.count--;
if (f.count > 0) {
return [f.key, f.salt];
}
f.count--;
}
const passphraseLen = 15 - passphrase.length;
const iteration = autoCalculateIterations ? ((passphraseLen > 0 ? passphraseLen : 0) * 1000) + 121 - passphraseLen : 100000;
const passphraseBin = new TextEncoder().encode(passphrase);
const digest = await webcrypto.subtle.digest({ name: "SHA-256" }, passphraseBin);
const keyMaterial = await webcrypto.subtle.importKey("raw", digest, { name: "PBKDF2" }, false, ["deriveKey"]);
const salt = webcrypto.getRandomValues(new Uint8Array(16));
const key = await webcrypto.subtle.deriveKey(
{
name: "PBKDF2",
salt,
iterations: iteration,
hash: "SHA-256",
},
keyMaterial,
{ name: "AES-GCM", length: 256 },
false,
["encrypt"]
);
KeyBuffs.set(buffKey, {
key,
salt,
count: KEY_RECYCLE_COUNT,
});
return [key, salt];
}
function getSemiStaticField(reset?: boolean) {
// return fixed field of iv.
if (semiStaticFieldBuffer != null && !reset) {
return semiStaticFieldBuffer;
}
semiStaticFieldBuffer = webcrypto.getRandomValues(new Uint8Array(12));
return semiStaticFieldBuffer;
}
function getNonce() {
// This is nonce, so do not send same thing.
nonceBuffer[0]++;
if (nonceBuffer[0] > 10000) {
// reset semi-static field.
getSemiStaticField(true);
}
return nonceBuffer;
}
function arrayBufferToBase64internalBrowser(buffer: DataView | Uint8Array): Promise<string> {
return new Promise((res, rej) => {
const blob = new Blob([buffer], { type: "application/octet-binary" });
const reader = new FileReader();
reader.onload = function (evt) {
const dataURI = evt.target?.result?.toString() || "";
if (buffer.byteLength != 0 && (dataURI == "" || dataURI == "data:")) return rej(new TypeError("Could not parse the encoded string"));
const result = dataURI.substring(dataURI.indexOf(",") + 1);
res(result);
};
reader.readAsDataURL(blob);
});
}
// Map for converting hexString
const revMap: { [key: string]: number } = {};
const numMap: { [key: number]: string } = {};
for (let i = 0; i < 256; i++) {
revMap[(`00${i.toString(16)}`.slice(-2))] = i;
numMap[i] = (`00${i.toString(16)}`.slice(-2));
}
function uint8ArrayToHexString(src: Uint8Array): string {
return [...src].map(e => numMap[e]).join("");
}
const QUANTUM = 32768;
async function arrayBufferToBase64Single(buffer: ArrayBuffer): Promise<string> {
const buf = buffer instanceof Uint8Array ? buffer : new Uint8Array(buffer);
if (buf.byteLength < QUANTUM) return btoa(String.fromCharCode.apply(null, [...buf]));
return await arrayBufferToBase64internalBrowser(buf);
}
export async function encrypt(input: string, passphrase: string, autoCalculateIterations: boolean) {
const [key, salt] = await getKeyForEncrypt(passphrase, autoCalculateIterations);
// Create initial vector with semi-fixed part and incremental part
// I think it's not good against related-key attacks.
const fixedPart = getSemiStaticField();
const invocationPart = getNonce();
const iv = new Uint8Array([...fixedPart, ...new Uint8Array(invocationPart.buffer)]);
const plainStringified = JSON.stringify(input);
// const plainStringBuffer: Uint8Array = tex.encode(plainStringified)
const plainStringBuffer: Uint8Array = writeString(plainStringified);
const encryptedDataArrayBuffer = await webcrypto.subtle.encrypt({ name: "AES-GCM", iv }, key, plainStringBuffer);
const encryptedData2 = (await arrayBufferToBase64Single(encryptedDataArrayBuffer));
//return data with iv and salt.
const ret = `["${encryptedData2}","${uint8ArrayToHexString(iv)}","${uint8ArrayToHexString(salt)}"]`;
return ret;
}
const URIBASE = "obsidian://setuplivesync?settings=";
async function main() {
const conf = {
"couchDB_URI": `${Deno.env.get("hostname")}`,
"couchDB_USER": `${Deno.env.get("username")}`,
"couchDB_PASSWORD": `${Deno.env.get("password")}`,
"couchDB_DBNAME": `${Deno.env.get("database")}`,
"syncOnStart": true,
"gcDelay": 0,
"periodicReplication": true,
"syncOnFileOpen": true,
"encrypt": true,
"passphrase": `${Deno.env.get("passphrase")}`,
"usePathObfuscation": true,
"batchSave": true,
"batch_size": 50,
"batches_limit": 50,
"useHistory": true,
"disableRequestURI": true,
"customChunkSize": 50,
"syncAfterMerge": false,
"concurrencyOfReadChunksOnline": 100,
"minimumIntervalOfReadChunksOnline": 100,
}
const encryptedConf = encodeURIComponent(await encrypt(JSON.stringify(conf), "welcome", false));
const theURI = `${URIBASE}${encryptedConf}`;
console.log(theURI);
}
await main();

30
utils/flyio/setenv.sh Executable file
View File

@@ -0,0 +1,30 @@
random_num() {
echo $RANDOM
}
random_noun() {
nouns=("waterfall" "river" "breeze" "moon" "rain" "wind" "sea" "morning" "snow" "lake" "sunset" "pine" "shadow" "leaf" "dawn" "glitter" "forest" "hill" "cloud" "meadow" "sun" "glade" "bird" "brook" "butterfly" "bush" "dew" "dust" "field" "fire" "flower" "firefly" "feather" "grass" "haze" "mountain" "night" "pond" "darkness" "snowflake" "silence" "sound" "sky" "shape" "surf" "thunder" "violet" "water" "wildflower" "wave" "water" "resonance" "sun" "log" "dream" "cherry" "tree" "fog" "frost" "voice" "paper" "frog" "smoke" "star")
echo ${nouns[$(($RANDOM % ${#nouns[*]}))]}
}
random_adjective() {
adjectives=("autumn" "hidden" "bitter" "misty" "silent" "empty" "dry" "dark" "summer" "icy" "delicate" "quiet" "white" "cool" "spring" "winter" "patient" "twilight" "dawn" "crimson" "wispy" "weathered" "blue" "billowing" "broken" "cold" "damp" "falling" "frosty" "green" "long" "late" "lingering" "bold" "little" "morning" "muddy" "old" "red" "rough" "still" "small" "sparkling" "thrumming" "shy" "wandering" "withered" "wild" "black" "young" "holy" "solitary" "fragrant" "aged" "snowy" "proud" "floral" "restless" "divine" "polished" "ancient" "purple" "lively" "nameless")
echo ${adjectives[$(($RANDOM % ${#adjectives[*]}))]}
}
cp ./fly.template.toml ./fly.toml
if [ "$1" = "renew" ]; then
unset appname
unset username
unset password
unset database
unset passphrase
unset region
fi
[ -z $appname ] && export appname=$(random_adjective)-$(random_noun)-$(random_num)
[ -z $username ] && export username=$(random_adjective)-$(random_noun)-$(random_num)
[ -z $password ] && export password=$(random_adjective)-$(random_noun)-$(random_num)
[ -z $database ] && export database="obsidiannotes"
[ -z $passphrase ] && export passphrase=$(random_adjective)-$(random_noun)-$(random_num)
[ -z $region ] && export region="nrt"

164
utils/readme.md Normal file
View File

@@ -0,0 +1,164 @@
<!-- For translation: 20240206r0 -->
# Utilities
Here are some useful things.
## couchdb
### couchdb-init.sh
This script can configure CouchDB with the necessary settings by REST APIs.
#### Materials
- Mandatory: curl
#### Usage
```sh
export hostname=http://localhost:5984/
export username=couchdb-admin-username
export password=couchdb-admin-password
./couchdb-init.sh
```
curl result will be shown, however, all of them can be ignored if the script has been run completely.
## fly.io
### deploy-server.sh
A fully automated CouchDB deployment script. We can deploy CouchDB onto fly.io. The only we need is an account of it.
All omitted configurations will be determined at random. (And, it is preferred). The region is configured to `nrt`.
If Japan is not close to you, please choose a region closer to you. However, the deployed database will work if you leave it at all.
#### Materials
- Mandatory: curl, flyctl
- Recommended: deno
#### Usage
```sh
#export appname=
#export username=
#export password=
#export database=
#export passphrase=
export region=nrt #pick your nearest location
./deploy-server.sh
```
The result of this command is as follows.
```
-- YOUR CONFIGURATION --
URL : https://young-darkness-25342.fly.dev
username: billowing-cherry-22580
password: misty-dew-13571
region : nrt
-- START DEPLOYING -->
An existing fly.toml file was found
Using build strategies '[the "couchdb:latest" docker image]'. Remove [build] from fly.toml to force a rescan
Creating app in /home/vorotamoroz/dev/obsidian-livesync/utils/flyio
We're about to launch your app on Fly.io. Here's what you're getting:
Organization: vorotamoroz (fly launch defaults to the personal org)
Name: young-darkness-25342 (specified on the command line)
Region: Tokyo, Japan (specified on the command line)
App Machines: shared-cpu-1x, 256MB RAM (specified on the command line)
Postgres: <none> (not requested)
Redis: <none> (not requested)
Created app 'young-darkness-25342' in organization 'personal'
Admin URL: https://fly.io/apps/young-darkness-25342
Hostname: young-darkness-25342.fly.dev
Wrote config file fly.toml
Validating /home/vorotamoroz/dev/obsidian-livesync/utils/flyio/fly.toml
Platform: machines
✓ Configuration is valid
Your app is ready! Deploy with `flyctl deploy`
Secrets are staged for the first deployment
==> Verifying app config
Validating /home/vorotamoroz/dev/obsidian-livesync/utils/flyio/fly.toml
Platform: machines
✓ Configuration is valid
--> Verified app config
==> Building image
Searching for image 'couchdb:latest' remotely...
image found: img_ox20prk63084j1zq
Watch your deployment at https://fly.io/apps/young-darkness-25342/monitoring
Provisioning ips for young-darkness-25342
Dedicated ipv6: 2a09:8280:1::37:fde9
Shared ipv4: 66.241.124.163
Add a dedicated ipv4 with: fly ips allocate-v4
Creating a 1 GB volume named 'couchdata' for process group 'app'. Use 'fly vol extend' to increase its size
This deployment will:
* create 1 "app" machine
No machines in group app, launching a new machine
WARNING The app is not listening on the expected address and will not be reachable by fly-proxy.
You can fix this by configuring your app to listen on the following addresses:
- 0.0.0.0:5984
Found these processes inside the machine with open listening sockets:
PROCESS | ADDRESSES
-----------------*---------------------------------------
/.fly/hallpass | [fdaa:0:73b9:a7b:22e:3851:7f28:2]:22
Finished launching new machines
NOTE: The machines for [app] have services with 'auto_stop_machines = true' that will be stopped when idling
-------
Checking DNS configuration for young-darkness-25342.fly.dev
Visit your newly deployed app at https://young-darkness-25342.fly.dev/
-- Configuring CouchDB by REST APIs... -->
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to young-darkness-25342.fly.dev:443
{"ok":true}
""
""
""
""
""
""
""
""
""
<-- Configuring CouchDB by REST APIs Done!
OK!
Setup finished! Also, we can set up Self-hosted LiveSync instantly, by the following setup uri.
Passphrase of setup-uri is `welcome`.
--- configured ---
database : obsidiannotes
E2EE passphrase: dark-wildflower-26467
--- setup uri ---
obsidian://setuplivesync?settings=%5B%22gZkBwjFbLqxbdSIbJymU%2FmTPBPAKUiHVGDRKYiNnKhW0auQeBgJOfvnxexZtMCn8sNiIUTAlxNaMGF2t%2BCEhpJoeCP%2FO%2BrwfN5LaNDQyky1Uf7E%2B64A5UWyjOYvZDOgq4iCKSdBAXp9oO%2BwKh4MQjUZ78vIVvJp8Mo6NWHfm5fkiWoAoddki1xBMvi%2BmmN%2FhZatQGcslVb9oyYWpZocduTl0a5Dv%2FQviGwlYQ%2F4NY0dVDIoOdvaYS%2FX4GhNAnLzyJKMXhPEJHo9FvR%2FEOBuwyfMdftV1SQUZ8YDCuiR3T7fh7Kn1c6OFgaFMpFm%2BWgIJ%2FZpmAyhZFpEcjpd7ty%2BN9kfd9gQsZM4%2BYyU9OwDd2DahVMBWkqoV12QIJ8OlJScHHdcUfMW5ex%2F4UZTWKNEHJsigITXBrtq11qGk3rBfHys8O0vY6sz%2FaYNM3iAOsR1aoZGyvwZm4O6VwtzK8edg0T15TL4O%2B7UajQgtCGxgKNYxb8EMOGeskv7NifYhjCWcveeTYOJzBhnIDyRbYaWbkAXQgHPBxzJRkkG%2FpBPfBBoJarj7wgjMvhLJ9xtL4FbP6sBNlr8jtAUCoq4L7LJcRNF4hlgvjJpL2BpFZMzkRNtUBcsRYR5J%2BM1X2buWi2BHncbSiRRDKEwNOQkc%2FmhMJjbAn%2F8eNKRuIICOLD5OvxD7FZNCJ0R%2BWzgrzcNV%22%2C%22ec7edc900516b4fcedb4c7cc01000000%22%2C%22fceb5fe54f6619ee266ed9a887634e07%22%5D
```
All we have to do is copy the setup-URI (`obsidian`://...`) and open it from Self-hosted LiveSync on Obsidian.
If you did not install Deno, configurations will be printed again, instead of the setup-URI. In this case, we should configure it manually.
### delete-server.sh
The pair script of `deploy-server.sh`. We can delete the deployed server by this with fly.toml.
#### Materials
- Mandatory: flyctl, jq
- Recommended: none
#### Usage
```sh
./delete-server.sh
```
```
App 'young-darkness-25342 is going to be scaled according to this plan:
-1 machines for group 'app' on region 'nrt' of size 'shared-cpu-1x'
Executing scale plan
Destroyed e28667eec57158 group:app region:nrt size:shared-cpu-1x
Destroyed app young-darkness-25342
```