public/obsidian-livesync
1
0
Fork 0
mirror of https://github.com/vrtmrz/obsidian-livesync.git synced 2026-05-12 18:41:52 +00:00
Code Issues Packages Projects Releases Wiki Activity

Compare commits

base: public:0.20.7
Branches Tags
public:main public:test_real_obsidian public:feat-userhashsalt public:p2p-rpc public:auto_conf_2 public:auto_conf public:beta public:alpha public:refactor_services_2 public:refactor_services public:dev public:svelteui public:forprivacy
public:0.25.60 public:0.25.59 public:0.25.58 public:0.25.57 public:0.25.56+patched5 public:0.25.56+patched4 public:0.25.56+patched3 public:0.25.56+patched2 public:0.25.56+patched1 public:0.25.56 public:0.25.55 public:0.25.54 public:0.25.53 public:0.25.52-patched-3 public:0.25.52-patched-2 public:0.25.52-patched-1 public:0.25.52 public:0.25.51 public:0.25.50 public:0.25.49 public:0.25.48 public:0.25.47 public:0.25.46 public:0.25.45 public:0.25.44 public:0.25.43-patched-9 public:0.25.43-patched-8 public:0.25.43-patched-7 public:0.25.43-patched-6 public:0.25.43-patched-5 public:0.25.43-patched-4 public:0.25.43-patched-3 public:0.25.43-patched-2 public:0.25.43-patched-1 public:0.25.43 public:0.25.42 public:0.25.41-patched-3 public:0.25.41-patched-2 public:0.25.41-patched-1 public:0.25.41 public:0.25.40 public:0.25.39 public:0.25.38 public:0.25.37 public:0.25.36 public:0.25.35 public:0.25.34 public:0.25.33 public:0.25.32 public:0.25.31 public:0.25.30 public:0.25.30-dev public:0.25.29 public:0.25.28 public:0.25.27 public:0.25.26 public:0.25.25 public:0.25.24 public:0.25.24.beta3 public:0.25.24.beta2 public:0.25.24.beta1 public:0.25.23 public:0.25.23.beta1 public:0.25.22 public:0.25.21 public:0.25.21.beta2 public:0.25.21.beta1 public:0.25.20 public:0.25.19 public:0.25.18 public:0.25.17 public:0.25.16 public:0.25.15 public:0.25.14 public:0.25.13 public:0.25.12 public:0.25.11 public:0.25.10 public:0.25.9 public:0.25.8 public:0.25.7 public:0.25.6 public:0.25.5 public:0.25.4 public:0.25.3 public:0.25.2 public:0.25.1 public:0.25.0 public:0.25.0-beta1 public:0.24.31 public:0.24.30 public:0.24.29 public:0.24.28 public:0.24.27 public:0.24.26 public:0.24.25 public:0.24.24 public:0.24.23 public:0.24.22 public:0.24.21 public:0.24.20 public:0.24.19 public:0.24.18 public:0.24.17 public:0.24.16 public:0.24.15 public:0.24.14 public:0.24.13 public:0.24.12 public:0.24.11 public:0.24.10 public:0.24.9 public:0.24.8 public:0.24.7 public:0.24.6 public:0.24.5 public:0.24.4 public:0.24.3 public:0.24.2 public:0.24.1 public:0.24.0 public:0.24.0.dev-rc8 public:0.24.0.dev-rc7 public:0.24.0.dev-rc6 public:0.24.0.dev-rc5 public:0.24.0.dev-rc4 public:0.24.0.dev-rc3 public:0.24.0.dev-rc2 public:0.23.23 public:0.23.22 public:0.23.21 public:0.23.20 public:0.23.19 public:0.23.18 public:0.23.17 public:0.23.16 public:0.23.15 public:0.23.14 public:0.23.13 public:0.23.12 public:0.23.11 public:0.23.10 public:0.23.9 public:0.23.8 public:0.23.7 public:0.23.6 public:0.23.5 public:0.23.4 public:0.23.3 public:0.23.2 public:0.23.1 public:0.23.0 public:0.22.19 public:0.22.18 public:0.22.17 public:0.22.16 public:0.22.15 public:0.22.14 public:0.22.13 public:0.22.12 public:0.22.11 public:0.22.10 public:0.22.9 public:0.22.8 public:0.22.7 public:0.22.6 public:0.22.5 public:0.22.4 public:0.22.3 public:0.22.2 public:0.22.1 public:0.22.0 public:0.21.5 public:0.21.4 public:0.21.3 public:0.21.2 public:0.21.1 public:0.21.0 public:0.20.7 public:0.20.6 public:0.20.5 public:0.20.4 public:0.20.3 public:0.20.2 public:0.20.1 public:0.20.0 public:0.19.23 public:0.19.22 public:0.19.21 public:0.19.20 public:0.19.19 public:0.19.18 public:0.19.17 public:0.19.16 public:0.19.15 public:0.19.14 public:0.19.13 public:0.19.12 public:0.19.11 public:0.19.10 public:0.19.9 public:0.19.8 public:0.19.7 public:0.19.6 public:0.19.5 public:0.19.4 public:0.19.3 public:0.19.2 public:0.19.1 public:0.19.0 public:0.18.6 public:0.18.5 public:0.18.4 public:0.18.3 public:0.18.2 public:0.18.1 public:0.18.0 public:0.17.34 public:0.17.33 public:0.17.32 public:0.17.31 public:0.17.30 public:0.17.29 public:0.17.28 public:0.17.27 public:0.17.26 public:0.17.25 public:0.17.24 public:0.17.23 public:0.17.22 public:0.17.21 public:0.17.20 public:0.17.19 public:0.17.18 public:0.17.17 public:0.17.16 public:0.17.15 public:0.17.14 public:0.17.13 public:0.17.12 public:0.17.11 public:0.17.10 public:0.17.9 public:0.17.8 public:0.17.7 public:0.17.6 public:0.17.5 public:0.17.4 public:0.17.3 public:0.17.2 public:0.17.1 public:0.17.0 public:0.16.8 public:0.16.7 public:0.16.6 public:0.16.5 public:0.16.4 public:0.16.3 public:0.16.2 public:0.16.1 public:0.16.0 public:0.15.10 public:0.15.9 public:0.15.8 public:0.15.7 public:0.15.6 public:0.15.5 public:0.15.4 public:0.15.3 public:0.15.2 public:0.15.1 public:0.15.0 public:0.14.8 public:0.14.7 public:0.14.6 public:0.14.5 public:0.14.4 public:0.14.3 public:0.14.2 public:0.14.1 public:0.14.0 public:0.13.4 public:0.13.3 public:0.13.2 public:0.13.1 public:0.13.0 public:0.12.5 public:0.12.4 public:0.12.3 public:0.12.2 public:0.12.1 public:0.12.0 public:0.11.10 public:0.11.9 public:0.11.8 public:0.11.7 public:0.11.6 public:0.11.5 public:0.11.4 public:0.11.3 public:0.11.2 public:0.11.1 public:0.11.0 public:0.10.1 public:0.10.0 public:0.9.0 public:0.8.8 public:0.8.7 public:0.8.6 public:0.8.5 public:0.8.3 public:0.8.2 public:0.8.1 public:0.8.0 public:0.7.2 public:0.7.1 public:0.7.0 public:0.6.1 public:0.6.0 public:0.5.0 public:0.4.1 public:0.4.0 public:0.3.1 public:0.3.0 public:0.2.1 public:0.2.0 public:0.1.26 public:0.1.25 public:0.1.24 public:0.1.23 public:0.1.22 public:0.1.21 public:0.1.20 public:0.1.19 public:0.1.18 public:0.1.17 public:0.1.16 public:0.1.15 public:0.1.14 public:0.1.13 public:0.1.12 public:0.1.11 public:0.1.10 public:0.1.9 public:0.1.8 public:0.1.7 public:0.1.6 public:0.1.5 public:0.1.4 public:0.1.3 public:0.1.2 public:0.1.1 public:0.1.0 public:0.0.9 public:0.0.8 public:0.0.7 public:0.0.6
...
compare: public:0.21.2
Branches Tags
public:test_real_obsidian public:main public:feat-userhashsalt public:p2p-rpc public:auto_conf_2 public:auto_conf public:beta public:alpha public:refactor_services_2 public:refactor_services public:dev public:svelteui public:forprivacy
public:0.25.60 public:0.25.59 public:0.25.58 public:0.25.57 public:0.25.56+patched5 public:0.25.56+patched4 public:0.25.56+patched3 public:0.25.56+patched2 public:0.25.56+patched1 public:0.25.56 public:0.25.55 public:0.25.54 public:0.25.53 public:0.25.52-patched-3 public:0.25.52-patched-2 public:0.25.52-patched-1 public:0.25.52 public:0.25.51 public:0.25.50 public:0.25.49 public:0.25.48 public:0.25.47 public:0.25.46 public:0.25.45 public:0.25.44 public:0.25.43-patched-9 public:0.25.43-patched-8 public:0.25.43-patched-7 public:0.25.43-patched-6 public:0.25.43-patched-5 public:0.25.43-patched-4 public:0.25.43-patched-3 public:0.25.43-patched-2 public:0.25.43-patched-1 public:0.25.43 public:0.25.42 public:0.25.41-patched-3 public:0.25.41-patched-2 public:0.25.41-patched-1 public:0.25.41 public:0.25.40 public:0.25.39 public:0.25.38 public:0.25.37 public:0.25.36 public:0.25.35 public:0.25.34 public:0.25.33 public:0.25.32 public:0.25.31 public:0.25.30 public:0.25.30-dev public:0.25.29 public:0.25.28 public:0.25.27 public:0.25.26 public:0.25.25 public:0.25.24 public:0.25.24.beta3 public:0.25.24.beta2 public:0.25.24.beta1 public:0.25.23 public:0.25.23.beta1 public:0.25.22 public:0.25.21 public:0.25.21.beta2 public:0.25.21.beta1 public:0.25.20 public:0.25.19 public:0.25.18 public:0.25.17 public:0.25.16 public:0.25.15 public:0.25.14 public:0.25.13 public:0.25.12 public:0.25.11 public:0.25.10 public:0.25.9 public:0.25.8 public:0.25.7 public:0.25.6 public:0.25.5 public:0.25.4 public:0.25.3 public:0.25.2 public:0.25.1 public:0.25.0 public:0.25.0-beta1 public:0.24.31 public:0.24.30 public:0.24.29 public:0.24.28 public:0.24.27 public:0.24.26 public:0.24.25 public:0.24.24 public:0.24.23 public:0.24.22 public:0.24.21 public:0.24.20 public:0.24.19 public:0.24.18 public:0.24.17 public:0.24.16 public:0.24.15 public:0.24.14 public:0.24.13 public:0.24.12 public:0.24.11 public:0.24.10 public:0.24.9 public:0.24.8 public:0.24.7 public:0.24.6 public:0.24.5 public:0.24.4 public:0.24.3 public:0.24.2 public:0.24.1 public:0.24.0 public:0.24.0.dev-rc8 public:0.24.0.dev-rc7 public:0.24.0.dev-rc6 public:0.24.0.dev-rc5 public:0.24.0.dev-rc4 public:0.24.0.dev-rc3 public:0.24.0.dev-rc2 public:0.23.23 public:0.23.22 public:0.23.21 public:0.23.20 public:0.23.19 public:0.23.18 public:0.23.17 public:0.23.16 public:0.23.15 public:0.23.14 public:0.23.13 public:0.23.12 public:0.23.11 public:0.23.10 public:0.23.9 public:0.23.8 public:0.23.7 public:0.23.6 public:0.23.5 public:0.23.4 public:0.23.3 public:0.23.2 public:0.23.1 public:0.23.0 public:0.22.19 public:0.22.18 public:0.22.17 public:0.22.16 public:0.22.15 public:0.22.14 public:0.22.13 public:0.22.12 public:0.22.11 public:0.22.10 public:0.22.9 public:0.22.8 public:0.22.7 public:0.22.6 public:0.22.5 public:0.22.4 public:0.22.3 public:0.22.2 public:0.22.1 public:0.22.0 public:0.21.5 public:0.21.4 public:0.21.3 public:0.21.2 public:0.21.1 public:0.21.0 public:0.20.7 public:0.20.6 public:0.20.5 public:0.20.4 public:0.20.3 public:0.20.2 public:0.20.1 public:0.20.0 public:0.19.23 public:0.19.22 public:0.19.21 public:0.19.20 public:0.19.19 public:0.19.18 public:0.19.17 public:0.19.16 public:0.19.15 public:0.19.14 public:0.19.13 public:0.19.12 public:0.19.11 public:0.19.10 public:0.19.9 public:0.19.8 public:0.19.7 public:0.19.6 public:0.19.5 public:0.19.4 public:0.19.3 public:0.19.2 public:0.19.1 public:0.19.0 public:0.18.6 public:0.18.5 public:0.18.4 public:0.18.3 public:0.18.2 public:0.18.1 public:0.18.0 public:0.17.34 public:0.17.33 public:0.17.32 public:0.17.31 public:0.17.30 public:0.17.29 public:0.17.28 public:0.17.27 public:0.17.26 public:0.17.25 public:0.17.24 public:0.17.23 public:0.17.22 public:0.17.21 public:0.17.20 public:0.17.19 public:0.17.18 public:0.17.17 public:0.17.16 public:0.17.15 public:0.17.14 public:0.17.13 public:0.17.12 public:0.17.11 public:0.17.10 public:0.17.9 public:0.17.8 public:0.17.7 public:0.17.6 public:0.17.5 public:0.17.4 public:0.17.3 public:0.17.2 public:0.17.1 public:0.17.0 public:0.16.8 public:0.16.7 public:0.16.6 public:0.16.5 public:0.16.4 public:0.16.3 public:0.16.2 public:0.16.1 public:0.16.0 public:0.15.10 public:0.15.9 public:0.15.8 public:0.15.7 public:0.15.6 public:0.15.5 public:0.15.4 public:0.15.3 public:0.15.2 public:0.15.1 public:0.15.0 public:0.14.8 public:0.14.7 public:0.14.6 public:0.14.5 public:0.14.4 public:0.14.3 public:0.14.2 public:0.14.1 public:0.14.0 public:0.13.4 public:0.13.3 public:0.13.2 public:0.13.1 public:0.13.0 public:0.12.5 public:0.12.4 public:0.12.3 public:0.12.2 public:0.12.1 public:0.12.0 public:0.11.10 public:0.11.9 public:0.11.8 public:0.11.7 public:0.11.6 public:0.11.5 public:0.11.4 public:0.11.3 public:0.11.2 public:0.11.1 public:0.11.0 public:0.10.1 public:0.10.0 public:0.9.0 public:0.8.8 public:0.8.7 public:0.8.6 public:0.8.5 public:0.8.3 public:0.8.2 public:0.8.1 public:0.8.0 public:0.7.2 public:0.7.1 public:0.7.0 public:0.6.1 public:0.6.0 public:0.5.0 public:0.4.1 public:0.4.0 public:0.3.1 public:0.3.0 public:0.2.1 public:0.2.0 public:0.1.26 public:0.1.25 public:0.1.24 public:0.1.23 public:0.1.22 public:0.1.21 public:0.1.20 public:0.1.19 public:0.1.18 public:0.1.17 public:0.1.16 public:0.1.15 public:0.1.14 public:0.1.13 public:0.1.12 public:0.1.11 public:0.1.10 public:0.1.9 public:0.1.8 public:0.1.7 public:0.1.6 public:0.1.5 public:0.1.4 public:0.1.3 public:0.1.2 public:0.1.1 public:0.1.0 public:0.0.9 public:0.0.8 public:0.0.7 public:0.0.6

6 Commits
0.20.7 ... 0.21.2

Author SHA1 Message Date
vorotamoroz
b337a05b5a bump 2023-11-27 07:13:15 +00:00
vorotamoroz
9ea6bee9d1 - Fixed: 
- No longer files are broken while rebuilding.
    - Now, Large binary files can be written correctly on a mobile platform.
    - Any decoding errors now make zero-byte files.
  - Modified:
    - All files are processed sequentially for each.
 2023-11-27 06:55:55 +00:00
vorotamoroz
9747c26d50 bump 2023-11-25 02:22:26 +09:00
vorotamoroz
bb4b764586 - Fixed: 
- No more infinity loops on larger files.
    - Show message on decode error.
  - Refactored:
    - Fixed to avoid obsolete global variables.
 2023-11-25 02:21:44 +09:00
vorotamoroz
279b4b41e5 bump 2023-11-24 10:32:46 +00:00
vorotamoroz
b644fb791d - Changes and performance improvements: 
- Now the saving files are processed by Blob.
    - The V2-Format has been reverted.
    - New encoding format has been enabled in default.
    - WARNING: Since this version, the compatibilities with older Filesystem LiveSync have been lost.
 2023-11-24 10:31:58 +00:00
18 changed files with 347 additions and 246 deletions
Expand all files Collapse all files

2
manifest.json
View File

@@ -1,7 +1,7 @@
{ {
"id": "obsidian-livesync", "id": "obsidian-livesync",
"name": "Self-hosted LiveSync", "name": "Self-hosted LiveSync",
"version": "0.20.7", "version": "0.21.2",
"minAppVersion": "0.9.12", "minAppVersion": "0.9.12",
"description": "Community implementation of self-hosted livesync. Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.", "description": "Community implementation of self-hosted livesync. Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.",
"author": "vorotamoroz", "author": "vorotamoroz",

4
package-lock.json generated
View File

@@ -1,12 +1,12 @@
{ {
"name": "obsidian-livesync", "name": "obsidian-livesync",
"version": "0.20.7", "version": "0.21.2",
"lockfileVersion": 2, "lockfileVersion": 2,
"requires": true, "requires": true,
"packages": { "packages": {
"": { "": {
"name": "obsidian-livesync", "name": "obsidian-livesync",
"version": "0.20.7", "version": "0.21.2",
"license": "MIT", "license": "MIT",
"dependencies": { "dependencies": {
"diff-match-patch": "^1.0.5", "diff-match-patch": "^1.0.5",

2
package.json
View File

@@ -1,6 +1,6 @@
{ {
"name": "obsidian-livesync", "name": "obsidian-livesync",
"version": "0.20.7", "version": "0.21.2",
"description": "Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.", "description": "Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.",
"main": "main.js", "main": "main.js",
"type": "module", "type": "module",

26
src/CmdConfigSync.ts
View File

@@ -1,13 +1,13 @@
import { writable } from 'svelte/store'; import { writable } from 'svelte/store';
import { Notice, type PluginManifest, parseYaml, normalizePath } from "./deps"; import { Notice, type PluginManifest, parseYaml, normalizePath } from "./deps";
import type { EntryDoc, LoadedEntry, InternalFileEntry, FilePathWithPrefix, FilePath, DocumentID, AnyEntry } from "./lib/src/types"; import type { EntryDoc, LoadedEntry, InternalFileEntry, FilePathWithPrefix, FilePath, DocumentID, AnyEntry, SavingEntry } from "./lib/src/types";
import { LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE, MODE_SELECTIVE } from "./lib/src/types"; import { LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE, MODE_SELECTIVE } from "./lib/src/types";
import { ICXHeader, PERIODIC_PLUGIN_SWEEP, } from "./types"; import { ICXHeader, PERIODIC_PLUGIN_SWEEP, } from "./types";
import { delay, getDocData } from "./lib/src/utils"; import { createTextBlob, delay, getDocData } from "./lib/src/utils";
import { Logger } from "./lib/src/logger"; import { Logger } from "./lib/src/logger";
import { WrappedNotice } from "./lib/src/wrapper"; import { WrappedNotice } from "./lib/src/wrapper";
import { readString, crc32CKHash, decodeBinary, encodeBinary } from "./lib/src/strbin"; import { readString, crc32CKHash, decodeBinary, arrayBufferToBase64 } from "./lib/src/strbin";
import { serialized } from "./lib/src/lock"; import { serialized } from "./lib/src/lock";
import { LiveSyncCommands } from "./LiveSyncCommands"; import { LiveSyncCommands } from "./LiveSyncCommands";
import { stripAllPrefixes } from "./lib/src/path"; import { stripAllPrefixes } from "./lib/src/path";
@@ -418,9 +418,9 @@ export class ConfigSync extends LiveSyncCommands {
await this.ensureDirectoryEx(path); await this.ensureDirectoryEx(path);
if (!content) { if (!content) {
const dt = decodeBinary(f.data); const dt = decodeBinary(f.data);
await this.app.vault.adapter.writeBinary(path, dt); await this.vaultAccess.adapterWrite(path, dt);
} else { } else {
await this.app.vault.adapter.write(path, content); await this.vaultAccess.adapterWrite(path, content);
} }
Logger(`Applying ${f.filename} of ${data.displayName || data.name}.. Done`); Logger(`Applying ${f.filename} of ${data.displayName || data.name}.. Done`);
@@ -540,16 +540,16 @@ export class ConfigSync extends LiveSyncCommands {
recentProcessedInternalFiles = [] as string[]; recentProcessedInternalFiles = [] as string[];
async makeEntryFromFile(path: FilePath): Promise<false | PluginDataExFile> { async makeEntryFromFile(path: FilePath): Promise<false | PluginDataExFile> {
const stat = await this.app.vault.adapter.stat(path); const stat = await this.vaultAccess.adapterStat(path);
let version: string | undefined; let version: string | undefined;
let displayName: string | undefined; let displayName: string | undefined;
if (!stat) { if (!stat) {
return false; return false;
} }
const contentBin = await this.app.vault.adapter.readBinary(path); const contentBin = await this.vaultAccess.adapterReadBinary(path);
let content: string[]; let content: string[];
try { try {
content = await encodeBinary(contentBin, this.settings.useV1); content = await arrayBufferToBase64(contentBin);
if (path.toLowerCase().endsWith("/manifest.json")) { if (path.toLowerCase().endsWith("/manifest.json")) {
const v = readString(new Uint8Array(contentBin)); const v = readString(new Uint8Array(contentBin));
try { try {
@@ -652,10 +652,10 @@ export class ConfigSync extends LiveSyncCommands {
return return
} }
const content = serialize(dt); const content = createTextBlob(serialize(dt));
try { try {
const old = await this.localDatabase.getDBEntryMeta(prefixedFileName, null, false); const old = await this.localDatabase.getDBEntryMeta(prefixedFileName, null, false);
let saveData: LoadedEntry; let saveData: SavingEntry;
if (old === false) { if (old === false) {
saveData = { saveData = {
_id: id, _id: id,
@@ -664,7 +664,7 @@ export class ConfigSync extends LiveSyncCommands {
mtime, mtime,
ctime: mtime, ctime: mtime,
datatype: "newnote", datatype: "newnote",
size: content.length, size: content.size,
children: [], children: [],
deleted: false, deleted: false,
type: "newnote", type: "newnote",
@@ -679,7 +679,7 @@ export class ConfigSync extends LiveSyncCommands {
...old, ...old,
data: content, data: content,
mtime, mtime,
size: content.length, size: content.size,
datatype: "newnote", datatype: "newnote",
children: [], children: [],
deleted: false, deleted: false,
@@ -701,7 +701,7 @@ export class ConfigSync extends LiveSyncCommands {
async watchVaultRawEventsAsync(path: FilePath) { async watchVaultRawEventsAsync(path: FilePath) {
if (!this.settings.usePluginSync) return false; if (!this.settings.usePluginSync) return false;
if (!this.isTargetPath(path)) return false; if (!this.isTargetPath(path)) return false;
const stat = await this.app.vault.adapter.stat(path); const stat = await this.vaultAccess.adapterStat(path);
// Make sure that target is a file. // Make sure that target is a file.
if (stat && stat.type != "file") if (stat && stat.type != "file")
return false; return false;

40
src/CmdHiddenFileSync.ts
View File

@@ -1,7 +1,7 @@
import { normalizePath, type PluginManifest } from "./deps"; import { normalizePath, type PluginManifest } from "./deps";
import { type EntryDoc, type LoadedEntry, type InternalFileEntry, type FilePathWithPrefix, type FilePath, LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE, MODE_SELECTIVE, MODE_PAUSED } from "./lib/src/types"; import { type EntryDoc, type LoadedEntry, type InternalFileEntry, type FilePathWithPrefix, type FilePath, LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE, MODE_SELECTIVE, MODE_PAUSED, type SavingEntry } from "./lib/src/types";
import { type InternalFileInfo, ICHeader, ICHeaderEnd } from "./types"; import { type InternalFileInfo, ICHeader, ICHeaderEnd } from "./types";
import { Parallels, delay, isDocContentSame } from "./lib/src/utils"; import { Parallels, createBinaryBlob, delay, isDocContentSame } from "./lib/src/utils";
import { Logger } from "./lib/src/logger"; import { Logger } from "./lib/src/logger";
import { PouchDB } from "./lib/src/pouchdb-browser.js"; import { PouchDB } from "./lib/src/pouchdb-browser.js";
import { scheduleTask, isInternalMetadata, PeriodicProcessor } from "./utils"; import { scheduleTask, isInternalMetadata, PeriodicProcessor } from "./utils";
@@ -103,7 +103,7 @@ export class HiddenFileSync extends LiveSyncCommands {
Logger(`Hidden file skipped: ${path} is synchronized in customization sync.`, LOG_LEVEL_VERBOSE); Logger(`Hidden file skipped: ${path} is synchronized in customization sync.`, LOG_LEVEL_VERBOSE);
return; return;
} }
const stat = await this.app.vault.adapter.stat(path); const stat = await this.vaultAccess.adapterStat(path);
// sometimes folder is coming. // sometimes folder is coming.
if (stat && stat.type != "file") if (stat && stat.type != "file")
return; return;
@@ -171,12 +171,12 @@ export class HiddenFileSync extends LiveSyncCommands {
if (result) { if (result) {
Logger(`Object merge:${path}`, LOG_LEVEL_INFO); Logger(`Object merge:${path}`, LOG_LEVEL_INFO);
const filename = stripAllPrefixes(path); const filename = stripAllPrefixes(path);
const isExists = await this.app.vault.adapter.exists(filename); const isExists = await this.plugin.vaultAccess.adapterExists(filename);
if (!isExists) { if (!isExists) {
await this.ensureDirectoryEx(filename); await this.ensureDirectoryEx(filename);
} }
await this.app.vault.adapter.write(filename, result); await this.plugin.vaultAccess.adapterWrite(filename, result);
const stat = await this.app.vault.adapter.stat(filename); const stat = await this.vaultAccess.adapterStat(filename);
await this.storeInternalFileToDatabase({ path: filename, ...stat }); await this.storeInternalFileToDatabase({ path: filename, ...stat });
await this.extractInternalFileFromDatabase(filename); await this.extractInternalFileFromDatabase(filename);
await this.localDatabase.removeRaw(id, revB); await this.localDatabase.removeRaw(id, revB);
@@ -408,10 +408,10 @@ export class HiddenFileSync extends LiveSyncCommands {
const id = await this.path2id(file.path, ICHeader); const id = await this.path2id(file.path, ICHeader);
const prefixedFileName = addPrefix(file.path, ICHeader); const prefixedFileName = addPrefix(file.path, ICHeader);
const contentBin = await this.app.vault.adapter.readBinary(file.path); const contentBin = await this.plugin.vaultAccess.adapterReadBinary(file.path);
let content: string[]; let content: Blob;
try { try {
content = await encodeBinary(contentBin, this.settings.useV1); content = createBinaryBlob(contentBin);
} catch (ex) { } catch (ex) {
Logger(`The file ${file.path} could not be encoded`); Logger(`The file ${file.path} could not be encoded`);
Logger(ex, LOG_LEVEL_VERBOSE); Logger(ex, LOG_LEVEL_VERBOSE);
@@ -421,7 +421,7 @@ export class HiddenFileSync extends LiveSyncCommands {
return await serialized("file-" + prefixedFileName, async () => { return await serialized("file-" + prefixedFileName, async () => {
try { try {
const old = await this.localDatabase.getDBEntry(prefixedFileName, null, false, false); const old = await this.localDatabase.getDBEntry(prefixedFileName, null, false, false);
let saveData: LoadedEntry; let saveData: SavingEntry;
if (old === false) { if (old === false) {
saveData = { saveData = {
_id: id, _id: id,
@@ -511,7 +511,7 @@ export class HiddenFileSync extends LiveSyncCommands {
} }
async extractInternalFileFromDatabase(filename: FilePath, force = false) { async extractInternalFileFromDatabase(filename: FilePath, force = false) {
const isExists = await this.app.vault.adapter.exists(filename); const isExists = await this.plugin.vaultAccess.adapterExists(filename);
const prefixedFileName = addPrefix(filename, ICHeader); const prefixedFileName = addPrefix(filename, ICHeader);
if (await this.plugin.isIgnoredByIgnoreFiles(filename)) { if (await this.plugin.isIgnoredByIgnoreFiles(filename)) {
return; return;
@@ -534,7 +534,7 @@ export class HiddenFileSync extends LiveSyncCommands {
Logger(`STORAGE <x- DB:${filename}: deleted (hidden) Deleted on DB, but the file is already not found on storage.`); Logger(`STORAGE <x- DB:${filename}: deleted (hidden) Deleted on DB, but the file is already not found on storage.`);
} else { } else {
Logger(`STORAGE <x- DB:${filename}: deleted (hidden).`); Logger(`STORAGE <x- DB:${filename}: deleted (hidden).`);
await this.app.vault.adapter.remove(filename); await this.plugin.vaultAccess.adapterRemove(filename);
try { try {
//@ts-ignore internalAPI //@ts-ignore internalAPI
await this.app.vault.adapter.reconcileInternalFile(filename); await this.app.vault.adapter.reconcileInternalFile(filename);
@@ -547,7 +547,7 @@ export class HiddenFileSync extends LiveSyncCommands {
} }
if (!isExists) { if (!isExists) {
await this.ensureDirectoryEx(filename); await this.ensureDirectoryEx(filename);
await this.app.vault.adapter.writeBinary(filename, decodeBinary(fileOnDB.data), { mtime: fileOnDB.mtime, ctime: fileOnDB.ctime }); await this.plugin.vaultAccess.adapterWrite(filename, decodeBinary(fileOnDB.data), { mtime: fileOnDB.mtime, ctime: fileOnDB.ctime });
try { try {
//@ts-ignore internalAPI //@ts-ignore internalAPI
await this.app.vault.adapter.reconcileInternalFile(filename); await this.app.vault.adapter.reconcileInternalFile(filename);
@@ -558,13 +558,13 @@ export class HiddenFileSync extends LiveSyncCommands {
Logger(`STORAGE <-- DB:${filename}: written (hidden,new${force ? ", force" : ""})`); Logger(`STORAGE <-- DB:${filename}: written (hidden,new${force ? ", force" : ""})`);
return true; return true;
} else { } else {
const contentBin = await this.app.vault.adapter.readBinary(filename); const contentBin = await this.plugin.vaultAccess.adapterReadBinary(filename);
const content = await encodeBinary(contentBin, this.settings.useV1); const content = await encodeBinary(contentBin);
if (isDocContentSame(content, fileOnDB.data) && !force) { if (isDocContentSame(content, fileOnDB.data) && !force) {
// Logger(`STORAGE <-- DB:${filename}: skipped (hidden) Not changed`, LOG_LEVEL_VERBOSE); // Logger(`STORAGE <-- DB:${filename}: skipped (hidden) Not changed`, LOG_LEVEL_VERBOSE);
return true; return true;
} }
await this.app.vault.adapter.writeBinary(filename, decodeBinary(fileOnDB.data), { mtime: fileOnDB.mtime, ctime: fileOnDB.ctime }); await this.plugin.vaultAccess.adapterWrite(filename, decodeBinary(fileOnDB.data), { mtime: fileOnDB.mtime, ctime: fileOnDB.ctime });
try { try {
//@ts-ignore internalAPI //@ts-ignore internalAPI
await this.app.vault.adapter.reconcileInternalFile(filename); await this.app.vault.adapter.reconcileInternalFile(filename);
@@ -613,12 +613,12 @@ export class HiddenFileSync extends LiveSyncCommands {
} }
} }
if (!keep && result) { if (!keep && result) {
const isExists = await this.app.vault.adapter.exists(filename); const isExists = await this.plugin.vaultAccess.adapterExists(filename);
if (!isExists) { if (!isExists) {
await this.ensureDirectoryEx(filename); await this.ensureDirectoryEx(filename);
} }
await this.app.vault.adapter.write(filename, result); await this.plugin.vaultAccess.adapterWrite(filename, result);
const stat = await this.app.vault.adapter.stat(filename); const stat = await this.plugin.vaultAccess.adapterStat(filename);
await this.storeInternalFileToDatabase({ path: filename, ...stat }, true); await this.storeInternalFileToDatabase({ path: filename, ...stat }, true);
try { try {
//@ts-ignore internalAPI //@ts-ignore internalAPI
@@ -657,7 +657,7 @@ export class HiddenFileSync extends LiveSyncCommands {
const files = filenames.filter(path => synchronisedInConfigSync.every(filterFile => !path.toLowerCase().startsWith(filterFile))).map(async (e) => { const files = filenames.filter(path => synchronisedInConfigSync.every(filterFile => !path.toLowerCase().startsWith(filterFile))).map(async (e) => {
return { return {
path: e as FilePath, path: e as FilePath,
stat: await this.app.vault.adapter.stat(e) stat: await this.plugin.vaultAccess.adapterStat(e)
}; };
}); });
const result: InternalFileInfo[] = []; const result: InternalFileInfo[] = [];

34
src/CmdPluginAndTheirSettings.ts
View File

@@ -1,8 +1,8 @@
import { normalizePath, type PluginManifest } from "./deps"; import { normalizePath, type PluginManifest } from "./deps";
import type { DocumentID, EntryDoc, FilePathWithPrefix, LoadedEntry } from "./lib/src/types"; import type { DocumentID, EntryDoc, FilePathWithPrefix, LoadedEntry, SavingEntry } from "./lib/src/types";
import { LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE } from "./lib/src/types"; import { LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE } from "./lib/src/types";
import { type PluginDataEntry, PERIODIC_PLUGIN_SWEEP, type PluginList, type DevicePluginList, PSCHeader, PSCHeaderEnd } from "./types"; import { type PluginDataEntry, PERIODIC_PLUGIN_SWEEP, type PluginList, type DevicePluginList, PSCHeader, PSCHeaderEnd } from "./types";
import { getDocData, isDocContentSame } from "./lib/src/utils"; import { createTextBlob, getDocData, isDocContentSame } from "./lib/src/utils";
import { Logger } from "./lib/src/logger"; import { Logger } from "./lib/src/logger";
import { PouchDB } from "./lib/src/pouchdb-browser.js"; import { PouchDB } from "./lib/src/pouchdb-browser.js";
import { isPluginMetadata, PeriodicProcessor } from "./utils"; import { isPluginMetadata, PeriodicProcessor } from "./utils";
@@ -186,18 +186,17 @@ export class PluginAndTheirSettings extends LiveSyncCommands {
} }
Logger(`Reading plugin:${m.name}(${m.id})`, LOG_LEVEL_VERBOSE); Logger(`Reading plugin:${m.name}(${m.id})`, LOG_LEVEL_VERBOSE);
const path = normalizePath(m.dir) + "/"; const path = normalizePath(m.dir) + "/";
const adapter = this.app.vault.adapter;
const files = ["manifest.json", "main.js", "styles.css", "data.json"]; const files = ["manifest.json", "main.js", "styles.css", "data.json"];
const pluginData: { [key: string]: string; } = {}; const pluginData: { [key: string]: string; } = {};
for (const file of files) { for (const file of files) {
const thePath = path + file; const thePath = path + file;
if (await adapter.exists(thePath)) { if (await this.plugin.vaultAccess.adapterExists(thePath)) {
pluginData[file] = await adapter.read(thePath); pluginData[file] = await this.plugin.vaultAccess.adapterRead(thePath);
} }
} }
let mtime = 0; let mtime = 0;
if (await adapter.exists(path + "/data.json")) { if (await this.plugin.vaultAccess.adapterExists(path + "/data.json")) {
mtime = (await adapter.stat(path + "/data.json")).mtime; mtime = (await this.plugin.vaultAccess.adapterStat(path + "/data.json")).mtime;
} }
const p: PluginDataEntry = { const p: PluginDataEntry = {
@@ -211,13 +210,14 @@ export class PluginAndTheirSettings extends LiveSyncCommands {
mtime: mtime, mtime: mtime,
type: "plugin", type: "plugin",
}; };
const d: LoadedEntry = { const blob = createTextBlob(JSON.stringify(p));
const d: SavingEntry = {
_id: p._id, _id: p._id,
path: p._id as string as FilePathWithPrefix, path: p._id as string as FilePathWithPrefix,
data: JSON.stringify(p), data: blob,
ctime: mtime, ctime: mtime,
mtime: mtime, mtime: mtime,
size: 0, size: blob.size,
children: [], children: [],
datatype: "plain", datatype: "plain",
type: "plain" type: "plain"
@@ -268,7 +268,6 @@ export class PluginAndTheirSettings extends LiveSyncCommands {
async applyPluginData(plugin: PluginDataEntry) { async applyPluginData(plugin: PluginDataEntry) {
await serialized("plugin-" + plugin.manifest.id, async () => { await serialized("plugin-" + plugin.manifest.id, async () => {
const pluginTargetFolderPath = normalizePath(plugin.manifest.dir) + "/"; const pluginTargetFolderPath = normalizePath(plugin.manifest.dir) + "/";
const adapter = this.app.vault.adapter;
// @ts-ignore // @ts-ignore
const stat = this.app.plugins.enabledPlugins.has(plugin.manifest.id) == true; const stat = this.app.plugins.enabledPlugins.has(plugin.manifest.id) == true;
if (stat) { if (stat) {
@@ -277,7 +276,7 @@ export class PluginAndTheirSettings extends LiveSyncCommands {
Logger(`Unload plugin:${plugin.manifest.id}`, LOG_LEVEL_NOTICE); Logger(`Unload plugin:${plugin.manifest.id}`, LOG_LEVEL_NOTICE);
} }
if (plugin.dataJson) if (plugin.dataJson)
await adapter.write(pluginTargetFolderPath + "data.json", plugin.dataJson); await this.plugin.vaultAccess.adapterWrite(pluginTargetFolderPath + "data.json", plugin.dataJson);
Logger("wrote:" + pluginTargetFolderPath + "data.json", LOG_LEVEL_NOTICE); Logger("wrote:" + pluginTargetFolderPath + "data.json", LOG_LEVEL_NOTICE);
if (stat) { if (stat) {
// @ts-ignore // @ts-ignore
@@ -298,14 +297,13 @@ export class PluginAndTheirSettings extends LiveSyncCommands {
} }
const pluginTargetFolderPath = normalizePath(plugin.manifest.dir) + "/"; const pluginTargetFolderPath = normalizePath(plugin.manifest.dir) + "/";
const adapter = this.app.vault.adapter; if ((await this.plugin.vaultAccess.adapterExists(pluginTargetFolderPath)) === false) {
if ((await adapter.exists(pluginTargetFolderPath)) === false) { await this.app.vault.adapter.mkdir(pluginTargetFolderPath);
await adapter.mkdir(pluginTargetFolderPath);
} }
await adapter.write(pluginTargetFolderPath + "main.js", plugin.mainJs); await this.plugin.vaultAccess.adapterWrite(pluginTargetFolderPath + "main.js", plugin.mainJs);
await adapter.write(pluginTargetFolderPath + "manifest.json", plugin.manifestJson); await this.plugin.vaultAccess.adapterWrite(pluginTargetFolderPath + "manifest.json", plugin.manifestJson);
if (plugin.styleCss) if (plugin.styleCss)
await adapter.write(pluginTargetFolderPath + "styles.css", plugin.styleCss); await this.plugin.vaultAccess.adapterWrite(pluginTargetFolderPath + "styles.css", plugin.styleCss);
if (stat) { if (stat) {
// @ts-ignore // @ts-ignore
await this.app.plugins.loadPlugin(plugin.manifest.id); await this.app.plugins.loadPlugin(plugin.manifest.id);

4
src/CmdSetupLiveSync.ts
View File

@@ -60,7 +60,7 @@ export class SetupLiveSync extends LiveSyncCommands {
delete setting[k]; delete setting[k];
} }
} }
const encryptedSetting = encodeURIComponent(await encrypt(JSON.stringify(setting), encryptingPassphrase, false, true)); const encryptedSetting = encodeURIComponent(await encrypt(JSON.stringify(setting), encryptingPassphrase, false));
const uri = `${configURIBase}${encryptedSetting}`; const uri = `${configURIBase}${encryptedSetting}`;
await navigator.clipboard.writeText(uri); await navigator.clipboard.writeText(uri);
Logger("Setup URI copied to clipboard", LOG_LEVEL_NOTICE); Logger("Setup URI copied to clipboard", LOG_LEVEL_NOTICE);
@@ -70,7 +70,7 @@ export class SetupLiveSync extends LiveSyncCommands {
if (encryptingPassphrase === false) if (encryptingPassphrase === false)
return; return;
const setting = { ...this.settings, configPassphraseStore: "", encryptedCouchDBConnection: "", encryptedPassphrase: "" }; const setting = { ...this.settings, configPassphraseStore: "", encryptedCouchDBConnection: "", encryptedPassphrase: "" };
const encryptedSetting = encodeURIComponent(await encrypt(JSON.stringify(setting), encryptingPassphrase, false, true)); const encryptedSetting = encodeURIComponent(await encrypt(JSON.stringify(setting), encryptingPassphrase, false));
const uri = `${configURIBase}${encryptedSetting}`; const uri = `${configURIBase}${encryptedSetting}`;
await navigator.clipboard.writeText(uri); await navigator.clipboard.writeText(uri);
Logger("Setup URI copied to clipboard", LOG_LEVEL_NOTICE); Logger("Setup URI copied to clipboard", LOG_LEVEL_NOTICE);

4
src/DocumentHistoryModal.ts
View File

@@ -200,11 +200,11 @@ export class DocumentHistoryModal extends Modal {
Logger("Path is not valid to write content.", LOG_LEVEL_INFO); Logger("Path is not valid to write content.", LOG_LEVEL_INFO);
} }
if (this.currentDoc?.datatype == "plain") { if (this.currentDoc?.datatype == "plain") {
await this.app.vault.adapter.write(pathToWrite, getDocData(this.currentDoc.data)); await this.plugin.vaultAccess.adapterWrite(pathToWrite, getDocData(this.currentDoc.data));
await focusFile(pathToWrite); await focusFile(pathToWrite);
this.close(); this.close();
} else if (this.currentDoc?.datatype == "newnote") { } else if (this.currentDoc?.datatype == "newnote") {
await this.app.vault.adapter.writeBinary(pathToWrite, decodeBinary(this.currentDoc.data)); await this.plugin.vaultAccess.adapterWrite(pathToWrite, decodeBinary(this.currentDoc.data));
await focusFile(pathToWrite); await focusFile(pathToWrite);
this.close(); this.close();
} else { } else {

11
src/GlobalHistory.svelte
View File

@@ -2,12 +2,11 @@
import ObsidianLiveSyncPlugin from "./main"; import ObsidianLiveSyncPlugin from "./main";
import { onDestroy, onMount } from "svelte"; import { onDestroy, onMount } from "svelte";
import type { AnyEntry, FilePathWithPrefix } from "./lib/src/types"; import type { AnyEntry, FilePathWithPrefix } from "./lib/src/types";
import { getDocData, isDocContentSame } from "./lib/src/utils"; import { createBinaryBlob, getDocData, isDocContentSame } from "./lib/src/utils";
import { diff_match_patch } from "./deps"; import { diff_match_patch } from "./deps";
import { DocumentHistoryModal } from "./DocumentHistoryModal"; import { DocumentHistoryModal } from "./DocumentHistoryModal";
import { isPlainText, stripAllPrefixes } from "./lib/src/path"; import { isPlainText, stripAllPrefixes } from "./lib/src/path";
import { TFile } from "./deps"; import { TFile } from "./deps";
import { encodeBinary } from "./lib/src/strbin";
export let plugin: ObsidianLiveSyncPlugin; export let plugin: ObsidianLiveSyncPlugin;
let showDiffInfo = false; let showDiffInfo = false;
@@ -108,15 +107,15 @@
} }
if (rev == docA._rev) { if (rev == docA._rev) {
if (checkStorageDiff) { if (checkStorageDiff) {
const abs = plugin.app.vault.getAbstractFileByPath(stripAllPrefixes(plugin.getPath(docA))); const abs = plugin.vaultAccess.getAbstractFileByPath(stripAllPrefixes(plugin.getPath(docA)));
if (abs instanceof TFile) { if (abs instanceof TFile) {
let result = false; let result = false;
if (isPlainText(docA.path)) { if (isPlainText(docA.path)) {
const data = await plugin.app.vault.read(abs); const data = await plugin.vaultAccess.adapterRead(abs);
result = isDocContentSame(data, doc.data); result = isDocContentSame(data, doc.data);
} else { } else {
const data = await plugin.app.vault.readBinary(abs); const data = await plugin.vaultAccess.adapterReadBinary(abs);
const dataEEncoded = await encodeBinary(data, plugin.settings.useV1); const dataEEncoded = createBinaryBlob(data);
result = isDocContentSame(dataEEncoded, doc.data); result = isDocContentSame(dataEEncoded, doc.data);
} }
if (result) { if (result) {

3
src/LiveSyncCommands.ts
View File

@@ -14,6 +14,9 @@ export abstract class LiveSyncCommands {
get localDatabase() { get localDatabase() {
return this.plugin.localDatabase; return this.plugin.localDatabase;
} }
get vaultAccess() {
return this.plugin.vaultAccess;
}
id2path(id: DocumentID, entry?: EntryHasPath, stripPrefix?: boolean): FilePathWithPrefix { id2path(id: DocumentID, entry?: EntryHasPath, stripPrefix?: boolean): FilePathWithPrefix {
return this.plugin.id2path(id, entry, stripPrefix); return this.plugin.id2path(id, entry, stripPrefix);
} }

9
src/ObsidianLiveSyncSettingTab.ts
View File

@@ -1832,15 +1832,6 @@ ${stringifyYaml(pluginConfig)}`;
await this.plugin.saveSettings(); await this.plugin.saveSettings();
}) })
); );
new Setting(containerHatchEl)
.setName("Use binary and encryption version 1")
.setDesc("Use the previous data format for other products which shares the remote database.")
.addToggle((toggle) =>
toggle.setValue(this.plugin.settings.useV1).onChange(async (value) => {
this.plugin.settings.useV1 = value;
await this.plugin.saveSettings();
})
);
addScreenElement("50", containerHatchEl); addScreenElement("50", containerHatchEl);

118
src/SerializedFileAccess.ts Normal file
View File

@@ -0,0 +1,118 @@
import { type App, TFile, type DataWriteOptions, TFolder, TAbstractFile } from "./deps";
import { serialized } from "./lib/src/lock";
import type { FilePath } from "./lib/src/types";
function getFileLockKey(file: TFile | TFolder | string) {
return `fl:${typeof (file) == "string" ? file : file.path}`;
}
function toArrayBuffer(arr: Uint8Array | ArrayBuffer | DataView): ArrayBufferLike {
if (arr instanceof Uint8Array) {
return arr.buffer;
}
if (arr instanceof DataView) {
return arr.buffer;
}
return arr;
}
export class SerializedFileAccess {
app: App
constructor(app: App) {
this.app = app;
}
async adapterStat(file: TFile | string) {
const path = file instanceof TFile ? file.path : file;
return await serialized(getFileLockKey(path), () => this.app.vault.adapter.stat(path));
}
async adapterExists(file: TFile | string) {
const path = file instanceof TFile ? file.path : file;
return await serialized(getFileLockKey(path), () => this.app.vault.adapter.exists(path));
}
async adapterRemove(file: TFile | string) {
const path = file instanceof TFile ? file.path : file;
return await serialized(getFileLockKey(path), () => this.app.vault.adapter.remove(path));
}
async adapterRead(file: TFile | string) {
const path = file instanceof TFile ? file.path : file;
return await serialized(getFileLockKey(path), () => this.app.vault.adapter.read(path));
}
async adapterReadBinary(file: TFile | string) {
const path = file instanceof TFile ? file.path : file;
return await serialized(getFileLockKey(path), () => this.app.vault.adapter.readBinary(path));
}
async adapterWrite(file: TFile | string, data: string | ArrayBuffer | Uint8Array, options?: DataWriteOptions) {
const path = file instanceof TFile ? file.path : file;
if (typeof (data) === "string") {
return await serialized(getFileLockKey(path), () => this.app.vault.adapter.write(path, data, options));
} else {
return await serialized(getFileLockKey(path), () => this.app.vault.adapter.writeBinary(path, toArrayBuffer(data), options));
}
}
async vaultCacheRead(file: TFile) {
return await serialized(getFileLockKey(file), () => this.app.vault.cachedRead(file));
}
async vaultRead(file: TFile) {
return await serialized(getFileLockKey(file), () => this.app.vault.read(file));
}
async vaultReadBinary(file: TFile) {
return await serialized(getFileLockKey(file), () => this.app.vault.readBinary(file));
}
async vaultModify(file: TFile, data: string | ArrayBuffer | Uint8Array, options?: DataWriteOptions) {
if (typeof (data) === "string") {
return await serialized(getFileLockKey(file), () => this.app.vault.modify(file, data, options));
} else {
return await serialized(getFileLockKey(file), () => this.app.vault.modifyBinary(file, toArrayBuffer(data), options));
}
}
async vaultCreate(path: string, data: string | ArrayBuffer | Uint8Array, options?: DataWriteOptions): Promise<TFile> {
if (typeof (data) === "string") {
return await serialized(getFileLockKey(path), () => this.app.vault.create(path, data, options));
} else {
return await serialized(getFileLockKey(path), () => this.app.vault.createBinary(path, toArrayBuffer(data), options));
}
}
async delete(file: TFile | TFolder, force = false) {
return await serialized(getFileLockKey(file), () => this.app.vault.delete(file, force));
}
async trash(file: TFile | TFolder, force = false) {
return await serialized(getFileLockKey(file), () => this.app.vault.trash(file, force));
}
getAbstractFileByPath(path: FilePath | string): TAbstractFile | null {
// Disabled temporary.
return this.app.vault.getAbstractFileByPath(path);
// // Hidden API but so useful.
// // @ts-ignore
// if ("getAbstractFileByPathInsensitive" in app.vault && (app.vault.adapter?.insensitive ?? false)) {
// // @ts-ignore
// return app.vault.getAbstractFileByPathInsensitive(path);
// } else {
// return app.vault.getAbstractFileByPath(path);
// }
}
touchedFiles: string[] = [];
touch(file: TFile | FilePath) {
const f = file instanceof TFile ? file : this.getAbstractFileByPath(file) as TFile;
const key = `${f.path}-${f.stat.mtime}-${f.stat.size}`;
this.touchedFiles.unshift(key);
this.touchedFiles = this.touchedFiles.slice(0, 100);
}
recentlyTouched(file: TFile) {
const key = `${file.path}-${file.stat.mtime}-${file.stat.size}`;
if (this.touchedFiles.indexOf(key) == -1) return false;
return true;
}
clearTouched() {
this.touchedFiles = [];
}
}

12
src/StorageEventManager.ts
View File

@@ -1,9 +1,9 @@
import type { SerializedFileAccess } from "./SerializedFileAccess";
import { Plugin, TAbstractFile, TFile, TFolder } from "./deps"; import { Plugin, TAbstractFile, TFile, TFolder } from "./deps";
import { isPlainText, shouldBeIgnored } from "./lib/src/path"; import { isPlainText, shouldBeIgnored } from "./lib/src/path";
import { getGlobalStore } from "./lib/src/store"; import { getGlobalStore } from "./lib/src/store";
import { type FilePath, type ObsidianLiveSyncSettings } from "./lib/src/types"; import { type FilePath, type ObsidianLiveSyncSettings } from "./lib/src/types";
import { type FileEventItem, type FileEventType, type FileInfo, type InternalFileInfo, type queueItem } from "./types"; import { type FileEventItem, type FileEventType, type FileInfo, type InternalFileInfo, type queueItem } from "./types";
import { recentlyTouched } from "./utils";
export abstract class StorageEventManager { export abstract class StorageEventManager {
@@ -16,6 +16,7 @@ type LiveSyncForStorageEventManager = Plugin &
{ {
settings: ObsidianLiveSyncSettings settings: ObsidianLiveSyncSettings
ignoreFiles: string[], ignoreFiles: string[],
vaultAccess: SerializedFileAccess
} & { } & {
isTargetFile: (file: string | TAbstractFile) => Promise<boolean>, isTargetFile: (file: string | TAbstractFile) => Promise<boolean>,
procFileEvent: (applyBatch?: boolean) => Promise<any>, procFileEvent: (applyBatch?: boolean) => Promise<any>,
@@ -105,15 +106,14 @@ export class StorageEventManagerObsidian extends StorageEventManager {
let cache: null | string | ArrayBuffer; let cache: null | string | ArrayBuffer;
// new file or something changed, cache the changes. // new file or something changed, cache the changes.
if (file instanceof TFile && (type == "CREATE" || type == "CHANGED")) { if (file instanceof TFile && (type == "CREATE" || type == "CHANGED")) {
if (recentlyTouched(file)) { if (this.plugin.vaultAccess.recentlyTouched(file)) {
continue; continue;
} }
if (!isPlainText(file.name)) { if (!isPlainText(file.name)) {
cache = await this.plugin.app.vault.readBinary(file); cache = await this.plugin.vaultAccess.vaultReadBinary(file);
} else { } else {
// cache = await this.app.vault.read(file); cache = await this.plugin.vaultAccess.vaultCacheRead(file);
cache = await this.plugin.app.vault.cachedRead(file); if (!cache) cache = await this.plugin.vaultAccess.vaultRead(file);
if (!cache) cache = await this.plugin.app.vault.read(file);
} }
} }
if (type == "DELETE" || type == "RENAME") { if (type == "DELETE" || type == "RENAME") {

2
src/lib

Submodule src/lib updated: 4a4e7a26b4...7e79c27035

119
src/main.ts
View File

@@ -2,21 +2,21 @@ const isDebug = false;
import { type Diff, DIFF_DELETE, DIFF_EQUAL, DIFF_INSERT, diff_match_patch } from "./deps"; import { type Diff, DIFF_DELETE, DIFF_EQUAL, DIFF_INSERT, diff_match_patch } from "./deps";
import { debounce, Notice, Plugin, TFile, addIcon, TFolder, normalizePath, TAbstractFile, Editor, MarkdownView, type RequestUrlParam, type RequestUrlResponse, requestUrl, type MarkdownFileInfo } from "./deps"; import { debounce, Notice, Plugin, TFile, addIcon, TFolder, normalizePath, TAbstractFile, Editor, MarkdownView, type RequestUrlParam, type RequestUrlResponse, requestUrl, type MarkdownFileInfo } from "./deps";
import { type EntryDoc, type LoadedEntry, type ObsidianLiveSyncSettings, type diff_check_result, type diff_result_leaf, type EntryBody, LOG_LEVEL, VER, DEFAULT_SETTINGS, type diff_result, FLAGMD_REDFLAG, SYNCINFO_ID, SALT_OF_PASSPHRASE, type ConfigPassphraseStore, type CouchDBConnection, FLAGMD_REDFLAG2, FLAGMD_REDFLAG3, PREFIXMD_LOGFILE, type DatabaseConnectingStatus, type EntryHasPath, type DocumentID, type FilePathWithPrefix, type FilePath, type AnyEntry, LOG_LEVEL_DEBUG, LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_URGENT, LOG_LEVEL_VERBOSE, } from "./lib/src/types"; import { type EntryDoc, type LoadedEntry, type ObsidianLiveSyncSettings, type diff_check_result, type diff_result_leaf, type EntryBody, LOG_LEVEL, VER, DEFAULT_SETTINGS, type diff_result, FLAGMD_REDFLAG, SYNCINFO_ID, SALT_OF_PASSPHRASE, type ConfigPassphraseStore, type CouchDBConnection, FLAGMD_REDFLAG2, FLAGMD_REDFLAG3, PREFIXMD_LOGFILE, type DatabaseConnectingStatus, type EntryHasPath, type DocumentID, type FilePathWithPrefix, type FilePath, type AnyEntry, LOG_LEVEL_DEBUG, LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_URGENT, LOG_LEVEL_VERBOSE, type SavingEntry, } from "./lib/src/types";
import { type InternalFileInfo, type queueItem, type CacheData, type FileEventItem, FileWatchEventQueueMax } from "./types"; import { type InternalFileInfo, type queueItem, type CacheData, type FileEventItem, FileWatchEventQueueMax } from "./types";
import { arrayToChunkedArray, getDocData, isDocContentSame } from "./lib/src/utils"; import { arrayToChunkedArray, createBinaryBlob, createTextBlob, getDocData, isDocContentSame } from "./lib/src/utils";
import { Logger, setGlobalLogFunction } from "./lib/src/logger"; import { Logger, setGlobalLogFunction } from "./lib/src/logger";
import { PouchDB } from "./lib/src/pouchdb-browser.js"; import { PouchDB } from "./lib/src/pouchdb-browser.js";
import { ConflictResolveModal } from "./ConflictResolveModal"; import { ConflictResolveModal } from "./ConflictResolveModal";
import { ObsidianLiveSyncSettingTab } from "./ObsidianLiveSyncSettingTab"; import { ObsidianLiveSyncSettingTab } from "./ObsidianLiveSyncSettingTab";
import { DocumentHistoryModal } from "./DocumentHistoryModal"; import { DocumentHistoryModal } from "./DocumentHistoryModal";
import { applyPatch, cancelAllPeriodicTask, cancelAllTasks, cancelTask, generatePatchObj, id2path, isObjectMargeApplicable, isSensibleMargeApplicable, flattenObject, path2id, scheduleTask, tryParseJSON, createFile, modifyFile, isValidPath, getAbstractFileByPath, touch, recentlyTouched, isInternalMetadata, isPluginMetadata, stripInternalMetadataPrefix, isChunk, askSelectString, askYesNo, askString, PeriodicProcessor, clearTouched, getPath, getPathWithoutPrefix, getPathFromTFile, performRebuildDB, memoIfNotExist, memoObject, retrieveMemoObject, disposeMemoObject, isCustomisationSyncMetadata } from "./utils"; import { applyPatch, cancelAllPeriodicTask, cancelAllTasks, cancelTask, generatePatchObj, id2path, isObjectMargeApplicable, isSensibleMargeApplicable, flattenObject, path2id, scheduleTask, tryParseJSON, isValidPath, isInternalMetadata, isPluginMetadata, stripInternalMetadataPrefix, isChunk, askSelectString, askYesNo, askString, PeriodicProcessor, getPath, getPathWithoutPrefix, getPathFromTFile, performRebuildDB, memoIfNotExist, memoObject, retrieveMemoObject, disposeMemoObject, isCustomisationSyncMetadata } from "./utils";
import { encrypt, tryDecrypt } from "./lib/src/e2ee_v2"; import { encrypt, tryDecrypt } from "./lib/src/e2ee_v2";
import { balanceChunkPurgedDBs, enableEncryption, isCloudantURI, isErrorOfMissingDoc, isValidRemoteCouchDBURI, purgeUnreferencedChunks } from "./lib/src/utils_couchdb"; import { balanceChunkPurgedDBs, enableEncryption, isCloudantURI, isErrorOfMissingDoc, isValidRemoteCouchDBURI, purgeUnreferencedChunks } from "./lib/src/utils_couchdb";
import { getGlobalStore, ObservableStore, observeStores } from "./lib/src/store"; import { getGlobalStore, ObservableStore, observeStores } from "./lib/src/store";
import { lockStore, logMessageStore, logStore, type LogEntry } from "./lib/src/stores"; import { lockStore, logMessageStore, logStore, type LogEntry } from "./lib/src/stores";
import { setNoticeClass } from "./lib/src/wrapper"; import { setNoticeClass } from "./lib/src/wrapper";
import { versionNumberString2Number, writeString, decodeBinary, encodeBinary, readString } from "./lib/src/strbin"; import { versionNumberString2Number, writeString, decodeBinary, readString } from "./lib/src/strbin";
import { addPrefix, isAcceptedAll, isPlainText, shouldBeIgnored, stripAllPrefixes } from "./lib/src/path"; import { addPrefix, isAcceptedAll, isPlainText, shouldBeIgnored, stripAllPrefixes } from "./lib/src/path";
import { isLockAcquired, serialized, skipIfDuplicated } from "./lib/src/lock"; import { isLockAcquired, serialized, skipIfDuplicated } from "./lib/src/lock";
import { Semaphore } from "./lib/src/semaphore"; import { Semaphore } from "./lib/src/semaphore";
@@ -33,6 +33,7 @@ import { GlobalHistoryView, VIEW_TYPE_GLOBAL_HISTORY } from "./GlobalHistoryView
import { LogPaneView, VIEW_TYPE_LOG } from "./LogPaneView"; import { LogPaneView, VIEW_TYPE_LOG } from "./LogPaneView";
import { mapAllTasksWithConcurrencyLimit, processAllTasksWithConcurrencyLimit } from "./lib/src/task"; import { mapAllTasksWithConcurrencyLimit, processAllTasksWithConcurrencyLimit } from "./lib/src/task";
import { LRUCache } from "./lib/src/LRUCache"; import { LRUCache } from "./lib/src/LRUCache";
import { SerializedFileAccess } from "./SerializedFileAccess";
setNoticeClass(Notice); setNoticeClass(Notice);
@@ -86,6 +87,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin
return !this.last_successful_post; return !this.last_successful_post;
} }
vaultAccess: SerializedFileAccess = new SerializedFileAccess(this.app);
_unloaded = false; _unloaded = false;
@@ -204,7 +206,7 @@ export default class ObsidianLiveSyncPlugin extends Plugin
const db: PouchDB.Database<EntryDoc> = new PouchDB<EntryDoc>(uri, conf); const db: PouchDB.Database<EntryDoc> = new PouchDB<EntryDoc>(uri, conf);
if (passphrase !== "false" && typeof passphrase === "string") { if (passphrase !== "false" && typeof passphrase === "string") {
enableEncryption(db, passphrase, useDynamicIterationCount, false, this.settings.useV1); enableEncryption(db, passphrase, useDynamicIterationCount, false);
} }
if (skipInfo) { if (skipInfo) {
return { db: db, info: { db_name: "", doc_count: 0, update_seq: "" } }; return { db: db, info: { db_name: "", doc_count: 0, update_seq: "" } };
@@ -293,36 +295,36 @@ export default class ObsidianLiveSyncPlugin extends Plugin
} }
isRedFlagRaised(): boolean { isRedFlagRaised(): boolean {
const redflag = getAbstractFileByPath(normalizePath(FLAGMD_REDFLAG)); const redflag = this.vaultAccess.getAbstractFileByPath(normalizePath(FLAGMD_REDFLAG));
if (redflag != null) { if (redflag != null) {
return true; return true;
} }
return false; return false;
} }
isRedFlag2Raised(): boolean { isRedFlag2Raised(): boolean {
const redflag = getAbstractFileByPath(normalizePath(FLAGMD_REDFLAG2)); const redflag = this.vaultAccess.getAbstractFileByPath(normalizePath(FLAGMD_REDFLAG2));
if (redflag != null) { if (redflag != null) {
return true; return true;
} }
return false; return false;
} }
async deleteRedFlag2() { async deleteRedFlag2() {
const redflag = getAbstractFileByPath(normalizePath(FLAGMD_REDFLAG2)); const redflag = this.vaultAccess.getAbstractFileByPath(normalizePath(FLAGMD_REDFLAG2));
if (redflag != null) { if (redflag != null && redflag instanceof TFile) {
await app.vault.delete(redflag, true); await this.vaultAccess.delete(redflag, true);
} }
} }
isRedFlag3Raised(): boolean { isRedFlag3Raised(): boolean {
const redflag = getAbstractFileByPath(normalizePath(FLAGMD_REDFLAG3)); const redflag = this.vaultAccess.getAbstractFileByPath(normalizePath(FLAGMD_REDFLAG3));
if (redflag != null) { if (redflag != null) {
return true; return true;
} }
return false; return false;
} }
async deleteRedFlag3() { async deleteRedFlag3() {
const redflag = getAbstractFileByPath(normalizePath(FLAGMD_REDFLAG3)); const redflag = this.vaultAccess.getAbstractFileByPath(normalizePath(FLAGMD_REDFLAG3));
if (redflag != null) { if (redflag != null && redflag instanceof TFile) {
await app.vault.delete(redflag, true); await this.vaultAccess.delete(redflag, true);
} }
} }
@@ -404,10 +406,6 @@ export default class ObsidianLiveSyncPlugin extends Plugin
Logger(`Checking expired file history done`); Logger(`Checking expired file history done`);
} }
async onLayoutReady() { async onLayoutReady() {
if (this.settings.useV1 === undefined) {
this.settings.useV1 = await this.askEnableV2();
await this.saveSettingData();
}
this.registerFileWatchEvents(); this.registerFileWatchEvents();
if (!this.localDatabase.isReady) { if (!this.localDatabase.isReady) {
Logger(`Something went wrong! The local database is not ready`, LOG_LEVEL_NOTICE); Logger(`Something went wrong! The local database is not ready`, LOG_LEVEL_NOTICE);
@@ -567,8 +565,13 @@ Note: We can always able to read V1 format. It will be progressively converted.
this.addCommand({ this.addCommand({
id: "livesync-dump", id: "livesync-dump",
name: "Dump information of this doc ", name: "Dump information of this doc ",
editorCallback: (editor: Editor, view: MarkdownView | MarkdownFileInfo) => { // editorCallback: (editor: Editor, view: MarkdownView | MarkdownFileInfo) => {
const file = view.file; // const file = view.file;
// if (!file) return;
// this.localDatabase.getDBEntry(getPathFromTFile(file), {}, true, false);
// },
callback: () => {
const file = this.app.workspace.getActiveFile();
if (!file) return; if (!file) return;
this.localDatabase.getDBEntry(getPathFromTFile(file), {}, true, false); this.localDatabase.getDBEntry(getPathFromTFile(file), {}, true, false);
}, },
@@ -826,7 +829,7 @@ Note: We can always able to read V1 format. It will be progressively converted.
async encryptConfigurationItem(src: string, settings: ObsidianLiveSyncSettings) { async encryptConfigurationItem(src: string, settings: ObsidianLiveSyncSettings) {
if (this.usedPassphrase != "") { if (this.usedPassphrase != "") {
return await encrypt(src, this.usedPassphrase + SALT_OF_PASSPHRASE, false, true); return await encrypt(src, this.usedPassphrase + SALT_OF_PASSPHRASE, false);
} }
const passphrase = await this.getPassphrase(settings); const passphrase = await this.getPassphrase(settings);
@@ -834,7 +837,7 @@ Note: We can always able to read V1 format. It will be progressively converted.
Logger("Could not determine passphrase to save data.json! You probably make the configuration sure again!", LOG_LEVEL_URGENT); Logger("Could not determine passphrase to save data.json! You probably make the configuration sure again!", LOG_LEVEL_URGENT);
return ""; return "";
} }
const dec = await encrypt(src, passphrase + SALT_OF_PASSPHRASE, false, true); const dec = await encrypt(src, passphrase + SALT_OF_PASSPHRASE, false);
if (dec) { if (dec) {
this.usedPassphrase = passphrase; this.usedPassphrase = passphrase;
return dec; return dec;
@@ -1063,7 +1066,7 @@ Note: We can always able to read V1 format. It will be progressively converted.
await this.addOnHiddenFileSync.watchVaultRawEventsAsync(file.path); await this.addOnHiddenFileSync.watchVaultRawEventsAsync(file.path);
await this.addOnConfigSync.watchVaultRawEventsAsync(file.path); await this.addOnConfigSync.watchVaultRawEventsAsync(file.path);
} else { } else {
const targetFile = this.app.vault.getAbstractFileByPath(file.path); const targetFile = this.vaultAccess.getAbstractFileByPath(file.path);
if (!(targetFile instanceof TFile)) { if (!(targetFile instanceof TFile)) {
Logger(`Target file was not found: ${file.path}`, LOG_LEVEL_INFO); Logger(`Target file was not found: ${file.path}`, LOG_LEVEL_INFO);
continue; continue;
@@ -1182,7 +1185,7 @@ Note: We can always able to read V1 format. It will be progressively converted.
if (this.settings?.writeLogToTheFile) { if (this.settings?.writeLogToTheFile) {
const time = now.toISOString().split("T")[0]; const time = now.toISOString().split("T")[0];
const logDate = `${PREFIXMD_LOGFILE}${time}.md`; const logDate = `${PREFIXMD_LOGFILE}${time}.md`;
const file = this.app.vault.getAbstractFileByPath(normalizePath(logDate)); const file = this.vaultAccess.getAbstractFileByPath(normalizePath(logDate));
if (!file) { if (!file) {
this.app.vault.adapter.append(normalizePath(logDate), "```\n"); this.app.vault.adapter.append(normalizePath(logDate), "```\n");
} }
@@ -1304,13 +1307,15 @@ Note: We can always able to read V1 format. It will be progressively converted.
try { try {
let outFile; let outFile;
if (mode == "create") { if (mode == "create") {
outFile = await createFile(normalizePath(path), writeData, { ctime: doc.ctime, mtime: doc.mtime, }); const normalizedPath = normalizePath(path);
await this.vaultAccess.vaultCreate(normalizedPath, writeData, { ctime: doc.ctime, mtime: doc.mtime, });
outFile = this.vaultAccess.getAbstractFileByPath(normalizedPath) as TFile;
} else { } else {
await modifyFile(file, writeData, { ctime: doc.ctime, mtime: doc.mtime }); await this.vaultAccess.vaultModify(file, writeData, { ctime: doc.ctime, mtime: doc.mtime });
outFile = getAbstractFileByPath(getPathFromTFile(file)) as TFile; outFile = this.vaultAccess.getAbstractFileByPath(getPathFromTFile(file)) as TFile;
} }
Logger(msg + path); Logger(msg + path);
touch(outFile); this.vaultAccess.touch(outFile);
this.app.vault.trigger(mode, outFile); this.app.vault.trigger(mode, outFile);
} catch (ex) { } catch (ex) {
@@ -1325,9 +1330,9 @@ Note: We can always able to read V1 format. It will be progressively converted.
} }
const dir = file.parent; const dir = file.parent;
if (this.settings.trashInsteadDelete) { if (this.settings.trashInsteadDelete) {
await this.app.vault.trash(file, false); await this.vaultAccess.trash(file, false);
} else { } else {
await this.app.vault.delete(file, true); await this.vaultAccess.delete(file, true);
} }
Logger(`xxx <- STORAGE (deleted) ${file.path}`); Logger(`xxx <- STORAGE (deleted) ${file.path}`);
Logger(`files: ${dir.children.length}`); Logger(`files: ${dir.children.length}`);
@@ -1392,7 +1397,7 @@ Note: We can always able to read V1 format. It will be progressively converted.
} }
async handleDBChangedAsync(change: EntryBody) { async handleDBChangedAsync(change: EntryBody) {
const targetFile = getAbstractFileByPath(this.getPathWithoutPrefix(change)); const targetFile = this.vaultAccess.getAbstractFileByPath(this.getPathWithoutPrefix(change));
if (targetFile == null) { if (targetFile == null) {
if (change._deleted || change.deleted) { if (change._deleted || change.deleted) {
return; return;
@@ -1516,7 +1521,7 @@ Note: We can always able to read V1 format. It will be progressively converted.
} }
if ((!isInternalMetadata(doc._id)) && skipOldFile) { if ((!isInternalMetadata(doc._id)) && skipOldFile) {
const info = getAbstractFileByPath(stripAllPrefixes(path)); const info = this.vaultAccess.getAbstractFileByPath(stripAllPrefixes(path));
if (info && info instanceof TFile) { if (info && info instanceof TFile) {
const localMtime = ~~(info.stat.mtime / 1000); const localMtime = ~~(info.stat.mtime / 1000);
@@ -2258,12 +2263,12 @@ Or if you are sure know what had been happened, we can unlock the database from
// remove conflicted revision. // remove conflicted revision.
await this.localDatabase.deleteDBEntry(path, { rev: conflictedRev }); await this.localDatabase.deleteDBEntry(path, { rev: conflictedRev });
const file = getAbstractFileByPath(stripAllPrefixes(path)) as TFile; const file = this.vaultAccess.getAbstractFileByPath(stripAllPrefixes(path)) as TFile;
if (file) { if (file) {
await this.app.vault.modify(file, p); await this.vaultAccess.vaultModify(file, p);
await this.updateIntoDB(file); await this.updateIntoDB(file);
} else { } else {
const newFile = await this.app.vault.create(path, p); const newFile = await this.vaultAccess.vaultCreate(path, p);
await this.updateIntoDB(newFile); await this.updateIntoDB(newFile);
} }
await this.pullFile(path); await this.pullFile(path);
@@ -2336,12 +2341,12 @@ Or if you are sure know what had been happened, we can unlock the database from
// delete conflicted revision and write a new file, store it again. // delete conflicted revision and write a new file, store it again.
const p = conflictCheckResult.diff.map((e) => e[1]).join(""); const p = conflictCheckResult.diff.map((e) => e[1]).join("");
await this.localDatabase.deleteDBEntry(filename, { rev: testDoc._conflicts[0] }); await this.localDatabase.deleteDBEntry(filename, { rev: testDoc._conflicts[0] });
const file = getAbstractFileByPath(stripAllPrefixes(filename)) as TFile; const file = this.vaultAccess.getAbstractFileByPath(stripAllPrefixes(filename)) as TFile;
if (file) { if (file) {
await this.app.vault.modify(file, p); await this.vaultAccess.vaultModify(file, p);
await this.updateIntoDB(file); await this.updateIntoDB(file);
} else { } else {
const newFile = await this.app.vault.create(filename, p); const newFile = await this.vaultAccess.vaultCreate(filename, p);
await this.updateIntoDB(newFile); await this.updateIntoDB(newFile);
} }
await this.pullFile(filename); await this.pullFile(filename);
@@ -2383,7 +2388,7 @@ Or if you are sure know what had been happened, we can unlock the database from
const checkFiles = JSON.parse(JSON.stringify(this.conflictedCheckFiles)) as FilePath[]; const checkFiles = JSON.parse(JSON.stringify(this.conflictedCheckFiles)) as FilePath[];
for (const filename of checkFiles) { for (const filename of checkFiles) {
try { try {
const file = getAbstractFileByPath(filename); const file = this.vaultAccess.getAbstractFileByPath(filename);
if (file != null && file instanceof TFile) { if (file != null && file instanceof TFile) {
await this.showIfConflicted(getPathFromTFile(file)); await this.showIfConflicted(getPathFromTFile(file));
} }
@@ -2418,7 +2423,7 @@ Or if you are sure know what had been happened, we can unlock the database from
} }
async pullFile(filename: FilePathWithPrefix, fileList?: TFile[], force?: boolean, rev?: string, waitForReady = true) { async pullFile(filename: FilePathWithPrefix, fileList?: TFile[], force?: boolean, rev?: string, waitForReady = true) {
const targetFile = getAbstractFileByPath(stripAllPrefixes(filename)); const targetFile = this.vaultAccess.getAbstractFileByPath(stripAllPrefixes(filename));
if (!await this.isTargetFile(filename)) return; if (!await this.isTargetFile(filename)) return;
if (targetFile == null) { if (targetFile == null) {
//have to create; //have to create;
@@ -2449,7 +2454,7 @@ Or if you are sure know what had been happened, we can unlock the database from
throw new Error(`Missing doc:${(file as any).path}`) throw new Error(`Missing doc:${(file as any).path}`)
} }
if (!(file instanceof TFile) && "path" in file) { if (!(file instanceof TFile) && "path" in file) {
const w = getAbstractFileByPath((file as any).path); const w = this.vaultAccess.getAbstractFileByPath((file as any).path);
if (w instanceof TFile) { if (w instanceof TFile) {
file = w; file = w;
} else { } else {
@@ -2497,15 +2502,15 @@ Or if you are sure know what had been happened, we can unlock the database from
if (shouldBeIgnored(file.path)) { if (shouldBeIgnored(file.path)) {
return true; return true;
} }
let content: string | string[]; let content: Blob;
let datatype: "plain" | "newnote" = "newnote"; let datatype: "plain" | "newnote" = "newnote";
if (!cache) { if (!cache) {
if (!isPlainText(file.name)) { if (!isPlainText(file.name)) {
Logger(`Reading : ${file.path}`, LOG_LEVEL_VERBOSE); Logger(`Reading : ${file.path}`, LOG_LEVEL_VERBOSE);
const contentBin = await this.app.vault.readBinary(file); const contentBin = await this.vaultAccess.vaultReadBinary(file);
Logger(`Processing: ${file.path}`, LOG_LEVEL_VERBOSE); Logger(`Processing: ${file.path}`, LOG_LEVEL_VERBOSE);
try { try {
content = await encodeBinary(contentBin, this.settings.useV1); content = createBinaryBlob(contentBin);
} catch (ex) { } catch (ex) {
Logger(`The file ${file.path} could not be encoded`); Logger(`The file ${file.path} could not be encoded`);
Logger(ex, LOG_LEVEL_VERBOSE); Logger(ex, LOG_LEVEL_VERBOSE);
@@ -2513,14 +2518,14 @@ Or if you are sure know what had been happened, we can unlock the database from
} }
datatype = "newnote"; datatype = "newnote";
} else { } else {
content = await this.app.vault.read(file); content = createTextBlob(await this.vaultAccess.vaultRead(file));
datatype = "plain"; datatype = "plain";
} }
} else { } else {
if (cache instanceof ArrayBuffer) { if (cache instanceof ArrayBuffer) {
Logger(`Processing: ${file.path}`, LOG_LEVEL_VERBOSE); Logger(`Cache Processing: ${file.path}`, LOG_LEVEL_VERBOSE);
try { try {
content = await encodeBinary(cache, this.settings.useV1); content = createBinaryBlob(cache);
} catch (ex) { } catch (ex) {
Logger(`The file ${file.path} could not be encoded`); Logger(`The file ${file.path} could not be encoded`);
Logger(ex, LOG_LEVEL_VERBOSE); Logger(ex, LOG_LEVEL_VERBOSE);
@@ -2528,13 +2533,13 @@ Or if you are sure know what had been happened, we can unlock the database from
} }
datatype = "newnote" datatype = "newnote"
} else { } else {
content = cache; content = createTextBlob(cache);
datatype = "plain"; datatype = "plain";
} }
} }
const fullPath = getPathFromTFile(file); const fullPath = getPathFromTFile(file);
const id = await this.path2id(fullPath); const id = await this.path2id(fullPath);
const d: LoadedEntry = { const d: SavingEntry = {
_id: id, _id: id,
path: getPathFromTFile(file), path: getPathFromTFile(file),
data: content, data: content,
@@ -2548,7 +2553,7 @@ Or if you are sure know what had been happened, we can unlock the database from
//upsert should locked //upsert should locked
const msg = `DB <- STORAGE (${datatype}) `; const msg = `DB <- STORAGE (${datatype}) `;
const isNotChanged = await serialized("file-" + fullPath, async () => { const isNotChanged = await serialized("file-" + fullPath, async () => {
if (recentlyTouched(file)) { if (this.vaultAccess.recentlyTouched(file)) {
return true; return true;
} }
try { try {
@@ -2572,7 +2577,11 @@ Or if you are sure know what had been happened, we can unlock the database from
} }
return false; return false;
}); });
if (isNotChanged) return true; if (isNotChanged) {
this.queuedFiles = this.queuedFiles.map((e) => ({ ...e, ...(e.entry._id == d._id ? { done: true } : {}) }));
Logger(msg + " Skip " + fullPath, LOG_LEVEL_VERBOSE);
return true;
}
const ret = await this.localDatabase.putDBEntry(d, initialScan); const ret = await this.localDatabase.putDBEntry(d, initialScan);
this.queuedFiles = this.queuedFiles.map((e) => ({ ...e, ...(e.entry._id == d._id ? { done: true } : {}) })); this.queuedFiles = this.queuedFiles.map((e) => ({ ...e, ...(e.entry._id == d._id ? { done: true } : {}) }));
@@ -2601,7 +2610,7 @@ Or if you are sure know what had been happened, we can unlock the database from
} }
async resetLocalDatabase() { async resetLocalDatabase() {
clearTouched(); this.vaultAccess.clearTouched();
await this.localDatabase.resetDatabase(); await this.localDatabase.resetDatabase();
} }
@@ -2641,10 +2650,6 @@ Or if you are sure know what had been happened, we can unlock the database from
return files.filter(file => !ignorePatterns.some(e => file.path.match(e))).filter(file => !targetFiles || (targetFiles && targetFiles.indexOf(file.path) !== -1)) return files.filter(file => !ignorePatterns.some(e => file.path.match(e))).filter(file => !targetFiles || (targetFiles && targetFiles.indexOf(file.path) !== -1))
} }
async applyMTimeToFile(file: InternalFileInfo) {
await this.app.vault.adapter.append(file.path, "", { ctime: file.ctime, mtime: file.mtime });
}
async resolveConflictByNewerEntry(path: FilePathWithPrefix) { async resolveConflictByNewerEntry(path: FilePathWithPrefix) {
const id = await this.path2id(path); const id = await this.path2id(path);
const doc = await this.localDatabase.getRaw<AnyEntry>(id, { conflicts: true }); const doc = await this.localDatabase.getRaw<AnyEntry>(id, { conflicts: true });
@@ -2671,7 +2676,7 @@ Or if you are sure know what had been happened, we can unlock the database from
ignoreFiles = [] as string[] ignoreFiles = [] as string[]
async readIgnoreFile(path: string) { async readIgnoreFile(path: string) {
try { try {
const file = await this.app.vault.adapter.read(path); const file = await this.vaultAccess.adapterRead(path);
const gitignore = file.split(/\r?\n/g); const gitignore = file.split(/\r?\n/g);
this.ignoreFileCache.set(path, gitignore); this.ignoreFileCache.set(path, gitignore);
return gitignore; return gitignore;

44
src/utils.ts
View File

@@ -1,4 +1,4 @@
import { type DataWriteOptions, normalizePath, TFile, Platform, TAbstractFile, App, Plugin, type RequestUrlParam, requestUrl } from "./deps"; import { normalizePath, TFile, Platform, TAbstractFile, App, Plugin, type RequestUrlParam, requestUrl } from "./deps";
import { path2id_base, id2path_base, isValidFilenameInLinux, isValidFilenameInDarwin, isValidFilenameInWidows, isValidFilenameInAndroid, stripAllPrefixes } from "./lib/src/path"; import { path2id_base, id2path_base, isValidFilenameInLinux, isValidFilenameInDarwin, isValidFilenameInWidows, isValidFilenameInAndroid, stripAllPrefixes } from "./lib/src/path";
import { Logger } from "./lib/src/logger"; import { Logger } from "./lib/src/logger";
@@ -303,20 +303,6 @@ export function flattenObject(obj: Record<string | number | symbol, any>, path:
return ret; return ret;
} }
export function modifyFile(file: TFile, data: string | ArrayBuffer, options?: DataWriteOptions) {
if (typeof (data) === "string") {
return app.vault.modify(file, data, options);
} else {
return app.vault.modifyBinary(file, data, options);
}
}
export function createFile(path: string, data: string | ArrayBuffer, options?: DataWriteOptions): Promise<TFile> {
if (typeof (data) === "string") {
return app.vault.create(path, data, options);
} else {
return app.vault.createBinary(path, data, options);
}
}
export function isValidPath(filename: string) { export function isValidPath(filename: string) {
if (Platform.isDesktop) { if (Platform.isDesktop) {
@@ -332,38 +318,10 @@ export function isValidPath(filename: string) {
return isValidFilenameInWidows(filename); return isValidFilenameInWidows(filename);
} }
let touchedFiles: string[] = [];
export function getAbstractFileByPath(path: FilePath): TAbstractFile | null {
// Disabled temporary.
return app.vault.getAbstractFileByPath(path);
// // Hidden API but so useful.
// // @ts-ignore
// if ("getAbstractFileByPathInsensitive" in app.vault && (app.vault.adapter?.insensitive ?? false)) {
// // @ts-ignore
// return app.vault.getAbstractFileByPathInsensitive(path);
// } else {
// return app.vault.getAbstractFileByPath(path);
// }
}
export function trimPrefix(target: string, prefix: string) { export function trimPrefix(target: string, prefix: string) {
return target.startsWith(prefix) ? target.substring(prefix.length) : target; return target.startsWith(prefix) ? target.substring(prefix.length) : target;
} }
export function touch(file: TFile | FilePath) {
const f = file instanceof TFile ? file : getAbstractFileByPath(file) as TFile;
const key = `${f.path}-${f.stat.mtime}-${f.stat.size}`;
touchedFiles.unshift(key);
touchedFiles = touchedFiles.slice(0, 100);
}
export function recentlyTouched(file: TFile) {
const key = `${file.path}-${file.stat.mtime}-${file.stat.size}`;
if (touchedFiles.indexOf(key) == -1) return false;
return true;
}
export function clearTouched() {
touchedFiles = [];
}
/** /**
* returns is internal chunk of file * returns is internal chunk of file

90
updates.md
View File

@@ -1,71 +1,31 @@
### 0.20.0 ### 0.21.0
The E2EE encryption V2 format has been reverted. That was probably the cause of the glitch.
Instead, to maintain efficiency, files are treated with Blob until just before saving. Along with this, the old-fashioned encryption format has also been discontinued.
There are both forward and backwards compatibilities, with recent versions. However, unfortunately, we lost compatibility with filesystem-livesync or some.
It will be addressed soon. Please be patient if you are using filesystem-livesync with E2EE.
At 0.20.0, Self-hosted LiveSync has changed the binary file format and encrypting format, for efficient synchronisation.
The dialogue will be shown and asks us to decide whether to keep v1 or use v2. Once we have enabled v2, all subsequent edits will be saved in v2. Therefore, devices running 0.19 or below cannot understand this and they might say that decryption error. Please update all devices.
Then we will have an impressive performance.
Of course, these are very impactful changes. If you have any questions or troubled things, please feel free to open an issue and mention me.
Note: if you want to roll it back to v1, please enable `Use binary and encryption version 1` on the `Hatch` pane and perform the `rebuild everything` once.
Extra but notable information:
This format change gives us the ability to detect some `marks` in the binary files as same as text files. Therefore, we can split binary files and some specific sort of them (i.e., PDF files) at the specific character. It means that editing the middle of files could be detected with marks.
Now only a few chunks are transferred, even if we add a comment to the PDF or put new files into the ZIP archives.
#### Version history #### Version history
- 0.20.7 - 0.21.2
- Fixed - IMPORTANT NOTICE: **0.21.1 CONTAINS A BUG WHILE REBUILDING THE DATABASE. IF YOU HAVE BEEN REBUILT, PLEASE MAKE SURE THAT ALL FILES ARE SANE.**
- To better replication, path obfuscation is now deterministic even if with E2EE. - This has been fixed in this version.
Note: Compatible with previous database without any conversion. Only new files will be obfuscated in deterministic.
- 0.20.6
- Fixed
- Now empty file could be decoded.
- Local files are no longer pre-saved before fetching from a remote database.
- No longer deadlock while applying customisation sync.
- Configuration with multiple files is now able to be applied correctly.
- Deleting folder propagation now works without enabling the use of a trash bin.
- 0.20.5
- Fixed
- Now the files which having digit or character prefixes in the path will not be ignored.
- 0.20.4
- Fixed
- The text-input-dialogue is no longer broken.
- Finally, we can use the Setup URI again on mobile.
- 0.20.3
- New feature:
- We can launch Customization sync from the Ribbon if we enabled it.
- Fixed: - Fixed:
- Setup URI is now back to the previous spec; be encrypted by V1. - No longer files are broken while rebuilding.
- It may avoid the trouble with iOS 17. - Now, Large binary files can be written correctly on a mobile platform.
- The Settings dialogue is now registered at the beginning of the start-up process. - Any decoding errors now make zero-byte files.
- We can change the configuration even though LiveSync could not be launched in normal. - Modified:
- Improved: - All files are processed sequentially for each.
- Enumerating documents has been faster. - 0.21.1
- 0.20.2
- New feature:
- We can delete all data of customization sync from the `Delete all customization sync data` on the `Hatch` pane.
- Fixed: - Fixed:
- Prevent keep restarting on iOS by yielding microtasks. - No more infinity loops on larger files.
- 0.20.1 - Show message on decode error.
- Fixed: - Refactored:
- No more UI freezing and keep restarting on iOS. - Fixed to avoid obsolete global variables.
- Diff of Non-markdown documents are now shown correctly. - 0.21.0
- Improved: - Changes and performance improvements:
- Performance has been a bit improved. - Now the saving files are processed by Blob.
- Customization sync has gotten faster. - The V2-Format has been reverted.
- However, We lost forward compatibility again (only for this feature). Please update all devices. - New encoding format has been enabled in default.
- Misc - WARNING: Since this version, the compatibilities with older Filesystem LiveSync have been lost.
- Terser configuration has been more aggressive.
- 0.20.0
- Improved:
- A New binary file handling implemented
- A new encrypted format has been implemented
- Now the chunk sizes will be adjusted for efficient sync
- Fixed:
- levels of exception in some logs have been fixed
- Tidied:
- Some Lint warnings have been suppressed.
... To continue on to `updates_old.md`. ... To continue on to `updates_old.md`.

69
updates_old.md
View File

@@ -1,3 +1,72 @@
## 0.20.0
At 0.20.0, Self-hosted LiveSync has changed the binary file format and encrypting format, for efficient synchronisation.
The dialogue will be shown and asks us to decide whether to keep v1 or use v2. Once we have enabled v2, all subsequent edits will be saved in v2. Therefore, devices running 0.19 or below cannot understand this and they might say that decryption error. Please update all devices.
Then we will have an impressive performance.
Of course, these are very impactful changes. If you have any questions or troubled things, please feel free to open an issue and mention me.
Note: if you want to roll it back to v1, please enable `Use binary and encryption version 1` on the `Hatch` pane and perform the `rebuild everything` once.
Extra but notable information:
This format change gives us the ability to detect some `marks` in the binary files as same as text files. Therefore, we can split binary files and some specific sort of them (i.e., PDF files) at the specific character. It means that editing the middle of files could be detected with marks.
Now only a few chunks are transferred, even if we add a comment to the PDF or put new files into the ZIP archives.
- 0.20.7
- Fixed
- To better replication, path obfuscation is now deterministic even if with E2EE.
Note: Compatible with previous database without any conversion. Only new files will be obfuscated in deterministic.
- 0.20.6
- Fixed
- Now empty file could be decoded.
- Local files are no longer pre-saved before fetching from a remote database.
- No longer deadlock while applying customisation sync.
- Configuration with multiple files is now able to be applied correctly.
- Deleting folder propagation now works without enabling the use of a trash bin.
- 0.20.5
- Fixed
- Now the files which having digit or character prefixes in the path will not be ignored.
- 0.20.4
- Fixed
- The text-input-dialogue is no longer broken.
- Finally, we can use the Setup URI again on mobile.
- 0.20.3
- New feature:
- We can launch Customization sync from the Ribbon if we enabled it.
- Fixed:
- Setup URI is now back to the previous spec; be encrypted by V1.
- It may avoid the trouble with iOS 17.
- The Settings dialogue is now registered at the beginning of the start-up process.
- We can change the configuration even though LiveSync could not be launched in normal.
- Improved:
- Enumerating documents has been faster.
- 0.20.2
- New feature:
- We can delete all data of customization sync from the `Delete all customization sync data` on the `Hatch` pane.
- Fixed:
- Prevent keep restarting on iOS by yielding microtasks.
- 0.20.1
- Fixed:
- No more UI freezing and keep restarting on iOS.
- Diff of Non-markdown documents are now shown correctly.
- Improved:
- Performance has been a bit improved.
- Customization sync has gotten faster.
- However, We lost forward compatibility again (only for this feature). Please update all devices.
- Misc
- Terser configuration has been more aggressive.
- 0.20.0
- Improved:
- A New binary file handling implemented
- A new encrypted format has been implemented
- Now the chunk sizes will be adjusted for efficient sync
- Fixed:
- levels of exception in some logs have been fixed
- Tidied:
- Some Lint warnings have been suppressed.
### 0.19.0 ### 0.19.0
#### Customization sync #### Customization sync