Compare commits

...

11 Commits

Author SHA1 Message Date
vorotamoroz
e76e7ae8ea bump 2024-03-19 17:59:38 +01:00
vorotamoroz
f7fbe85d65 - New feature:
- We can disable the status bar in the setting dialogue.
- Improved:
  - Now some files are handled as correct data type.
  - Customisation sync now uses the digest of each file for better performance.
  - The status in the Editor now works performant.
- Refactored:
  - Common functions have been ready and the codebase has been organised.
  - Stricter type checking following TypeScript updates.
  - Remove old iOS workaround for simplicity and performance.
2024-03-19 17:58:55 +01:00
vorotamoroz
0313443b29 Merge pull request #389 from Seeker0472/fix-command
Fixed docker-compose command in docs
2024-03-19 14:06:23 +09:00
seeker0472
755c30f468 fix docker-compose command 2024-03-17 14:30:35 +08:00
vorotamoroz
b00b0cc5e5 bump 2024-03-15 10:37:15 +01:00
vorotamoroz
d7985a6b41 Improved:
- Now using HTTP for the remote database URI warns of an error (on mobile) or notice (on desktop).
2024-03-15 10:36:00 +01:00
vorotamoroz
486e816902 Update dependencies 2024-03-15 10:35:41 +01:00
vorotamoroz
ef9b19c24b bump 2024-03-04 04:07:51 +00:00
vorotamoroz
4ed9494176 Changed:
- The default settings has been changed.
Improved:
- Default and preferred settings are applied on completion of the wizard.
Fixed:
- Now Initialisation `Fetch` will be performed smoothly and there will be fewer conflicts.
- No longer stuck while Handling transferred or initialised documents.
2024-03-04 04:07:11 +00:00
vorotamoroz
fcd56d59d5 bump 2024-03-01 08:33:37 +00:00
vorotamoroz
1cabfcfd19 Fixed:
- `Verify and repair all files` is no longer broken.
New feature::
- Now `Verify and repair all files` can restore or show history
Improved:
- Performance improved
2024-03-01 08:32:48 +00:00
19 changed files with 2008 additions and 1633 deletions

View File

@@ -91,7 +91,7 @@ services:
最后, 创建并启动容器:
```
# -d will launch detached so the container runs in background
docker compose up -d
docker-compose up -d
```
## 创建数据库

View File

@@ -1,7 +1,7 @@
{
"id": "obsidian-livesync",
"name": "Self-hosted LiveSync",
"version": "0.22.10",
"version": "0.22.14",
"minAppVersion": "0.9.12",
"description": "Community implementation of self-hosted livesync. Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.",
"author": "vorotamoroz",

2646
package-lock.json generated

File diff suppressed because it is too large Load Diff

View File

@@ -1,6 +1,6 @@
{
"name": "obsidian-livesync",
"version": "0.22.10",
"version": "0.22.14",
"description": "Reflect your vault changes to some other devices immediately. Please make sure to disable other synchronize solutions to avoid content corruption or duplication.",
"main": "main.js",
"type": "module",
@@ -13,29 +13,29 @@
"author": "vorotamoroz",
"license": "MIT",
"devDependencies": {
"@tsconfig/svelte": "^5.0.0",
"@types/diff-match-patch": "^1.0.32",
"@types/node": "^20.2.5",
"@types/pouchdb": "^6.4.0",
"@types/pouchdb-adapter-http": "^6.1.3",
"@types/pouchdb-adapter-idb": "^6.1.4",
"@types/pouchdb-browser": "^6.1.3",
"@types/pouchdb-core": "^7.0.11",
"@types/pouchdb-mapreduce": "^6.1.7",
"@types/pouchdb-replication": "^6.4.4",
"@types/transform-pouch": "^1.0.2",
"@typescript-eslint/eslint-plugin": "^6.2.1",
"@typescript-eslint/parser": "^6.2.1",
"@tsconfig/svelte": "^5.0.2",
"@types/diff-match-patch": "^1.0.36",
"@types/node": "^20.11.28",
"@types/pouchdb": "^6.4.2",
"@types/pouchdb-adapter-http": "^6.1.6",
"@types/pouchdb-adapter-idb": "^6.1.7",
"@types/pouchdb-browser": "^6.1.5",
"@types/pouchdb-core": "^7.0.14",
"@types/pouchdb-mapreduce": "^6.1.10",
"@types/pouchdb-replication": "^6.4.7",
"@types/transform-pouch": "^1.0.6",
"@typescript-eslint/eslint-plugin": "^7.2.0",
"@typescript-eslint/parser": "^7.2.0",
"builtin-modules": "^3.3.0",
"esbuild": "0.18.17",
"esbuild-svelte": "^0.7.4",
"eslint": "^8.46.0",
"esbuild": "0.20.2",
"esbuild-svelte": "^0.8.0",
"eslint": "^8.57.0",
"eslint-config-airbnb-base": "^15.0.0",
"eslint-plugin-import": "^2.28.0",
"eslint-plugin-import": "^2.29.1",
"events": "^3.3.0",
"obsidian": "^1.4.11",
"postcss": "^8.4.27",
"postcss-load-config": "^4.0.1",
"obsidian": "^1.5.7",
"postcss": "^8.4.35",
"postcss-load-config": "^5.0.3",
"pouchdb-adapter-http": "^8.0.1",
"pouchdb-adapter-idb": "^8.0.1",
"pouchdb-adapter-indexeddb": "^8.0.1",
@@ -46,16 +46,16 @@
"pouchdb-merge": "^8.0.1",
"pouchdb-replication": "^8.0.1",
"pouchdb-utils": "^8.0.1",
"svelte": "^4.1.2",
"svelte-preprocess": "^5.0.4",
"terser": "^5.19.2",
"svelte": "^4.2.12",
"svelte-preprocess": "^5.1.3",
"terser": "^5.29.2",
"transform-pouch": "^2.0.0",
"tslib": "^2.6.1",
"typescript": "^5.1.6"
"tslib": "^2.6.2",
"typescript": "^5.4.2"
},
"dependencies": {
"diff-match-patch": "^1.0.5",
"idb": "^7.1.1",
"idb": "^8.0.0",
"minimatch": "^9.0.3",
"xxhash-wasm": "0.4.2",
"xxhash-wasm-102": "npm:xxhash-wasm@^1.0.2"

View File

@@ -4,10 +4,9 @@ import { Notice, type PluginManifest, parseYaml, normalizePath, type ListedFiles
import type { EntryDoc, LoadedEntry, InternalFileEntry, FilePathWithPrefix, FilePath, DocumentID, AnyEntry, SavingEntry } from "./lib/src/types";
import { LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE, MODE_SELECTIVE } from "./lib/src/types";
import { ICXHeader, PERIODIC_PLUGIN_SWEEP, } from "./types";
import { createTextBlob, delay, getDocData, isDocContentSame, sendSignal, waitForSignal } from "./lib/src/utils";
import { createSavingEntryFromLoadedEntry, createTextBlob, delay, fireAndForget, getDocData, isDocContentSame, sendSignal, waitForSignal } from "./lib/src/utils";
import { Logger } from "./lib/src/logger";
import { WrappedNotice } from "./lib/src/wrapper";
import { readString, decodeBinary, arrayBufferToBase64, sha1 } from "./lib/src/strbin";
import { readString, decodeBinary, arrayBufferToBase64, digestHash } from "./lib/src/strbin";
import { serialized } from "./lib/src/lock";
import { LiveSyncCommands } from "./LiveSyncCommands";
import { stripAllPrefixes } from "./lib/src/path";
@@ -31,7 +30,8 @@ function serialize(data: PluginDataEx): string {
ret += data.mtime + d2;
for (const file of data.files) {
ret += file.filename + d + (file.displayName ?? "") + d + (file.version ?? "") + d2;
ret += file.mtime + d + file.size + d2;
const hash = digestHash((file.data ?? []).join());
ret += file.mtime + d + file.size + d + hash + d2;
for (const data of file.data ?? []) {
ret += data + d
}
@@ -95,6 +95,7 @@ function deserialize2(str: string): PluginDataEx {
tokens.nextLine();
const mtime = Number(tokens.next());
const size = Number(tokens.next());
const hash = tokens.next();
tokens.nextLine();
const data = [] as string[];
let piece = "";
@@ -110,7 +111,8 @@ function deserialize2(str: string): PluginDataEx {
version,
mtime,
size,
data
data,
hash
}
)
tokens.nextLine();
@@ -137,10 +139,11 @@ export const pluginIsEnumerating = writable(false);
export type PluginDataExFile = {
filename: string,
data?: string[],
data: string[],
mtime: number,
size: number,
version?: string,
hash?: string,
displayName?: string,
}
export type PluginDataExDisplay = {
@@ -169,17 +172,16 @@ export class ConfigSync extends LiveSyncCommands {
pluginScanningCount.onChanged((e) => {
const total = e.value;
pluginIsEnumerating.set(total != 0);
if (total == 0) {
Logger(`Processing configurations done`, LOG_LEVEL_INFO, "get-plugins");
}
// if (total == 0) {
// Logger(`Processing configurations done`, LOG_LEVEL_INFO, "get-plugins");
// }
})
}
confirmPopup: WrappedNotice = null;
get kvDB() {
return this.plugin.kvDB;
}
pluginDialog: PluginDialogModal = null;
pluginDialog?: PluginDialogModal = undefined;
periodicPluginSweepProcessor = new PeriodicProcessor(this.plugin, async () => await this.scanAllConfigFiles(false));
pluginList: PluginDataExDisplay[] = [];
@@ -187,7 +189,7 @@ export class ConfigSync extends LiveSyncCommands {
if (!this.settings.usePluginSync) {
return;
}
if (this.pluginDialog != null) {
if (this.pluginDialog) {
this.pluginDialog.open();
} else {
this.pluginDialog = new PluginDialogModal(this.app, this.plugin);
@@ -198,7 +200,7 @@ export class ConfigSync extends LiveSyncCommands {
hidePluginSyncModal() {
if (this.pluginDialog != null) {
this.pluginDialog.close();
this.pluginDialog = null;
this.pluginDialog = undefined;
}
}
onunload() {
@@ -273,16 +275,28 @@ export class ConfigSync extends LiveSyncCommands {
await this.updatePluginList(showMessage);
}
async loadPluginData(path: FilePathWithPrefix): Promise<PluginDataExDisplay | false> {
const wx = await this.localDatabase.getDBEntry(path, null, false, false);
const wx = await this.localDatabase.getDBEntry(path, undefined, false, false);
if (wx) {
const data = deserialize(getDocData(wx.data), {}) as PluginDataEx;
const xFiles = [] as PluginDataExFile[];
let missingHash = false;
for (const file of data.files) {
const work = { ...file };
const tempStr = getDocData(work.data);
work.data = [await sha1(tempStr)];
const work = { ...file, data: [] as string[] };
if (!file.hash) {
// debugger;
const tempStr = getDocData(work.data);
const hash = digestHash(tempStr);
file.hash = hash;
missingHash = true;
}
work.data = [file.hash];
xFiles.push(work);
}
if (missingHash) {
Logger(`Digest created for ${path} to improve checking`, LOG_LEVEL_VERBOSE);
wx.data = serialize(data);
fireAndForget(() => this.localDatabase.putDBEntry(createSavingEntryFromLoadedEntry(wx)));
}
return ({
...data,
documentPath: this.getPath(wx),
@@ -317,21 +331,21 @@ export class ConfigSync extends LiveSyncCommands {
const plugin = v[0];
const path = plugin.path || this.getPath(plugin);
const oldEntry = (this.pluginList.find(e => e.documentPath == path));
if (oldEntry && oldEntry.mtime == plugin.mtime) return;
if (oldEntry && oldEntry.mtime == plugin.mtime) return [];
try {
const pluginData = await this.loadPluginData(path);
if (pluginData) {
return [pluginData];
}
// Failed to load
return;
return [];
} catch (ex) {
Logger(`Something happened at enumerating customization :${path}`, LOG_LEVEL_NOTICE);
Logger(ex, LOG_LEVEL_VERBOSE);
}
return;
}, { suspended: true, batchSize: 1, concurrentLimit: 5, delay: 300, yieldThreshold: 10 }).pipeTo(
return [];
}, { suspended: false, batchSize: 1, concurrentLimit: 5, delay: 100, yieldThreshold: 10, maintainDelay: false }).pipeTo(
new QueueProcessor(
async (pluginDataList) => {
// Concurrency is two, therefore, we can unlock the previous awaiting.
@@ -349,8 +363,8 @@ export class ConfigSync extends LiveSyncCommands {
}
return;
}
, { suspended: true, batchSize: 10, concurrentLimit: 2, delay: 250, yieldThreshold: 25, totalRemainingReactiveSource: pluginScanningCount })).startPipeline().root.onIdle(() => {
Logger(`All files enumerated`, LOG_LEVEL_INFO, "get-plugins");
, { suspended: false, batchSize: 10, concurrentLimit: 2, delay: 100, yieldThreshold: 25, totalRemainingReactiveSource: pluginScanningCount, maintainDelay: false })).startPipeline().root.onIdle(() => {
// Logger(`All files enumerated`, LOG_LEVEL_INFO, "get-plugins");
this.createMissingConfigurationEntry();
});
@@ -502,9 +516,9 @@ export class ConfigSync extends LiveSyncCommands {
if (this.plugin.settings.usePluginSync && this.plugin.settings.notifyPluginOrSettingUpdated) {
if (!this.pluginDialog || (this.pluginDialog && !this.pluginDialog.isOpened())) {
const fragment = createFragment((doc) => {
doc.createEl("span", null, (a) => {
doc.createEl("span", undefined, (a) => {
a.appendText(`Some configuration has been arrived, Press `);
a.appendChild(a.createEl("a", null, (anchor) => {
a.appendChild(a.createEl("a", undefined, (anchor) => {
anchor.text = "HERE";
anchor.addEventListener("click", () => {
this.showPluginSyncModal();
@@ -670,7 +684,7 @@ export class ConfigSync extends LiveSyncCommands {
const content = createTextBlob(serialize(dt));
try {
const old = await this.localDatabase.getDBEntryMeta(prefixedFileName, null, false);
const old = await this.localDatabase.getDBEntryMeta(prefixedFileName, undefined, false);
let saveData: SavingEntry;
if (old === false) {
saveData = {
@@ -694,7 +708,7 @@ export class ConfigSync extends LiveSyncCommands {
if (oldC) {
const d = await deserialize(getDocData(oldC.data), {}) as PluginDataEx;
const diffs = (d.files.map(previous => ({ prev: previous, curr: dt.files.find(e => e.filename == previous.filename) })).map(async e => {
try { return await isDocContentSame(e.curr.data, e.prev.data) } catch (_) { return false }
try { return await isDocContentSame(e.curr?.data ?? [], e.prev.data) } catch (_) { return false }
}))
const isSame = (await Promise.all(diffs)).every(e => e == true);
if (isSame) {
@@ -767,7 +781,11 @@ export class ConfigSync extends LiveSyncCommands {
const filesOnDB = ((await this.localDatabase.allDocsRaw({ startkey: ICXHeader + "", endkey: `${ICXHeader}\u{10ffff}`, include_docs: true })).rows.map(e => e.doc) as InternalFileEntry[]).filter(e => !e.deleted);
let deleteCandidate = filesOnDB.map(e => this.getPath(e)).filter(e => e.startsWith(`${ICXHeader}${term}/`));
for (const vp of virtualPathsOfLocalFiles) {
const p = files.find(e => e.key == vp).file;
const p = files.find(e => e.key == vp)?.file;
if (!p) {
Logger(`scanAllConfigFiles - File not found: ${vp}`, LOG_LEVEL_VERBOSE);
continue;
}
await this.storeCustomizationFiles(p);
deleteCandidate = deleteCandidate.filter(e => e != vp);
}
@@ -782,10 +800,11 @@ export class ConfigSync extends LiveSyncCommands {
const mtime = new Date().getTime();
await serialized("file-x-" + prefixedFileName, async () => {
try {
const old = await this.localDatabase.getDBEntryMeta(prefixedFileName, null, false) as InternalFileEntry | false;
const old = await this.localDatabase.getDBEntryMeta(prefixedFileName, undefined, false) as InternalFileEntry | false;
let saveData: InternalFileEntry;
if (old === false) {
Logger(`STORAGE -x> DB:${prefixedFileName}: (config) already deleted (Not found on database)`);
return;
} else {
if (old.deleted) {
Logger(`STORAGE -x> DB:${prefixedFileName}: (config) already deleted`);

View File

@@ -1,12 +1,10 @@
import { normalizePath, type PluginManifest, type ListedFiles } from "./deps";
import { type EntryDoc, type LoadedEntry, type InternalFileEntry, type FilePathWithPrefix, type FilePath, LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE, MODE_SELECTIVE, MODE_PAUSED, type SavingEntry, type DocumentID } from "./lib/src/types";
import { type InternalFileInfo, ICHeader, ICHeaderEnd } from "./types";
import { createBinaryBlob, isDocContentSame, sendSignal } from "./lib/src/utils";
import { readAsBlob, isDocContentSame, sendSignal, readContent, createBlob } from "./lib/src/utils";
import { Logger } from "./lib/src/logger";
import { PouchDB } from "./lib/src/pouchdb-browser.js";
import { isInternalMetadata, PeriodicProcessor } from "./utils";
import { WrappedNotice } from "./lib/src/wrapper";
import { decodeBinary, encodeBinary } from "./lib/src/strbin";
import { serialized } from "./lib/src/lock";
import { JsonResolveModal } from "./JsonResolveModal";
import { LiveSyncCommands } from "./LiveSyncCommands";
@@ -16,7 +14,7 @@ import { hiddenFilesEventCount, hiddenFilesProcessingCount } from "./lib/src/sto
export class HiddenFileSync extends LiveSyncCommands {
periodicInternalFileScanProcessor: PeriodicProcessor = new PeriodicProcessor(this.plugin, async () => this.settings.syncInternalFiles && this.localDatabase.isReady && await this.syncInternalFilesAndDatabase("push", false));
confirmPopup: WrappedNotice = null;
get kvDB() {
return this.plugin.kvDB;
}
@@ -67,11 +65,11 @@ export class HiddenFileSync extends LiveSyncCommands {
realizeSettingSyncMode(): Promise<void> {
this.periodicInternalFileScanProcessor?.disable();
if (this.plugin.suspended)
return;
return Promise.resolve();
if (!this.plugin.isReady)
return;
return Promise.resolve();
this.periodicInternalFileScanProcessor.enable(this.settings.syncInternalFiles && this.settings.syncInternalFilesInterval ? (this.settings.syncInternalFilesInterval * 1000) : 0);
return;
return Promise.resolve();
}
procInternalFile(filename: string) {
@@ -125,7 +123,7 @@ export class HiddenFileSync extends LiveSyncCommands {
if (storageMTime == 0) {
await this.deleteInternalFileOnDatabase(path);
} else {
await this.storeInternalFileToDatabase({ path: path, ...stat });
await this.storeInternalFileToDatabase({ path: path, mtime, ctime: stat?.ctime ?? mtime, size: stat?.size ?? 0 });
}
}
@@ -162,7 +160,7 @@ export class HiddenFileSync extends LiveSyncCommands {
await this.localDatabase.removeRevision(id, delRev);
Logger(`Older one has been deleted:${path}`);
const cc = await this.localDatabase.getRaw(id, { conflicts: true });
if (cc._conflicts.length == 0) {
if (cc._conflicts?.length === 0) {
await this.extractInternalFileFromDatabase(stripAllPrefixes(path))
} else {
this.conflictResolutionProcessor.enqueue(path);
@@ -177,11 +175,12 @@ export class HiddenFileSync extends LiveSyncCommands {
// Retrieve data
const id = await this.path2id(path, ICHeader);
const doc = await this.localDatabase.getRaw(id, { conflicts: true });
// If there is no conflict, return with false.
if (!("_conflicts" in doc))
return;
// if (!("_conflicts" in doc)){
// return [];
// }
if (doc._conflicts === undefined) return [];
if (doc._conflicts.length == 0)
return;
return [];
Logger(`Hidden file conflicted:${path}`);
const conflicts = doc._conflicts.sort((a, b) => Number(a.split("-")[0]) - Number(b.split("-")[0]));
const revA = doc._rev;
@@ -192,7 +191,7 @@ export class HiddenFileSync extends LiveSyncCommands {
const conflictedRevNo = Number(conflictedRev.split("-")[0]);
//Search
const revFrom = (await this.localDatabase.getRaw<EntryDoc>(id, { revs_info: true }));
const commonBase = revFrom._revs_info.filter(e => e.status == "available" && Number(e.rev.split("-")[0]) < conflictedRevNo).first()?.rev ?? "";
const commonBase = revFrom._revs_info?.filter(e => e.status == "available" && Number(e.rev.split("-")[0]) < conflictedRevNo).first()?.rev ?? "";
const result = await this.plugin.mergeObject(path, commonBase, doc._rev, conflictedRev);
if (result) {
Logger(`Object merge:${path}`, LOG_LEVEL_INFO);
@@ -203,11 +202,14 @@ export class HiddenFileSync extends LiveSyncCommands {
}
await this.plugin.vaultAccess.adapterWrite(filename, result);
const stat = await this.vaultAccess.adapterStat(filename);
if (!stat) {
throw new Error(`conflictResolutionProcessor: Failed to stat file ${filename}`);
}
await this.storeInternalFileToDatabase({ path: filename, ...stat });
await this.extractInternalFileFromDatabase(filename);
await this.localDatabase.removeRevision(id, revB);
this.conflictResolutionProcessor.enqueue(path);
return;
return [];
} else {
Logger(`Object merge is not applicable.`, LOG_LEVEL_VERBOSE);
}
@@ -215,11 +217,11 @@ export class HiddenFileSync extends LiveSyncCommands {
}
// When not JSON file, resolve conflicts by choosing a newer one.
await this.resolveByNewerEntry(id, path, doc, revA, revB);
return;
return [];
} catch (ex) {
Logger(`Failed to resolve conflict (Hidden): ${path}`);
Logger(ex, LOG_LEVEL_VERBOSE);
return;
return [];
}
}, {
suspended: false, batchSize: 1, concurrentLimit: 5, delay: 10, keepResultUntilDownstreamConnected: true, yieldThreshold: 10,
@@ -312,11 +314,11 @@ export class HiddenFileSync extends LiveSyncCommands {
if (processed % 100 == 0) {
Logger(`Hidden file: ${processed}/${fileCount}`, logLevel, "sync_internal");
}
if (!filename) return;
if (!filename) return [];
if (ignorePatterns.some(e => filename.match(e)))
return;
return [];
if (await this.plugin.isIgnoredByIgnoreFiles(filename)) {
return;
return [];
}
const fileOnStorage = filename in filesMap ? filesMap[filename] : undefined;
@@ -403,7 +405,7 @@ export class HiddenFileSync extends LiveSyncCommands {
const enabledPlugins = this.app.plugins.enabledPlugins as Set<string>;
const enabledPluginManifests = manifests.filter(e => enabledPlugins.has(e.id));
for (const manifest of enabledPluginManifests) {
if (manifest.dir in updatedFolders) {
if (manifest.dir && manifest.dir in updatedFolders) {
// If notified about plug-ins, reloading Obsidian may not be necessary.
updatedCount -= updatedFolders[manifest.dir];
const updatePluginId = manifest.id;
@@ -451,19 +453,11 @@ export class HiddenFileSync extends LiveSyncCommands {
const id = await this.path2id(file.path, ICHeader);
const prefixedFileName = addPrefix(file.path, ICHeader);
const contentBin = await this.plugin.vaultAccess.adapterReadBinary(file.path);
let content: Blob;
try {
content = createBinaryBlob(contentBin);
} catch (ex) {
Logger(`The file ${file.path} could not be encoded`);
Logger(ex, LOG_LEVEL_VERBOSE);
return false;
}
const content = createBlob(await this.plugin.vaultAccess.adapterReadAuto(file.path));
const mtime = file.mtime;
return await serialized("file-" + prefixedFileName, async () => {
try {
const old = await this.localDatabase.getDBEntry(prefixedFileName, null, false, false);
const old = await this.localDatabase.getDBEntry(prefixedFileName, undefined, false, false);
let saveData: SavingEntry;
if (old === false) {
saveData = {
@@ -479,7 +473,7 @@ export class HiddenFileSync extends LiveSyncCommands {
type: "newnote",
};
} else {
if (await isDocContentSame(createBinaryBlob(decodeBinary(old.data)), content) && !forceWrite) {
if (await isDocContentSame(readAsBlob(old), content) && !forceWrite) {
// Logger(`STORAGE --> DB:${file.path}: (hidden) Not changed`, LOG_LEVEL_VERBOSE);
return;
}
@@ -489,10 +483,10 @@ export class HiddenFileSync extends LiveSyncCommands {
data: content,
mtime,
size: file.size,
datatype: "newnote",
datatype: old.datatype,
children: [],
deleted: false,
type: "newnote",
type: old.datatype,
};
}
const ret = await this.localDatabase.putDBEntry(saveData);
@@ -515,7 +509,7 @@ export class HiddenFileSync extends LiveSyncCommands {
}
await serialized("file-" + prefixedFileName, async () => {
try {
const old = await this.localDatabase.getDBEntryMeta(prefixedFileName, null, true) as InternalFileEntry | false;
const old = await this.localDatabase.getDBEntryMeta(prefixedFileName, undefined, true) as InternalFileEntry | false;
let saveData: InternalFileEntry;
if (old === false) {
saveData = {
@@ -531,7 +525,7 @@ export class HiddenFileSync extends LiveSyncCommands {
} else {
// Remove all conflicted before deleting.
const conflicts = await this.localDatabase.getRaw(old._id, { conflicts: true });
if ("_conflicts" in conflicts) {
if (conflicts._conflicts !== undefined) {
for (const conflictRev of conflicts._conflicts) {
await this.localDatabase.removeRevision(old._id, conflictRev);
Logger(`STORAGE -x> DB:${filename}: (hidden) conflict removed ${old._rev} => ${conflictRev}`, LOG_LEVEL_VERBOSE);
@@ -581,7 +575,7 @@ export class HiddenFileSync extends LiveSyncCommands {
const deleted = fileOnDB.deleted || fileOnDB._deleted || false;
if (deleted) {
if (!isExists) {
Logger(`STORAGE <x- DB:${filename}: deleted (hidden) Deleted on DB, but the file is already not found on storage.`);
Logger(`STORAGE <x- DB:${filename}: deleted (hidden) Deleted on DB, but the file is already not found on storage.`);
} else {
Logger(`STORAGE <x- DB:${filename}: deleted (hidden).`);
await this.plugin.vaultAccess.adapterRemove(filename);
@@ -597,7 +591,7 @@ export class HiddenFileSync extends LiveSyncCommands {
}
if (!isExists) {
await this.vaultAccess.ensureDirectory(filename);
await this.plugin.vaultAccess.adapterWrite(filename, decodeBinary(fileOnDB.data), { mtime: fileOnDB.mtime, ctime: fileOnDB.ctime });
await this.plugin.vaultAccess.adapterWrite(filename, readContent(fileOnDB), { mtime: fileOnDB.mtime, ctime: fileOnDB.ctime });
try {
//@ts-ignore internalAPI
await this.app.vault.adapter.reconcileInternalFile(filename);
@@ -608,13 +602,13 @@ export class HiddenFileSync extends LiveSyncCommands {
Logger(`STORAGE <-- DB:${filename}: written (hidden,new${force ? ", force" : ""})`);
return true;
} else {
const contentBin = await this.plugin.vaultAccess.adapterReadBinary(filename);
const content = await encodeBinary(contentBin);
if (await isDocContentSame(content, fileOnDB.data) && !force) {
const content = await this.plugin.vaultAccess.adapterReadAuto(filename);
const docContent = readContent(fileOnDB);
if (await isDocContentSame(content, docContent) && !force) {
// Logger(`STORAGE <-- DB:${filename}: skipped (hidden) Not changed`, LOG_LEVEL_VERBOSE);
return true;
}
await this.plugin.vaultAccess.adapterWrite(filename, decodeBinary(fileOnDB.data), { mtime: fileOnDB.mtime, ctime: fileOnDB.ctime });
await this.plugin.vaultAccess.adapterWrite(filename, docContent, { mtime: fileOnDB.mtime, ctime: fileOnDB.ctime });
try {
//@ts-ignore internalAPI
await this.app.vault.adapter.reconcileInternalFile(filename);
@@ -669,7 +663,11 @@ export class HiddenFileSync extends LiveSyncCommands {
}
await this.plugin.vaultAccess.adapterWrite(filename, result);
const stat = await this.plugin.vaultAccess.adapterStat(filename);
await this.storeInternalFileToDatabase({ path: filename, ...stat }, true);
if (!stat) {
throw new Error("Stat failed");
}
const mtime = stat?.mtime ?? 0;
await this.storeInternalFileToDatabase({ path: filename, mtime, ctime: stat?.ctime ?? mtime, size: stat?.size ?? 0 }, true);
try {
//@ts-ignore internalAPI
await this.app.vault.adapter.reconcileInternalFile(filename);
@@ -703,7 +701,7 @@ export class HiddenFileSync extends LiveSyncCommands {
const root = this.app.vault.getRoot();
const findRoot = root.path;
const filenames = (await this.getFiles(findRoot, [], null, ignoreFilter)).filter(e => e.startsWith(".")).filter(e => !e.startsWith(".trash"));
const filenames = (await this.getFiles(findRoot, [], undefined, ignoreFilter)).filter(e => e.startsWith(".")).filter(e => !e.startsWith(".trash"));
const files = filenames.filter(path => synchronisedInConfigSync.every(filterFile => !path.toLowerCase().startsWith(filterFile))).map(async (e) => {
return {
path: e as FilePath,
@@ -716,9 +714,12 @@ export class HiddenFileSync extends LiveSyncCommands {
if (await this.plugin.isIgnoredByIgnoreFiles(w.path)) {
continue
}
const mtime = w.stat?.mtime ?? 0
const ctime = w.stat?.ctime ?? mtime;
const size = w.stat?.size ?? 0;
result.push({
...w,
...w.stat
mtime, ctime, size
});
}
return result;
@@ -729,8 +730,8 @@ export class HiddenFileSync extends LiveSyncCommands {
async getFiles(
path: string,
ignoreList: string[],
filter: RegExp[],
ignoreFilter: RegExp[]
filter?: RegExp[],
ignoreFilter?: RegExp[]
) {
let w: ListedFiles;
try {

View File

@@ -134,7 +134,7 @@ export class SetupLiveSync extends LiveSyncCommands {
} else if (setupType == setupAsMerge) {
this.plugin.settings = newSettingW;
this.plugin.usedPassphrase = "";
await this.fetchLocalWithKeepLocal();
await this.fetchLocalWithRebuild();
} else if (setupType == setupAgain) {
const confirm = "I know this operation will rebuild all my databases with files on this device, and files that are on the remote database and I didn't synchronize to any other devices will be lost and want to proceed indeed.";
if (await askSelectString(this.app, "Do you really want to do this?", ["Cancel", confirm]) != confirm) {
@@ -377,11 +377,13 @@ Of course, we are able to disable these features.`
await this.plugin.replicateAllFromServer(true);
await delay(1000);
await this.plugin.replicateAllFromServer(true);
await this.fetchRemoteChunks();
// if (!tryLessFetching) {
// await this.fetchRemoteChunks();
// }
await this.resumeReflectingDatabase();
await this.askHiddenFileConfiguration({ enableFetch: true });
}
async fetchLocalWithKeepLocal() {
async fetchLocalWithRebuild() {
return await this.fetchLocal(true);
}
async rebuildRemote() {

View File

@@ -5,7 +5,7 @@ import ObsidianLiveSyncPlugin from "./main";
import { type DocumentID, type FilePathWithPrefix, type LoadedEntry, LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE } from "./lib/src/types";
import { Logger } from "./lib/src/logger";
import { isErrorOfMissingDoc } from "./lib/src/utils_couchdb";
import { getDocData } from "./lib/src/utils";
import { getDocData, readContent } from "./lib/src/utils";
import { isPlainText, stripPrefix } from "./lib/src/path";
function isImage(path: string) {
@@ -66,7 +66,7 @@ export class DocumentHistoryModal extends Modal {
}
}
async loadFile(initialRev: string) {
async loadFile(initialRev?: string) {
if (!this.id) {
this.id = await this.plugin.path2id(this.file);
}
@@ -109,7 +109,7 @@ export class DocumentHistoryModal extends Modal {
if (v) {
URL.revokeObjectURL(v);
}
this.BlobURLs.set(key, undefined);
this.BlobURLs.delete(key);
}
generateBlobURL(key: string, data: Uint8Array) {
this.revokeURL(key);
@@ -247,12 +247,10 @@ export class DocumentHistoryModal extends Modal {
Logger(`Old content copied to clipboard`, LOG_LEVEL_NOTICE);
});
});
async function focusFile(path: string) {
const targetFile = app.vault
.getFiles()
.find((f) => f.path === path);
const focusFile = async (path: string) => {
const targetFile = this.plugin.app.vault.getFileByPath(path);
if (targetFile) {
const leaf = app.workspace.getLeaf(false);
const leaf = this.plugin.app.workspace.getLeaf(false);
await leaf.openFile(targetFile);
} else {
Logger("The file could not view on the editor", LOG_LEVEL_NOTICE)
@@ -265,19 +263,16 @@ export class DocumentHistoryModal extends Modal {
const pathToWrite = stripPrefix(this.file);
if (!isValidPath(pathToWrite)) {
Logger("Path is not valid to write content.", LOG_LEVEL_INFO);
return;
}
if (this.currentDoc?.datatype == "plain") {
await this.plugin.vaultAccess.adapterWrite(pathToWrite, getDocData(this.currentDoc.data));
await focusFile(pathToWrite);
this.close();
} else if (this.currentDoc?.datatype == "newnote") {
await this.plugin.vaultAccess.adapterWrite(pathToWrite, decodeBinary(this.currentDoc.data));
await focusFile(pathToWrite);
this.close();
} else {
Logger(`Could not parse entry`, LOG_LEVEL_NOTICE);
if (!this.currentDoc) {
Logger("No active file loaded.", LOG_LEVEL_INFO);
return;
}
const d = readContent(this.currentDoc);
await this.plugin.vaultAccess.adapterWrite(pathToWrite, d);
await focusFile(pathToWrite);
this.close();
});
});
}

View File

@@ -2,12 +2,11 @@
import ObsidianLiveSyncPlugin from "./main";
import { onDestroy, onMount } from "svelte";
import type { AnyEntry, FilePathWithPrefix } from "./lib/src/types";
import { createBinaryBlob, getDocData, isDocContentSame } from "./lib/src/utils";
import { getDocData, isDocContentSame, readAsBlob } from "./lib/src/utils";
import { diff_match_patch } from "./deps";
import { DocumentHistoryModal } from "./DocumentHistoryModal";
import { isPlainText, stripAllPrefixes } from "./lib/src/path";
import { TFile } from "./deps";
import { decodeBinary } from "./lib/src/strbin";
export let plugin: ObsidianLiveSyncPlugin;
let showDiffInfo = false;
@@ -107,15 +106,9 @@
if (checkStorageDiff) {
const abs = plugin.vaultAccess.getAbstractFileByPath(stripAllPrefixes(plugin.getPath(docA)));
if (abs instanceof TFile) {
let result = false;
if (isPlainText(docA.path)) {
const data = await plugin.vaultAccess.adapterRead(abs);
result = await isDocContentSame(data, doc.data);
} else {
const data = await plugin.vaultAccess.adapterReadBinary(abs);
const dataEEncoded = createBinaryBlob(data);
result = await isDocContentSame(dataEEncoded, createBinaryBlob(decodeBinary(doc.data)));
}
const data = await plugin.vaultAccess.adapterReadAuto(abs);
const d = readAsBlob(doc);
const result = await isDocContentSame(data, d);
if (result) {
diffDetail += " ⚖️";
} else {

View File

@@ -1,13 +1,14 @@
import { App, PluginSettingTab, Setting, sanitizeHTMLToDom, TextAreaComponent, MarkdownRenderer, stringifyYaml } from "./deps";
import { DEFAULT_SETTINGS, type ObsidianLiveSyncSettings, type ConfigPassphraseStore, type RemoteDBSettings, type FilePathWithPrefix, type HashAlgorithm, type DocumentID, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE, LOG_LEVEL_INFO } from "./lib/src/types";
import { createBinaryBlob, createTextBlob, delay, isDocContentSame } from "./lib/src/utils";
import { DEFAULT_SETTINGS, type ObsidianLiveSyncSettings, type ConfigPassphraseStore, type RemoteDBSettings, type FilePathWithPrefix, type HashAlgorithm, type DocumentID, LOG_LEVEL_NOTICE, LOG_LEVEL_VERBOSE, LOG_LEVEL_INFO, type LoadedEntry, PREFERRED_SETTING_CLOUDANT, PREFERRED_SETTING_SELF_HOSTED } from "./lib/src/types";
import { createBlob, delay, isDocContentSame, readAsBlob } from "./lib/src/utils";
import { versionNumberString2Number } from "./lib/src/strbin";
import { Logger } from "./lib/src/logger";
import { checkSyncInfo, isCloudantURI } from "./lib/src/utils_couchdb";
import { testCrypt } from "./lib/src/e2ee_v2";
import ObsidianLiveSyncPlugin from "./main";
import { askYesNo, performRebuildDB, requestToCouchDB, scheduleTask } from "./utils";
import { request, type ButtonComponent } from "obsidian";
import { request, type ButtonComponent, TFile } from "obsidian";
import { shouldBeIgnored } from "./lib/src/path";
export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
@@ -85,11 +86,11 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
}
w.querySelectorAll(`.sls-setting-label`).forEach((element) => {
element.removeClass("selected");
(element.querySelector<HTMLInputElement>("input[type=radio]")).checked = false;
(element.querySelector<HTMLInputElement>("input[type=radio]"))!.checked = false;
});
w.querySelectorAll(`.sls-setting-label.c-${screen}`).forEach((element) => {
element.addClass("selected");
(element.querySelector<HTMLInputElement>("input[type=radio]")).checked = true;
(element.querySelector<HTMLInputElement>("input[type=radio]"))!.checked = true;
});
this.selectedScreen = screen;
};
@@ -119,7 +120,7 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
tmpDiv.innerHTML = `<button> OK, I read all. </button>`;
if (lastVersion > this.plugin.settings.lastReadUpdates) {
const informationButtonDiv = h3El.appendChild(tmpDiv);
informationButtonDiv.querySelector("button").addEventListener("click", async () => {
informationButtonDiv.querySelector("button")?.addEventListener("click", async () => {
this.plugin.settings.lastReadUpdates = lastVersion;
await this.plugin.saveSettings();
informationButtonDiv.remove();
@@ -229,23 +230,23 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
let remoteTroubleShootMDSrc = "";
try {
remoteTroubleShootMDSrc = await request(`${rawRepoURI}${basePath}/${filename}`);
} catch (ex) {
} catch (ex: any) {
remoteTroubleShootMDSrc = "Error Occurred!!\n" + ex.toString();
}
const remoteTroubleShootMD = remoteTroubleShootMDSrc.replace(/\((.*?(.png)|(.jpg))\)/g, `(${rawRepoURI}${basePath}/$1)`)
// Render markdown
await MarkdownRenderer.render(this.plugin.app, `<a class='sls-troubleshoot-anchor'></a> [Tips and Troubleshooting](${topPath}) [PageTop](${filename})\n\n${remoteTroubleShootMD}`, troubleShootEl, `${rawRepoURI}`, this.plugin);
// Menu
troubleShootEl.querySelector<HTMLAnchorElement>(".sls-troubleshoot-anchor")
.parentElement.setCssStyles({ position: "sticky", top: "-1em", backgroundColor: "var(--modal-background)" });
troubleShootEl.querySelector<HTMLAnchorElement>(".sls-troubleshoot-anchor")?.parentElement?.setCssStyles({ position: "sticky", top: "-1em", backgroundColor: "var(--modal-background)" });
// Trap internal links.
troubleShootEl.querySelectorAll<HTMLAnchorElement>("a.internal-link").forEach((anchorEl) => {
anchorEl.addEventListener("click", async (evt) => {
const uri = anchorEl.getAttr("data-href");
if (!uri) return;
if (uri.startsWith("#")) {
evt.preventDefault();
const elements = Array.from(troubleShootEl.querySelectorAll<HTMLHeadingElement>("[data-heading]"))
const p = elements.find(e => e.getAttr("data-heading").toLowerCase().split(" ").join("-") == uri.substring(1).toLowerCase());
const p = elements.find(e => e.getAttr("data-heading")?.toLowerCase().split(" ").join("-") == uri.substring(1).toLowerCase());
if (p) {
p.setCssStyles({ scrollMargin: "3em" });
p.scrollIntoView({ behavior: "instant", block: "start" });
@@ -266,6 +267,16 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
const containerRemoteDatabaseEl = containerEl.createDiv();
containerRemoteDatabaseEl.createEl("h3", { text: "Remote Database configuration" });
const syncWarn = containerRemoteDatabaseEl.createEl("div", { text: `These settings are kept locked while any synchronization options are enabled. Disable these options in the "Sync Settings" tab to unlock.` });
if (this.plugin.settings.couchDB_URI.startsWith("http://")) {
if (this.plugin.isMobile) {
containerRemoteDatabaseEl.createEl("div", { text: `Configured as using plain HTTP. We cannot connect to the remote. Please set up the credentials and use HTTPS for the remote URI.` })
.addClass("op-warn");
} else {
containerRemoteDatabaseEl.createEl("div", { text: `Configured as using plain HTTP. We might fail on mobile devices.` })
.addClass("op-warn-info");
}
}
syncWarn.addClass("op-warn-info");
syncWarn.addClass("sls-hidden");
@@ -403,7 +414,7 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
tmpDiv.addClass("ob-btn-config-fix");
tmpDiv.innerHTML = `<label>${title}</label><button>Fix</button>`;
const x = checkResultDiv.appendChild(tmpDiv);
x.querySelector("button").addEventListener("click", async () => {
x.querySelector("button")?.addEventListener("click", async () => {
Logger(`CouchDB Configuration: ${title} -> Set ${key} to ${value}`)
const res = await requestToCouchDB(this.plugin.settings.couchDB_URI, this.plugin.settings.couchDB_USER, this.plugin.settings.couchDB_PASSWORD, undefined, key, value);
if (res.status == 200) {
@@ -499,15 +510,11 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
const origins = ["app://obsidian.md", "capacitor://localhost", "http://localhost"];
for (const org of origins) {
const rr = await requestToCouchDB(this.plugin.settings.couchDB_URI, this.plugin.settings.couchDB_USER, this.plugin.settings.couchDB_PASSWORD, org);
const responseHeaders = Object.entries(rr.headers)
const responseHeaders = Object.fromEntries(Object.entries(rr.headers)
.map((e) => {
e[0] = (e[0] + "").toLowerCase();
e[0] = `${e[0]}`.toLowerCase();
return e;
})
.reduce((obj, [key, val]) => {
obj[key] = val;
return obj;
}, {} as { [key: string]: string });
}));
addResult(`Origin check:${org}`);
if (responseHeaders["access-control-allow-credentials"] != "true") {
addResult("❗ CORS is not allowing credential");
@@ -523,7 +530,7 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
addResult("--Done--", ["ob-btn-config-head"]);
addResult("If you have some trouble with Connection-check even though all Config-check has been passed, Please check your reverse proxy's configuration.", ["ob-btn-config-info"]);
Logger(`Checking configuration done`, LOG_LEVEL_INFO);
} catch (ex) {
} catch (ex: any) {
if (ex?.status == 401) {
addResult(`❗ Access forbidden.`);
addResult(`We could not continue the test.`);
@@ -663,7 +670,7 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
.setWarning()
.setDisabled(false)
.onClick(async () => {
await rebuildDB("localOnlyWithChunks");
await rebuildDB("localOnly");
})
)
.addButton((button) =>
@@ -773,9 +780,10 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
this.plugin.settings.passphrase = "";
}
if (isCloudantURI(this.plugin.settings.couchDB_URI)) {
this.plugin.settings.customChunkSize = 0;
// this.plugin.settings.customChunkSize = 0;
this.plugin.settings = { ...this.plugin.settings, ...PREFERRED_SETTING_CLOUDANT };
} else {
this.plugin.settings.customChunkSize = 50;
this.plugin.settings = { ...this.plugin.settings, ...PREFERRED_SETTING_SELF_HOSTED };
}
changeDisplay("30")
})
@@ -791,7 +799,7 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
new Setting(containerGeneralSettingsEl)
.setName("Show status inside the editor")
.setDesc("")
.setDesc("Reflected after reboot")
.addToggle((toggle) =>
toggle.setValue(this.plugin.settings.showStatusOnEditor).onChange(async (value) => {
this.plugin.settings.showStatusOnEditor = value;
@@ -810,7 +818,16 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
})
);
}
new Setting(containerGeneralSettingsEl)
.setName("Show status on the status bar")
.setDesc("Reflected after reboot.")
.addToggle((toggle) =>
toggle.setValue(this.plugin.settings.showStatusOnStatusbar).onChange(async (value) => {
this.plugin.settings.showStatusOnStatusbar = value;
await this.plugin.saveSettings();
this.display();
})
);
containerGeneralSettingsEl.createEl("h4", { text: "Logging" });
new Setting(containerGeneralSettingsEl)
.setName("Show only notifications")
@@ -1053,7 +1070,7 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
if (!this.plugin.settings.isConfigured) {
this.plugin.settings.isConfigured = true;
await this.plugin.saveSettings();
await rebuildDB("localOnlyWithChunks");
await rebuildDB("localOnly");
Logger("All done! Please set up subsequent devices with 'Copy current settings as a new setup URI' and 'Use the copied setup URI'.", LOG_LEVEL_NOTICE);
await this.plugin.addOnSetup.command_copySetupURI();
} else {
@@ -1325,7 +1342,7 @@ export class ObsidianLiveSyncSettingTab extends PluginSettingTab {
});
text.inputEl.setAttribute("type", "number");
});
let skipPatternTextArea: TextAreaComponent = null;
let skipPatternTextArea: TextAreaComponent;
const defaultSkipPattern = "\\/node_modules\\/, \\/\\.git\\/, \\/obsidian-livesync\\/";
const defaultSkipPatternXPlat = defaultSkipPattern + ",\\/workspace$ ,\\/workspace.json$,\\/workspace-mobile.json$";
new Setting(containerSyncSettingEl)
@@ -1698,6 +1715,52 @@ ${stringifyYaml(pluginConfig)}`;
const hatchWarn = containerHatchEl.createEl("div", { text: `To stop the boot up sequence for fixing problems on databases, you can put redflag.md on top of your vault (Rebooting obsidian is required).` });
hatchWarn.addClass("op-warn-info");
const addResult = (path: string, file: TFile | false, fileOnDB: LoadedEntry | false) => {
resultArea.appendChild(resultArea.createEl("div", {}, el => {
el.appendChild(el.createEl("h6", { text: path }));
el.appendChild(el.createEl("div", {}, infoGroupEl => {
infoGroupEl.appendChild(infoGroupEl.createEl("div", { text: `Storage : Modified: ${!file ? `Missing:` : `${new Date(file.stat.mtime).toLocaleString()}, Size:${file.stat.size}`}` }))
infoGroupEl.appendChild(infoGroupEl.createEl("div", { text: `Database: Modified: ${!fileOnDB ? `Missing:` : `${new Date(fileOnDB.mtime).toLocaleString()}, Size:${fileOnDB.size}`}` }))
}));
if (fileOnDB && file) {
el.appendChild(el.createEl("button", { text: "Show history" }, buttonEl => {
buttonEl.onClickEvent(() => {
this.plugin.showHistory(file, fileOnDB._id);
})
}))
}
if (file) {
el.appendChild(el.createEl("button", { text: "Storage -> Database" }, buttonEl => {
buttonEl.onClickEvent(() => {
this.plugin.updateIntoDB(file, undefined, true);
el.remove();
})
}))
}
if (fileOnDB) {
el.appendChild(el.createEl("button", { text: "Database -> Storage" }, buttonEl => {
buttonEl.onClickEvent(() => {
this.plugin.pullFile(this.plugin.getPath(fileOnDB), [], true, undefined, false);
el.remove();
})
}))
}
return el;
}))
}
const checkBetweenStorageAndDatabase = async (file: TFile, fileOnDB: LoadedEntry) => {
const dataContent = readAsBlob(fileOnDB);
const content = createBlob(await this.plugin.vaultAccess.vaultReadAuto(file))
if (await isDocContentSame(content, dataContent)) {
Logger(`Compare: SAME: ${file.path}`)
} else {
Logger(`Compare: CONTENT IS NOT MATCHED! ${file.path}`, LOG_LEVEL_NOTICE);
addResult(file.path, file, fileOnDB)
}
}
new Setting(containerHatchEl)
.setName("Verify and repair all files")
.setDesc("Compare the content of files between on local database and storage. If not matched, you will asked which one want to keep.")
@@ -1708,47 +1771,36 @@ ${stringifyYaml(pluginConfig)}`;
.setWarning()
.onClick(async () => {
const files = this.app.vault.getFiles();
const documents = [] as FilePathWithPrefix[];
const adn = this.plugin.localDatabase.findAllNormalDocs()
for await (const i of adn) documents.push(this.plugin.getPath(i));
const allPaths = [...new Set([...documents, ...files.map(e => e.path as FilePathWithPrefix)])];
let i = 0;
for (const file of files) {
for (const path of allPaths) {
i++;
Logger(`${i}/${files.length}\n${file.path}`, LOG_LEVEL_NOTICE, "verify");
if (!await this.plugin.isTargetFile(file)) continue;
const fileOnDB = await this.plugin.localDatabase.getDBEntry(file.path as FilePathWithPrefix);
if (!fileOnDB) {
Logger(`Compare: Not found on local database: ${file.path}`, LOG_LEVEL_NOTICE);
Logger(`${i}/${files.length}\n${path}`, LOG_LEVEL_NOTICE, "verify");
if (shouldBeIgnored(path)) continue;
const abstractFile = this.plugin.vaultAccess.getAbstractFileByPath(path);
const fileOnStorage = abstractFile instanceof TFile ? abstractFile : false;
if (!await this.plugin.isTargetFile(path)) continue;
if (fileOnStorage && this.plugin.isFileSizeExceeded(fileOnStorage.stat.size)) continue;
const fileOnDB = await this.plugin.localDatabase.getDBEntry(path);
if (fileOnDB && this.plugin.isFileSizeExceeded(fileOnDB.size)) continue;
if (!fileOnDB && fileOnStorage) {
Logger(`Compare: Not found on the local database: ${path}`, LOG_LEVEL_NOTICE);
addResult(path, fileOnStorage, false)
continue;
}
let content: Blob;
if (fileOnDB.type == "newnote") {
content = createBinaryBlob(await this.plugin.vaultAccess.vaultReadBinary(file));
} else {
content = createTextBlob(await this.plugin.vaultAccess.vaultRead(file));
if (fileOnDB && !fileOnStorage) {
Logger(`Compare: Not found on the storage: ${path}`, LOG_LEVEL_NOTICE);
addResult(path, false, fileOnDB)
continue;
}
if (isDocContentSame(content, fileOnDB.data)) {
Logger(`Compare: SAME: ${file.path}`)
} else {
Logger(`Compare: CONTENT IS NOT MATCHED! ${file.path}`, LOG_LEVEL_NOTICE);
resultArea.appendChild(resultArea.createEl("div", {}, el => {
el.appendChild(el.createEl("h6", { text: file.path }));
el.appendChild(el.createEl("div", {}, infoGroupEl => {
infoGroupEl.appendChild(infoGroupEl.createEl("div", { text: `Storage : Modified: ${new Date(file.stat.mtime).toLocaleString()}, Size:${file.stat.size}` }))
infoGroupEl.appendChild(infoGroupEl.createEl("div", { text: `Database: Modified: ${new Date(fileOnDB.mtime).toLocaleString()}, Size:${content.size}` }))
}));
el.appendChild(el.createEl("button", { text: "Storage -> Database" }, buttonEl => {
buttonEl.onClickEvent(() => {
this.plugin.updateIntoDB(file, undefined, true);
el.remove();
})
}))
el.appendChild(el.createEl("button", { text: "Database -> Storage" }, buttonEl => {
buttonEl.onClickEvent(() => {
this.plugin.pullFile(file.path as FilePathWithPrefix, [], true, undefined, false);
el.remove();
})
}))
return el;
}))
if (fileOnStorage && fileOnDB) {
await checkBetweenStorageAndDatabase(fileOnStorage, fileOnDB)
}
}
Logger("done", LOG_LEVEL_NOTICE, "verify");
@@ -1777,6 +1829,7 @@ ${stringifyYaml(pluginConfig)}`;
//Prepare converted data
newDoc._id = idEncoded;
newDoc.path = docName as FilePathWithPrefix;
// @ts-ignore
delete newDoc._rev;
try {
const obfuscatedDoc = await this.plugin.localDatabase.getRaw(idEncoded, { revs_info: true });
@@ -1802,7 +1855,7 @@ ${stringifyYaml(pluginConfig)}`;
Logger(`Converting ${docName} Failed!`, LOG_LEVEL_NOTICE);
Logger(ret, LOG_LEVEL_VERBOSE);
}
} catch (ex) {
} catch (ex: any) {
if (ex?.status == 404) {
// We can perform this safely
if ((await this.plugin.localDatabase.putRaw(newDoc)).ok) {
@@ -1923,7 +1976,7 @@ ${stringifyYaml(pluginConfig)}`;
toggle.setValue(!this.plugin.settings.useIndexedDBAdapter).onChange(async (value) => {
this.plugin.settings.useIndexedDBAdapter = !value;
await this.plugin.saveSettings();
await rebuildDB("localOnlyWithChunks");
await rebuildDB("localOnly");
})
);
@@ -2121,20 +2174,20 @@ ${stringifyYaml(pluginConfig)}`;
.setWarning()
.setDisabled(false)
.onClick(async () => {
await rebuildDB("localOnlyWithChunks");
await rebuildDB("localOnly");
})
)
new Setting(containerMaintenanceEl)
.setName("Fetch rebuilt DB with all remote chunks")
.setDesc("Restore or reconstruct local database from remote database but use remote chunk .")
.setName("Fetch rebuilt DB (Save local documents before)")
.setDesc("Restore or reconstruct local database from remote database but use local chunks.")
.addButton((button) =>
button
.setButtonText("Fetch all")
.setButtonText("Save and Fetch")
.setWarning()
.setDisabled(false)
.onClick(async () => {
await rebuildDB("localOnly");
await rebuildDB("localOnlyWithChunks");
})
)

View File

@@ -1,6 +1,7 @@
import { type App, TFile, type DataWriteOptions, TFolder, TAbstractFile } from "./deps";
import { serialized } from "./lib/src/lock";
import { Logger } from "./lib/src/logger";
import { isPlainText } from "./lib/src/path";
import type { FilePath } from "./lib/src/types";
import { createBinaryBlob, isDocContentSame } from "./lib/src/utils";
import type { InternalFileInfo } from "./types";
@@ -56,6 +57,12 @@ export class SerializedFileAccess {
return await processReadFile(file, () => this.app.vault.adapter.readBinary(path));
}
async adapterReadAuto(file: TFile | string) {
const path = file instanceof TFile ? file.path : file;
if (isPlainText(path)) return await processReadFile(file, () => this.app.vault.adapter.read(path));
return await processReadFile(file, () => this.app.vault.adapter.readBinary(path));
}
async adapterWrite(file: TFile | string, data: string | ArrayBuffer | Uint8Array, options?: DataWriteOptions) {
const path = file instanceof TFile ? file.path : file;
if (typeof (data) === "string") {
@@ -77,12 +84,19 @@ export class SerializedFileAccess {
return await processReadFile(file, () => this.app.vault.readBinary(file));
}
async vaultReadAuto(file: TFile) {
const path = file.path;
if (isPlainText(path)) return await processReadFile(file, () => this.app.vault.read(file));
return await processReadFile(file, () => this.app.vault.readBinary(file));
}
async vaultModify(file: TFile, data: string | ArrayBuffer | Uint8Array, options?: DataWriteOptions) {
if (typeof (data) === "string") {
return await processWriteFile(file, async () => {
const oldData = await this.app.vault.read(file);
if (data === oldData) {
markChangesAreSame(file, file.stat.mtime, options.mtime);
if (options && options.mtime) markChangesAreSame(file, file.stat.mtime, options.mtime);
return false
}
await this.app.vault.modify(file, data, options)
@@ -93,7 +107,7 @@ export class SerializedFileAccess {
return await processWriteFile(file, async () => {
const oldData = await this.app.vault.readBinary(file);
if (await isDocContentSame(createBinaryBlob(oldData), createBinaryBlob(data))) {
markChangesAreSame(file, file.stat.mtime, options.mtime);
if (options && options.mtime) markChangesAreSame(file, file.stat.mtime, options.mtime);
return false;
}
await this.app.vault.modifyBinary(file, toArrayBuffer(data), options)
@@ -149,10 +163,9 @@ export class SerializedFileAccess {
c += v;
try {
await this.app.vault.adapter.mkdir(c);
} catch (ex) {
// basically skip exceptions.
if (ex.message && ex.message == "Folder already exists.") {
// especially this message is.
} catch (ex: any) {
if (ex?.message == "Folder already exists.") {
// Skip if already exists.
} else {
Logger("Folder Create Error");
Logger(ex);

View File

@@ -1,7 +1,7 @@
import type { SerializedFileAccess } from "./SerializedFileAccess";
import { Plugin, TAbstractFile, TFile, TFolder } from "./deps";
import { Logger } from "./lib/src/logger";
import { isPlainText, shouldBeIgnored } from "./lib/src/path";
import { shouldBeIgnored } from "./lib/src/path";
import type { KeyedQueueProcessor } from "./lib/src/processor";
import { LOG_LEVEL_NOTICE, type FilePath, type ObsidianLiveSyncSettings } from "./lib/src/types";
import { delay } from "./lib/src/utils";
@@ -109,7 +109,8 @@ export class StorageEventManagerObsidian extends StorageEventManager {
if (file instanceof TFolder) continue;
if (!await this.plugin.isTargetFile(file.path)) continue;
let cache: null | string | ArrayBuffer;
// Stop cache using to prevent the corruption;
// let cache: null | string | ArrayBuffer;
// new file or something changed, cache the changes.
if (file instanceof TFile && (type == "CREATE" || type == "CHANGED")) {
// Wait for a bit while to let the writer has marked `touched` at the file.
@@ -117,12 +118,13 @@ export class StorageEventManagerObsidian extends StorageEventManager {
if (this.plugin.vaultAccess.recentlyTouched(file)) {
continue;
}
if (!isPlainText(file.name)) {
cache = await this.plugin.vaultAccess.vaultReadBinary(file);
} else {
cache = await this.plugin.vaultAccess.vaultCacheRead(file);
if (!cache) cache = await this.plugin.vaultAccess.vaultRead(file);
}
// cache = await this.plugin.vaultAccess.vaultReadAuto(file);
// if (!isPlainText(file.name)) {
// cache = await this.plugin.vaultAccess.vaultReadBinary(file);
// } else {
// cache = await this.plugin.vaultAccess.vaultCacheRead(file);
// if (!cache) cache = await this.plugin.vaultAccess.vaultRead(file);
// }
}
const fileInfo = file instanceof TFile ? {
ctime: file.stat.ctime,
@@ -137,7 +139,7 @@ export class StorageEventManagerObsidian extends StorageEventManager {
args: {
file: fileInfo,
oldPath,
cache,
// cache,
ctx
},
key: atomicKey

View File

@@ -7,10 +7,9 @@ import PluginPane from "./PluginPane.svelte";
export class PluginDialogModal extends Modal {
plugin: ObsidianLiveSyncPlugin;
logEl: HTMLDivElement;
component: PluginPane = null;
component: PluginPane | undefined;
isOpened() {
return this.component != null;
return this.component != undefined;
}
constructor(app: App, plugin: ObsidianLiveSyncPlugin) {
@@ -21,7 +20,7 @@ export class PluginDialogModal extends Modal {
onOpen() {
const { contentEl } = this;
this.titleEl.setText("Customization Sync (Beta2)")
if (this.component == null) {
if (!this.component) {
this.component = new PluginPane({
target: contentEl,
props: { plugin: this.plugin },
@@ -30,9 +29,9 @@ export class PluginDialogModal extends Modal {
}
onClose() {
if (this.component != null) {
if (this.component) {
this.component.$destroy();
this.component = null;
this.component = undefined;
}
}
}
@@ -94,13 +93,13 @@ export class InputStringDialog extends Modal {
}
export class PopoverSelectString extends FuzzySuggestModal<string> {
app: App;
callback: (e: string) => void = () => { };
callback: ((e: string) => void) | undefined = () => { };
getItemsFun: () => string[] = () => {
return ["yes", "no"];
}
constructor(app: App, note: string, placeholder: string | null, getItemsFun: () => string[], callback: (e: string) => void) {
constructor(app: App, note: string, placeholder: string | undefined, getItemsFun: (() => string[]) | undefined, callback: (e: string) => void) {
super(app);
this.app = app;
this.setPlaceholder((placeholder ?? "y/n) ") + note);
@@ -118,13 +117,14 @@ export class PopoverSelectString extends FuzzySuggestModal<string> {
onChooseItem(item: string, evt: MouseEvent | KeyboardEvent): void {
// debugger;
this.callback(item);
this.callback = null;
this.callback?.(item);
this.callback = undefined;
}
onClose(): void {
setTimeout(() => {
if (this.callback != null) {
if (this.callback) {
this.callback("");
this.callback = undefined;
}
}, 100);
}
@@ -136,16 +136,16 @@ export class MessageBox extends Modal {
title: string;
contentMd: string;
buttons: string[];
result: string;
result: string | false = false;
isManuallyClosed = false;
defaultAction: string | undefined;
timeout: number | undefined;
timer: ReturnType<typeof setInterval> = undefined;
timer: ReturnType<typeof setInterval> | undefined = undefined;
defaultButtonComponent: ButtonComponent | undefined;
onSubmit: (result: string | false) => void;
constructor(plugin: Plugin, title: string, contentMd: string, buttons: string[], defaultAction: (typeof buttons)[number], timeout: number, onSubmit: (result: (typeof buttons)[number] | false) => void) {
constructor(plugin: Plugin, title: string, contentMd: string, buttons: string[], defaultAction: (typeof buttons)[number], timeout: number | undefined, onSubmit: (result: (typeof buttons)[number] | false) => void) {
super(plugin.app);
this.plugin = plugin;
this.title = title;
@@ -156,6 +156,7 @@ export class MessageBox extends Modal {
this.timeout = timeout;
if (this.timeout) {
this.timer = setInterval(() => {
if (this.timeout === undefined) return;
this.timeout--;
if (this.timeout < 0) {
if (this.timer) {
@@ -166,7 +167,7 @@ export class MessageBox extends Modal {
this.isManuallyClosed = true;
this.close();
} else {
this.defaultButtonComponent.setButtonText(`( ${this.timeout} ) ${defaultAction}`);
this.defaultButtonComponent?.setButtonText(`( ${this.timeout} ) ${defaultAction}`);
}
}, 1000);
}
@@ -223,7 +224,7 @@ export class MessageBox extends Modal {
}
export function confirmWithMessage(plugin: Plugin, title: string, contentMd: string, buttons: string[], defaultAction?: (typeof buttons)[number], timeout?: number): Promise<(typeof buttons)[number] | false> {
export function confirmWithMessage(plugin: Plugin, title: string, contentMd: string, buttons: string[], defaultAction: (typeof buttons)[number], timeout?: number): Promise<(typeof buttons)[number] | false> {
return new Promise((res) => {
const dialog = new MessageBox(plugin, title, contentMd, buttons, defaultAction, timeout, (result) => res(result));
dialog.open();

Submodule src/lib updated: 8a8177c1f0...29e23f5763

View File

@@ -4,7 +4,7 @@ import { type Diff, DIFF_DELETE, DIFF_EQUAL, DIFF_INSERT, diff_match_patch, stri
import { Notice, Plugin, TFile, addIcon, TFolder, normalizePath, TAbstractFile, Editor, MarkdownView, type RequestUrlParam, type RequestUrlResponse, requestUrl, type MarkdownFileInfo } from "./deps";
import { type EntryDoc, type LoadedEntry, type ObsidianLiveSyncSettings, type diff_check_result, type diff_result_leaf, type EntryBody, LOG_LEVEL, VER, DEFAULT_SETTINGS, type diff_result, FLAGMD_REDFLAG, SYNCINFO_ID, SALT_OF_PASSPHRASE, type ConfigPassphraseStore, type CouchDBConnection, FLAGMD_REDFLAG2, FLAGMD_REDFLAG3, PREFIXMD_LOGFILE, type DatabaseConnectingStatus, type EntryHasPath, type DocumentID, type FilePathWithPrefix, type FilePath, type AnyEntry, LOG_LEVEL_DEBUG, LOG_LEVEL_INFO, LOG_LEVEL_NOTICE, LOG_LEVEL_URGENT, LOG_LEVEL_VERBOSE, type SavingEntry, MISSING_OR_ERROR, NOT_CONFLICTED, AUTO_MERGED, CANCELLED, LEAVE_TO_SUBSEQUENT, FLAGMD_REDFLAG2_HR, FLAGMD_REDFLAG3_HR, } from "./lib/src/types";
import { type InternalFileInfo, type CacheData, type FileEventItem, FileWatchEventQueueMax } from "./types";
import { arrayToChunkedArray, createBinaryBlob, createTextBlob, fireAndForget, getDocData, isDocContentSame, isObjectDifferent, sendValue } from "./lib/src/utils";
import { arrayToChunkedArray, createBlob, fireAndForget, getDocData, isDocContentSame, isObjectDifferent, readContent, sendValue } from "./lib/src/utils";
import { Logger, setGlobalLogFunction } from "./lib/src/logger";
import { PouchDB } from "./lib/src/pouchdb-browser.js";
import { ConflictResolveModal } from "./ConflictResolveModal";
@@ -211,6 +211,15 @@ export default class ObsidianLiveSyncPlugin extends Plugin
this.last_successful_post = true;
}
Logger(`HTTP:${method}${size} to:${localURL} -> ${response.status}`, LOG_LEVEL_DEBUG);
if (Math.floor(response.status / 100) !== 2) {
const r = response.clone();
Logger(`The request may have failed. The reason sent by the server: ${r.status}: ${r.statusText}`);
try {
Logger(await (await r.blob()).text(), LOG_LEVEL_VERBOSE);
} catch (_) {
Logger("Cloud not parse response", LOG_LEVEL_VERBOSE);
}
}
return response;
} catch (ex) {
Logger(`HTTP:${method}${size} to:${localURL} -> failed`, LOG_LEVEL_VERBOSE);
@@ -786,8 +795,10 @@ Note: We can always able to read V1 format. It will be progressively converted.
const lsKey = "obsidian-live-sync-ver" + this.getVaultName();
const last_version = localStorage.getItem(lsKey);
this.observeForLogs();
this.statusBar = this.addStatusBarItem();
this.statusBar.addClass("syncstatusbar");
if (this.settings.showStatusOnStatusbar) {
this.statusBar = this.addStatusBarItem();
this.statusBar.addClass("syncstatusbar");
}
const lastVersion = ~~(versionNumberString2Number(manifestVersion) / 1000);
if (lastVersion > this.settings.lastReadUpdates && this.settings.isConfigured) {
Logger("Self-hosted LiveSync has undergone a major upgrade. Please open the setting dialog, and check the information pane.", LOG_LEVEL_NOTICE);
@@ -1329,12 +1340,12 @@ We can perform a command in this file.
return;
}
const cache = queue.args.cache;
// const cache = queue.args.cache;
if (queue.type == "CREATE" || queue.type == "CHANGED") {
fireAndForget(() => this.checkAndApplySettingFromMarkdown(queue.args.file.path, true));
const keyD1 = `file-last-proc-DELETED-${file.path}`;
await this.kvDB.set(keyD1, mtime);
if (!await this.updateIntoDB(targetFile, cache)) {
if (!await this.updateIntoDB(targetFile, undefined)) {
Logger(`STORAGE -> DB: failed, cancel the relative operations: ${targetFile.path}`, LOG_LEVEL_INFO);
// cancel running queues and remove one of atomic operation
this.cancelRelativeEvent(queue);
@@ -1551,7 +1562,7 @@ We can perform a command in this file.
Logger(msg + "ERROR, invalid path: " + path, LOG_LEVEL_NOTICE);
return;
}
const writeData = doc.datatype == "newnote" ? decodeBinary(doc.data) : getDocData(doc.data);
const writeData = readContent(doc);
await this.vaultAccess.ensureDirectory(path);
try {
let outFile;
@@ -1631,7 +1642,7 @@ We can perform a command in this file.
}
databaseQueueCount = reactiveSource(0);
databaseQueuedProcessor = new KeyedQueueProcessor(async (docs: EntryBody[]) => {
databaseQueuedProcessor = new QueueProcessor(async (docs: EntryBody[]) => {
const dbDoc = docs[0];
const path = this.getPath(dbDoc);
// If `Read chunks online` is disabled, chunks should be transferred before here.
@@ -1658,13 +1669,13 @@ We can perform a command in this file.
storageApplyingProcessor = new KeyedQueueProcessor(async (docs: LoadedEntry[]) => {
const entry = docs[0];
const path = this.getPath(entry);
Logger(`Processing ${path} (${entry._id.substring(0, 8)}: ${entry._rev?.substring(0, 5)}) change...`, LOG_LEVEL_VERBOSE);
Logger(`Processing ${path} (${entry._id.substring(0, 8)}: ${entry._rev?.substring(0, 5)}) :Started...`, LOG_LEVEL_VERBOSE);
const targetFile = this.vaultAccess.getAbstractFileByPath(this.getPathWithoutPrefix(entry));
if (targetFile instanceof TFolder) {
Logger(`${this.getPath(entry)} is already exist as the folder`);
} else {
await this.processEntryDoc(entry, targetFile instanceof TFile ? targetFile : undefined);
Logger(`Processing ${path} (${entry._id.substring(0, 8)}:${entry._rev?.substring(0, 5)}) `);
Logger(`Processing ${path} (${entry._id.substring(0, 8)} :${entry._rev?.substring(0, 5)}) : Done`);
}
return;
@@ -1708,7 +1719,7 @@ We can perform a command in this file.
Logger(`Processing ${change.path} has been skipped due to file size exceeding the limit`, LOG_LEVEL_NOTICE);
return;
}
this.databaseQueuedProcessor.enqueueWithKey(change.path, change);
this.databaseQueuedProcessor.enqueue(change);
}
return;
}, { batchSize: 1, suspended: true, concurrentLimit: 100, delay: 0, totalRemainingReactiveSource: this.replicationResultCount }).startPipeline().onUpdateProgress(() => {
@@ -1853,22 +1864,22 @@ We can perform a command in this file.
const newLog = log;
// scheduleTask("update-display", 50, () => {
this.statusBar?.setText(newMsg.split("\n")[0]);
const selector = `.CodeMirror-wrap,` +
`.markdown-preview-view.cm-s-obsidian,` +
`.markdown-source-view.cm-s-obsidian,` +
`.canvas-wrapper,` +
`.empty-state`
;
// const selector = `.CodeMirror-wrap,` +
// `.markdown-preview-view.cm-s-obsidian,` +
// `.markdown-source-view.cm-s-obsidian,` +
// `.canvas-wrapper,` +
// `.empty-state`
// ;
if (this.settings.showStatusOnEditor) {
const root = activeDocument.documentElement;
const q = root.querySelectorAll(selector);
q.forEach(e => e.setAttr("data-log", '' + (newMsg + "\n" + newLog) + ''))
root.style.setProperty("--sls-log-text", "'" + (newMsg + "\\A " + newLog) + "'");
} else {
const root = activeDocument.documentElement;
const q = root.querySelectorAll(selector);
q.forEach(e => e.setAttr("data-log", ''))
// const root = activeDocument.documentElement;
// root.style.setProperty("--log-text", "'" + (newMsg + "\\A " + newLog) + "'");
}
// }, true);
scheduleTask("log-hide", 3000, () => { this.statusLog.value = "" });
}
@@ -1902,7 +1913,7 @@ Even if you choose to clean up, you will see this option again if you exit Obsid
const CHOICE_DISMISS = "Dismiss";
const ret = await confirmWithMessage(this, "Cleaned", message, [CHOICE_FETCH, CHOICE_CLEAN, CHOICE_DISMISS], CHOICE_DISMISS, 30);
if (ret == CHOICE_FETCH) {
await performRebuildDB(this, "localOnlyWithChunks");
await performRebuildDB(this, "localOnly");
}
if (ret == CHOICE_CLEAN) {
const remoteDB = await this.getReplicator().connectRemoteCouchDBWithSetting(this.settings, this.getIsMobile(), true);
@@ -1936,7 +1947,7 @@ Or if you are sure know what had been happened, we can unlock the database from
const CHOICE_DISMISS = "Dismiss";
const ret = await confirmWithMessage(this, "Locked", message, [CHOICE_FETCH, CHOICE_DISMISS], CHOICE_DISMISS, 10);
if (ret == CHOICE_FETCH) {
await performRebuildDB(this, "localOnlyWithChunks");
await performRebuildDB(this, "localOnly");
}
}
}
@@ -2113,7 +2124,7 @@ Or if you are sure know what had been happened, we can unlock the database from
new QueueProcessor(
async (pairs) => {
const docs = await this.localDatabase.allDocsRaw<EntryDoc>({ keys: pairs.map(e => e.id), include_docs: true });
const docsMap = docs.rows.reduce((p, c) => ({ ...p, [c.id]: c.doc }), {} as Record<DocumentID, EntryDoc>);
const docsMap = Object.fromEntries(docs.rows.map(e => [e.id, e.doc]));
const syncFilesToSync = pairs.map((e) => ({ file: e.file, doc: docsMap[e.id] as LoadedEntry }));
return syncFilesToSync;
}
@@ -2675,41 +2686,24 @@ Or if you are sure know what had been happened, we can unlock the database from
if (shouldBeIgnored(file.path)) {
return true;
}
let content: Blob;
let datatype: "plain" | "newnote" = "newnote";
if (!cache) {
if (!isPlainText(file.name)) {
Logger(`Reading : ${file.path}`, LOG_LEVEL_VERBOSE);
const contentBin = await this.vaultAccess.vaultReadBinary(file);
Logger(`Processing: ${file.path}`, LOG_LEVEL_VERBOSE);
try {
content = createBinaryBlob(contentBin);
} catch (ex) {
Logger(`The file ${file.path} could not be encoded`);
Logger(ex, LOG_LEVEL_VERBOSE);
return false;
}
datatype = "newnote";
} else {
content = createTextBlob(await this.vaultAccess.vaultRead(file));
datatype = "plain";
}
} else {
if (cache instanceof ArrayBuffer) {
Logger(`Cache Processing: ${file.path}`, LOG_LEVEL_VERBOSE);
try {
content = createBinaryBlob(cache);
} catch (ex) {
Logger(`The file ${file.path} could not be encoded`);
Logger(ex, LOG_LEVEL_VERBOSE);
return false;
}
datatype = "newnote"
} else {
content = createTextBlob(cache);
datatype = "plain";
}
}
// let content: Blob;
// let datatype: "plain" | "newnote" = "newnote";
const isPlain = isPlainText(file.name);
const possiblyLarge = !isPlain;
// if (!cache) {
if (possiblyLarge) Logger(`Reading : ${file.path}`, LOG_LEVEL_VERBOSE);
const content = createBlob(await this.vaultAccess.vaultReadAuto(file));
const datatype = isPlain ? "plain" : "newnote";
// }
// else if (cache instanceof ArrayBuffer) {
// Logger(`Cache Reading: ${file.path}`, LOG_LEVEL_VERBOSE);
// content = createBinaryBlob(cache);
// datatype = "newnote"
// } else {
// content = createTextBlob(cache);
// datatype = "plain";
// }
if (possiblyLarge) Logger(`Processing: ${file.path}`, LOG_LEVEL_VERBOSE);
const fullPath = getPathFromTFile(file);
const id = await this.path2id(fullPath);
const d: SavingEntry = {

View File

@@ -242,14 +242,11 @@ export function mergeObject(
ret[key] = v;
}
}
const retSorted = Object.fromEntries(Object.entries(ret).sort((a, b) => a[0] < b[0] ? -1 : a[0] > b[0] ? 1 : 0));
if (Array.isArray(objA) && Array.isArray(objB)) {
return Object.values(Object.entries(ret)
.sort()
.reduce((p, [key, value]) => ({ ...p, [key]: value }), {}));
return Object.values(retSorted);
}
return Object.entries(ret)
.sort()
.reduce((p, [key, value]) => ({ ...p, [key]: value }), {});
return retSorted;
}
export function flattenObject(obj: Record<string | number | symbol, any>, path: string[] = []): [string, any][] {
@@ -313,7 +310,7 @@ export function isCustomisationSyncMetadata(str: string): boolean {
export const askYesNo = (app: App, message: string): Promise<"yes" | "no"> => {
return new Promise((res) => {
const popover = new PopoverSelectString(app, message, null, null, (result) => res(result as "yes" | "no"));
const popover = new PopoverSelectString(app, message, undefined, undefined, (result) => res(result as "yes" | "no"));
popover.open();
});
};
@@ -327,7 +324,7 @@ export const askSelectString = (app: App, message: string, items: string[]): Pro
};
export const askString = (app: App, title: string, key: string, placeholder: string, isPassword?: boolean): Promise<string | false> => {
export const askString = (app: App, title: string, key: string, placeholder: string, isPassword: boolean = false): Promise<string | false> => {
return new Promise((res) => {
const dialog = new InputStringDialog(app, title, key, placeholder, isPassword, (result) => res(result));
dialog.open();
@@ -400,7 +397,7 @@ export const _requestToCouchDB = async (baseUri: string, username: string, passw
};
return await requestUrl(requestParam);
}
export const requestToCouchDB = async (baseUri: string, username: string, password: string, origin: string, key?: string, body?: string, method?: string) => {
export const requestToCouchDB = async (baseUri: string, username: string, password: string, origin: string = "", key?: string, body?: string, method?: string) => {
const uri = `_node/_local/_config${key ? "/" + key : ""}`;
return await _requestToCouchDB(baseUri, username, password, origin, uri, body, method);
};
@@ -440,7 +437,7 @@ export function compareMTime(baseMTime: number, targetMTime: number): typeof BAS
export function markChangesAreSame(file: TFile | AnyEntry | string, mtime1: number, mtime2: number) {
if (mtime1 === mtime2) return true;
const key = typeof file == "string" ? file : file instanceof TFile ? file.path : file.path ?? file._id;
const pairs = sameChangePairs.get(key, []);
const pairs = sameChangePairs.get(key, []) || [];
if (pairs.some(e => e == mtime1 || e == mtime2)) {
sameChangePairs.set(key, [...new Set([...pairs, mtime1, mtime2])]);
} else {
@@ -449,7 +446,7 @@ export function markChangesAreSame(file: TFile | AnyEntry | string, mtime1: numb
}
export function isMarkedAsSameChanges(file: TFile | AnyEntry | string, mtimes: number[]) {
const key = typeof file == "string" ? file : file instanceof TFile ? file.path : file.path ?? file._id;
const pairs = sameChangePairs.get(key, []);
const pairs = sameChangePairs.get(key, []) || [];
if (mtimes.every(e => pairs.indexOf(e) !== -1)) {
return EVEN;
}

View File

@@ -77,14 +77,6 @@
border-top: 1px solid var(--background-modifier-border);
}
/* .sls-table-head{
width:50%;
}
.sls-table-tail{
width:50%;
} */
.sls-header-button {
margin-left: 2em;
}
@@ -94,7 +86,7 @@
}
:root {
--slsmessage: "";
--sls-log-text: "";
}
.sls-troubleshoot-preview {
@@ -110,7 +102,7 @@
.markdown-source-view.cm-s-obsidian::before,
.canvas-wrapper::before,
.empty-state::before {
content: attr(data-log);
content: var(--sls-log-text, "");
text-align: right;
white-space: pre-wrap;
position: absolute;

View File

@@ -10,134 +10,43 @@ Note: we got a very performance improvement.
Note at 0.22.2: **Now, to rescue mobile devices, Maximum file size is set to 50 by default**. Please configure the limit as you need. If you do not want to limit the sizes, set zero manually, please.
#### Version history
- 0.22.14:
- New feature:
- We can disable the status bar in the setting dialogue.
- Improved:
- Now some files are handled as correct data type.
- Customisation sync now uses the digest of each file for better performance.
- The status in the Editor now works performant.
- Refactored:
- Common functions have been ready and the codebase has been organised.
- Stricter type checking following TypeScript updates.
- Remove old iOS workaround for simplicity and performance.
- 0.22.13:
- Improved:
- Now using HTTP for the remote database URI warns of an error (on mobile) or notice (on desktop).
- Refactored:
- Dependencies have been polished.
- 0.22.12:
- Changed:
- The default settings has been changed.
- Improved:
- Default and preferred settings are applied on completion of the wizard.
- Fixed:
- Now Initialisation `Fetch` will be performed smoothly and there will be fewer conflicts.
- No longer stuck while Handling transferred or initialised documents.
- 0.22.11:
- Fixed:
- `Verify and repair all files` is no longer broken.
- New feature:
- Now `Verify and repair all files` is able to...
- Restore if the file only in the local database.
- Show the history.
- Improved:
- Performance improved.
- 0.22.10
- Fixed:
- No longer unchanged hidden files and customisations are saved and transferred now.
- File integrity of vault history indicates the integrity correctly.
- Improved:
- In the report, the schema of the remote database URI is now printed.
- 0.22.9
- Fixed:
- Fixed a bug on `fetch chunks on demand` that could not fetch the chunks on demand.
- Improved:
- `fetch chunks on demand` works more smoothly.
- Initialisation `Fetch` is now more efficient.
- Tidied:
- Removed some meaningless codes.
- 0.22.8
- Fixed:
- Now fetch and unlock the locked remote database works well again.
- No longer crash on symbolic links inside hidden folders.
- Improved:
- Chunks are now created more efficiently.
- Splitting old notes into a larger chunk.
- Better performance in saving notes.
- Network activities are indicated as an icon.
- Less memory used for binary processing.
- Tidied:
- Cleaned unused functions up.
- Sorting out the codes that have become nonsense.
- Changed:
- Now no longer `fetch chunks on demand` needs `Pacing replication`
- The setting `Do not pace synchronization` has been deleted.
- 0.22.7
- Fixed:
- No longer deleted hidden files were ignored.
- The document history dialogue is now able to process the deleted revisions.
- Deletion of a hidden file is now surely performed even if the file is already conflicted.
- 0.22.6
- Fixed:
- Fixed a problem with synchronisation taking a long time to start in some cases.
- The first synchronisation after update might take a bit longer.
- Now we can disable E2EE encryption.
- Improved:
- `Setup Wizard` is now more clear.
- `Minimal Setup` is now more simple.
- Self-hosted LiveSync now be able to use even if there are vaults with the same name.
- Database suffix will automatically added.
- Now Self-hosted LiveSync waits until set-up is complete.
- Show reload prompts when possibly recommended while settings.
- New feature:
- A guidance dialogue prompting for settings will be shown after the installation.
- Changed
- `Open setup URI` is now `Use the copied setup URI`
- `Copy setup URI` is now `Copy current settings as a new setup URI`
- `Setup Wizard` is now `Minimal Setup`
- `Check database configuration` is now `Check and Fix database configuration`
- 0.22.5
- Fixed:
- Some description of settings have been refined
- New feature:
- TroubleShooting is now shown in the setting dialogue.
- 0.22.4
- Fixed:
- Now the result of conflict resolution could be surely written into the storage.
- Deleted files can be handled correctly again in the history dialogue and conflict dialogue.
- Some wrong log messages were fixed.
- Change handling now has become more stable.
- Some event handling became to be safer.
- Improved:
- Dumping document information shows conflicts and revisions.
- The timestamp-only differences can be surely cached.
- Timestamp difference detection can be rounded by two seconds.
- Refactored:
- A bit of organisation to write the test.
- 0.22.3
- Fixed:
- No longer detects storage changes which have been caused by Self-hosted LiveSync itself.
- Setting sync file will be detected only if it has been configured now.
- And its log will be shown only while the verbose log is enabled.
- Customisation file enumeration has got less blingy.
- Deletion of files is now reliably synchronised.
- Fixed and improved:
- In-editor-status is now shown in the following areas:
- Note editing pane (Source mode and live-preview mode).
- New tab pane.
- Canvas pane.
- 0.22.2
- Fixed:
- Now the results of resolving conflicts are surely synchronised.
- Modified:
- Some setting items got new clear names. (`Sync Settings` -> `Targets`).
- New feature:
- We can limit the synchronising files by their size. (`Sync Settings` -> `Targets` -> `Maximum file size`).
- It depends on the size of the newer one.
- At Obsidian 1.5.3 on mobile, we should set this to around 50MB to avoid restarting Obsidian.
- Now the settings could be stored in a specific markdown file to synchronise or switch it (`General Setting` -> `Share settings via markdown`).
- [Screwdriver](https://github.com/vrtmrz/obsidian-screwdriver) is quite good, but mostly we only need this.
- Customisation of the obsoleted device is now able to be deleted at once.
- We have to put the maintenance mode in at the Customisation sync dialogue.
- 0.22.1
- New feature:
- We can perform automatic conflict resolution for inactive files, and postpone only manual ones by `Postpone manual resolution of inactive files`.
- Now we can see the image in the document history dialogue.
- We can see the difference of the image, in the document history dialogue.
- And also we can highlight differences.
- Improved:
- Hidden file sync has been stabilised.
- Now automatically reloads the conflict-resolution dialogue when new conflicted revisions have arrived.
- Fixed:
- No longer periodic process runs after unloading the plug-in.
- Now the modification of binary files is surely stored in the storage.
- 0.22.0
- Refined:
- Task scheduling logics has been rewritten.
- Screen updates are also now efficient.
- Possibly many bugs and fragile behaviour has been fixed.
- Status updates and logging have been thinned out to display.
- Fixed:
- Remote-chunk-fetching now works with keeping request intervals
- New feature:
- We can show only the icons in the editor.
- Progress indicators have been more meaningful:
- 📥 Unprocessed transferred items
- 📄 Working database operation
- 💾 Working write storage processes
- ⏳ Working read storage processes
- 🛫 Pending read storage processes
- ⚙️ Working or pending storage processes of hidden files
- 🧩 Waiting chunks
- 🔌 Working Customisation items (Configuration, snippets and plug-ins)
... To continue on to `updates_old.md`.

View File

@@ -1,3 +1,141 @@
### 0.22.0
A few years passed since Self-hosted LiveSync was born, and our codebase had been very complicated. This could be patient now, but it should be a tremendous hurt.
Therefore at v0.22.0, for future maintainability, I refined task scheduling logic totally.
Of course, I think this would be our suffering in some cases. However, I would love to ask you for your cooperation and contribution.
Sorry for being absent so much long. And thank you for your patience!
Note: we got a very performance improvement.
Note at 0.22.2: **Now, to rescue mobile devices, Maximum file size is set to 50 by default**. Please configure the limit as you need. If you do not want to limit the sizes, set zero manually, please.
#### Version history
- 0.22.9
- Fixed:
- Fixed a bug on `fetch chunks on demand` that could not fetch the chunks on demand.
- Improved:
- `fetch chunks on demand` works more smoothly.
- Initialisation `Fetch` is now more efficient.
- Tidied:
- Removed some meaningless codes.
- 0.22.8
- Fixed:
- Now fetch and unlock the locked remote database works well again.
- No longer crash on symbolic links inside hidden folders.
- Improved:
- Chunks are now created more efficiently.
- Splitting old notes into a larger chunk.
- Better performance in saving notes.
- Network activities are indicated as an icon.
- Less memory used for binary processing.
- Tidied:
- Cleaned unused functions up.
- Sorting out the codes that have become nonsense.
- Changed:
- Now no longer `fetch chunks on demand` needs `Pacing replication`
- The setting `Do not pace synchronization` has been deleted.
- 0.22.7
- Fixed:
- No longer deleted hidden files were ignored.
- The document history dialogue is now able to process the deleted revisions.
- Deletion of a hidden file is now surely performed even if the file is already conflicted.
- 0.22.6
- Fixed:
- Fixed a problem with synchronisation taking a long time to start in some cases.
- The first synchronisation after update might take a bit longer.
- Now we can disable E2EE encryption.
- Improved:
- `Setup Wizard` is now more clear.
- `Minimal Setup` is now more simple.
- Self-hosted LiveSync now be able to use even if there are vaults with the same name.
- Database suffix will automatically added.
- Now Self-hosted LiveSync waits until set-up is complete.
- Show reload prompts when possibly recommended while settings.
- New feature:
- A guidance dialogue prompting for settings will be shown after the installation.
- Changed
- `Open setup URI` is now `Use the copied setup URI`
- `Copy setup URI` is now `Copy current settings as a new setup URI`
- `Setup Wizard` is now `Minimal Setup`
- `Check database configuration` is now `Check and Fix database configuration`
- 0.22.5
- Fixed:
- Some description of settings have been refined
- New feature:
- TroubleShooting is now shown in the setting dialogue.
- 0.22.4
- Fixed:
- Now the result of conflict resolution could be surely written into the storage.
- Deleted files can be handled correctly again in the history dialogue and conflict dialogue.
- Some wrong log messages were fixed.
- Change handling now has become more stable.
- Some event handling became to be safer.
- Improved:
- Dumping document information shows conflicts and revisions.
- The timestamp-only differences can be surely cached.
- Timestamp difference detection can be rounded by two seconds.
- Refactored:
- A bit of organisation to write the test.
- 0.22.3
- Fixed:
- No longer detects storage changes which have been caused by Self-hosted LiveSync itself.
- Setting sync file will be detected only if it has been configured now.
- And its log will be shown only while the verbose log is enabled.
- Customisation file enumeration has got less blingy.
- Deletion of files is now reliably synchronised.
- Fixed and improved:
- In-editor-status is now shown in the following areas:
- Note editing pane (Source mode and live-preview mode).
- New tab pane.
- Canvas pane.
- 0.22.2
- Fixed:
- Now the results of resolving conflicts are surely synchronised.
- Modified:
- Some setting items got new clear names. (`Sync Settings` -> `Targets`).
- New feature:
- We can limit the synchronising files by their size. (`Sync Settings` -> `Targets` -> `Maximum file size`).
- It depends on the size of the newer one.
- At Obsidian 1.5.3 on mobile, we should set this to around 50MB to avoid restarting Obsidian.
- Now the settings could be stored in a specific markdown file to synchronise or switch it (`General Setting` -> `Share settings via markdown`).
- [Screwdriver](https://github.com/vrtmrz/obsidian-screwdriver) is quite good, but mostly we only need this.
- Customisation of the obsoleted device is now able to be deleted at once.
- We have to put the maintenance mode in at the Customisation sync dialogue.
- 0.22.1
- New feature:
- We can perform automatic conflict resolution for inactive files, and postpone only manual ones by `Postpone manual resolution of inactive files`.
- Now we can see the image in the document history dialogue.
- We can see the difference of the image, in the document history dialogue.
- And also we can highlight differences.
- Improved:
- Hidden file sync has been stabilised.
- Now automatically reloads the conflict-resolution dialogue when new conflicted revisions have arrived.
- Fixed:
- No longer periodic process runs after unloading the plug-in.
- Now the modification of binary files is surely stored in the storage.
- 0.22.0
- Refined:
- Task scheduling logics has been rewritten.
- Screen updates are also now efficient.
- Possibly many bugs and fragile behaviour has been fixed.
- Status updates and logging have been thinned out to display.
- Fixed:
- Remote-chunk-fetching now works with keeping request intervals
- New feature:
- We can show only the icons in the editor.
- Progress indicators have been more meaningful:
- 📥 Unprocessed transferred items
- 📄 Working database operation
- 💾 Working write storage processes
- ⏳ Working read storage processes
- 🛫 Pending read storage processes
- ⚙️ Working or pending storage processes of hidden files
- 🧩 Waiting chunks
- 🔌 Working Customisation items (Configuration, snippets and plug-ins)
... To continue on to `updates_old.md`.
### 0.21.0
The E2EE encryption V2 format has been reverted. That was probably the cause of the glitch.
Instead, to maintain efficiency, files are treated with Blob until just before saving. Along with this, the old-fashioned encryption format has also been discontinued.