diff --git a/docs/design_docs/path_component_length_compatibility.md b/docs/design_docs/path_component_length_compatibility.md new file mode 100644 index 00000000..367d7d42 --- /dev/null +++ b/docs/design_docs/path_component_length_compatibility.md @@ -0,0 +1,70 @@ +--- +date: 2026-09-04 +commonlib-version: "0.1.21" +self-hosted-livesync-version: "1.0.24" +status: unreleased +--- + +# Path component length compatibility + +## Purpose + +File systems place limits on each file or folder name, rather than applying one +common limit to an entire Vault-relative path. Those limits are also expressed +in different units. Self-hosted LiveSync therefore treats 255 UTF-8 bytes as a +focused Android and Linux compatibility warning, not as a universal definition +of a valid path. + +## Basis for the 255-byte warning + +- The Linux kernel documentation gives ext4 a maximum file-name length of + [255 bytes](https://www.kernel.org/doc/html/latest/filesystems/ext4/directory.html). +- The F2FS on-disk header defines + [`F2FS_NAME_LEN` as 255](https://android.googlesource.com/kernel/common/+/88d92fb1c034922572bab93482ac9cc61d4ba43c/include/linux/f2fs_fs.h) + and stores names in byte arrays. +- Android's MediaProvider uses a + [`MAX_FILENAME_BYTES` value of 255](https://android.googlesource.com/platform/packages/providers/MediaProvider/+/bae279463/src/com/android/providers/media/util/FileUtils.java) + when building file names. Its source notes that emulated storage can write to + ext4 through FUSE, where names are encoded as UTF-8. +- Android 11 and later use + [FUSE for emulated storage](https://source.android.com/docs/core/storage/fuse-passthrough), + with requests passing through to the underlying file system. + +Together, these provide a conservative compatibility boundary for file names +which may reach Android or Linux storage. They do not show that every Android +device, storage provider, or Linux file system has the same limit. + +## Why the rule is not universal + +Other platforms describe component limits differently. Microsoft's file-system +comparison documents limits in +[Unicode characters](https://learn.microsoft.com/en-us/windows/win32/fileio/filesystem-functionality-comparison), +not UTF-8 bytes. Apple's HFS Plus format stores a name as up to +[255 16-bit `UniChar` values](https://developer.apple.com/library/archive/technotes/tn/tn1150.html). +Apple's APFS guidance discusses valid UTF-8 names, normalisation, and case +sensitivity, but does not establish a universal +[255-byte component rule](https://developer.apple.com/library/archive/documentation/FileManagement/Conceptual/APFS_Guide/FAQ/FAQ.html). + +A name can consequently exceed 255 UTF-8 bytes and still work on one platform, +or fail for another platform-specific reason while remaining below this +boundary. + +## Product policy + +Self-hosted LiveSync applies the warning as follows: + +1. split the Vault-relative path on `/` and inspect each non-empty component; +2. measure each component after UTF-8 encoding; +3. accept 255 bytes without this warning and warn at 256 bytes or more; +4. identify every over-limit file or folder name in the active-file status; +5. do not reject, truncate, or rename the path; and +6. treat the result of the real storage operation as authoritative. + +If a scan cannot process an individual file, its path is recorded in the +verbose log and remains eligible for a later retry. Ordinary start-up may still +become ready so that unaffected files can synchronise. Explicit Fetch and +Rebuild operations retain strict scan completion because they establish an +authoritative local or remote state. + +This policy does not replace the existing checks for reserved characters, +case collisions, ignore rules, or configured file-size limits. diff --git a/docs/tech_info.md b/docs/tech_info.md index d2cafc6c..9a881e6f 100644 --- a/docs/tech_info.md +++ b/docs/tech_info.md @@ -22,6 +22,9 @@ Note: The figure is drawn as single-directional, between two devices for demonst defines the current revision-tree and file-provenance rules. - [Chunk Retrieval and Waiting](design_docs/chunk_retrieval_and_waiting.md) defines missing-Chunk arrival and quiescence handling. +- [Path component length compatibility](design_docs/path_component_length_compatibility.md) + explains why 255 UTF-8 bytes is an Android and Linux compatibility warning, + rather than a universal rule for deciding whether a path is valid. - [Data Compression](specs_data_compression.md) and [Garbage Collection V3](specs_garbage_collection.md) describe their respective storage and maintenance contracts. diff --git a/src/common/messages/combinedMessages.prod.ts b/src/common/messages/combinedMessages.prod.ts index 27d97349..74ded0d5 100644 --- a/src/common/messages/combinedMessages.prod.ts +++ b/src/common/messages/combinedMessages.prod.ts @@ -4212,6 +4212,9 @@ export const allMessages: Readonly- + A file or folder name exceeds ${maxBytes} UTF-8 bytes and may not work on + some Android and Linux file systems: ${components} showLog: Show Log moduleMigration: fix0256: @@ -2126,6 +2129,7 @@ xxhash64 (Fastest): xxhash64 (Fastest) "This feature enables direct synchronisation between devices. No server is required, but both devices must be online at the same time for synchronisation to occur, and some features may be limited. Internet connection is only required to signalling (detecting peers) and not for data transfer.": "This feature enables direct synchronisation between devices. No server is required, but both devices must be online at the same time for synchronisation to occur, and some features may be limited. Internet connection is only required to signalling (detecting peers) and not for data transfer." Ui: Common: + LocalDatabaseInitialisationFailed: Local database initialisation did not complete. See the log for details. Signal: Caution: CAUTION Danger: DANGER diff --git a/src/common/pathCompatibility.ts b/src/common/pathCompatibility.ts new file mode 100644 index 00000000..69ad7254 --- /dev/null +++ b/src/common/pathCompatibility.ts @@ -0,0 +1,26 @@ +export const ANDROID_LINUX_PATH_COMPONENT_UTF8_WARNING_BOUNDARY = 255; + +export interface OversizedPathComponent { + component: string; + utf8Bytes: number; +} + +const utf8Encoder = new TextEncoder(); + +/** + * Return path components which exceed the conservative Android/Linux + * compatibility boundary. + * + * Obsidian paths use forward slashes. The limit applies to each file or + * folder name, not to the combined Vault-relative path. + */ +export function findPathComponentsExceedingUtf8Limit( + path: string, + maxBytes: number = ANDROID_LINUX_PATH_COMPONENT_UTF8_WARNING_BOUNDARY +): OversizedPathComponent[] { + return path + .split("/") + .filter((component) => component.length > 0) + .map((component) => ({ component, utf8Bytes: utf8Encoder.encode(component).byteLength })) + .filter(({ utf8Bytes }) => utf8Bytes > maxBytes); +} diff --git a/src/common/pathCompatibility.unit.spec.ts b/src/common/pathCompatibility.unit.spec.ts new file mode 100644 index 00000000..33502939 --- /dev/null +++ b/src/common/pathCompatibility.unit.spec.ts @@ -0,0 +1,46 @@ +import { describe, expect, it } from "vitest"; +import { + ANDROID_LINUX_PATH_COMPONENT_UTF8_WARNING_BOUNDARY, + findPathComponentsExceedingUtf8Limit, +} from "./pathCompatibility.ts"; + +describe("findPathComponentsExceedingUtf8Limit", () => { + it("accepts 255 UTF-8 bytes and reports 256 UTF-8 bytes", () => { + expect(findPathComponentsExceedingUtf8Limit("a".repeat(255))).toEqual([]); + expect(findPathComponentsExceedingUtf8Limit("a".repeat(256))).toEqual([ + { + component: "a".repeat(256), + utf8Bytes: 256, + }, + ]); + }); + + it("counts UTF-8 bytes rather than JavaScript characters", () => { + expect(findPathComponentsExceedingUtf8Limit("界".repeat(85))).toEqual([]); + expect(findPathComponentsExceedingUtf8Limit(`${"界".repeat(85)}a`)).toEqual([ + { + component: `${"界".repeat(85)}a`, + utf8Bytes: 256, + }, + ]); + }); + + it("does not apply the component limit to the whole path", () => { + const path = `${"a".repeat(200)}/${"b".repeat(200)}`; + + expect(new TextEncoder().encode(path).byteLength).toBeGreaterThan( + ANDROID_LINUX_PATH_COMPONENT_UTF8_WARNING_BOUNDARY + ); + expect(findPathComponentsExceedingUtf8Limit(path)).toEqual([]); + }); + + it("reports an oversized folder component as well as an oversized file name", () => { + const folder = "界".repeat(86); + const file = `${"b".repeat(256)}.md`; + + expect(findPathComponentsExceedingUtf8Limit(`parent/${folder}/${file}`)).toEqual([ + { component: folder, utf8Bytes: 258 }, + { component: file, utf8Bytes: 259 }, + ]); + }); +}); diff --git a/src/modules/features/ModuleLog.ts b/src/modules/features/ModuleLog.ts index 485014b1..4b7e4fec 100644 --- a/src/modules/features/ModuleLog.ts +++ b/src/modules/features/ModuleLog.ts @@ -49,6 +49,10 @@ import { MARK_LOG_NETWORK_ERROR, MARK_LOG_SEPARATOR } from "@vrtmrz/livesync-com import { NetworkWarningStyles } from "@vrtmrz/livesync-commonlib/compat/common/models/setting.const"; import { compatGlobal } from "@vrtmrz/livesync-commonlib/compat/common/coreEnvFunctions"; import { generateReport } from "@/common/reportTool.ts"; +import { + ANDROID_LINUX_PATH_COMPONENT_UTF8_WARNING_BOUNDARY, + findPathComponentsExceedingUtf8Limit, +} from "@/common/pathCompatibility.ts"; // This module cannot be a core module because it depends on the Obsidian UI. @@ -293,6 +297,18 @@ export class ModuleLog extends AbstractObsidianModule { reasonWarn.push("Some platforms may be unable to process this file correctly: " + labels.join(" ")); } } + const oversizedPathComponents = findPathComponentsExceedingUtf8Limit(thisFile.path); + if (oversizedPathComponents.length > 0) { + const components = oversizedPathComponents + .map(({ component, utf8Bytes }) => `${component} (${utf8Bytes} bytes)`) + .join(", "); + reasonWarn.push( + $msg("moduleLog.pathComponentTooLong", { + maxBytes: `${ANDROID_LINUX_PATH_COMPONENT_UTF8_WARNING_BOUNDARY}`, + components, + }) + ); + } // Case Sensitivity if (this.services.vault.shouldCheckCaseInsensitively()) { const f = (await this.core.storageAccess.getFiles()) diff --git a/src/modules/features/SettingDialogue/PaneMaintenance.ts b/src/modules/features/SettingDialogue/PaneMaintenance.ts index a016bb8f..9767eb71 100644 --- a/src/modules/features/SettingDialogue/PaneMaintenance.ts +++ b/src/modules/features/SettingDialogue/PaneMaintenance.ts @@ -412,7 +412,9 @@ export function paneMaintenance( .setDisabled(false) .onClick(async () => { await this.services.database.resetDatabase(); - await this.services.databaseEvents.initialiseDatabase(); + if (!(await this.services.databaseEvents.initialiseDatabase())) { + Logger($msg("Ui.Common.LocalDatabaseInitialisationFailed"), LOG_LEVEL_NOTICE); + } }) ); }); diff --git a/src/modules/features/SettingDialogue/PaneMaintenance.unit.spec.ts b/src/modules/features/SettingDialogue/PaneMaintenance.unit.spec.ts index 8b6e8f1e..7f5bf8b5 100644 --- a/src/modules/features/SettingDialogue/PaneMaintenance.unit.spec.ts +++ b/src/modules/features/SettingDialogue/PaneMaintenance.unit.spec.ts @@ -93,7 +93,7 @@ afterEach(() => { vi.clearAllMocks(); }); -describe("paneMaintenance Fresh Start Wipe", () => { +describe("paneMaintenance", () => { it("does not announce success when the remote wipe reports failure", async () => { const updateCheckPointInfo = vi.fn(async () => undefined); const resetRemoteBucket = vi.fn(async () => false); @@ -140,4 +140,49 @@ describe("paneMaintenance Fresh Start Wipe", () => { ); expect(maintenanceHarness.logger).not.toHaveBeenCalledWith("Deleted all data on remote server", "notice"); }); + + it("reports when database initialisation after a local reset does not complete", async () => { + const resetDatabase = vi.fn(async () => undefined); + const initialiseDatabase = vi.fn(async () => false); + const addPanel = vi.fn((_parent: HTMLElement, heading: string) => ({ + then(callback: (paneEl: HTMLElement) => void) { + if (heading === "Reset") { + callback({} as HTMLElement); + } + return Promise.resolve(); + }, + })); + const host = { + core: {}, + createEl: vi.fn(), + editingSettings: {}, + isConfiguredAs: vi.fn(), + onlyOnCouchDB: vi.fn(), + onlyOnCouchDBOrMinIO: vi.fn(), + onlyOnMinIO: vi.fn(), + services: { + appLifecycle: { askRestart: vi.fn() }, + database: { resetDatabase }, + databaseEvents: { initialiseDatabase }, + setting: { saveSettingData: vi.fn() }, + }, + }; + + paneMaintenance.call(host as never, {} as HTMLElement, { addPanel } as never); + const deleteLocalDatabase = maintenanceHarness.createdSettings.find( + ({ name }) => name === "Delete local database to reset or uninstall Self-hosted LiveSync" + ); + if (!deleteLocalDatabase?.click) { + throw new Error("Delete local database action was not registered"); + } + + await deleteLocalDatabase.click(); + + expect(resetDatabase).toHaveBeenCalledOnce(); + expect(initialiseDatabase).toHaveBeenCalledOnce(); + expect(maintenanceHarness.logger).toHaveBeenCalledWith( + "Ui.Common.LocalDatabaseInitialisationFailed", + "notice" + ); + }); }); diff --git a/src/modules/features/SettingDialogue/PanePatches.ts b/src/modules/features/SettingDialogue/PanePatches.ts index 87a08c21..dbc99b7e 100644 --- a/src/modules/features/SettingDialogue/PanePatches.ts +++ b/src/modules/features/SettingDialogue/PanePatches.ts @@ -15,6 +15,7 @@ import { PouchDB } from "@vrtmrz/livesync-commonlib/compat/pouchdb/pouchdb-brows import { ExtraSuffixIndexedDB } from "@vrtmrz/livesync-commonlib/compat/common/types"; import { migrateDatabases } from "./settingUtils.ts"; import { usesLegacyIndexedDBAdapter } from "@/common/compatibilitySettings.ts"; +import { $msg } from "@/common/translation"; export function panePatches(this: ObsidianLiveSyncSettingTab, paneEl: HTMLElement, { addPanel }: PageFunctions): void { void addPanel(paneEl, "Compatibility (Metadata)").then((paneEl) => { @@ -142,7 +143,9 @@ export function panePatches(this: ObsidianLiveSyncSettingTab, paneEl: HTMLElemen this.addOnSaved("additionalSuffixOfDatabaseName", async (key) => { Logger("Suffix has been changed. Reopening database...", LOG_LEVEL_NOTICE); - await this.services.databaseEvents.initialiseDatabase(); + if (!(await this.services.databaseEvents.initialiseDatabase())) { + Logger($msg("Ui.Common.LocalDatabaseInitialisationFailed"), LOG_LEVEL_NOTICE); + } }); new Setting(paneEl).autoWireDropDown("hashAlg", { diff --git a/src/modules/features/SettingDialogue/PanePatches.unit.spec.ts b/src/modules/features/SettingDialogue/PanePatches.unit.spec.ts index bcdf5904..efb5cb71 100644 --- a/src/modules/features/SettingDialogue/PanePatches.unit.spec.ts +++ b/src/modules/features/SettingDialogue/PanePatches.unit.spec.ts @@ -1,4 +1,5 @@ import { afterEach, describe, expect, it, vi } from "vitest"; +import { LOG_LEVEL_NOTICE } from "@vrtmrz/livesync-commonlib/compat/common/types"; import { panePatches } from "./PanePatches.ts"; const remediationHarness = vi.hoisted(() => { @@ -14,11 +15,13 @@ const remediationHarness = vi.hoisted(() => { }; const setButtonClassState = vi.fn(); const setSettingClassState = vi.fn(); + const logger = vi.fn(); return { createSpan, dateElement, inputEl, + logger, setButtonClassState, setSettingClassState, textComponent, @@ -59,9 +62,25 @@ vi.mock("./LiveSyncSetting.ts", () => ({ autoWireToggle(): this { return this; } + + autoWireText(): this { + return this; + } + + autoWireDropDown(): this { + return this; + } }, })); +vi.mock("@/common/translation", () => ({ + $msg: (message: string) => message, +})); + +vi.mock("@vrtmrz/livesync-commonlib/compat/common/logger", () => ({ + Logger: remediationHarness.logger, +})); + afterEach(() => { Reflect.deleteProperty(globalThis, "activeDocument"); vi.clearAllMocks(); @@ -69,7 +88,7 @@ afterEach(() => { remediationHarness.inputEl.type = ""; }); -describe("panePatches remediation setting", () => { +describe("panePatches", () => { it("creates the status element in the setting control instead of the document", () => { const hierarchyError = new DOMException( "Failed to execute 'appendChild' on 'Node': Only one element on document allowed.", @@ -115,4 +134,36 @@ describe("panePatches remediation setting", () => { ); expect(remediationHarness.setButtonClassState).toHaveBeenCalledWith("sls-setting-additional-action", true); }); + + it("reports when database reinitialisation after a suffix change does not complete", async () => { + const initialiseDatabase = vi.fn(async () => false); + let onSuffixSaved: (() => Promise) | undefined; + const host = { + addOnSaved: vi.fn((key: string, callback: () => Promise) => { + if (key === "additionalSuffixOfDatabaseName") onSuffixSaved = callback; + }), + services: { + databaseEvents: { initialiseDatabase }, + }, + }; + const addPanel = vi.fn((_paneEl: HTMLElement, title: string) => ({ + then(callback: (paneEl: HTMLElement) => void) { + if (title === "Edge case addressing (Database)") { + callback({} as HTMLElement); + } + return Promise.resolve(); + }, + })); + + panePatches.call(host as never, {} as HTMLElement, { addPanel } as never); + if (!onSuffixSaved) throw new Error("Database suffix save handler was not registered"); + + await onSuffixSaved(); + + expect(initialiseDatabase).toHaveBeenCalledOnce(); + expect(remediationHarness.logger).toHaveBeenCalledWith( + "Ui.Common.LocalDatabaseInitialisationFailed", + LOG_LEVEL_NOTICE + ); + }); }); diff --git a/src/modules/main/ModuleLiveSyncMain.ts b/src/modules/main/ModuleLiveSyncMain.ts index 2dc32c2f..2daed5c5 100644 --- a/src/modules/main/ModuleLiveSyncMain.ts +++ b/src/modules/main/ModuleLiveSyncMain.ts @@ -42,8 +42,11 @@ export class ModuleLiveSyncMain extends AbstractModule { return false; } } - const isInitialized = await this.services.databaseEvents.initialiseDatabase(false, false); + // Ordinary start-up may continue when individual files could not be + // processed. Explicit Fetch and Rebuild flows retain the strict default. + const isInitialized = await this.services.databaseEvents.initialiseDatabase(false, false, false, true); if (!isInitialized) { + this._log($msg("Ui.Common.LocalDatabaseInitialisationFailed"), LOG_LEVEL_NOTICE); //TODO:stop all sync. return false; } diff --git a/src/modules/main/ModuleLiveSyncMain.unit.spec.ts b/src/modules/main/ModuleLiveSyncMain.unit.spec.ts new file mode 100644 index 00000000..1fc6f06a --- /dev/null +++ b/src/modules/main/ModuleLiveSyncMain.unit.spec.ts @@ -0,0 +1,50 @@ +import { describe, expect, it, vi } from "vitest"; +import { LOG_LEVEL_NOTICE } from "@vrtmrz/livesync-commonlib/compat/common/types"; + +vi.mock("@/common/events.ts", () => ({ + EVENT_LAYOUT_READY: "layout-ready", + EVENT_PLUGIN_LOADED: "plugin-loaded", + EVENT_REQUEST_RELOAD_SETTING_TAB: "reload-setting-tab", + EVENT_SETTING_SAVED: "setting-saved", + eventHub: { + emitEvent: vi.fn(), + onEvent: vi.fn(), + }, +})); + +vi.mock("@/common/translation", () => ({ + $msg: (message: string) => message, + setLang: vi.fn(), +})); + +import { ModuleLiveSyncMain } from "./ModuleLiveSyncMain.ts"; + +describe("ModuleLiveSyncMain", () => { + it("reports a database preparation failure at the application boundary", async () => { + const initialiseDatabase = vi.fn(async () => false); + const log = vi.fn(); + const host = { + core: { + services: { + appLifecycle: { + onLayoutReady: vi.fn(async () => true), + }, + }, + }, + services: { + databaseEvents: { initialiseDatabase }, + }, + settings: { + suspendFileWatching: false, + suspendParseReplicationResult: false, + }, + _log: log, + }; + + const result = await ModuleLiveSyncMain.prototype._onLiveSyncReady.call(host as never); + + expect(result).toBe(false); + expect(initialiseDatabase).toHaveBeenCalledWith(false, false, false, true); + expect(log).toHaveBeenCalledWith("Ui.Common.LocalDatabaseInitialisationFailed", LOG_LEVEL_NOTICE); + }); +}); diff --git a/updates.md b/updates.md index 521cb64e..0efeec04 100644 --- a/updates.md +++ b/updates.md @@ -17,10 +17,13 @@ Earlier releases remain available in the 1.0 release history, the 1.0 preview hi #### Fixed - Conflict resolution dialogues now close when the same file is resolved elsewhere or the plug-in unloads. Requests for different files are shown one at a time, while a newer request for the same file replaces the stale dialogue. +- An individual file-processing failure during ordinary start-up no longer keeps the entire application unready. The affected path is recorded in verbose logs and remains eligible for retry, while explicit Fetch and Rebuild operations retain strict completion. +- Replication readiness diagnostics now state that application initialisation is incomplete instead of reporting only 'Not ready'. Database-preparation failures show a short notice, with the failed stage available in verbose logs. #### Improved - Start-up now keeps unconfigured Vaults on the onboarding path without running configured-only checks or accepting Config Doctor and incomplete-document repair requests. Returning a configured Vault to an unconfigured state also retires those requests for the current plug-in process, so completing setup admits them only after the requested restart. +- The active-file warning now identifies file or folder names longer than 255 UTF-8 bytes as an Android and Linux compatibility risk, without rejecting or changing the path. ### Testing