diff --git a/docs/settings.md b/docs/settings.md index 91101117..0885589c 100644 --- a/docs/settings.md +++ b/docs/settings.md @@ -820,9 +820,13 @@ Open the dialogue ## 7. Hatch -### 1. Reporting Issue +### 1. Reporting Tools -#### Make report to inform the issue +This panel groups the five actions that collect reports or inspect database state. These actions provide diagnostic information; they do not repair or reset the database. + +#### Prepare a report to inform an issue + +Select **Copy Report to clipboard** to collect diagnostic information and recent logs for a support report. Review the report before sharing it because it can contain sensitive information. #### Copy database information for a file @@ -830,20 +834,48 @@ Select a file to copy its local database information. The command **Copy databas The report includes the Vault-relative path, document and chunk identifiers, local database revisions, conflicts, and local chunk availability. It does not query the remote or include file contents. Paths and identifiers can still be private metadata, so review the report before sharing it. +#### Find a file from a report pseudonym + +Paste a path pseudonym from a diagnostic report, then select **Find file**. The lookup checks the current Vault and local normal-file Metadata only; it does not contact the remote database. It returns every matching local candidate, and shows each candidate's real Vault path and document ID only in this local dialogue. Multiple local document IDs for one path remain separate candidates. The follow-up for a Vault-only candidate marks Metadata as not inspected, because no local document was selected. No match can mean that the file was renamed or deleted, or that the matching Metadata exists only remotely. + +For a selected candidate, open an existing Vault file or prepare a shareable local report using the original report's pseudonym salt and the exact selected document ID. The report includes current, conflict, and history status, Chunk availability, merge bases, and Vault size facts; it does not include file contents or credentials. Its history status listing does not scan every historical body. + +Path pseudonyms use the report's public eight-character salt. Someone with the report can test a guessed path against a pseudonym. The same salt reproduces path and identifier pseudonyms in a targeted follow-up; older ordinal pseudonyms cannot be looked up with this tool. + #### Inspect database integrity -Select **Inspect** to choose original names or temporary pseudonyms and inspect normal-file Metadata and current conflicts locally and in CouchDB. The command **Inspect database integrity** opens the same dialogue. The inspection is read-only; its findings do not authorise deletion or perform repairs. See [Collect a report](troubleshooting.md#collect-a-report) for coverage, report formats, privacy, and cancellation. +Select **Inspect** to choose original names or temporary pseudonyms and inspect normal-file Metadata and current conflicts. **Include remote database** is selected by default; clear it to inspect only the local database and Vault. This choice applies to the current run only, and the report marks remote coverage as not checked when excluded. The command **Inspect database integrity** opens the same dialogue. The inspection is read-only; its findings do not authorise deletion or perform repairs. See [Collect a report](troubleshooting.md#collect-a-report) for coverage, report formats, privacy, and cancellation, and the [compact Markdown reading guide](troubleshooting.md#read-the-compact-markdown-integrity-report) for interpreting its state and review index. #### Analyse database usage Select **Analyse** to choose a name policy and report local revision and Chunk usage. The command **Analyse Database Usage (advanced)** opens the same dialogue when [advanced features](#enable-advanced-features) are enabled. Complete runs offer TSV and JSON; incomplete runs retain their status and reasons in JSON. Existing on-demand Chunk fetching stays unchanged. See [Collect a report](troubleshooting.md#collect-a-report) for accounting limits and export instructions. +### 2. Checks and Recovery + +This panel groups checks that may lead to changes to settings or database contents. + +#### Setting Doctor + +Run Setting Doctor to check for configuration mismatches or suboptimal settings. Applying its recommendations can change settings or request a database rebuild, so review the recommendations before proceeding. + +#### Scan for broken files + +Scan for files which are not stored correctly in the database. The scan can offer repair actions that change database contents; review each proposed action before confirming it. + +### 3. TroubleShooting + +This panel keeps the remote storage notification check and file logging controls separate from report collection and recovery checks. + +#### Reset notification threshold and check the remote database usage + +Reset the remote storage size threshold and check the remote storage size again. This changes the saved notification threshold. + #### Write logs into the file Setting key: writeLogToTheFile Warning! This will have a serious impact on performance. And the logs will not be synchronised under the default name. Please be careful with logs; they often contain your confidential information. -### 2. Scram Switches +### 4. Scram Switches Emergency controls to suspend synchronisation processes in order to prevent database corruption. If a critical mismatch or sync error occurs, the plug-in may automatically enter a Scram state and suspend operations. @@ -859,7 +891,7 @@ Setting key: suspendParseReplicationResult Stop reflecting database changes to storage files. -### 3. Recovery and Repair +### 5. Recovery and Repair #### Recreate chunks for current Vault files @@ -879,7 +911,7 @@ After confirmation, resolve every conflict by modification time. This logically #### Check and convert non-path-obfuscated files -### 4. Reset +### 6. Reset #### Back to non-configured diff --git a/docs/to_issue_reporting.md b/docs/to_issue_reporting.md index 1643f1c3..dc64edc1 100644 --- a/docs/to_issue_reporting.md +++ b/docs/to_issue_reporting.md @@ -4,6 +4,16 @@ Thank you for helping improve Self-hosted LiveSync. A concise report with the ri Use the [issue report template](https://github.com/vrtmrz/obsidian-livesync/issues/new?template=issue-report.md) for the report itself. Use [Troubleshooting](troubleshooting.md) to diagnose a symptom or choose a recovery action. +If a maintainer requests additional evidence, follow the relevant procedure below. These reports supplement the required information in this guide. + +| Requested evidence | Procedure | +| --------------------------------------------------------------------- | ------------------------------------- | +| Database integrity, current conflicts, or differences between sources | [Integrity report](#integrity-report) | +| Local revision and Chunk usage | [Usage report](#usage-report) | +| More information about a path pseudonym in an existing report | [File follow-up](#file-follow-up) | + +Maintainers can link directly to each procedure and adapt the [example requests](#requesting-diagnostics). + ## Preserve the original symptom Do not reset a database, rebuild a remote, change transport, or enable P2P merely to see whether the problem disappears. These actions can change the evidence and may make the original cause harder to identify. @@ -46,7 +56,7 @@ For a large report, you may share a GitHub Gist after reviewing and redacting it ### A problem involving database integrity or usage -Use **Hatch** → **Inspect database integrity** for database-wide consistency findings, or **Analyse database usage** for local revision and Chunk accounting. Follow [Collect a report](troubleshooting.md#collect-a-report) for the name choice, formats, and inspection limits. Temporary pseudonyms are the default; review the report before sharing it. Retain the status and reported reasons when sharing a partial or cancelled result. +Follow the [Integrity report](#integrity-report) procedure for database-wide consistency findings, or the [Usage report](#usage-report) procedure for local revision and Chunk accounting. [Collect a report](troubleshooting.md#collect-a-report) explains the inspection limits and how to read the results. ### A problem involving one file @@ -54,6 +64,8 @@ Run `Copy database information for the active file`, or use **Hatch** → **Copy This report describes only the local database on that device. It includes the Vault-relative path, document and chunk identifiers, local revisions, conflicts, and local chunk availability. It does not query the remote or include file contents. Review paths and identifiers as private metadata before sharing them. +If the request identifies a path pseudonym from an existing report, use [File follow-up](#file-follow-up) to retain that report's pseudonyms. + ### A problem which crosses a restart Use `Write logs into the file` under **Hatch** only when the in-memory report cannot cover the restart. Persistent logging affects performance and can record private information. Disable it after reproducing the problem, review the log before sharing it, and remove the log file when it is no longer needed. @@ -62,8 +74,89 @@ Use `Write logs into the file` under **Hatch** only when the in-memory report ca Include network evidence only when the ordinary LiveSync log cannot show the rejected response. Follow [Inspect a network failure](troubleshooting.md#inspect-a-network-failure), and remove request paths, remote addresses, authority and authorisation values, cookies, credentials, payload identifiers, and response secrets before sharing screenshots or copied data. +## Requested diagnostic reports + +Use the device and Vault named in the request. Label each result with a device role such as 'Device A: edits originate here', and include when it was collected. If an action below is unavailable, report the installed plug-in version and the missing action. + +These reporting tools inspect existing state without repairing or resetting it. Preserve the original symptom as described above. + +### Integrity report + +1. Open Self-hosted LiveSync settings → **Hatch** → **Reporting Tools** → **Inspect database integrity**, then select **Inspect**. +2. Keep **Pseudonyms (recommended)** selected. +3. Leave **Include remote database** selected unless the request specifies a local-only inspection. For a local-only inspection, clear it. This controls diagnostic requests for this run; it does not change normal synchronisation settings. +4. Select **Begin inspection**. If you cancel, retain the resulting report and its cancellation status. +5. Choose **MARKDOWN**, then **Copy report** or **Save report**. Share the complete output after reviewing it. Use **JSON** instead if the maintainer requests it. + +Keep the status, versions, settings, scope, coverage, counts, findings, and errors together. A partial or cancelled result is useful evidence. If the report is too large for a comment, share the saved report or a reviewed Gist, and identify its device role in the issue. + +The remote inspection currently supports CouchDB. An excluded or unavailable remote remains visible in the report; changing remote settings is not part of this procedure. Content checks cover suspect groups, so `not-checked` is meaningful. A completed inspection does not establish that every file is healthy. See [Read the compact Markdown integrity report](troubleshooting.md#read-the-compact-markdown-integrity-report) for the state notation and Review index. + +### Usage report + +1. Open Self-hosted LiveSync settings → **Hatch** → **Reporting Tools** → **Analyse database usage**, then select **Analyse**. +2. Keep **Pseudonyms (recommended)** selected, then select **Begin inspection**. +3. For a complete result, choose **TSV**, unless the maintainer requests **JSON**. Partial or cancelled results use **JSON** to retain their status and reasons. +4. Select **Copy report** or **Save report**, review the output, and share it with the device role and collection time. + +This analysis reads the local database. Its revision-reading total can grow during discovery, followed by a separate accounting stage. Reported sizes describe stored string units and do not measure the physical database file. Cancelling during accounting omits unfinished statistics. + +### File follow-up + +Use this procedure when a maintainer identifies a path token from an Integrity or Usage report. The complete token has the form `$`. Copy the token itself, including the salt and `$`, without surrounding Markdown backticks. An observation reference such as `local:12` or `remote:12`, a document pseudonym, or a revision pseudonym is not a path token. + +1. On the requested device, open Self-hosted LiveSync settings → **Hatch** → **Reporting Tools** → **Find a file from a report pseudonym**, then select **Find file**. +2. Paste the complete token into **Report path pseudonym**, then select **Find file** in the dialogue. +3. For a matching candidate, select **Prepare follow-up report**. +4. Choose **MARKDOWN** or **JSON**, then **Copy report** or **Save report**. Review the generated output before sharing it, and identify the original report, token, and device role. +5. If several candidates match the path, use **Back to matches** and prepare a separate report for each, unless the maintainer has specified a particular candidate. Keep the reports separate so their document IDs can be compared. + +Use the token from the original request throughout. The follow-up reuses its salt, which keeps pseudonyms comparable with the original report. A new database-wide report generates a new salt, so its pseudonyms will differ. + +The candidate list displays real paths and document IDs locally. Share the generated pseudonymised report rather than a screenshot of that list. A Metadata candidate can still be inspected when its current Vault file is missing. A Vault-only candidate reports that local Metadata was not inspected. + +If there are no matches, report that outcome and the device role. The lookup searches current Vault paths and current local normal-file Metadata; it does not search the remote or older history. A rename, deletion, or remote-only path can therefore have no local match. An older ordinal pseudonym cannot be used for this lookup. + +The follow-up report inspects the selected local document, current conflicts, and selected merge bases, and lists available revision-history status. It omits file contents and credentials, does not query the remote, and does not scan every historical file body. Pseudonyms use a public salt, so someone with the report can test guessed paths against a token. + ## Sharing the report Paste reports into the matching collapsible sections in the issue template, or provide a link to an already-redacted Gist. A separate plug-in log is normally unnecessary because the full LiveSync report already contains the recent verbose log history. +Integrity and file follow-up Markdown already include a collapsible section and a code block. Paste that output directly into an issue comment, outside any existing code block. Keep each device's report separate, and preserve the original token when adding a follow-up. + If a maintainer asks for a more specialised diagnostic, collect only that additional evidence and review it again before publishing it. + +## Requesting diagnostics + +For maintainers: choose the procedure which answers the current question, specify the device role, and replace the placeholders before posting a request. The examples use direct links to the relevant steps. For Integrity, choose one remote setting explicitly; inclusion is the default. If the requested action requires a newer build, identify the version to use. + +### Request an integrity report + +```markdown +Would you be willing to collect an [Integrity report](https://github.com/vrtmrz/obsidian-livesync/blob/main/docs/to_issue_reporting.md#integrity-report) on **[DEVICE ROLE]**, using **Pseudonyms (recommended)** with **Include remote database [selected / cleared]**? + +Please share the complete Markdown output after reviewing it, including its status, scope, coverage, and errors. A partial or cancelled result is useful too. Include the device role and collection time. +``` + +### Request a usage report + +```markdown +Would you be willing to collect a [Usage report](https://github.com/vrtmrz/obsidian-livesync/blob/main/docs/to_issue_reporting.md#usage-report) on **[DEVICE ROLE]**, using **Pseudonyms (recommended)**? + +Please share the reviewed TSV output for a complete result, or JSON for a partial or cancelled result, with the device role and collection time. +``` + +### Request a file follow-up + +Include the original report link or label, its observation reference where available, and the complete path token. Keep the revision generation and document pseudonym available when discussing a particular branch. A reference such as `remote:12` locates an observation within that report; the path token is what the user enters into the lookup. + +```markdown +Would you be willing to follow the [File follow-up steps](https://github.com/vrtmrz/obsidian-livesync/blob/main/docs/to_issue_reporting.md#file-follow-up) on **[DEVICE ROLE]** for this entry? + +- Original report: [REPORT LINK OR LABEL] +- Observation: `[REFERENCE, IF AVAILABLE]` +- Path token: `[COMPLETE SALT$HASH TOKEN]` + +Please share the generated pseudonymised Markdown or JSON report. If several candidates match, share a separate report for each. If none match, tell us that outcome. Include the device role and collection time; the generated reports are sufficient without sharing real paths or a screenshot of the candidate list. +``` diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index e8649667..c68fe8fc 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -163,9 +163,37 @@ Enable Obsidian's `Detect all file extensions`, then check LiveSync selectors, i ## Collect a report -For a database-wide inspection, use **Hatch** → **Inspect database integrity**. Choose **Pseudonyms (recommended)** or **Original names** before beginning. Both choices are visible as radio buttons, with pseudonyms selected initially. The inspection reads normal-file Metadata and current conflict branches locally and in CouchDB, then compares bytes within suspect groups and against the current Vault files. It reports unavailable sources, unreadable content, and detected changes during the run. Older revision history and other remote types are outside this inspection. A completed run means the stated inspection finished; it does not mean every entry is healthy or safe to delete. +For a database-wide inspection, use **Hatch** → **Inspect database integrity**. Choose **Pseudonyms (recommended)** or **Original names** before beginning. Both choices are visible as radio buttons, with pseudonyms selected initially. **Include remote database** is selected by default; clear it to inspect the local database and current Vault files without making diagnostic requests to the remote. This choice applies to one run, and the report marks the remote as excluded with coverage not checked. When included, the inspection reads normal-file Metadata and current conflict branches locally and in CouchDB, then compares bytes within suspect groups and against the current Vault files. It reports unavailable sources, unreadable content, and detected changes during the run. Older revision history and other remote types are outside this inspection. A completed run means the stated inspection finished; it does not mean every entry is healthy or safe to delete. -The result dialogue provides Markdown and JSON reports with the same name policy. Available formats are displayed as radio buttons. Pseudonyms preserve exact path equality across sources while keeping different document IDs distinct. They are temporary and are not saved as a global setting. Counts and relationships remain visible. Both modes omit file contents and credentials; review the report before sharing it. Cancelling keeps the evidence collected so far and marks the result as cancelled. +The result dialogue provides Markdown and JSON reports with the same name policy. Available formats are displayed as radio buttons. Pseudonyms preserve exact path equality across sources while keeping different document IDs distinct. Each report uses a random eight-character public salt; path tokens are derived from the exact stored Vault-relative path without lower-casing or Unicode NFC normalisation. The same salt reproduces path, document, revision, and Chunk pseudonyms for a targeted follow-up. Because the salt is public, a guessed path can be checked against its token. Earlier ordinal pseudonyms cannot be looked up this way. Pseudonyms are temporary and are not saved as a global setting. Counts and relationships remain visible. Both modes omit file contents and credentials; review the report before sharing it. Cancelling keeps the evidence collected so far and marks the result as cancelled. + +Integrity observations include recorded Metadata type, size, and total, unique, and embedded Chunk-reference counts. When suspect content has already been read, they also include measured decoded and Vault sizes, their differences from recorded or reconstructed sizes, and the Vault-to-Metadata modification-time difference. Case-only and NFC-equivalent path groups are listed once. The inspection uses content already needed for its comparisons and does not add database, HTTP, or Vault reads. Values which were not inspected remain absent or `not-checked`. + +### Read the compact Markdown integrity report + +Markdown keeps its status, settings, scope, coverage, and count summary, and adds separate counts for content and Vault checks which were not performed. **State counts** count observations, not unique files; each local or remote observation contributes to its state total. The **Review index** lists references to observations which match review conditions, before **Findings**. Those conditions include identity mismatches, current conflicts, case- or NFC-equivalent paths, non-NFC paths, a missing group in the other inspected database, missing or unreadable content or Vault files, unavailable Chunks, different or unavailable comparisons, non-zero measured differences, and absent Metadata. These are review candidates, not proof of corruption. A report with no review-index entry still does not confirm that unchecked content is healthy. JSON data and its schema are unchanged. + +Each observation is shown in a compact form, while retaining every observation and its full paths, IDs, revisions, and references for follow-up: + +```text +### `remote:4410` source=remote +path: `$` +doc: `` rev=`` expected=self +state: consistent-winner-logicallyLive-NFC-newnote conflicts=0 +paths: case=none nfcPeers=none | otherDB=present +data: recorded=47584B chunks=1/1/0 read=0 +check: content=not-checked vault=not-checked@`vault:remote:4410` +``` + +The `state` values always appear in this order: identity, current branch, logical deletion, exact stored path normalisation, and Metadata type. Identity is `consistent`, `documentIdMismatch`, or `namespaceMismatch`; JSON retains `document-id-mismatch` and `namespace-mismatch`. `winner` is the selected current branch and can coexist with conflicts, shown by `conflicts`. `logicallyLive` and `logicallyDeleted` describe the deletion marker. `NFC` means the stored path is NFC-normalised; `nonNFC` means it is not. The Metadata type is `plain`, `newnote`, `notes`, or `unknown`; `notRecorded` means Metadata is absent. A `consistent` identity says only that the ID matches its expected form; it does not show that content is readable or equal across sources. + +In `paths`, `case` and `nfcPeers` describe case-equivalent and NFC-equivalent paths: `none` means no equivalent path was found, `yes` means one was found, and `yes@` followed by a reference identifies the available group. `otherDB` says whether the same identity and path group occurs in the other inspected source. It does not compare content. If that source was excluded or unavailable, the field is `not-checked`. + +In `data`, `chunks` is total/unique/embedded Chunk references, and `read` counts Chunk reads, not HTTP requests. A `?` means a value was not recorded; it is distinct from zero, missing, unreadable, and `not-checked`. If Metadata is absent, its recorded size and Chunk counts are `?`. In `doc`, `expected=self` means the actual document ID exactly matches the expected ID. Otherwise, the report shows the expected ID or `?` when it was not recorded. + +The `check` statuses distinguish `available`, `missing`, `unreadable`, and `not-checked`; a recorded reason follows the status in parentheses. An optional `measure` line includes only available values: decoded content size and Vault size in bytes, `deltaContent` (decoded content minus recorded Metadata size), `deltaVault` (Vault minus decoded content), and `deltaMtime` (Vault minus Metadata modification time) in milliseconds. Positive differences have a `+` prefix, negative differences retain `-`, and zero is `0`. These measurements describe the evidence collected and do not select an authoritative version. Unavailable Chunk names, comparison results, path-equivalence groups, and errors retain their reasons and references. All observations remain in the report's collapsible section and fenced copy. + +Usage progress first follows available-revision discovery, so its `n/m` total can grow and remains provisional until discovery ends. It then shows separate progress for accounting. Cancellation during revision reading keeps the collected evidence; cancellation during accounting omits incomplete statistics. For storage accounting, use **Hatch** → **Analyse database usage** → **Analyse**. It offers the same name choice and reads existing local revisions without changing on-demand Chunk fetching. Complete runs retain the existing TSV columns and first-reference unique/shared Chunk accounting. Sizes count stored string units, not physical database size. Incomplete runs provide JSON so their status and reasons remain attached to the rows. **Save report** requests a download; **Copy report** remains available on devices which do not support downloads. @@ -175,6 +203,10 @@ Run `Generate full report for opening the issue with debug info` to copy the cur When a problem concerns one file, run **Copy database information for the active file**, or use **Hatch** → **Copy database information for a file** to select another file. The report describes this device's local database view, including the Vault-relative path, document and chunk identifiers, local database revisions, conflicts, and local chunk availability. It does not query the remote server or include file contents. Treat paths and identifiers as private metadata before sharing. +To follow up on a path pseudonym from a report, use **Hatch** → **Find a file from a report pseudonym**, paste the token, and select **Find file**. The lookup checks the current Vault and local normal-file Metadata only; it does not contact the remote database. It lists every local candidate, with real paths and document IDs visible only in the local dialogue. Multiple IDs for the same path remain distinct. A Vault-only candidate means local Metadata was not inspected. No match may mean the file was renamed or deleted, or that the matching Metadata exists only remotely. + +For a selected candidate, the tool can open an existing Vault file or prepare a shareable local report using the original salt and selected document ID. That report shows current, conflict, and history status, Chunk availability, merge bases, and Vault size facts, without file contents or credentials. Its history status is a listing; it does not inspect every historical body. + Use `Show log` for live inspection. Logs are intentionally kept in memory for a limited time to reduce accidental disclosure. Enable `Write logs into the file` only while reproducing a problem, then disable it and remove the file after review because persistent logging affects performance and may contain private data. ![Write logs into the file](../images/write_logs_into_the_file.png) diff --git a/src/common/messages/LiveSyncProvisionalMessages.ts b/src/common/messages/LiveSyncProvisionalMessages.ts index 073cbac7..34ae100b 100644 --- a/src/common/messages/LiveSyncProvisionalMessages.ts +++ b/src/common/messages/LiveSyncProvisionalMessages.ts @@ -7,6 +7,43 @@ * remove it from this map in the same change. */ export const liveSyncProvisionalEnglishMessages = { + "Preparing follow-up report…": "Preparing follow-up report…", + "Cancelling local search. Checked ${COUNT} entries.": "Cancelling local search. Checked ${COUNT} entries.", + "Search current Vault paths and current local Metadata only. Older history and remote-only files are not searched.": + "Search current Vault paths and current local Metadata only. Older history and remote-only files are not searched.", + "The lookup does not read file contents or contact a remote database. Follow-up reports include current conflicts and merge bases.": + "The lookup does not read file contents or contact a remote database. Follow-up reports include current conflicts and merge bases.", + "Local file diagnostic report": "Local file diagnostic report", + "Checked ${COUNT} local entries.": "Checked ${COUNT} local entries.", + "Enter a valid report path pseudonym.": "Enter a valid report path pseudonym.", + "Search cancelled. Matches found so far are shown.": "Search cancelled. Matches found so far are shown.", + "No matches were found in current Vault paths or current local Metadata. Older history and remote-only files are not searched.": + "No matches were found in current Vault paths or current local Metadata. Older history and remote-only files are not searched.", + "Found ${COUNT} local matches.": "Found ${COUNT} local matches.", + "The local search could not finish.": "The local search could not finish.", + "The file could not be opened.": "The file could not be opened.", + "The follow-up report could not be prepared.": "The follow-up report could not be prepared.", + "Report path pseudonym": "Report path pseudonym", + Source: "Source", + Metadata: "Metadata", + Vault: "Vault", + "Metadata document ID": "Metadata document ID", + "No current Metadata document": "No current Metadata document", + "Current Vault file": "Current Vault file", + Present: "Present", + Missing: "Missing", + "Open file": "Open file", + "Prepare follow-up report": "Prepare follow-up report", + "This report uses pseudonyms. Review its coverage before sharing.": + "This report uses pseudonyms. Review its coverage before sharing.", + "Back to matches": "Back to matches", + "Reading revisions": "Reading revisions", + "Calculating database usage": "Calculating database usage", + "Discovering revisions": "Discovering revisions", + "Find a file from a report pseudonym": "Find a file from a report pseudonym", + "Find matching paths in this Vault and local Metadata, then prepare a follow-up report using the same pseudonyms.": + "Find matching paths in this Vault and local Metadata, then prepare a follow-up report using the same pseudonyms.", + "Find file": "Find file", "⚠️ Initialise or overwrite the remote": "⚠️ Initialise or overwrite the remote", "🔗 Join this device": "🔗 Join this device", "⚙️ Apply settings only (advanced)": "⚙️ Apply settings only (advanced)", @@ -268,12 +305,13 @@ export const liveSyncProvisionalEnglishMessages = { "Copy failed. Select the report text and copy it manually.", "Download requested. You can also copy the report if downloads are unavailable on this device.": "Download requested. You can also copy the report if downloads are unavailable on this device.", - "Choose how names appear in this run. Pseudonyms replace paths and identifiers consistently. Original names include private filenames. File contents and credentials are omitted in either mode.": - "Choose how names appear in this run. Pseudonyms replace paths and identifiers consistently. Original names include private filenames. File contents and credentials are omitted in either mode.", + "Pseudonyms include a public salt, so guessed paths can be checked. Original names include private filenames. Reports omit file contents and credentials.": + "Pseudonyms include a public salt, so guessed paths can be checked. Original names include private filenames. Reports omit file contents and credentials.", "Read available local revisions and report their Chunk references. Settings are left unchanged.": "Read available local revisions and report their Chunk references. Settings are left unchanged.", - "Read normal-file Metadata, current conflicts, and content for suspect groups from this device and CouchDB. Other remote types and older revision history are outside this inspection. Results do not authorise deletion.": - "Read normal-file Metadata, current conflicts, and content for suspect groups from this device and CouchDB. Other remote types and older revision history are outside this inspection. Results do not authorise deletion.", + "Inspect normal-file Metadata, current conflicts, and suspect content against the Vault. Remote inspection supports CouchDB. Older history is excluded; findings do not authorise deletion.": + "Inspect normal-file Metadata, current conflicts, and suspect content against the Vault. Remote inspection supports CouchDB. Older history is excluded; findings do not authorise deletion.", + "Include remote database": "Include remote database", "Names in this report": "Names in this report", "Pseudonyms (recommended)": "Pseudonyms (recommended)", "Original names": "Original names", diff --git a/src/modules/features/SettingDialogue/PaneHatch.ts b/src/modules/features/SettingDialogue/PaneHatch.ts index 711556ff..48f6e4de 100644 --- a/src/modules/features/SettingDialogue/PaneHatch.ts +++ b/src/modules/features/SettingDialogue/PaneHatch.ts @@ -10,7 +10,7 @@ import { import { createBlob, escapeMarkdownValue, readAsBlob } from "@vrtmrz/livesync-commonlib/compat/common/utils"; import { Logger } from "@vrtmrz/livesync-commonlib/compat/common/logger"; import { shouldBeIgnored } from "@vrtmrz/livesync-commonlib/compat/string_and_binary/path"; -import { Menu, diff_match_patch, setIcon } from "@/deps.ts"; +import { Menu, TFile, diff_match_patch, setIcon } from "@/deps.ts"; import { $msg } from "@/common/translation"; import { Semaphore } from "octagonal-wheels/concurrency/semaphore"; import { LiveSyncSetting as Setting } from "./LiveSyncSetting.ts"; @@ -62,10 +62,70 @@ import { executeMetadataIdentityRepair, MetadataIdentityRepairExecutions, } from "@/serviceFeatures/metadataIdentityRepair.ts"; +import { showDiagnosticFileLookup } from "@/serviceFeatures/diagnosticFileLookup.ts"; export function paneHatch(this: ObsidianLiveSyncSettingTab, paneEl: HTMLElement, { addPanel }: PageFunctions): void { // const hatchWarn = this.createEl(paneEl, "div", { text: `To stop the boot up sequence for fixing problems on databases, you can put redflag.md on top of your vault (Rebooting obsidian is required).` }); // hatchWarn.addClass("op-warn-info"); - void addPanel(paneEl, $msg("Setting.TroubleShooting")).then((paneEl) => { + void addPanel(paneEl, "Reporting Tools").then((paneEl) => { + new Setting(paneEl).setName($msg("Prepare the 'report' to create an issue")).addButton((button) => + button + .setButtonText($msg("Copy Report to clipboard")) + .setCta() + .setDisabled(false) + .onClick(async () => { + await this.app.commands.executeCommandById("obsidian-livesync:dump-debug-info"); + }) + ); + new Setting(paneEl) + .setName($msg("Copy database information for a file")) + .setDesc( + $msg( + "Copy revision, conflict, and local chunk availability information, including document and chunk identifiers but not file contents." + ) + ) + .addButton((button) => + button.setButtonText($msg("Choose file")).onClick(async () => { + await chooseAndCopyFileDatabaseInfo(this.core); + }) + ); + new Setting(paneEl) + .setName($msg("Find a file from a report pseudonym")) + .setDesc( + $msg( + "Find matching paths in this Vault and local Metadata, then prepare a follow-up report using the same pseudonyms." + ) + ) + .addButton((button) => + button.setButtonText($msg("Find file")).onClick(async () => { + await showDiagnosticFileLookup(this.core, async (path) => { + const file = this.app.vault.getAbstractFileByPath(path); + if (file instanceof TFile) await this.app.workspace.getLeaf(false).openFile(file); + }); + }) + ); + new Setting(paneEl) + .setName($msg("Inspect database integrity")) + .setDesc( + $msg("Read normal-file Metadata and current conflicts locally and in CouchDB, then compare content for suspect groups. Choose original names or pseudonyms before starting.") + ) + .addButton((button) => + button.setButtonText($msg("Inspect")).onClick(() => eventHub.emitEvent(EVENT_INSPECT_DB_INTEGRITY)) + ); + new Setting(paneEl) + .setName($msg("Analyse database usage")) + .setDesc( + $msg( + "Analyse database usage and generate a TSV report for diagnosis yourself. You can paste the generated report with any spreadsheet you like." + ) + ) + .addButton((button) => + button.setButtonText($msg("Analyse")).onClick(() => { + eventHub.emitEvent(EVENT_ANALYSE_DB_USAGE); + }) + ); + }); + + void addPanel(paneEl, "Checks and Recovery").then((paneEl) => { new Setting(paneEl) .setName($msg("Setting.TroubleShooting.Doctor")) .setDesc($msg("Setting.TroubleShooting.Doctor.Desc")) @@ -92,48 +152,9 @@ export function paneHatch(this: ObsidianLiveSyncSettingTab, paneEl: HTMLElement, eventHub.emitEvent(EVENT_REQUEST_RUN_FIX_INCOMPLETE); }) ); + }); - new Setting(paneEl).setName($msg("Prepare the 'report' to create an issue")).addButton((button) => - button - .setButtonText($msg("Copy Report to clipboard")) - .setCta() - .setDisabled(false) - .onClick(async () => { - await this.app.commands.executeCommandById("obsidian-livesync:dump-debug-info"); - }) - ); - new Setting(paneEl) - .setName($msg("Copy database information for a file")) - .setDesc( - $msg( - "Copy revision, conflict, and local chunk availability information, including document and chunk identifiers but not file contents." - ) - ) - .addButton((button) => - button.setButtonText($msg("Choose file")).onClick(async () => { - await chooseAndCopyFileDatabaseInfo(this.core); - }) - ); - new Setting(paneEl) - .setName($msg("Inspect database integrity")) - .setDesc( - $msg("Read normal-file Metadata and current conflicts locally and in CouchDB, then compare content for suspect groups. Choose original names or pseudonyms before starting.") - ) - .addButton((button) => - button.setButtonText($msg("Inspect")).onClick(() => eventHub.emitEvent(EVENT_INSPECT_DB_INTEGRITY)) - ); - new Setting(paneEl) - .setName($msg("Analyse database usage")) - .setDesc( - $msg( - "Analyse database usage and generate a TSV report for diagnosis yourself. You can paste the generated report with any spreadsheet you like." - ) - ) - .addButton((button) => - button.setButtonText($msg("Analyse")).onClick(() => { - eventHub.emitEvent(EVENT_ANALYSE_DB_USAGE); - }) - ); + void addPanel(paneEl, $msg("Setting.TroubleShooting")).then((paneEl) => { new Setting(paneEl) .setName($msg("Reset notification threshold and check the remote database usage")) .setDesc($msg("Reset the remote storage size threshold and check the remote storage size again.")) diff --git a/src/modules/services/LiveSyncUI/dialogues/DatabaseDiagnosticDialog.svelte b/src/modules/services/LiveSyncUI/dialogues/DatabaseDiagnosticDialog.svelte index b7b56b5f..46ed9056 100644 --- a/src/modules/services/LiveSyncUI/dialogues/DatabaseDiagnosticDialog.svelte +++ b/src/modules/services/LiveSyncUI/dialogues/DatabaseDiagnosticDialog.svelte @@ -1,21 +1,31 @@ -
+
{#if phase === "choice"} -

{translate("Choose how names appear in this run. Pseudonyms replace paths and identifiers consistently. Original names include private filenames. File contents and credentials are omitted in either mode.")}

+

{translate("Pseudonyms include a public salt, so guessed paths can be checked. Original names include private filenames. Reports omit file contents and credentials.")}

{options?.kind === "usage" ? translate("Read available local revisions and report their Chunk references. Settings are left unchanged.") - : translate("Read normal-file Metadata, current conflicts, and content for suspect groups from this device and CouchDB. Other remote types and older revision history are outside this inspection. Results do not authorise deletion.")} + : translate("Inspect normal-file Metadata, current conflicts, and suspect content against the Vault. Remote inspection supports CouchDB. Older history is excluded; findings do not authorise deletion.")}

{translate("Names in this report")} @@ -103,14 +189,21 @@
+ {#if options?.kind === "integrity"} + + {/if}
{:else if phase === "running"} -

- {cancelling ? translate("Cancelling inspection…") : `${translate("Items inspected")}: ${processed}`} -

+
+ {#if cancelling} +

{translate("Cancelling inspection…")}

+ {/if} @@ -149,16 +242,48 @@
diff --git a/src/modules/services/LiveSyncUI/dialogues/DiagnosticFileLookupDialog.svelte b/src/modules/services/LiveSyncUI/dialogues/DiagnosticFileLookupDialog.svelte new file mode 100644 index 00000000..75cab5cd --- /dev/null +++ b/src/modules/services/LiveSyncUI/dialogues/DiagnosticFileLookupDialog.svelte @@ -0,0 +1,367 @@ + + + +
+ {#if phase !== "report"} + {#if phase !== "results"} +

+ {translate("Search current Vault paths and current local Metadata only. Older history and remote-only files are not searched.")} +

+

+ {translate( + "The lookup does not read file contents or contact a remote database. Follow-up reports include current conflicts and merge bases." + )} +

+ {/if} + +
+ + +
+ + {#if phase === "running"} + + {/if} + +
+
+ {:else if completedToken} +

{translate("Report path pseudonym")}: {completedToken}

+ {/if} + + {#if phase === "input" && actionStatus} +

{actionStatus}

+ {/if} + + {#if phase === "running"} +

{lookupStatus}

+ {:else if phase === "input" && lookupResult?.valid === false} +

{lookupStatus}

+ {:else if phase === "results" && lookupResult?.valid} +

{lookupStatus}

+ {#if lookupResult.candidates.length > 0} +
    + {#each lookupResult.candidates as candidate} +
  • +
    + {candidate.path} +
    +
    {translate("Source")}
    +
    {candidate.source === "metadata" ? translate("Metadata") : translate("Vault")}
    +
    {translate("Metadata document ID")}
    +
    {candidate.documentId ?? translate("No current Metadata document")}
    +
    {translate("Current Vault file")}
    +
    {candidate.storageExists ? translate("Present") : translate("Missing")}
    +
    +
    +
    + + +
    +
    • + {/each} +
+ {/if} +

{preparingReport ? translate("Preparing follow-up report…") : actionStatus}

+ {:else if phase === "report" && output} +

{translate("This report uses pseudonyms. Review its coverage before sharing.")}

+
+ {translate("Report format")} +
+ {#each output.artifacts as artifact} + + {/each} +
+
+ +

{actionStatus}

+
+ + + + +
+ {/if} +
+ + diff --git a/src/serviceFeatures/databaseDiagnosticDialogTypes.ts b/src/serviceFeatures/databaseDiagnosticDialogTypes.ts index ea142bd6..c7686214 100644 --- a/src/serviceFeatures/databaseDiagnosticDialogTypes.ts +++ b/src/serviceFeatures/databaseDiagnosticDialogTypes.ts @@ -1,9 +1,17 @@ import type { DiagnosticNameMode } from "@vrtmrz/livesync-commonlib/diagnostics"; -import type { DatabaseDiagnosticKind, DatabaseDiagnosticOutput } from "./databaseDiagnostics"; +import type { + DatabaseDiagnosticKind, + DatabaseDiagnosticOutput, + DatabaseDiagnosticProgress, +} from "./databaseDiagnostics"; /** The dialogue can run or cancel an inspection without receiving the service container. */ export interface DatabaseDiagnosticDialogOptions { kind: DatabaseDiagnosticKind; - run(nameMode: DiagnosticNameMode, onProgress: (processed: number) => void): Promise; + run( + nameMode: DiagnosticNameMode, + onProgress: (progress: DatabaseDiagnosticProgress) => void, + includeRemote?: boolean + ): Promise; cancel(): void; } diff --git a/src/serviceFeatures/databaseDiagnostics.ts b/src/serviceFeatures/databaseDiagnostics.ts index 779a8aef..7519746b 100644 --- a/src/serviceFeatures/databaseDiagnostics.ts +++ b/src/serviceFeatures/databaseDiagnostics.ts @@ -17,6 +17,7 @@ import { REMOTE_COUCHDB, type ObsidianLiveSyncSettings } from "@vrtmrz/livesync- import type { StorageAccess } from "@vrtmrz/livesync-commonlib/compat/interfaces/StorageAccess"; import type { NecessaryServicesInterfaces } from "@vrtmrz/livesync-commonlib/compat/interfaces/ServiceModule"; import commonlibPackage from "@vrtmrz/livesync-commonlib/package.json"; +import { LOG_LEVEL_INFO, Logger } from "octagonal-wheels/common/logger"; export type DatabaseDiagnosticKind = "usage" | "integrity"; export type DatabaseDiagnosticHost = NecessaryServicesInterfaces< @@ -36,6 +37,15 @@ export interface DatabaseDiagnosticOutput { artifacts: DatabaseDiagnosticArtifact[]; } +export interface DatabaseDiagnosticProgress { + phase: "usage" | "accounting" | "metadata" | "content"; + source?: "local" | "remote"; + processed: number; + total?: number; + totalIsFinal?: boolean; + elapsedMs: number; +} + export async function readDiagnosticStorage( storage: Pick, path: string @@ -59,7 +69,7 @@ export async function readDiagnosticStorage( ) { return { status: "unreadable", reason: "storage-changed" }; } - return { status: "available", bytes }; + return { status: "available", bytes, mtime: after.mtime }; } catch { return { status: "unreadable", reason: "content-unreadable" }; } @@ -70,8 +80,13 @@ export async function runDatabaseDiagnostic( host: DatabaseDiagnosticHost, kind: DatabaseDiagnosticKind, nameMode: DiagnosticNameMode, - options: { signal?: AbortSignal; onProgress?(processed: number): void } = {} + options: { + signal?: AbortSignal; + onProgress?(progress: DatabaseDiagnosticProgress): void; + includeRemote?: boolean; + } = {} ): Promise { + const started = performance.now(); const configuration = JSON.stringify(host.services.setting.currentSettings()); const settings = JSON.parse(configuration) as ObsidianLiveSyncSettings; const database = host.services.database.localDatabase.localDatabase; @@ -79,6 +94,12 @@ export async function runDatabaseDiagnostic( JSON.stringify(host.services.setting.currentSettings()) === configuration && host.services.database.localDatabase.localDatabase === database; const mapper = createDiagnosticNameMapper(nameMode); + const httpRequestCounters = { httpRequests: 0, httpFailedRequests: 0, httpRequestDurationMs: 0 }; + const getHttpRequestCounters = () => ({ + httpRequests: httpRequestCounters.httpRequests, + httpFailedRequests: httpRequestCounters.httpFailedRequests, + httpRequestDurationMs: Math.round(httpRequestCounters.httpRequestDurationMs), + }); const versions = { commonlib: commonlibPackage.version, plugin: host.services.API.getPluginVersion(), @@ -89,7 +110,14 @@ export async function runDatabaseDiagnostic( const result = await analyseDatabaseUsage({ source: local, signal: options.signal, - onProgress: (progress) => options.onProgress?.(progress.revisionsProcessed), + onProgress: (progress) => + options.onProgress?.({ + phase: progress.phase === "accounting" ? "accounting" : "usage", + processed: progress.processed, + total: progress.total, + totalIsFinal: progress.totalIsFinal, + elapsedMs: Math.round(performance.now() - started), + }), }); if (!isConfigurationCurrent()) { result.errors.push({ code: "configuration_changed" }); @@ -110,23 +138,41 @@ export async function runDatabaseDiagnostic( } return { status: result.status, nameMode, artifacts }; } - const result = await analyseDatabaseIntegrity({ + const includeRemote = options.includeRemote !== false; + const integrityOptions: Parameters[0] = { local, remote: - settings.remoteType === REMOTE_COUCHDB + includeRemote && settings.remoteType === REMOTE_COUCHDB ? () => Promise.resolve( createCouchDBDiagnosticSource({ settings, signal: options.signal, - fetch: (request, requestOptions) => - settings.useRequestAPI - ? host.services.API.nativeFetch(request, requestOptions) - : host.services.API.webCompatFetch(request, requestOptions), + fetch: async (request, requestOptions) => { + const requestStarted = performance.now(); + httpRequestCounters.httpRequests++; + try { + const response = settings.useRequestAPI + ? await host.services.API.nativeFetch(request, requestOptions) + : await host.services.API.webCompatFetch(request, requestOptions); + if (!response.ok) httpRequestCounters.httpFailedRequests++; + return response; + } catch (error) { + httpRequestCounters.httpFailedRequests++; + throw error; + } finally { + httpRequestCounters.httpRequestDurationMs += performance.now() - requestStarted; + } + }, }) ) : undefined, - remoteUnavailableReason: settings.remoteType === REMOTE_COUCHDB ? "source-unavailable" : "unsupported-remote", + remoteUnavailableReason: includeRemote + ? settings.remoteType === REMOTE_COUCHDB + ? "source-unavailable" + : "unsupported-remote" + : undefined, + includeRemote, path: { getPath: (entry) => host.services.path.getPath(entry), path2id: (path, prefix) => host.services.path.path2idWithSettings(path, settings, prefix), @@ -141,10 +187,17 @@ export async function runDatabaseDiagnostic( readStorage: (path) => readDiagnosticStorage(host.serviceModules.storageAccess, path), isConfigurationCurrent, signal: options.signal, - onProgress: (progress) => options.onProgress?.(progress.processed), - }); + onProgress: (progress) => options.onProgress?.(progress), + onLog: (event) => + Logger( + `Database integrity inspection: ${JSON.stringify({ ...event, ...getHttpRequestCounters() })}`, + LOG_LEVEL_INFO + ), + }; + const result = await analyseDatabaseIntegrity(integrityOptions); + const formattingStarted = performance.now(); const report = projectIntegrityReport(result, mapper, versions); - return { + const output: DatabaseDiagnosticOutput = { status: result.status, nameMode, artifacts: [ @@ -160,4 +213,13 @@ export async function runDatabaseDiagnostic( }, ], }; + const requestCounters = getHttpRequestCounters(); + Logger( + `Database integrity report: status=${result.status}, observations=${result.observations.length}, ` + + `httpRequests=${requestCounters.httpRequests}, httpFailedRequests=${requestCounters.httpFailedRequests}, ` + + `httpRequestDurationMs=${requestCounters.httpRequestDurationMs}, ` + + `formattingMs=${Math.round(performance.now() - formattingStarted)}, elapsedMs=${Math.round(performance.now() - started)}`, + LOG_LEVEL_INFO + ); + return output; } diff --git a/src/serviceFeatures/databaseDiagnostics.unit.spec.ts b/src/serviceFeatures/databaseDiagnostics.unit.spec.ts index b572832d..3d17ea88 100644 --- a/src/serviceFeatures/databaseDiagnostics.unit.spec.ts +++ b/src/serviceFeatures/databaseDiagnostics.unit.spec.ts @@ -1,12 +1,31 @@ import { describe, expect, it, vi } from "vitest"; -import { DEFAULT_SETTINGS, REMOTE_P2P } from "@vrtmrz/livesync-commonlib/compat/common/types"; +import { + DEFAULT_SETTINGS, + REMOTE_COUCHDB, + REMOTE_P2P, + type ObsidianLiveSyncSettings, +} from "@vrtmrz/livesync-commonlib/compat/common/types"; + +const diagnosticsMocks = vi.hoisted(() => ({ analyseOptions: vi.fn() })); + +vi.mock("@vrtmrz/livesync-commonlib/diagnostics", async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + analyseDatabaseIntegrity: (options: Parameters[0]) => { + diagnosticsMocks.analyseOptions(options); + return actual.analyseDatabaseIntegrity(options); + }, + }; +}); + import { readDiagnosticStorage, runDatabaseDiagnostic, type DatabaseDiagnosticHost } from "./databaseDiagnostics"; function fixture() { const path = "Private/日本語.md"; const body = "private file contents"; const bytes = new TextEncoder().encode(body); - const settings = { + const settings: ObsidianLiveSyncSettings = { ...DEFAULT_SETTINGS, remoteType: REMOTE_P2P, readChunksOnline: true, @@ -81,6 +100,41 @@ function fixture() { } describe("database diagnostic host composition", () => { + it("passes growing Usage totals and the separate accounting phase to the dialogue", async () => { + const onProgress = vi.fn(); + await runDatabaseDiagnostic(fixture().host, "usage", "pseudonym", { onProgress }); + const progress = onProgress.mock.calls.map(([value]) => value); + expect(progress).toContainEqual( + expect.objectContaining({ + phase: "usage", + processed: 0, + total: 0, + totalIsFinal: false, + }) + ); + expect(progress).toContainEqual( + expect.objectContaining({ + phase: "usage", + processed: 2, + total: 2, + totalIsFinal: true, + }) + ); + const accounting = progress.filter((value) => value.phase === "accounting"); + expect(accounting.length).toBeGreaterThan(1); + const final = accounting[accounting.length - 1]; + expect(final).toMatchObject({ processed: final.total, totalIsFinal: true }); + }); + + it("preserves phase, source, and totals for the inspection dialogue", async () => { + const onProgress = vi.fn(); + await runDatabaseDiagnostic(fixture().host, "integrity", "pseudonym", { onProgress }); + expect(onProgress).toHaveBeenCalledWith( + expect.objectContaining({ phase: "metadata", source: "local", processed: 0 }) + ); + expect(onProgress).toHaveBeenCalledWith(expect.objectContaining({ phase: "content", processed: 1, total: 1 })); + }); + it("keeps on-demand fetching enabled while analysing only existing local revisions", async () => { const data = fixture(); const before = JSON.stringify(data.settings); @@ -113,6 +167,9 @@ describe("database diagnostic host composition", () => { it("keeps local findings when the remote type is unsupported and projects both formats consistently", async () => { const data = fixture(); const result = await runDatabaseDiagnostic(data.host, "integrity", "pseudonym"); + expect(diagnosticsMocks.analyseOptions).toHaveBeenLastCalledWith( + expect.objectContaining({ includeRemote: true }) + ); expect(result.status).toBe("partial"); expect(data.fetch).not.toHaveBeenCalled(); expect(data.put).not.toHaveBeenCalled(); @@ -135,6 +192,82 @@ describe("database diagnostic host composition", () => { } }); + it("excludes the remote source for a local integrity run", async () => { + const data = fixture(); + data.settings.remoteType = REMOTE_COUCHDB; + data.settings.couchDB_DBNAME = "private-database"; + data.settings.useRequestAPI = true; + data.fetch + .mockImplementationOnce(async () => { + await new Promise((resolve) => setTimeout(resolve, 2)); + return { ok: false, status: 503 } as Response; + }) + .mockRejectedValue(new Error("simulated diagnostic request failure")); + + diagnosticsMocks.analyseOptions.mockClear(); + const result = await runDatabaseDiagnostic(data.host, "integrity", "pseudonym", { + includeRemote: false, + }); + + expect(data.fetch).not.toHaveBeenCalled(); + const passedOptions = diagnosticsMocks.analyseOptions.mock.calls[0][0]; + expect(passedOptions.includeRemote).toBe(false); + expect(passedOptions.remote).toBeUndefined(); + expect(result.artifacts.map((artifact) => artifact.format)).toEqual(["markdown", "json"]); + const report = JSON.parse(result.artifacts.find((artifact) => artifact.format === "json")!.content); + expect(report.scope.remote).toBe("excluded"); + expect(report.coverage.remote.status).toBe("not-checked"); + expect(report.errors.some((error: { source: string }) => error.source === "remote")).toBe(false); + }); + + it("logs cumulative HTTP dispatch, failure, and duration counters for remote inspection", async () => { + const log = vi.spyOn(console, "log").mockImplementation(() => {}); + + try { + const nonOk = fixture(); + nonOk.settings.remoteType = REMOTE_COUCHDB; + nonOk.settings.couchDB_DBNAME = "private-database"; + nonOk.settings.useRequestAPI = true; + nonOk.fetch.mockImplementation(async () => { + await new Promise((resolve) => setTimeout(resolve, 2)); + return { ok: false, status: 503 } as Response; + }); + await runDatabaseDiagnostic(nonOk.host, "integrity", "pseudonym"); + + const thrown = fixture(); + thrown.settings.remoteType = REMOTE_COUCHDB; + thrown.settings.couchDB_DBNAME = "private-database"; + thrown.settings.useRequestAPI = true; + thrown.fetch.mockRejectedValue(new Error("simulated diagnostic request failure")); + await runDatabaseDiagnostic(thrown.host, "integrity", "pseudonym"); + + const messages = log.mock.calls.map(([message]) => String(message)); + const inspectionLogs = messages.filter((message) => message.includes("Database integrity inspection:")); + expect(inspectionLogs.length).toBeGreaterThan(0); + const inspectionEvents = inspectionLogs.map((message) => { + const json = message.slice(message.indexOf("{")); + return JSON.parse(json); + }); + expect(inspectionEvents.some((event) => event.httpRequests > 0)).toBe(true); + expect(inspectionEvents.some((event) => event.httpFailedRequests > 0)).toBe(true); + expect(inspectionEvents.every((event) => typeof event.httpRequestDurationMs === "number")).toBe(true); + + const summaries = messages.filter((message) => message.includes("Database integrity report:")); + expect(summaries).toHaveLength(2); + for (const [index, summary] of summaries.entries()) { + const match = /httpRequests=(\d+), httpFailedRequests=(\d+), httpRequestDurationMs=(\d+)/.exec(summary); + expect(match).not.toBeNull(); + const data = index === 0 ? nonOk : thrown; + expect(Number(match![1])).toBe(data.fetch.mock.calls.length); + expect(Number(match![2])).toBe(data.fetch.mock.calls.length); + expect(Number(match![3])).toBeGreaterThanOrEqual(0); + if (index === 0) expect(Number(match![3])).toBeGreaterThan(0); + } + } finally { + log.mockRestore(); + } + }); + it("includes names only when requested and never includes credentials or file contents", async () => { const data = fixture(); const result = await runDatabaseDiagnostic(data.host, "usage", "original"); @@ -172,6 +305,17 @@ describe("database diagnostic host composition", () => { }); describe("diagnostic Vault reads", () => { + it("reuses the validated snapshot timestamp without adding stat or content reads", async () => { + const data = fixture(); + expect(await readDiagnosticStorage(data.storage, data.path)).toEqual({ + status: "available", + bytes: data.bytes, + mtime: 1, + }); + expect(data.storage.statHidden).toHaveBeenCalledTimes(2); + expect(data.storage.readHiddenFileBinary).toHaveBeenCalledTimes(1); + }); + it("distinguishes an absent file, a read error, and a file changing during its read", async () => { const data = fixture(); data.storage.statHidden.mockResolvedValueOnce(null as never); diff --git a/src/serviceFeatures/diagnosticFileLookup.ts b/src/serviceFeatures/diagnosticFileLookup.ts new file mode 100644 index 00000000..f9bdf7f3 --- /dev/null +++ b/src/serviceFeatures/diagnosticFileLookup.ts @@ -0,0 +1,346 @@ +import { $msg } from "@/common/translation"; +import { delay } from "octagonal-wheels/promises"; +import { createDiagnosticNameMapper, parseDiagnosticPathToken } from "@vrtmrz/livesync-commonlib/diagnostics"; +import { EntryTypes, type FilePath, type FilePathWithPrefix } from "@vrtmrz/livesync-commonlib/compat/common/types"; +import { ICHeader, ICXHeader, PSCHeader } from "@vrtmrz/livesync-commonlib/compat/common/models/fileaccess.const"; +import { stripAllPrefixes } from "@vrtmrz/livesync-commonlib/compat/string_and_binary/path"; +import type { ComponentHasResult } from "@/modules/services/LiveSyncUI/svelteDialog"; +import DiagnosticFileLookupDialog from "@/modules/services/LiveSyncUI/dialogues/DiagnosticFileLookupDialog.svelte"; +import type { + FileDatabaseInfo, + FileDatabaseInfoCore, + FileDatabaseMergeBaseInfo, + InspectFileDatabaseInfoOptions, + RevisionDatabaseInfo, +} from "./fileDatabaseInfo"; +import { inspectFileDatabaseInfo } from "./fileDatabaseInfo"; +import type { + DiagnosticFileCandidate, + DiagnosticFileLookupDialogOptions, + DiagnosticFileLookupResult, + FindDiagnosticFilesOptions, +} from "./diagnosticFileLookupTypes"; +import type { DatabaseDiagnosticArtifact, DatabaseDiagnosticOutput } from "./databaseDiagnostics"; + +type CurrentFileMetadata = { + _id: string; + path: FilePathWithPrefix | FilePath; + type: string; + _deleted?: boolean; +}; + +type FileLookupDialogManager = { + open( + component: ComponentHasResult, + initialData?: TInitial + ): Promise; +}; + +type FileLookupUI = FileDatabaseInfoCore["services"]["UI"] & { + dialogManager: FileLookupDialogManager; +}; + +function isCurrentFileMetadata(value: unknown): value is CurrentFileMetadata { + if (value === null || typeof value !== "object") return false; + const entry = value as Record; + if ( + typeof entry._id !== "string" || + typeof entry.path !== "string" || + (entry.type !== EntryTypes.NOTE_BINARY && entry.type !== EntryTypes.NOTE_PLAIN) || + entry._deleted === true + ) { + return false; + } + const path = entry.path; + return ![ICHeader, ICXHeader, PSCHeader].some((prefix) => path.startsWith(prefix)); +} + +function cancelledResult(processed: number, candidates: DiagnosticFileCandidate[]): DiagnosticFileLookupResult { + return { valid: true, status: "cancelled", processed, candidates }; +} + +async function yieldToEventLoop(): Promise { + await delay(0); +} + +export async function findDiagnosticFiles( + core: FileDatabaseInfoCore, + token: string, + options: FindDiagnosticFilesOptions = {} +): Promise { + const parsedToken = parseDiagnosticPathToken(token); + if (!parsedToken) { + return { valid: false, status: "invalid", processed: 0, candidates: [] }; + } + const mapper = createDiagnosticNameMapper("pseudonym", parsedToken.salt); + const candidates: DiagnosticFileCandidate[] = []; + let processed = 0; + const reportProgress = () => options.onProgress?.(processed); + reportProgress(); + if (options.signal?.aborted) { + reportProgress(); + return cancelledResult(processed, candidates); + } + const vaultPaths = new Set(await core.storageAccess.getFileNames()); + const matchingMetadataPaths = new Set(); + if (options.signal?.aborted) { + reportProgress(); + return cancelledResult(processed, candidates); + } + + for await (const entry of core.localDatabase.findAllDocs()) { + processed++; + if (isCurrentFileMetadata(entry)) { + const databasePath = entry.path; + const path = stripAllPrefixes(databasePath); + if (mapper.path(path) === token) { + matchingMetadataPaths.add(path); + candidates.push({ + path, + databasePath, + documentId: entry._id, + storageExists: vaultPaths.has(path), + source: "metadata", + }); + } + } + if (processed % 128 === 0) { + reportProgress(); + await yieldToEventLoop(); + if (options.signal?.aborted) { + reportProgress(); + return cancelledResult(processed, candidates); + } + } + } + if (options.signal?.aborted) { + reportProgress(); + return cancelledResult(processed, candidates); + } + + for (const path of vaultPaths) { + processed++; + if (mapper.path(path) === token && !matchingMetadataPaths.has(path)) { + candidates.push({ path, storageExists: true, source: "vault" }); + } + if (processed % 128 === 0) { + reportProgress(); + await yieldToEventLoop(); + if (options.signal?.aborted) { + reportProgress(); + return cancelledResult(processed, candidates); + } + } + } + reportProgress(); + if (options.signal?.aborted) return cancelledResult(processed, candidates); + return { valid: true, status: "complete", processed, candidates }; +} + +function finiteMetric(value: number | undefined, fallback = 0): number { + return typeof value === "number" && Number.isFinite(value) ? value : fallback; +} + +function nonnegativeMetric(value: number | undefined): number { + return Math.max(0, finiteMetric(value)); +} + +function revisionStatus(value: string): "available" | "missing" | "unknown" { + if (value === "available" || value === "missing") return value; + return "unknown"; +} + +function storageType(value: string): "absent" | "notes" | "newnote" | "plain" | "unknown" { + if (value === "absent" || value === "notes" || value === "newnote" || value === "plain") return value; + return "unknown"; +} + +function localChunkState(value: string): "available" | "deleted" | "missing" | "unknown" { + if (value === "available" || value === "deleted" || value === "missing") return value; + return "unknown"; +} + +function projectRevisionInfo(info: RevisionDatabaseInfo, mapper: ReturnType) { + return { + documentId: mapper.id(info.documentId), + revision: info.revision === null ? null : mapper.revision(info.revision), + current: info.current, + deleted: info.deleted, + storageType: storageType(info.storageType), + storageLayout: info.storageLayout === "legacy-inline" ? "legacy-inline" : "chunked", + ctime: finiteMetric(info.ctime), + mtime: finiteMetric(info.mtime), + recordedSize: nonnegativeMetric(info.recordedSize), + revisionHistory: info.revisionHistory.map(({ revision, status }) => ({ + revision: mapper.revision(revision), + status: revisionStatus(status), + })), + chunkReferences: nonnegativeMetric(info.chunkReferences), + uniqueChunkReferences: nonnegativeMetric(info.uniqueChunkReferences), + embeddedChunkReferences: nonnegativeMetric(info.embeddedChunkReferences), + locallyStoredChunkReferences: nonnegativeMetric(info.locallyStoredChunkReferences), + contentAvailableLocally: info.contentAvailableLocally, + chunks: info.chunks.map((chunk) => ({ + id: mapper.chunk(chunk.id), + referenceCount: nonnegativeMetric(chunk.referenceCount), + embedded: chunk.embedded, + storedInLocalDatabase: chunk.storedInLocalDatabase, + localDatabaseState: localChunkState(chunk.localDatabaseState), + localDatabaseRevision: + chunk.localDatabaseRevision === null ? null : mapper.revision(chunk.localDatabaseRevision), + })), + }; +} + +function projectMergeBase(info: FileDatabaseMergeBaseInfo, mapper: ReturnType) { + return { + winnerRevision: mapper.revision(info.winnerRevision), + conflictRevision: mapper.revision(info.conflictRevision), + revision: info.revision === null ? null : mapper.revision(info.revision), + metadataAvailableLocally: info.metadataAvailableLocally, + contentAvailableLocally: info.contentAvailableLocally, + missingChunkIds: info.missingChunkIds.map((id) => mapper.chunk(id)), + unavailableSharedRevisions: info.unavailableSharedRevisions.map((revision) => mapper.revision(revision)), + }; +} + +function projectStorage(info: FileDatabaseInfo["storage"]) { + return { + exists: info.exists, + ...(info.ctime === undefined ? {} : { ctime: finiteMetric(info.ctime) }), + ...(info.mtime === undefined ? {} : { mtime: finiteMetric(info.mtime) }), + ...(info.size === undefined ? {} : { size: nonnegativeMetric(info.size) }), + }; +} + +function projectFileInfo( + info: FileDatabaseInfo, + source: DiagnosticFileCandidate["source"], + mapper: ReturnType +) { + return { + path: mapper.path(info.path), + databasePath: mapper.path(info.databasePath), + source, + storage: projectStorage(info.storage), + database: { + source: "local database on this device" as const, + remoteQueried: false as const, + inspected: true, + exists: info.database.exists, + ...(info.database.documentId === undefined ? {} : { documentId: mapper.id(info.database.documentId) }), + currentRevision: + info.database.currentRevision === null ? null : mapper.revision(info.database.currentRevision), + conflictCount: nonnegativeMetric(info.database.conflictCount), + conflictRevisions: info.database.conflictRevisions.map((revision) => mapper.revision(revision)), + unavailableConflictRevisions: info.database.unavailableConflictRevisions.map((revision) => + mapper.revision(revision) + ), + revisions: info.database.revisions.map((revision) => projectRevisionInfo(revision, mapper)), + mergeBases: info.database.mergeBases.map((mergeBase) => projectMergeBase(mergeBase, mapper)), + }, + }; +} + +async function inspectVaultOnlyCandidate( + core: FileDatabaseInfoCore, + candidate: DiagnosticFileCandidate, + mapper: ReturnType, + signal?: AbortSignal +) { + const exists = await core.storageAccess.isExistsIncludeHidden(candidate.path); + if (signal?.aborted) throw new DOMException("File inspection cancelled", "AbortError"); + const stat = exists ? await core.storageAccess.statHidden(candidate.path) : null; + return { + path: mapper.path(candidate.path), + source: "vault" as const, + storage: projectStorage( + stat ? { exists: true, ctime: stat.ctime, mtime: stat.mtime, size: stat.size } : { exists: false } + ), + database: { + source: "local database on this device" as const, + remoteQueried: false as const, + inspected: false, + reason: "no-local-target" as const, + }, + }; +} + +export async function buildPseudonymousFileReport( + core: FileDatabaseInfoCore, + candidate: DiagnosticFileCandidate, + salt: string, + signal?: AbortSignal +): Promise { + if (signal?.aborted) throw new DOMException("File inspection cancelled", "AbortError"); + const mapper = createDiagnosticNameMapper("pseudonym", salt); + if (!mapper.salt) throw new Error("Invalid diagnostic path token"); + if (candidate.source === "metadata" && candidate.documentId === undefined) { + throw new Error("A Metadata candidate requires its exact document ID"); + } + const info = + candidate.source === "metadata" && candidate.documentId !== undefined + ? await inspectFileDatabaseInfo(core, candidate.path, { + documentId: candidate.documentId, + signal, + ...(candidate.databasePath === undefined ? {} : { databasePath: candidate.databasePath }), + } satisfies InspectFileDatabaseInfoOptions) + : undefined; + const file = info + ? projectFileInfo(info, candidate.source, mapper) + : await inspectVaultOnlyCandidate(core, candidate, mapper, signal); + if (signal?.aborted) throw new DOMException("File inspection cancelled", "AbortError"); + const report = { + schemaVersion: 1, + kind: "file", + status: "complete", + nameMode: "pseudonym", + salt: mapper.salt, + scope: { + source: "local database on this device", + remoteQueried: false, + vaultContentRead: false, + fileContents: "omitted", + revisions: "current revision, current conflict revisions, and selected merge-base revisions", + revisionHistory: "status metadata is listed for current and conflict revisions where available", + historicalFileBodiesScanned: false, + }, + file, + }; + const json = JSON.stringify(report, null, 2); + const markdown = `
\n${$msg("Local file diagnostic report")}\n\n\`\`\`json\n${json}\n\`\`\`\n
`; + const artifacts: DatabaseDiagnosticArtifact[] = [ + { format: "json", filename: "diagnostic-file-report.json", content: json }, + { format: "markdown", filename: "diagnostic-file-report.md", content: markdown }, + ]; + return { status: "complete", nameMode: "pseudonym", artifacts }; +} + +export async function showDiagnosticFileLookup( + core: FileDatabaseInfoCore, + openFile?: (path: string) => Promise | void +): Promise { + let activeController: AbortController | undefined; + const options: DiagnosticFileLookupDialogOptions = { + find: async (token, onProgress) => { + activeController?.abort(); + activeController = new AbortController(); + return await findDiagnosticFiles(core, token, { signal: activeController.signal, onProgress }); + }, + prepareReport: async (candidate, token) => { + const parsed = parseDiagnosticPathToken(token); + if (!parsed) throw new Error("Invalid diagnostic path token"); + activeController?.abort(); + activeController = new AbortController(); + return await buildPseudonymousFileReport(core, candidate, parsed.salt, activeController.signal); + }, + openFile, + cancel: () => activeController?.abort(), + }; + const ui = core.services.UI as FileLookupUI; + try { + await ui.dialogManager.open(DiagnosticFileLookupDialog, options); + } finally { + activeController?.abort(); + } +} diff --git a/src/serviceFeatures/diagnosticFileLookup.unit.spec.ts b/src/serviceFeatures/diagnosticFileLookup.unit.spec.ts new file mode 100644 index 00000000..3079df4a --- /dev/null +++ b/src/serviceFeatures/diagnosticFileLookup.unit.spec.ts @@ -0,0 +1,362 @@ +import { createDiagnosticNameMapper } from "@vrtmrz/livesync-commonlib/diagnostics"; +import { describe, expect, it, vi } from "vitest"; +import { buildPseudonymousFileReport, findDiagnosticFiles, showDiagnosticFileLookup } from "./diagnosticFileLookup"; +import type { DiagnosticFileLookupDialogOptions } from "./diagnosticFileLookupTypes"; + +type TestDocument = { + _id: string; + _rev?: string; + _conflicts?: string[]; + _revs_info?: Array<{ rev: string; status: string }>; + _deleted?: boolean; + children?: string[]; + ctime?: number; + data?: string; + deleted?: boolean; + eden?: Record; + mtime?: number; + path?: string; + size?: number; + type?: string; +}; + +async function* entries(documents: TestDocument[]) { + for (const document of documents) yield document; +} + +function createCore(vaultPaths: string[] = [], documents: TestDocument[] = []) { + const existingPaths = new Set(vaultPaths); + const chunkRevisions = new Map([["h:secret-chunk", "1-secret-chunk-revision"]]); + const core = { + settings: {}, + storageAccess: { + getFileNames: vi.fn(async () => [...vaultPaths]), + getFilesIncludeHidden: vi.fn(async () => [...vaultPaths]), + isExistsIncludeHidden: vi.fn(async (path: string) => existingPaths.has(path)), + statHidden: vi.fn(async (path: string) => + existingPaths.has(path) ? { ctime: 10, mtime: 20, size: 30, type: "file" } : undefined + ), + }, + localDatabase: { + allDocsRaw: vi.fn(async ({ keys }: { keys: string[] }) => ({ + rows: keys.flatMap((key) => { + const rev = chunkRevisions.get(key); + return rev ? [{ id: key, key, value: { rev } }] : []; + }), + })), + findAllDocs: vi.fn(() => entries(documents)), + getDBEntryFromMeta: vi.fn(), + getDBEntry: vi.fn(), + localDatabase: { + get: vi.fn(async (id: string) => { + const document = documents.find(({ _id }) => _id === id); + if (document) return document; + throw Object.assign(new Error("missing"), { status: 404 }); + }), + }, + }, + services: { + path: { path2id: vi.fn(async () => "f:path-derived") }, + UI: { + promptCopyToClipboard: vi.fn(async () => true), + dialogManager: { open: vi.fn() }, + }, + }, + }; + return core; +} + +function diagnosticPath(path: string, salt = "salt1234"): string { + return createDiagnosticNameMapper("pseudonym", salt).path(path); +} + +describe("diagnostic file lookup", () => { + it.each(["metadata", "chunks"] as const)( + "stops follow-up reads when the dialogue closes during a %s read", + async (pendingRead) => { + const path = "cancelled.md"; + const selected = { + _id: "f:cancelled", + _rev: "3-current", + _conflicts: ["3-conflict"], + path, + type: "plain", + children: ["h:secret-chunk"], + }; + const core = createCore([path], [selected]); + let markStarted!: () => void; + const started = new Promise((resolve) => { + markStarted = resolve; + }); + let releaseRead!: () => void; + const pending = new Promise((resolve) => { + releaseRead = resolve; + }); + if (pendingRead === "metadata") { + core.localDatabase.localDatabase.get.mockImplementationOnce(async () => { + markStarted(); + await pending; + return selected; + }); + } else { + core.localDatabase.allDocsRaw.mockImplementationOnce(async () => { + markStarted(); + await pending; + return { rows: [] }; + }); + } + core.services.UI.dialogManager.open.mockImplementation(async (...args: unknown[]) => { + const options = args[1] as DiagnosticFileLookupDialogOptions; + const preparation = options.prepareReport( + { path, documentId: selected._id, storageExists: true, source: "metadata" }, + diagnosticPath(path) + ); + await started; + const cancelled = expect(preparation).rejects.toMatchObject({ name: "AbortError" }); + options.cancel(); + releaseRead(); + await cancelled; + }); + + await showDiagnosticFileLookup(core as never); + + expect(core.localDatabase.localDatabase.get).toHaveBeenCalledTimes(1); + expect(core.localDatabase.allDocsRaw).toHaveBeenCalledTimes(pendingRead === "metadata" ? 0 : 1); + } + ); + + it("rejects an invalid path token before reading storage or the database", async () => { + const core = createCore(["note.md"], [{ _id: "f:note", path: "note.md", type: "plain" }]); + + await expect(findDiagnosticFiles(core as never, "not-a-path-token")).resolves.toEqual({ + valid: false, + status: "invalid", + processed: 0, + candidates: [], + }); + + expect(core.storageAccess.getFileNames).not.toHaveBeenCalled(); + expect(core.localDatabase.findAllDocs).not.toHaveBeenCalled(); + expect(core.localDatabase.localDatabase.get).not.toHaveBeenCalled(); + }); + + it("matches path spelling exactly without case folding or Unicode normalisation", async () => { + const decomposed = "Cafe\u0301.md"; + const core = createCore( + [decomposed, "Café.md", "CAFÉ.md"], + [ + { _id: "f:decomposed", path: decomposed, type: "plain" }, + { _id: "f:composed", path: "Café.md", type: "plain" }, + { _id: "f:case", path: "CAFÉ.md", type: "plain" }, + ] + ); + const progressCounts: number[] = []; + + const result = await findDiagnosticFiles(core as never, diagnosticPath(decomposed), { + onProgress: (processed) => progressCounts.push(processed), + }); + + expect(result).toMatchObject({ valid: true, status: "complete" }); + expect(result.candidates).toEqual([ + expect.objectContaining({ + path: decomposed, + documentId: "f:decomposed", + source: "metadata", + storageExists: true, + }), + ]); + expect(core.localDatabase.localDatabase.get).not.toHaveBeenCalled(); + expect(progressCounts).toEqual([0, 6]); + }); + + it("keeps distinct Metadata documents with the same exact path as separate candidates", async () => { + const path = "duplicate.md"; + const core = createCore( + [path], + [ + { _id: "f:duplicate-one", path, type: "plain" }, + { _id: "f:duplicate-two", path, type: "plain" }, + ] + ); + + const result = await findDiagnosticFiles(core as never, diagnosticPath(path)); + + expect(result.candidates).toEqual([ + expect.objectContaining({ documentId: "f:duplicate-one", path, storageExists: true, source: "metadata" }), + expect.objectContaining({ documentId: "f:duplicate-two", path, storageExists: true, source: "metadata" }), + ]); + expect(result.candidates).toHaveLength(2); + }); + + it("keeps logically deleted current file Metadata as an exact candidate", async () => { + const path = "vault-only.md"; + const core = createCore( + [path], + [ + { _id: "f:deleted", path, type: "plain", deleted: true }, + { _id: "f:chunk", path, type: "chunk" }, + { _id: "i:internal", path: "i:.obsidian/config.json", type: "plain" }, + ] + ); + + const result = await findDiagnosticFiles(core as never, diagnosticPath(path)); + expect(result.candidates).toEqual([ + expect.objectContaining({ path, documentId: "f:deleted", storageExists: true, source: "metadata" }), + ]); + }); + + it("returns a Vault-only candidate when there is no current normal-file Metadata", async () => { + const path = "vault-only.md"; + const core = createCore( + [path], + [ + { _id: "f:chunk", path, type: "chunk" }, + { _id: "f:legacy", path, type: "notes" }, + { _id: "i:internal", path: "i:.obsidian/config.json", type: "plain" }, + ] + ); + + const result = await findDiagnosticFiles(core as never, diagnosticPath(path)); + expect(result.candidates).toEqual([expect.objectContaining({ path, storageExists: true, source: "vault" })]); + expect(result.candidates[0]).not.toHaveProperty("documentId"); + + const report = await buildPseudonymousFileReport(core as never, result.candidates[0], "salt1234"); + const jsonText = report.artifacts.find(({ format }) => format === "json")!.content; + const json = JSON.parse(jsonText); + + expect(json.file.path).toBe(diagnosticPath(path)); + expect(jsonText).not.toContain(path); + expect(json.file.database).toEqual({ + source: "local database on this device", + remoteQueried: false, + inspected: false, + reason: "no-local-target", + }); + expect(json.scope).toEqual({ + source: "local database on this device", + remoteQueried: false, + vaultContentRead: false, + fileContents: "omitted", + revisions: "current revision, current conflict revisions, and selected merge-base revisions", + revisionHistory: "status metadata is listed for current and conflict revisions where available", + historicalFileBodiesScanned: false, + }); + expect(core.localDatabase.localDatabase.get).not.toHaveBeenCalled(); + }); + + it("returns partial exact matches when cancellation arrives during enumeration", async () => { + const path = "match.md"; + const documents = [ + { _id: "f:match", path, type: "plain" }, + ...Array.from({ length: 299 }, (_, index) => ({ + _id: `f:other-${index}`, + path: `other-${index}.md`, + type: "plain", + })), + ]; + const core = createCore([], documents); + const controller = new AbortController(); + const progressCounts: number[] = []; + + const result = await findDiagnosticFiles(core as never, diagnosticPath(path), { + signal: controller.signal, + onProgress: (processed) => { + progressCounts.push(processed); + if (processed >= 128) controller.abort(); + }, + }); + + expect(result).toMatchObject({ valid: true, status: "cancelled", processed: 128 }); + expect(result.candidates).toEqual([ + expect.objectContaining({ documentId: "f:match", path, storageExists: false, source: "metadata" }), + ]); + expect(progressCounts).toEqual([0, 128, 128]); + }); + + it("projects an exact target with the token salt, preserves revision generations, and omits raw values", async () => { + const path = "private-name.md"; + const salt = "salt1234"; + const token = diagnosticPath(path, salt); + const selected = { + _id: "f:selected-private-id", + _rev: "3-private-revision", + _conflicts: [], + _revs_info: [{ rev: "3-private-revision", status: "private-status-value" }], + path, + ctime: 11, + mtime: 22, + size: 33, + type: "plain", + data: "private file contents", + children: ["h:private-chunk-id"], + eden: {}, + unreported: "private-extra-value", + }; + const decoy = { ...selected, _id: "f:path-derived", path: "different.md", _rev: "9-decoy-revision" }; + const core = createCore([path], [selected, decoy]); + core.localDatabase.localDatabase.get.mockImplementation(async (id: string) => { + if (id === selected._id) return selected; + if (id === decoy._id) return decoy; + throw Object.assign(new Error("missing"), { status: 404 }); + }); + const candidate = { + path, + databasePath: path as never, + documentId: selected._id, + storageExists: true, + source: "metadata" as const, + }; + + const output = await buildPseudonymousFileReport(core as never, candidate, salt); + const jsonText = output.artifacts.find(({ format }) => format === "json")!.content; + const json = JSON.parse(jsonText); + const revision = json.file.database.revisions[0]; + + expect(output.nameMode).toBe("pseudonym"); + expect(json.kind).toBe("file"); + expect(json.nameMode).toBe("pseudonym"); + expect(json.salt).toBe(salt); + expect(json.file.path).toBe(token); + expect(json.scope).toMatchObject({ vaultContentRead: false, fileContents: "omitted" }); + expect(revision.documentId).toMatch(/^document-[0-9a-f]{32}$/); + expect(revision.revision).toMatch(/^3-psu[0-9a-f]{32}$/); + expect(revision.chunks[0].id).toMatch(/^chunk-[0-9a-f]{32}$/); + expect(revision.revisionHistory[0].status).toBe("unknown"); + expect(json.file.database.inspected).toBe(true); + expect(output.artifacts.map(({ format }) => format)).toEqual(["json", "markdown"]); + expect(output.artifacts.find(({ format }) => format === "markdown")?.content).toContain("```json"); + expect(jsonText).not.toContain(path); + expect(jsonText).not.toContain(selected._id); + expect(jsonText).not.toContain("3-private-revision"); + expect(jsonText).not.toContain("h:private-chunk-id"); + expect(jsonText).not.toContain("private-status-value"); + expect(jsonText).not.toContain("private file contents"); + expect(jsonText).not.toContain("private-extra-value"); + expect(core.localDatabase.localDatabase.get.mock.calls.map(([id]) => id)).toEqual([selected._id]); + expect(core.services.path.path2id).not.toHaveBeenCalled(); + }); + + it("keeps an exact Metadata target visibly missing without substituting a path-derived document", async () => { + const path = "vanished.md"; + const salt = "salt1234"; + const selected = { _id: "f:vanished", path, type: "plain" }; + const decoy = { _id: "f:path-derived", path: "wrong.md", type: "plain", _rev: "1-decoy" }; + const core = createCore([path], [selected, decoy]); + const lookup = await findDiagnosticFiles(core as never, diagnosticPath(path, salt)); + core.localDatabase.localDatabase.get.mockImplementation(async (id: string) => { + if (id === decoy._id) return decoy; + throw Object.assign(new Error("missing"), { status: 404 }); + }); + + const output = await buildPseudonymousFileReport(core as never, lookup.candidates[0], salt); + const jsonText = output.artifacts.find(({ format }) => format === "json")!.content; + const json = JSON.parse(jsonText); + + expect(json.file.database).toMatchObject({ inspected: true, exists: false }); + expect(json.file.database.documentId).toMatch(/^document-[0-9a-f]{32}$/); + expect(jsonText).not.toContain("f:vanished"); + expect(jsonText).not.toContain("f:path-derived"); + expect(core.localDatabase.localDatabase.get.mock.calls.map(([id]) => id)).toEqual([selected._id]); + expect(core.services.path.path2id).not.toHaveBeenCalled(); + }); +}); diff --git a/src/serviceFeatures/diagnosticFileLookupTypes.ts b/src/serviceFeatures/diagnosticFileLookupTypes.ts new file mode 100644 index 00000000..6a08cc2a --- /dev/null +++ b/src/serviceFeatures/diagnosticFileLookupTypes.ts @@ -0,0 +1,36 @@ +import type { FilePath, FilePathWithPrefix } from "@vrtmrz/livesync-commonlib/compat/common/types"; +import type { DatabaseDiagnosticOutput } from "./databaseDiagnostics"; + +export type DiagnosticFileCandidate = { + path: string; + databasePath?: FilePathWithPrefix | FilePath; + documentId?: string; + storageExists: boolean; + source: "metadata" | "vault"; +}; + +export type DiagnosticFileLookupResult = + | { + valid: false; + status: "invalid"; + processed: 0; + candidates: []; + } + | { + valid: true; + status: "cancelled" | "complete"; + processed: number; + candidates: DiagnosticFileCandidate[]; + }; + +export interface FindDiagnosticFilesOptions { + signal?: AbortSignal; + onProgress?(processed: number): void; +} + +export interface DiagnosticFileLookupDialogOptions { + find(token: string, onProgress: (processed: number) => void): Promise; + prepareReport(candidate: DiagnosticFileCandidate, token: string): Promise; + openFile?(path: string): Promise | void; + cancel(): void; +} diff --git a/src/serviceFeatures/fileDatabaseInfo.ts b/src/serviceFeatures/fileDatabaseInfo.ts index 300ab4e3..8fbf06e3 100644 --- a/src/serviceFeatures/fileDatabaseInfo.ts +++ b/src/serviceFeatures/fileDatabaseInfo.ts @@ -91,6 +91,7 @@ export type FileDatabaseInfo = { source: "local database on this device"; remoteQueried: false; exists: boolean; + documentId?: string; currentRevision: string | null; conflictCount: number; conflictRevisions: string[]; @@ -100,6 +101,12 @@ export type FileDatabaseInfo = { }; }; +export type InspectFileDatabaseInfoOptions = { + documentId?: string; + databasePath?: FilePathWithPrefix | FilePath; + signal?: AbortSignal; +}; + const REPORT_WARNING = "All revisions and chunk availability below are a snapshot of this device's local database; the remote is not queried. Review the Vault-relative path, document identifier, content-derived chunk identifiers, and metadata before sharing this report. File contents are omitted."; @@ -125,25 +132,35 @@ type RawDatabaseDocument = { data?: string | string[]; eden?: Record; mtime?: number; + path?: FilePathWithPrefix | FilePath; size?: number; type?: string; }; +function checkInspectionCancelled(signal?: AbortSignal): void { + if (signal?.aborted) throw new DOMException("File inspection cancelled", "AbortError"); +} + async function getLocalDatabaseMeta( core: FileDatabaseInfoCore, path: FilePathWithPrefix | FilePath, - options: PouchDB.Core.GetOptions + options: PouchDB.Core.GetOptions, + target?: InspectFileDatabaseInfoOptions ): Promise { - const documentId = await core.services.path.path2id(path); + checkInspectionCancelled(target?.signal); + const documentId = target?.documentId ?? (await core.services.path.path2id(path)); + checkInspectionCancelled(target?.signal); let raw: RawDatabaseDocument; try { raw = await core.localDatabase.localDatabase.get(documentId, options); } catch (error) { + checkInspectionCancelled(target?.signal); if (isNotFoundError(error)) { return false; } throw error; } + checkInspectionCancelled(target?.signal); if (raw.type === "leaf") { return false; @@ -156,13 +173,15 @@ async function getLocalDatabaseMeta( const legacy = rawStorageType === null || rawStorageType === "notes"; const type = legacy ? "notes" : rawStorageType; const legacyBodyPresent = - legacy && (typeof raw.data === "string" || (Array.isArray(raw.data) && raw.data.every((item) => typeof item === "string"))); + legacy && + (typeof raw.data === "string" || + (Array.isArray(raw.data) && raw.data.every((item) => typeof item === "string"))); return { _id: raw._id, _rev: raw._rev, _conflicts: raw._conflicts, _revs_info: raw._revs_info, - path, + path: target ? (raw.path ?? path) : path, data: legacyBodyPresent ? raw.data : "", ctime: raw.ctime ?? 0, mtime: raw.mtime ?? 0, @@ -180,8 +199,10 @@ async function getLocalDatabaseMeta( async function collectRevisionDatabaseInfo( core: FileDatabaseInfoCore, meta: DatabaseMeta, - current: boolean + current: boolean, + signal?: AbortSignal ): Promise { + checkInspectionCancelled(signal); const legacy = meta._rawStorageType === null || meta._rawStorageType === "notes"; const children = legacy ? [] : "children" in meta ? meta.children : []; const uniqueChildren = [...new Set(children)]; @@ -201,6 +222,7 @@ async function collectRevisionDatabaseInfo( include_docs: false, }) ).rows; + checkInspectionCancelled(signal); const localChunkStates = new Map( localRows .filter((row) => "value" in row) @@ -236,9 +258,7 @@ async function collectRevisionDatabaseInfo( locallyStoredChunkReferences: children.filter((id) => localChunkStates.get(id)?.state === "available").length, contentAvailableLocally: legacy ? meta._legacyBodyPresent - : uniqueChildren.every( - (id) => embeddedChildren.has(id) || localChunkStates.get(id)?.state === "available" - ), + : uniqueChildren.every((id) => embeddedChildren.has(id) || localChunkStates.get(id)?.state === "available"), chunks: uniqueChildren.map((id) => { const localState = localChunkStates.get(id); return { @@ -273,15 +293,27 @@ function missingChunkIds(info: RevisionDatabaseInfo): string[] { .map(({ id }) => id); } -export async function inspectFileDatabaseInfo(core: FileDatabaseInfoCore, path: string): Promise { +export async function inspectFileDatabaseInfo( + core: FileDatabaseInfoCore, + path: string, + options?: InspectFileDatabaseInfoOptions +): Promise { + checkInspectionCancelled(options?.signal); const storageExists = await core.storageAccess.isExistsIncludeHidden(path); + checkInspectionCancelled(options?.signal); const storageStat = storageExists ? await core.storageAccess.statHidden(path) : null; - const databasePath = toDatabasePath(path); - const currentMeta = await getLocalDatabaseMeta(core, databasePath, { - conflicts: true, - revs: true, - revs_info: true, - }); + const requestedDatabasePath = options?.databasePath ?? toDatabasePath(path); + const currentMeta = await getLocalDatabaseMeta( + core, + requestedDatabasePath, + { + conflicts: true, + revs: true, + revs_info: true, + }, + options + ); + const databasePath = options && currentMeta ? currentMeta.path : requestedDatabasePath; const revisions: RevisionDatabaseInfo[] = []; const conflictRevisions = currentMeta === false ? [] : (currentMeta._conflicts ?? []); @@ -292,25 +324,31 @@ export async function inspectFileDatabaseInfo(core: FileDatabaseInfoCore, path: metadataByRevision.set(currentMeta._rev, currentMeta); } const getRevisionMeta = async (revision: string): Promise => { + checkInspectionCancelled(options?.signal); const cached = metadataByRevision.get(revision); if (cached !== undefined) { return cached; } - const meta = await getLocalDatabaseMeta(core, databasePath, { - rev: revision, - revs: true, - revs_info: true, - }); + const meta = await getLocalDatabaseMeta( + core, + requestedDatabasePath, + { + rev: revision, + revs: true, + revs_info: true, + }, + options + ); metadataByRevision.set(revision, meta); return meta; }; if (currentMeta) { - revisions.push(await collectRevisionDatabaseInfo(core, currentMeta, true)); + revisions.push(await collectRevisionDatabaseInfo(core, currentMeta, true, options?.signal)); for (const revision of conflictRevisions) { const conflictMeta = await getRevisionMeta(revision); if (conflictMeta) { - revisions.push(await collectRevisionDatabaseInfo(core, conflictMeta, false)); + revisions.push(await collectRevisionDatabaseInfo(core, conflictMeta, false, options?.signal)); const winnerHistory = revisionHistory(currentMeta); const conflictHistory = revisionHistory(conflictMeta); const conflictHistoryByRevision = new Map( @@ -323,13 +361,12 @@ export async function inspectFileDatabaseInfo(core: FileDatabaseInfoCore, path: const unavailableSharedRevisions = sharedHistory .filter( ({ revision: historyRevision, status }) => - status !== "available" || - conflictHistoryByRevision.get(historyRevision) !== "available" + status !== "available" || conflictHistoryByRevision.get(historyRevision) !== "available" ) .map(({ revision: historyRevision }) => historyRevision); const sharedMeta = sharedRevision ? await getRevisionMeta(sharedRevision) : false; const sharedInfo = sharedMeta - ? await collectRevisionDatabaseInfo(core, sharedMeta, false) + ? await collectRevisionDatabaseInfo(core, sharedMeta, false, options?.signal) : undefined; mergeBases.push({ winnerRevision: currentMeta._rev ?? "", @@ -346,6 +383,7 @@ export async function inspectFileDatabaseInfo(core: FileDatabaseInfoCore, path: } } + checkInspectionCancelled(options?.signal); const report: FileDatabaseInfo = { path, databasePath, @@ -363,6 +401,7 @@ export async function inspectFileDatabaseInfo(core: FileDatabaseInfoCore, path: source: "local database on this device", remoteQueried: false, exists: currentMeta !== false, + ...(options?.documentId !== undefined ? { documentId: options.documentId } : {}), currentRevision: currentMeta ? (currentMeta._rev ?? null) : null, conflictCount: conflictRevisions.length, conflictRevisions, diff --git a/src/serviceFeatures/fileDatabaseInfo.unit.spec.ts b/src/serviceFeatures/fileDatabaseInfo.unit.spec.ts index 1d1b6207..a9c8dcb1 100644 --- a/src/serviceFeatures/fileDatabaseInfo.unit.spec.ts +++ b/src/serviceFeatures/fileDatabaseInfo.unit.spec.ts @@ -198,8 +198,10 @@ describe("file database information", () => { it("reports the exact shared ancestor and its missing chunks for each conflict", async () => { const { conflict, core, current } = createCore(); + const selectedCurrent = { ...current, _id: "f:selected" }; + const selectedConflict = { ...conflict, _id: "f:selected" }; const parent = { - ...current, + ...selectedCurrent, _rev: "2-parent", _conflicts: undefined, _revs_info: [ @@ -212,7 +214,7 @@ describe("file database information", () => { core.localDatabase.localDatabase.get.mockImplementation(async (_id: string, options?: { rev?: string }) => { if (options?.rev === "2-conflict") { return { - ...conflict, + ...selectedConflict, _revs_info: [ { rev: "2-conflict", status: "available" }, { rev: "2-parent", status: "available" }, @@ -224,7 +226,7 @@ describe("file database information", () => { return parent; } return { - ...current, + ...selectedCurrent, _revs_info: [ { rev: "3-current", status: "available" }, { rev: "2-parent", status: "available" }, @@ -233,7 +235,10 @@ describe("file database information", () => { }; }); - const info = await inspectFileDatabaseInfo(core as never, "note.md"); + const info = await inspectFileDatabaseInfo(core as never, "note.md", { + documentId: "f:selected", + databasePath: "note.md" as never, + }); expect(info.database.mergeBases).toEqual([ { @@ -246,6 +251,64 @@ describe("file database information", () => { unavailableSharedRevisions: ["1-root"], }, ]); + expect(core.localDatabase.localDatabase.get.mock.calls.map(([id]) => id)).toEqual([ + "f:selected", + "f:selected", + "f:selected", + ]); + }); + + it("inspects the exact selected document when its ID differs from the path-derived ID", async () => { + const { core, current } = createCore(); + const selected = { + ...current, + _id: "f:selected", + _rev: "4-selected", + _conflicts: [], + path: "stored-name.md", + children: [], + }; + core.localDatabase.localDatabase.get.mockImplementation(async (id: string) => + id === "f:selected" ? selected : current + ); + + const info = await inspectFileDatabaseInfo(core as never, "note.md", { + documentId: "f:selected", + databasePath: "listed-name.md" as never, + }); + + expect(info.database.exists).toBe(true); + expect(info.database.revisions).toEqual([ + expect.objectContaining({ + documentId: "f:selected", + revision: "4-selected", + }), + ]); + expect(info.databasePath).toBe("stored-name.md"); + expect(info.database.documentId).toBe("f:selected"); + expect(core.services.path.path2id).not.toHaveBeenCalled(); + }); + + it("reports a missing exact document without falling back to the path-derived ID", async () => { + const { core, current } = createCore(); + core.localDatabase.localDatabase.get.mockImplementation(async (id: string) => { + if (id === "f:missing") { + throw Object.assign(new Error("missing"), { status: 404 }); + } + return current; + }); + + const info = await inspectFileDatabaseInfo(core as never, "note.md", { + documentId: "f:missing", + databasePath: "requested-name.md" as never, + }); + + expect(info.database.exists).toBe(false); + expect(info.database.revisions).toEqual([]); + expect(info.databasePath).toBe("requested-name.md"); + expect(info.database.documentId).toBe("f:missing"); + expect(core.localDatabase.localDatabase.get.mock.calls.map(([id]) => id)).toEqual(["f:missing"]); + expect(core.services.path.path2id).not.toHaveBeenCalled(); }); it("does not decode a revision whose chunks are not all available locally", async () => { @@ -293,13 +356,7 @@ describe("file database information", () => { await retryReadFileDatabaseRevision(core as never, "note.md", "2-conflict"); - expect(core.localDatabase.getDBEntry).toHaveBeenCalledWith( - "note.md", - { rev: "2-conflict" }, - false, - true, - true - ); + expect(core.localDatabase.getDBEntry).toHaveBeenCalledWith("note.md", { rev: "2-conflict" }, false, true, true); }); it("reports the exact revision as locally available after retry recovers its missing chunk", async () => { @@ -322,14 +379,10 @@ describe("file database information", () => { : [], })); - await expect( - retryReadFileDatabaseRevision(core as never, "note.md", "2-conflict") - ).resolves.not.toBe(false); + await expect(retryReadFileDatabaseRevision(core as never, "note.md", "2-conflict")).resolves.not.toBe(false); const information = await inspectFileDatabaseInfo(core as never, "note.md"); - expect( - information.database.revisions.find(({ revision }) => revision === "2-conflict") - ).toEqual( + expect(information.database.revisions.find(({ revision }) => revision === "2-conflict")).toEqual( expect.objectContaining({ contentAvailableLocally: true, chunks: [ diff --git a/src/serviceFeatures/useDatabaseDiagnostics.ts b/src/serviceFeatures/useDatabaseDiagnostics.ts index 71021b1c..2fa974c8 100644 --- a/src/serviceFeatures/useDatabaseDiagnostics.ts +++ b/src/serviceFeatures/useDatabaseDiagnostics.ts @@ -23,11 +23,12 @@ export const useDatabaseDiagnosticsFeature = createServiceFeature((host: Databas try { await services.UI.dialogManager.open<"close", DatabaseDiagnosticDialogOptions>(DatabaseDiagnosticDialog, { kind, - run: async (nameMode, onProgress) => { + run: async (nameMode, onProgress, includeRemote = true) => { if (!services.database.isDatabaseReady()) throw new Error("local-database-unavailable"); return await runDatabaseDiagnostic(host, kind, nameMode, { signal: controller.signal, onProgress, + includeRemote, }); }, cancel: () => controller.abort(), diff --git a/src/serviceFeatures/useDatabaseDiagnostics.unit.spec.ts b/src/serviceFeatures/useDatabaseDiagnostics.unit.spec.ts index 0e9669e0..8372b5ef 100644 --- a/src/serviceFeatures/useDatabaseDiagnostics.unit.spec.ts +++ b/src/serviceFeatures/useDatabaseDiagnostics.unit.spec.ts @@ -119,4 +119,29 @@ describe("database diagnostics serviceFeature", () => { data.close(); await opened; }); + + it("forwards the per-run remote inspection choice and defaults existing callers to include the remote database", async () => { + vi.mocked(runDatabaseDiagnostic).mockClear(); + const data = fixture(); + const opened = data.view.open("integrity"); + const options = data.openDialog.mock.calls[0][1]; + + await options.run("pseudonym", () => {}, false); + expect(runDatabaseDiagnostic).toHaveBeenLastCalledWith( + data.host, + "integrity", + "pseudonym", + expect.objectContaining({ includeRemote: false, signal: expect.any(AbortSignal) }) + ); + + await options.run("pseudonym", () => {}); + expect(runDatabaseDiagnostic).toHaveBeenLastCalledWith( + data.host, + "integrity", + "pseudonym", + expect.objectContaining({ includeRemote: true, signal: expect.any(AbortSignal) }) + ); + data.close(); + await opened; + }); }); diff --git a/test/e2e-obsidian/README.md b/test/e2e-obsidian/README.md index 0a31cae4..05aadf06 100644 --- a/test/e2e-obsidian/README.md +++ b/test/e2e-obsidian/README.md @@ -87,9 +87,11 @@ npm run test:e2e:obsidian:focused -- couchdb-http-cache The wrapper accepts only maintained real-Obsidian scenario names; run it with `--help` for the current list. It deliberately does not manage CouchDB, Object Storage, or the P2P signalling relay. Start the required fixture first, or use the complete service-managed suite. -`database-diagnostics` needs no remote service. It checks the Usage command and Hatch integrity action with an isolated Vault, including visible radio choices for original names and pseudonyms, their default and keyboard navigation, retained local findings when the remote is unavailable, cancellation during inspection with earlier rows retained, and unchanged settings, local revisions, and Vault files. It captures desktop results and the choice and result dialogues in mobile emulation, checking layout, radio choices for available formats, their touch targets, and export controls. Copy and download payloads are captured inside the isolated renderer, so the scenario leaves the host clipboard and download directory unchanged. +`database-diagnostics` needs no remote service. It checks the Usage command and Hatch integrity action with an isolated Vault, including visible radio choices for original names and pseudonyms, their default and keyboard navigation, the default-on integrity-only remote option, retained local findings when the remote is unavailable, cancellation during inspection with earlier rows retained, and unchanged settings, local revisions, and Vault files. It captures desktop results and the choice and result dialogues in mobile emulation, checking layout at 800×600 and 375×667, radio choices for available formats, their touch targets, and export controls. Copy and download payloads are captured inside the isolated renderer, so the scenario leaves the host clipboard and download directory unchanged. -`database-diagnostics-couchdb` needs the CouchDB fixture with CORS enabled. It checks unencrypted and encrypted databases, both browser fetch and Obsidian's native Request API, and both name policies through the real integrity dialogue. A remote-only document with a mismatched ID must produce a finding with readable Chunks and equal content. JSON and Markdown must omit file contents and credentials. The scenario observes real requests and requires GET-only access, unchanged local settings, revisions, and Vault contents, and unchanged remote documents and Security Seed. Commonlib's separate CouchDB integration tests additionally cover a missing Security Seed without creating one. +The same scenario checks the five actions in Reporting Tools, the separate Checks and Recovery panel, Metadata progress with its document total and gauge, and collapsible Markdown. It also observes Usage revision progress with a count and discovery note during a gated read. Local lookup checks cover multiple document IDs for one path, selection of an exact ID, and a follow-up report using the original salt. The compact Markdown must retain every JSON observation reference and pseudonym, and its copy and save payloads must match the full output. Results and export controls are checked at 800×600 on desktop and 375×667 in mobile emulation, including after copy and save feedback appears. + +`database-diagnostics-couchdb` needs the CouchDB fixture with CORS enabled. It checks that clearing the remote option completes a local database and Vault inspection with remote coverage marked not checked and makes no HTTP requests, then confirms that each new run includes the remote database by default. It also checks unencrypted and encrypted databases, both browser fetch and Obsidian's native Request API, and both name policies through the real integrity dialogue. A remote-only document with a mismatched ID must produce a finding with readable Chunks and equal content. JSON and Markdown must omit file contents and credentials. Included inspections observe real requests and require GET-only access, unchanged local settings, revisions, and Vault contents, and unchanged remote documents and Security Seed. Commonlib's separate CouchDB integration tests additionally cover a missing Security Seed without creating one. `folder-batch` needs no remote service. It creates 24 notes and imports three notes with colons in their names into nested folders, reflects the database content into the existing files, then renames and deletes the parent through the Obsidian Vault API. It checks exact paths, descendant events, content, Chunks, deletion markers, provenance, and the absence of unexpected files. A note outside the parent must remain writable. The colon fixtures use the adapter to represent externally created files because Obsidian's Vault creation API rejects those names. The scenario also seeds a database-only colon-named note and verifies that Obsidian's refusal to create it preserves its Metadata without writing a differently named Vault file. It does not establish successful restoration of that absent note or behaviour on other operating systems. @@ -326,6 +328,8 @@ E2E_OBSIDIAN_ONLY_INDEPENDENT_IDS=true npm run test:e2e:obsidian:two-vault-sync The first scenario needs no service; the other two need CouchDB. The last checks an encrypted note round trip between two real Obsidian Vaults, with Path Obfuscation and independent IDs, and rejection of incompatible keys. These checks complement Commonlib's classification, conflict, privacy, and cancellation tests. +For a real-Obsidian follow-up check, copy a path token from a pseudonym Usage or integrity report and use Hatch → Reporting Tools → **Find a file from a report pseudonym**. Confirm that **Find file** searches only the current Vault and local normal-file Metadata, returns every matching local document ID, and shows real paths and IDs only in the local dialogue. Include cases with several IDs for one path, a Vault-only candidate whose Metadata is marked not inspected, and a path which no longer matches locally. Choose a result to open an existing Vault file or prepare a shareable local report; check that it reuses the report salt and exact document ID, retains the selected pseudonym mode, and contains no file contents or credentials. Check that the lookup makes no remote requests. Its history status listing is not a scan of every historical body. + Set `E2E_OBSIDIAN_ARTIFACT_ROOT` to a directory containing the exact release `main.js`, `manifest.json`, and `styles.css` when testing downloaded artefacts. These three scenarios accept that directory. Invoke their direct commands above, because the focused wrapper rebuilds local artefacts. Record the commit, Obsidian version, artefact hashes, and results. If testing a local build, compare its hashes with the published release before relying on that evidence for BRAT acceptance. When these checks pass for the release artefacts, BRAT acceptance can be limited to installation or update, enabling the plug-in, and a cold start. Check any other changes included in that release using their own regression scenarios. The diagnostic export checks verify report payloads; they do not exercise the operating system's clipboard or download implementation on iOS or Android. diff --git a/test/e2e-obsidian/scripts/database-diagnostics-couchdb.ts b/test/e2e-obsidian/scripts/database-diagnostics-couchdb.ts index d51279bf..c7b39756 100644 --- a/test/e2e-obsidian/scripts/database-diagnostics-couchdb.ts +++ b/test/e2e-obsidian/scripts/database-diagnostics-couchdb.ts @@ -36,11 +36,16 @@ const timeout = 20000; interface IntegrityReport { status: string; + scope: { remote: "included" | "excluded" }; coverage: { local: { status: string }; remote: { status: string; contentReads: number } }; observations: { source: string; identity: string; - content: { status: string; chunksRead: number }; + path: string; + revision?: string; + metadata?: { recordedSize?: number; chunkReferences: number; uniqueChunkReferences: number }; + vault: { status: string; size?: number; contentSizeDifference?: number; mtimeDifferenceMs?: number }; + content: { status: string; chunksRead: number; size?: number; recordedSizeDifference?: number }; comparisons: { relation: string }[]; }[]; errors: unknown[]; @@ -165,6 +170,73 @@ async function runCase(config: CouchDbConfig, encrypted: boolean) { let calls: { transport: string; method: string }[] = []; try { await withObsidianPage(session.remoteDebuggingPort, async (page) => { + await page.setViewportSize({ width: 800, height: 600 }); + const opened = await page.evaluate( + "app.commands.executeCommandById('obsidian-livesync:inspect-database-integrity')" + ); + assert.equal(opened, true); + const dialogue = page.locator(".database-diagnostic"); + await dialogue.waitFor({ state: "visible", timeout }); + const remoteChoice = dialogue.getByRole("checkbox", { + name: "Include remote database", + exact: true, + }); + assert.equal(await remoteChoice.isChecked(), true, "Remote inspection must be selected by default"); + const modal = page.locator(".modal-container").filter({ has: dialogue }).locator(".modal"); + const layout = await modal.evaluate((element) => { + const modalBounds = element.getBoundingClientRect(); + const choice = element.querySelector(".diagnostic-remote-choice"); + if (!choice) + return { contained: false, outerScroll: element.scrollHeight - element.clientHeight }; + const choiceBounds = choice.getBoundingClientRect(); + return { + contained: + choiceBounds.top >= modalBounds.top - 1 && + choiceBounds.bottom <= modalBounds.bottom + 1, + outerScroll: Math.max(0, element.scrollHeight - element.clientHeight), + }; + }); + assert.equal(layout.contained, true, "The remote option must remain inside the desktop modal"); + assert.ok( + layout.outerScroll <= 1, + `The choice modal must not need outer scrolling; got ${layout.outerScroll}px` + ); + await remoteChoice.uncheck(); + assert.equal(await remoteChoice.isChecked(), false); + await dialogue.getByRole("button", { name: "Begin inspection", exact: true }).click(); + await page.locator('[data-diagnostic-phase="result"]').waitFor({ timeout }); + await dialogue.getByRole("radio", { name: "JSON", exact: true }).check(); + const report = JSON.parse(await dialogue.locator("textarea").inputValue()) as IntegrityReport; + assert.equal(report.status, "complete"); + assert.equal(report.scope.remote, "excluded"); + assert.equal(report.coverage.local.status, "complete"); + assert.equal(report.coverage.remote.status, "not-checked"); + assert.deepEqual(report.errors, []); + assert.equal( + report.observations.some((row) => row.source === "remote"), + false + ); + const localObservations = report.observations.filter((row) => row.source === "local"); + assert.ok(localObservations.length > 0); + assert.ok(localObservations.every((row) => row.identity === "consistent")); + assert.ok( + localObservations.every( + (row) => row.content.status === "not-checked" && row.vault.status === "not-checked" + ), + "Only the remote has a discrepancy in this fixture; excluding it leaves no suspect content group" + ); + await dialogue.getByRole("button", { name: "Close", exact: true }).click(); + await dialogue.waitFor({ state: "hidden", timeout }); + }); + } finally { + calls = await finishRequests(cli, session); + } + assert.equal(calls.length, 0, "Excluding the remote database must make no HTTP requests"); + + await observeRequests(cli, session); + try { + await withObsidianPage(session.remoteDebuggingPort, async (page) => { + await page.setViewportSize({ width: 800, height: 600 }); for (const names of ["pseudonym", "original"]) { const opened = await page.evaluate( "app.commands.executeCommandById('obsidian-livesync:inspect-database-integrity')" @@ -172,6 +244,15 @@ async function runCase(config: CouchDbConfig, encrypted: boolean) { assert.equal(opened, true); const dialogue = page.locator(".database-diagnostic"); await dialogue.waitFor({ state: "visible", timeout }); + const remoteChoice = dialogue.getByRole("checkbox", { + name: "Include remote database", + exact: true, + }); + assert.equal( + await remoteChoice.isChecked(), + true, + "Each run must include the remote database by default" + ); const nameOptions = dialogue.getByRole("group", { name: "Names in this report", exact: true }); assert.equal(await nameOptions.getByRole("radio").count(), 2); const nameLabel = names === "pseudonym" ? "Pseudonyms (recommended)" : "Original names"; @@ -187,6 +268,7 @@ async function runCase(config: CouchDbConfig, encrypted: boolean) { const text = await dialogue.locator("textarea").inputValue(); const report = JSON.parse(text) as IntegrityReport; assert.equal(report.status, "complete"); + assert.equal(report.scope.remote, "included"); assert.equal(report.coverage.local.status, "complete"); assert.equal(report.coverage.remote.status, "complete"); assert.ok(report.coverage.remote.contentReads > 0); @@ -197,6 +279,22 @@ async function runCase(config: CouchDbConfig, encrypted: boolean) { assert.ok(suspect); assert.equal(suspect.content.status, "available"); assert.ok(suspect.content.chunksRead > 0); + const byteLength = Buffer.byteLength(fixtureContent); + assert.equal( + suspect.metadata?.recordedSize, + byteLength, + "Metadata properties must be decoded before reporting" + ); + assert.equal(suspect.metadata?.chunkReferences, entry.children.length); + assert.equal(suspect.content.size, byteLength); + assert.equal(suspect.content.recordedSizeDifference, 0); + assert.equal(suspect.vault.size, byteLength); + assert.equal(suspect.vault.contentSizeDifference, 0); + assert.ok(Number.isFinite(suspect.vault.mtimeDifferenceMs)); + if (names === "pseudonym") { + assert.match(suspect.path, /^[A-Za-z0-9_-]{8}\$[0-9a-f]{32}$/); + assert.match(suspect.revision ?? "", /^1-psu[0-9a-f]{32}$/); + } assert.ok(suspect.comparisons.some((comparison) => comparison.relation === "equal")); await formats.getByRole("radio", { name: "MARKDOWN", exact: true }).check(); assert.equal(await formats.getByRole("radio", { checked: true }).count(), 1); diff --git a/test/e2e-obsidian/scripts/database-diagnostics.ts b/test/e2e-obsidian/scripts/database-diagnostics.ts index d4f44ae9..49080c5c 100644 --- a/test/e2e-obsidian/scripts/database-diagnostics.ts +++ b/test/e2e-obsidian/scripts/database-diagnostics.ts @@ -6,6 +6,7 @@ import { assertNoHorizontalOverflow, } from "@vrtmrz/obsidian-test-session"; import { DoctorRegulation } from "@vrtmrz/livesync-commonlib/compat/common/configForDoc"; +import { createDiagnosticNameMapper, parseDiagnosticPathToken } from "@vrtmrz/livesync-commonlib/diagnostics"; import { requireObsidianBinary, requireObsidianCli } from "../runner/environment.ts"; import { evalObsidianJson } from "../runner/cli.ts"; import { @@ -49,11 +50,35 @@ async function openCommand(page: Page, kind: "usage" | "integrity") { assert.equal(await names.getByRole("radio", { name, exact: true }).isVisible(), true); assert.equal(await names.getByRole("radio", { name: "Pseudonyms (recommended)", exact: true }).isChecked(), true); assert.equal(await names.getByRole("radio", { checked: true }).count(), 1); + const remoteChoice = dialogue.getByRole("checkbox", { name: "Include remote database", exact: true }); + if (kind === "integrity") { + assert.equal(await remoteChoice.count(), 1); + assert.equal(await remoteChoice.isChecked(), true, "The remote database must be included by default"); + } else assert.equal(await remoteChoice.count(), 0, "Usage analysis has no remote database option"); assert.equal(await dialogue.getAttribute("data-diagnostic-phase"), "choice"); assert.equal(await dialogue.locator("textarea").count(), 0); return dialogue; } +async function assertDesktopDiagnosticChoiceLayout(page: Page, dialogue: Locator): Promise { + const modal = page.locator(".modal-container").filter({ has: dialogue }).locator(".modal"); + const layout = await modal.evaluate((element) => { + const modalBounds = element.getBoundingClientRect(); + const choice = element.querySelector(".diagnostic-remote-choice"); + if (!choice) return { contained: false, outerScroll: Math.max(0, element.scrollHeight - element.clientHeight) }; + const choiceBounds = choice.getBoundingClientRect(); + return { + contained: choiceBounds.top >= modalBounds.top - 1 && choiceBounds.bottom <= modalBounds.bottom + 1, + outerScroll: Math.max(0, element.scrollHeight - element.clientHeight), + }; + }); + assert.equal(layout.contained, true, "The remote database option must remain inside the desktop modal"); + assert.ok( + layout.outerScroll <= 1, + `The diagnostic choice must not need outer scrolling; got ${layout.outerScroll}px` + ); +} + async function assertFormatChoices(dialogue: Locator, formats: string[]): Promise { const group = dialogue.getByRole("group", { name: "Report format", exact: true }); assert.equal(await group.getByRole("radio").count(), formats.length); @@ -69,6 +94,31 @@ async function chooseFormat(dialogue: Locator, format: string): Promise { assert.equal(await group.getByRole("radio", { checked: true }).count(), 1); } +async function assertDiagnosticExportActionsWithinModal(page: Page, dialogue: Locator, action: string): Promise { + for (const name of ["Copy report", "Save report"]) { + const button = dialogue.getByRole("button", { name, exact: true }); + assert.equal(await button.isVisible(), true, `${name} must remain visible after ${action}`); + const contained = await button.evaluate((element) => { + const modalElement = element.closest(".modal"); + if (!modalElement || !modalElement.contains(element)) return false; + const buttonBounds = element.getBoundingClientRect(); + const modalBounds = modalElement.getBoundingClientRect(); + return buttonBounds.top >= modalBounds.top - 1 && buttonBounds.bottom <= modalBounds.bottom + 1; + }); + assert.equal(contained, true, `${name} must remain inside the modal after ${action}`); + } +} + +async function assertDesktopDiagnosticResultLayout(page: Page, dialogue: Locator, action: string): Promise { + const modal = page.locator(".modal-container").filter({ has: dialogue }).locator(".modal"); + const outerScroll = await modal.evaluate((element) => Math.max(0, element.scrollHeight - element.clientHeight)); + assert.ok( + outerScroll <= 1, + `The diagnostic modal must not need outer scrolling after ${action}; got ${outerScroll}px` + ); + await assertDiagnosticExportActionsWithinModal(page, dialogue, action); +} + async function assertMobileRadioTargets(page: Page, dialogue: Locator, name: string): Promise { const radios = dialogue.getByRole("group", { name, exact: true }).getByRole("radio"); for (let index = 0; index < (await radios.count()); index++) { @@ -113,11 +163,174 @@ async function installDiagnosticExportInterception(page: Page): Promise { async function restoreDiagnosticReadGate(cli: string, session: ObsidianLiveSyncSession): Promise { await evalObsidianJson( cli, - `(async()=>{const gate=globalThis.databaseDiagnosticReadGate;if(gate){gate.release();const db=app.plugins.plugins['obsidian-livesync'].core.localDatabase.localDatabase;db.get=gate.originalGet;delete globalThis.databaseDiagnosticReadGate;}return JSON.stringify(true);})()`, + `(async()=>{const gate=globalThis.databaseDiagnosticReadGate;if(gate){gate.release();if(gate.reached&&gate.drainPromise)await gate.drainPromise;const db=app.plugins.plugins['obsidian-livesync'].core.localDatabase.localDatabase;db.get=gate.originalGet;delete globalThis.databaseDiagnosticReadGate;}return JSON.stringify(true);})()`, session.cliEnv ); } +async function installDiagnosticReadGate( + cli: string, + session: ObsidianLiveSyncSession, + documentId: string, + readKind: "metadata" | "revision" +): Promise { + const condition = readKind === "metadata" ? "options?.conflicts" : "options?.rev"; + await evalObsidianJson( + cli, + `(async()=>{const db=app.plugins.plugins['obsidian-livesync'].core.localDatabase.localDatabase;const gate={reached:false,release:null,releasePromise:null,drain:null,drainPromise:null,originalGet:db.get};gate.releasePromise=new Promise(resolve=>{gate.release=resolve;});gate.drainPromise=new Promise(resolve=>{gate.drain=resolve;});globalThis.databaseDiagnosticReadGate=gate;db.get=async function(id,options){if(id===${JSON.stringify(documentId)}&&${condition}&&!gate.reached){gate.reached=true;try{await gate.releasePromise;return await gate.originalGet.call(this,id,options);}finally{gate.drain();}}return await gate.originalGet.call(this,id,options);};return JSON.stringify(true);})()`, + session.cliEnv + ); +} + +async function assertIntegrityProgress(dialogue: Locator): Promise { + const progress = dialogue.locator(".diagnostic-progress"); + await progress.getByText("Metadata · local", { exact: true }).waitFor({ state: "visible", timeout }); + const progressBar = progress.getByRole("progressbar"); + assert.equal(await progressBar.count(), 1, "Metadata inspection must expose a progressbar"); + assert.equal(await progressBar.isVisible(), true); + const numericText = progress.getByText(/^[1-9]\d* \/ \d+ documents?$/); + await numericText.waitFor({ state: "visible", timeout }); + const numeric = await numericText.innerText(); + const match = /^(\d+) \/ (\d+) documents?$/.exec(numeric); + assert.ok(match, `Metadata progress must report its document count; got ${numeric}`); + const [, processedText, totalText] = match; + const processed = Number(processedText); + const total = Number(totalText); + assert.ok(processed > 0, `Metadata progress must report a positive processed count; got ${numeric}`); + assert.ok(total >= processed, `Metadata progress must report its total; got ${numeric}`); +} + +async function assertHatchReportingGroups(hatch: Locator): Promise { + const reportingHeading = hatch.getByRole("heading", { name: "Reporting Tools", exact: true }); + const checksHeading = hatch.getByRole("heading", { name: "Checks and Recovery", exact: true }); + assert.equal(await reportingHeading.count(), 1); + assert.equal(await checksHeading.count(), 1); + const reportingPanel = reportingHeading.locator(".."); + const checksPanel = checksHeading.locator(".."); + const reportingNames = [ + "Prepare the 'report' to create an issue", + "Copy database information for a file", + "Find a file from a report pseudonym", + "Inspect database integrity", + "Analyse database usage", + ]; + assert.equal(await reportingPanel.locator(".setting-item").count(), reportingNames.length); + for (const name of reportingNames) { + assert.equal( + await reportingPanel.getByText(name, { exact: true }).count(), + 1, + `${name} belongs in Reporting Tools` + ); + assert.equal( + await checksPanel.getByText(name, { exact: true }).count(), + 0, + `${name} must remain outside Checks and Recovery` + ); + } + assert.equal(await checksPanel.locator(".setting-item").count(), 2); + for (const name of ["Run Doctor", "Scan for Broken files"]) { + const button = checksPanel.getByRole("button", { name, exact: true }); + assert.equal(await button.count(), 1, `${name} belongs in Checks and Recovery`); + assert.equal(await reportingPanel.getByRole("button", { name, exact: true }).count(), 0); + } +} + +async function assertDiagnosticFileLookup( + page: Page, + hatch: Locator, + token: string, + shadowRevision: string, + session: ObsidianLiveSyncSession, + mobile = false +): Promise { + const setting = hatch.locator(".setting-item").filter({ + has: page.getByText("Find a file from a report pseudonym", { exact: true }), + }); + await setting.getByRole("button", { name: "Find file", exact: true }).click(); + const dialogue = page.locator(".diagnostic-file-lookup"); + await dialogue.waitFor({ state: "visible", timeout }); + const input = dialogue.getByLabel("Report path pseudonym", { exact: true }); + await input.fill("file-001"); + await dialogue.getByRole("button", { name: "Find file", exact: true }).click(); + await dialogue.getByText("Enter a valid report path pseudonym.", { exact: true }).waitFor({ timeout }); + await input.fill(token); + await dialogue.getByRole("button", { name: "Find file", exact: true }).click(); + const candidates = dialogue.locator("[data-diagnostic-candidate]"); + await candidates.first().waitFor({ timeout }); + assert.equal(await candidates.count(), 2, "Distinct Metadata IDs claiming the same path must remain selectable"); + if (mobile) { + await assertNoHorizontalOverflow(page, dialogue, { label: "Pseudonym file lookup matches" }); + await assertNoHorizontalOverflow(page, dialogue.locator(".lookup-candidates"), { + label: "Pseudonym file lookup candidate list", + }); + assert.ok( + (await dialogue.locator(".lookup-candidates").evaluate((element) => element.clientHeight)) >= 160, + "The mobile lookup must leave enough room to read the matching file details" + ); + await captureObsidianElement( + session.remoteDebuggingPort, + "database-diagnostics-file-matches-mobile.png", + (page) => page.locator(".modal-container").filter({ has: page.locator(".diagnostic-file-lookup") }) + ); + } + const shadow = candidates.filter({ hasText: shadowId }); + assert.equal(await shadow.count(), 1); + assert.ok((await shadow.textContent())?.includes(fixturePath)); + await shadow.getByRole("button", { name: "Open file", exact: true }).click(); + await page.waitForFunction(`app.workspace.getActiveFile()?.path === ${JSON.stringify(fixturePath)}`, undefined, { + timeout, + }); + await shadow.getByRole("button", { name: "Prepare follow-up report", exact: true }).click(); + await dialogue.locator("textarea").waitFor({ timeout }); + const json = await dialogue.locator("textarea").inputValue(); + const report = JSON.parse(json); + const parsed = parseDiagnosticPathToken(token); + assert.ok(parsed); + const mapper = createDiagnosticNameMapper("pseudonym", parsed.salt); + assert.equal(report.file.path, token); + assert.equal(report.file.database.documentId, mapper.id(shadowId)); + assert.equal(report.file.database.currentRevision, mapper.revision(shadowRevision)); + assert.equal(report.file.database.remoteQueried, false); + for (const privateValue of [fixturePath, shadowId, fixtureContent.trim()]) { + assert.ok(!json.includes(privateValue), "A follow-up report must preserve the original report's pseudonyms"); + } + await installDiagnosticExportInterception(page); + await dialogue.getByRole("button", { name: "Copy report", exact: true }).click(); + assert.equal(await page.evaluate(() => (globalThis as DiagnosticPageGlobal).diagnosticClipboard), json); + await dialogue.getByRole("button", { name: "Save report", exact: true }).click(); + await page.waitForFunction(() => !!(globalThis as DiagnosticPageGlobal).diagnosticDownload, undefined, { timeout }); + assert.deepEqual(await page.evaluate(() => (globalThis as DiagnosticPageGlobal).diagnosticDownload), { + name: "diagnostic-file-report.json", + content: json, + }); + await assertDesktopDiagnosticResultLayout(page, dialogue, "saving a follow-up report"); + if (mobile) { + const modal = page.locator(".modal-container").filter({ has: dialogue }); + await assertMobileDialogueLayout(page, modal, "Pseudonym file follow-up report"); + await assertNoHorizontalOverflow(page, dialogue, { label: "Pseudonym file follow-up content" }); + await assertNoHorizontalOverflow(page, dialogue.locator(".report-token"), { + label: "Pseudonym file follow-up token", + }); + } + await captureObsidianElement( + session.remoteDebuggingPort, + mobile ? "database-diagnostics-file-follow-up-mobile.png" : "database-diagnostics-file-follow-up.png", + (page) => page.locator(".modal-container").filter({ has: page.locator(".diagnostic-file-lookup") }) + ); + await dialogue.getByRole("radio", { name: "MARKDOWN", exact: true }).check(); + const markdown = await dialogue.locator("textarea").inputValue(); + assert.ok(markdown.startsWith("
\n")); + assert.ok(markdown.includes(json)); + await dialogue.getByRole("button", { name: "Back to matches", exact: true }).click(); + await candidates.first().waitFor({ timeout }); + await input.fill(createDiagnosticNameMapper("pseudonym", parsed.salt).path("not-present.md")); + await dialogue.getByRole("button", { name: "Find file", exact: true }).click(); + await dialogue.getByText(/No matches were found in current Vault paths/).waitFor({ timeout }); + assert.equal(await candidates.count(), 0); + await dialogue.getByRole("button", { name: "Close", exact: true }).click(); + await dialogue.waitFor({ state: "hidden", timeout }); +} + async function main(): Promise { const binary = requireObsidianBinary(); const cli = requireObsidianCli(); @@ -161,15 +374,17 @@ async function main(): Promise { session.cliEnv ); await waitForLocalDatabaseEntry(cli, session.cliEnv, fixturePath); - await evalObsidianJson( + const rawFixture = await evalObsidianJson<{ id: string; revision: string; shadowRevision: string }>( cli, - `(async()=>{const core=app.plugins.plugins['obsidian-livesync'].core;await core.services.fileProcessing.commitPendingFileEvents();const db=core.localDatabase.localDatabase;const id=await core.services.path.path2id(${JSON.stringify(fixturePath)});const original=await db.get(id);const shadow={...original,_id:${JSON.stringify(shadowId)}};delete shadow._rev;await db.put(shadow);return JSON.stringify(true);})()`, + `(async()=>{const core=app.plugins.plugins['obsidian-livesync'].core;await core.services.fileProcessing.commitPendingFileEvents();const db=core.localDatabase.localDatabase;const id=await core.services.path.path2id(${JSON.stringify(fixturePath)});const original=await db.get(id);const shadow={...original,_id:${JSON.stringify(shadowId)}};delete shadow._rev;const stored=await db.put(shadow);return JSON.stringify({id,revision:original._rev,shadowRevision:stored.rev});})()`, session.cliEnv ); const before = await readDatabaseState(cli, session); + const activeSession = session; let usageJSON = ""; let usageFixtureRow: { path: string; id: string; revision: string } | undefined; await withObsidianPage(session.remoteDebuggingPort, async (page) => { + await page.setViewportSize({ width: 800, height: 600 }); const choice = await openCommand(page, "usage"); await choice.getByRole("radio", { name: "Pseudonyms (recommended)", exact: true }).focus(); await page.keyboard.press("ArrowRight"); @@ -188,6 +403,7 @@ async function main(): Promise { ); await dialogue.getByRole("button", { name: "Begin inspection", exact: true }).click(); await page.locator('[data-diagnostic-phase="result"]').waitFor({ timeout }); + await assertDesktopDiagnosticResultLayout(page, dialogue, "the result appearing"); await assertFormatChoices(dialogue, ["TSV", "JSON"]); assert.equal(await dialogue.getByRole("radio", { name: "TSV", exact: true }).isChecked(), true); const tsv = await dialogue.locator("textarea").inputValue(); @@ -202,17 +418,24 @@ async function main(): Promise { const report = JSON.parse(usageJSON); assert.equal(report.status, "complete"); assert.ok(report.versions.commonlib); - assert.ok(report.rows.some((row: { path: string }) => row.path.startsWith("file-"))); - const fixtureRow = report.rows.find((row: { path: string; id: string; revision: string }) => - row.path.startsWith("file-") + const token = report.rows.find((row: { path: string }) => parseDiagnosticPathToken(row.path))?.path; + const parsedToken = parseDiagnosticPathToken(token ?? ""); + assert.ok(parsedToken, "The report must contain searchable salted path pseudonyms"); + const mapper = createDiagnosticNameMapper("pseudonym", parsedToken.salt); + const fixtureRow = report.rows.find( + (row: { path: string; id: string; revision: string }) => row.id === mapper.id(rawFixture.id) ); assert.ok(fixtureRow); + assert.equal(fixtureRow.path, mapper.path(fixturePath)); + assert.equal(fixtureRow.revision, mapper.revision(rawFixture.revision)); + assert.equal(fixtureRow.revision, `${fixtureRow.rev}-${fixtureRow.revHash}`); usageFixtureRow = { path: fixtureRow.path, id: fixtureRow.id, revision: fixtureRow.revision }; // Capture exports inside the isolated renderer without changing the // host clipboard or leaving downloads in a user's download directory. await installDiagnosticExportInterception(page); await dialogue.getByRole("button", { name: "Copy report", exact: true }).click(); + await assertDesktopDiagnosticResultLayout(page, dialogue, "copying the report"); assert.equal( await page.evaluate(() => (globalThis as DiagnosticPageGlobal).diagnosticClipboard), usageJSON @@ -221,6 +444,7 @@ async function main(): Promise { await page.waitForFunction(() => !!(globalThis as DiagnosticPageGlobal).diagnosticDownload, undefined, { timeout, }); + await assertDesktopDiagnosticResultLayout(page, dialogue, "saving the report"); assert.deepEqual(await page.evaluate(() => (globalThis as DiagnosticPageGlobal).diagnosticDownload), { name: "database-usage-complete.json", content: usageJSON, @@ -240,10 +464,69 @@ async function main(): Promise { await original.waitFor({ state: "hidden", timeout }); const settingsNavigator = await openLiveSyncSettings(page, timeout); - const hatch = await settingsNavigator.openPage("Hatch"); + let hatch = await settingsNavigator.openPage("Hatch"); + await assertHatchReportingGroups(hatch); + assert.ok(usageFixtureRow); + await assertDiagnosticFileLookup( + page, + hatch, + usageFixtureRow.path, + rawFixture.shadowRevision, + activeSession + ); + hatch = await (await openLiveSyncSettings(page, timeout)).openPage("Hatch"); + await captureObsidianElement(activeSession.remoteDebuggingPort, "database-diagnostics-hatch.png", (page) => + page.locator(".modal.mod-settings") + ); const setting = hatch .locator(".setting-item") .filter({ has: page.getByText("Inspect database integrity", { exact: true }) }); + await setting.getByRole("button", { name: "Inspect", exact: true }).click(); + const progressDialogue = page.locator(".database-diagnostic"); + assert.equal( + await progressDialogue + .getByRole("radio", { name: "Pseudonyms (recommended)", exact: true }) + .isChecked(), + true + ); + assert.equal( + await progressDialogue + .getByRole("checkbox", { name: "Include remote database", exact: true }) + .isChecked(), + true + ); + await assertDesktopDiagnosticChoiceLayout(page, progressDialogue); + await installDiagnosticReadGate(cli, activeSession, shadowId, "metadata"); + try { + await progressDialogue.getByRole("button", { name: "Begin inspection", exact: true }).click(); + await page.waitForFunction( + () => (globalThis as DiagnosticPageGlobal).databaseDiagnosticReadGate?.reached === true, + undefined, + { timeout } + ); + assert.equal(await progressDialogue.getAttribute("data-diagnostic-phase"), "running"); + await assertIntegrityProgress(progressDialogue); + await captureObsidianElement( + activeSession.remoteDebuggingPort, + "database-diagnostics-progress.png", + (page) => page.locator(".modal-container").filter({ has: page.locator(".database-diagnostic") }) + ); + await progressDialogue.getByRole("button", { name: "Cancel inspection", exact: true }).click(); + await evalObsidianJson( + cli, + `(async()=>{globalThis.databaseDiagnosticReadGate?.release();return JSON.stringify(true);})()`, + activeSession.cliEnv + ); + await page.locator('[data-diagnostic-phase="result"]').waitFor({ timeout }); + await assertFormatChoices(progressDialogue, ["MARKDOWN", "JSON"]); + await chooseFormat(progressDialogue, "json"); + assert.equal(JSON.parse(await progressDialogue.locator("textarea").inputValue()).status, "cancelled"); + await progressDialogue.getByRole("button", { name: "Close", exact: true }).click(); + await progressDialogue.waitFor({ state: "hidden", timeout }); + } finally { + await restoreDiagnosticReadGate(cli, activeSession); + } + await setting.getByRole("button", { name: "Inspect", exact: true }).click(); const integrity = page.locator(".database-diagnostic"); assert.equal( @@ -254,10 +537,40 @@ async function main(): Promise { await page.locator('[data-diagnostic-phase="result"]').waitFor({ timeout }); await assertFormatChoices(integrity, ["MARKDOWN", "JSON"]); assert.equal(await integrity.getByRole("radio", { name: "MARKDOWN", exact: true }).isChecked(), true); + const markdownReport = await integrity.locator("textarea").inputValue(); + const markdownDetailsWithCodeBlock = + /
\s*Database integrity report \(partial\)<\/summary>\s*(`{3,})text\r?\n[\s\S]*?\1\s*<\/details>/; + assert.match(markdownReport, markdownDetailsWithCodeBlock); + await installDiagnosticExportInterception(page); + await integrity.getByRole("button", { name: "Copy report", exact: true }).click(); + assert.equal( + await page.evaluate(() => (globalThis as DiagnosticPageGlobal).diagnosticClipboard), + markdownReport + ); + await integrity.getByRole("button", { name: "Save report", exact: true }).click(); + await page.waitForFunction(() => !!(globalThis as DiagnosticPageGlobal).diagnosticDownload, undefined, { + timeout, + }); + assert.deepEqual(await page.evaluate(() => (globalThis as DiagnosticPageGlobal).diagnosticDownload), { + name: "database-integrity-partial.md", + content: markdownReport, + }); await chooseFormat(integrity, "json"); const reportText = await integrity.locator("textarea").inputValue(); const integrityReport = JSON.parse(reportText); assert.equal(integrityReport.status, "partial"); + assert.ok(markdownReport.includes("## Reading the findings")); + assert.ok(markdownReport.includes("## State counts")); + assert.ok(markdownReport.includes("## Review index")); + assert.ok(markdownReport.includes("chunks=total/unique/embedded")); + assert.equal(markdownReport.match(/^### /gm)?.length, integrityReport.observations.length); + for (const observation of integrityReport.observations) { + assert.ok(markdownReport.includes(`### \`${observation.reference}\` source=${observation.source}`)); + assert.ok(markdownReport.includes(`path: \`${observation.path}\``)); + assert.ok(markdownReport.includes(`doc: \`${observation.id}\``)); + if (observation.revision !== undefined) + assert.ok(markdownReport.includes(`rev=\`${observation.revision}\``)); + } assert.ok( integrityReport.errors.some((error: { reason: string }) => error.reason === "source-unavailable") ); @@ -282,6 +595,27 @@ async function main(): Promise { await withObsidianPage(session.remoteDebuggingPort, async (page) => { await page.locator(".database-diagnostic").getByRole("button", { name: "Close", exact: true }).click(); await page.keyboard.press("Escape"); + const localOnly = await openCommand(page, "integrity"); + await localOnly.getByRole("checkbox", { name: "Include remote database", exact: true }).uncheck(); + await localOnly.getByRole("button", { name: "Begin inspection", exact: true }).click(); + await page.locator('[data-diagnostic-phase="result"]').waitFor({ timeout }); + await chooseFormat(localOnly, "json"); + const report = JSON.parse(await localOnly.locator("textarea").inputValue()); + assert.equal(report.status, "complete"); + assert.equal(report.scope.remote, "excluded"); + assert.equal(report.coverage.remote.status, "not-checked"); + assert.deepEqual(report.errors, []); + assert.ok( + report.observations.some( + (row: { identity: string; vault: { status: string }; comparisons: { relation: string }[] }) => + row.identity === "document-id-mismatch" && + row.vault.status === "available" && + row.comparisons.some((comparison) => comparison.relation === "equal") + ), + "Local discrepancies must retain Vault comparisons when the remote is excluded" + ); + await localOnly.getByRole("button", { name: "Close", exact: true }).click(); + await localOnly.waitFor({ state: "hidden", timeout }); }); const cancellationOrder = await evalObsidianJson<{ fixtureId: string; orderedIds: string[] }>( cli, @@ -294,15 +628,10 @@ async function main(): Promise { assert.ok(gatePosition >= 0, "The cancellation read gate must exist in the local database"); assert.ok(fixturePosition < gatePosition, "The cancellation read gate must follow the report fixture"); assert.ok(usageFixtureRow, "The complete report must contain the cancellation fixture"); - const expectedFixtureRow = usageFixtureRow; const cancellationEnv = session.cliEnv; try { - await evalObsidianJson( - cli, - `(async()=>{const db=app.plugins.plugins['obsidian-livesync'].core.localDatabase.localDatabase;const gate={reached:false,release:null,releasePromise:null,originalGet:db.get};gate.releasePromise=new Promise(resolve=>{gate.release=resolve;});globalThis.databaseDiagnosticReadGate=gate;db.get=async function(id,options){if(id===${JSON.stringify(shadowId)}&&options?.rev&&!gate.reached){gate.reached=true;await gate.releasePromise;}return await gate.originalGet.call(this,id,options);};return JSON.stringify(true);})()`, - session.cliEnv - ); + await installDiagnosticReadGate(cli, session, shadowId, "revision"); await withObsidianPage(session.remoteDebuggingPort, async (page) => { const dialogue = await openCommand(page, "usage"); await dialogue.getByRole("button", { name: "Begin inspection", exact: true }).click(); @@ -312,6 +641,10 @@ async function main(): Promise { { timeout } ); assert.equal(await dialogue.getAttribute("data-diagnostic-phase"), "running"); + const usageProgress = dialogue.locator(".diagnostic-progress"); + await usageProgress.getByText("Reading revisions", { exact: true }).waitFor({ timeout }); + await usageProgress.getByText(/^\d+ \/ \d+ revisions$/).waitFor({ timeout }); + await usageProgress.getByText(/Discovering revisions/).waitFor({ timeout }); await dialogue.getByRole("button", { name: "Cancel inspection", exact: true }).click(); await dialogue.getByText("Cancelling inspection…", { exact: true }).waitFor({ timeout }); await evalObsidianJson( @@ -324,12 +657,18 @@ async function main(): Promise { const cancelledReport = JSON.parse(cancelledText); assert.equal(cancelledReport.status, "cancelled"); await assertFormatChoices(dialogue, ["JSON"]); + const cancelledToken = cancelledReport.rows.find((row: { path: string }) => + parseDiagnosticPathToken(row.path) + )?.path; + const parsedToken = parseDiagnosticPathToken(cancelledToken ?? ""); + assert.ok(parsedToken); + const mapper = createDiagnosticNameMapper("pseudonym", parsedToken.salt); assert.ok( cancelledReport.rows.some( (row: { path: string; id: string; revision: string }) => - row.path === expectedFixtureRow.path && - row.id === expectedFixtureRow.id && - row.revision === expectedFixtureRow.revision + row.path === mapper.path(fixturePath) && + row.id === mapper.id(rawFixture.id) && + row.revision === mapper.revision(rawFixture.revision) ), "Cancellation must retain report rows collected before the gated database read" ); @@ -351,6 +690,7 @@ async function main(): Promise { await setObsidianMobileTestMode(session.remoteDebuggingPort, true, timeout); await waitForLiveSyncCoreReady(cli, session.cliEnv); await withObsidianPage(session.remoteDebuggingPort, async (page) => { + await page.setViewportSize({ width: 375, height: 667 }); await openCommand(page, "integrity"); }); const mobileScreenshot = await captureObsidianElement( @@ -359,18 +699,31 @@ async function main(): Promise { (page) => page.locator(".modal-container").filter({ has: page.locator(".database-diagnostic") }) ); await withObsidianPage(session.remoteDebuggingPort, async (page) => { + await page.setViewportSize({ width: 375, height: 667 }); const modal = page.locator(".modal-container").filter({ has: page.locator(".database-diagnostic") }); await assertMobileDialogueLayout(page, modal, "Database diagnostic choice"); await assertMobileRadioTargets(page, modal, "Names in this report"); + const remoteChoice = modal.getByRole("checkbox", { name: "Include remote database", exact: true }); + assert.equal(await remoteChoice.isChecked(), true); + await assertLocatorHasMinimumTouchTarget(page, remoteChoice.locator(".."), { + label: "Include remote database", + }); + await assertLocatorWithinSafeArea(page, remoteChoice.locator(".."), { + label: "Include remote database", + safeAreaInsets: iPhoneSafeArea, + }); + await assertNoHorizontalOverflow(page, remoteChoice.locator(".."), { label: "Include remote database" }); await modal.getByRole("button", { name: "Cancel", exact: true }).click(); await modal.waitFor({ state: "hidden", timeout }); }); await withObsidianPage(session.remoteDebuggingPort, async (page) => { + await page.setViewportSize({ width: 375, height: 667 }); const dialogue = await openCommand(page, "usage"); await dialogue.getByRole("button", { name: "Begin inspection", exact: true }).click(); await page.locator('[data-diagnostic-phase="result"]').waitFor({ timeout }); const modal = page.locator(".modal-container").filter({ has: page.locator(".database-diagnostic") }); await assertMobileDialogueLayout(page, modal, "Database diagnostic result"); + await assertDiagnosticExportActionsWithinModal(page, dialogue, "the mobile result appearing"); await assertFormatChoices(dialogue, ["TSV", "JSON"]); await assertMobileRadioTargets(page, dialogue, "Report format"); @@ -394,6 +747,7 @@ async function main(): Promise { (page) => page.locator(".modal-container").filter({ has: page.locator(".database-diagnostic") }) ); await withObsidianPage(session.remoteDebuggingPort, async (page) => { + await page.setViewportSize({ width: 375, height: 667 }); const dialogue = page.locator(".database-diagnostic"); await installDiagnosticExportInterception(page); const reportText = await dialogue.locator("textarea").inputValue(); @@ -401,6 +755,7 @@ async function main(): Promise { await dialogue.getByText("Report copied.", { exact: true }).waitFor({ timeout }); const modal = page.locator(".modal-container").filter({ has: page.locator(".database-diagnostic") }); await assertMobileDialogueLayout(page, modal, "Database diagnostic copied result"); + await assertDiagnosticExportActionsWithinModal(page, dialogue, "copying the mobile report"); assert.equal( await page.evaluate(() => (globalThis as DiagnosticPageGlobal).diagnosticClipboard), reportText @@ -414,9 +769,35 @@ async function main(): Promise { content: reportText, }); await assertMobileDialogueLayout(page, modal, "Database diagnostic saved result"); + await assertDiagnosticExportActionsWithinModal(page, dialogue, "saving the mobile report"); + await captureObsidianElement( + activeSession.remoteDebuggingPort, + "database-diagnostics-mobile-saved.png", + (page) => page.locator(".modal-container").filter({ has: page.locator(".database-diagnostic") }) + ); await dialogue.getByRole("button", { name: "Close", exact: true }).click(); await dialogue.waitFor({ state: "hidden", timeout }); }); + await withObsidianPage(session.remoteDebuggingPort, async (page) => { + await page.setViewportSize({ width: 375, height: 667 }); + const settings = await openLiveSyncSettings(page, timeout); + const hatch = await settings.openPage("Hatch"); + assert.ok(usageFixtureRow); + await assertDiagnosticFileLookup( + page, + hatch, + usageFixtureRow.path, + rawFixture.shadowRevision, + activeSession, + true + ); + await settings.close(); + }); + assert.deepEqual( + await readDatabaseState(cli, session), + before, + "File lookup and follow-up reports remain read-only" + ); console.log( JSON.stringify({ status: "passed", diff --git a/updates.md b/updates.md index dcdc8352..394b9323 100644 --- a/updates.md +++ b/updates.md @@ -21,8 +21,12 @@ Earlier releases remain available in the 1.0 release history, the 1.0 preview hi ### Diagnostics -- We can inspect normal-file Metadata and current conflicts in the local database and CouchDB together, with byte comparisons for suspect groups. Partial results retain the available evidence when a source is unavailable. The inspection leaves databases and settings unchanged. -- Database Usage analysis and integrity inspection now offer temporary pseudonyms or original names before each run. Both name modes and the available report formats stay visible as radio buttons. Reports omit file contents and credentials in both modes. Complete Usage runs retain the existing TSV columns; partial runs use JSON to retain their status and reasons. +- We can inspect normal-file Metadata and current conflicts in the local database and CouchDB together, with byte comparisons for suspect groups. A per-run option excludes the remote database when a local database and Vault inspection is enough; reports identify that scope. Partial results retain the available evidence when a source is unavailable. The inspection leaves databases and settings unchanged. +- Database Usage analysis and integrity inspection now offer temporary pseudonyms or original names before each run. Both name modes and the available report formats stay visible as radio buttons. Pseudonym reports can be followed up with the same public salt; the local Hatch lookup searches the current Vault and local normal-file Metadata and can prepare a report for a selected document. Reports omit file contents and credentials in both modes. Complete Usage runs retain the existing TSV columns; partial runs use JSON to retain their status and reasons. +- Revision pseudonyms retain their generation number. Integrity reports include existing size, modification-time, Chunk-reference, and case/NFC relationship evidence without additional database or Vault reads. +- We can review an integrity report more quickly in Markdown: a Review index groups references by identity, conflict, path, availability, and measured-difference findings before the observations. The compact display keeps each observation and its full follow-up references; the JSON report is unchanged. +- Usage progress shows the growing number of available revisions during reading, then a separate cancellable accounting stage. Integrity inspection reduces remote requests with bulk reads and bounded concurrency, shows progress counts and a gauge for each stage, and records document-read and HTTP-request timing and counts in the log. Markdown reports include a collapsible code block for sharing in issues. +- Hatch groups the five reporting tools separately from checks, recovery, and reset operations. Diagnostic dialogues resize their report preview to keep the controls visible in smaller windows. ## 1.0.38