public/obsidian-livesync
1
0
Fork 0
mirror of https://github.com/vrtmrz/obsidian-livesync.git synced 2026-07-01 10:25:19 +00:00
Code Issues Packages Projects Releases Wiki Activity

(test): add local Obsidian E2E suite

Browse Source
This commit is contained in:
vorotamoroz
2026-06-30 08:36:29 +00:00
parent e916683b8d
commit 54e6a761e5
18 changed files with 975 additions and 119 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/adr/2026_06_real_obsidian_e2e.md
+37 -17
View File
@@ -125,12 +125,12 @@ Initial discovery on Linux ARM64 found that:
Current implementation status:
- Added `test/e2e-obsidian/runner` helpers for Obsidian discovery, CLI discovery, temporary vault creation, plug-in installation, process launch, CLI execution, and readiness polling.
- Added `test:e2e:obsidian:discover`, `test:e2e:obsidian:cli-help`, `test:e2e:obsidian:smoke`, `test:e2e:obsidian:vault-reflection`, `test:e2e:obsidian:couchdb-upload`, `test:e2e:obsidian:startup-scan`, `test:e2e:obsidian:two-vault-sync`, `test:e2e:obsidian:hidden-file-snippet-sync`, `test:e2e:obsidian:customisation-sync`, `test:e2e:obsidian:setting-markdown-export`, and `test:e2e:obsidian:install-appimage`.
- Added `startObsidianLiveSyncSession()` so future workflows can reuse the launch, vault open, community plug-in enablement, plug-in reload, and readiness sequence without duplicating smoke runner code.
- Added `test:e2e:obsidian:discover`, `test:e2e:obsidian:cli-help`, `test:e2e:obsidian:smoke`, `test:e2e:obsidian:vault-reflection`, `test:e2e:obsidian:couchdb-upload`, `test:e2e:obsidian:minio-upload`, `test:e2e:obsidian:startup-scan`, `test:e2e:obsidian:two-vault-sync`, `test:e2e:obsidian:hidden-file-snippet-sync`, `test:e2e:obsidian:customisation-sync`, `test:e2e:obsidian:setting-markdown-export`, `test:e2e:obsidian:local-suite`, `test:e2e:obsidian:local-suite:services`, and `test:e2e:obsidian:install-appimage`.
- Added `startObsidianLiveSyncSession()` so future workflows can reuse the launch, trusted temporary vault state, vault open, community plug-in reload, and readiness sequence without duplicating smoke runner code.
- Added CouchDB runner utilities that reuse `.test.env`/process environment values, create unique temporary databases, query uploaded documents directly, and clean up the database unless `E2E_OBSIDIAN_KEEP_COUCHDB=true` is set.
- Added a manual AppImage installer that downloads Obsidian `1.12.7` for `arm64` or `x86_64`, stores it under `_testdata/obsidian`, and extracts it for FUSE-free execution.
- Confirmed the smoke runner on Linux ARM64 with the extracted Obsidian `1.12.7` AppImage, `xvfb-run`, and the built Self-hosted LiveSync bundle.
- Confirmed the runner can enable the Obsidian CLI through isolated `obsidian.json` state, open the temporary vault through `obsidian-cli`, enable community plug-ins through `app.plugins.setEnable(true)`, reload Self-hosted LiveSync, and verify readiness through `obsidian-cli eval`.
- Confirmed the runner can enable the Obsidian CLI through isolated `obsidian.json` state, pre-seed the temporary Chromium local storage so the generated vault ID is trusted for community plug-ins, open the temporary vault through `obsidian-cli`, reload Self-hosted LiveSync, and verify readiness through `obsidian-cli eval`.
- Removed the first test-only ready-marker bridge from the plug-in bundle. The current runner observes readiness from outside the plug-in through Obsidian's own CLI, so normal user vaults do not receive E2E marker files.
Current verification:
@@ -141,18 +141,22 @@ Current verification:
- `E2E_OBSIDIAN_SMOKE_TIMEOUT_MS=1000 npm run test:e2e:obsidian:smoke` passes locally.
- `npm run test:e2e:obsidian:vault-reflection` creates a note through Obsidian's vault API, verifies the reflected file on disk, and reads it back through Obsidian.
- `npm run test:e2e:obsidian:couchdb-upload` configures a unique CouchDB database, creates a note through Obsidian, commits it into the local database, runs one-shot synchronisation, and verifies that CouchDB contains the metadata document and all referenced chunk documents.
- `npm run test:e2e:obsidian:minio-upload` configures a unique Object Storage bucket prefix, creates a note through Obsidian, runs one-shot Journal Sync, and verifies through the AWS SDK that objects were written to the S3-compatible bucket.
- `npm run test:e2e:obsidian:startup-scan` verifies that a file written while Obsidian is stopped is picked up during the next real Obsidian boot and uploaded to CouchDB after one-shot synchronisation.
- `npm run test:e2e:obsidian:two-vault-sync` verifies two-vault note synchronisation: creation, update, deletion, Markdown conflict automatic merging with the merged result propagated by a second synchronisation, and per-device target-filter differences.
- `npm run test:e2e:obsidian:two-vault-sync` verifies two-vault note synchronisation: creation, update, rename, deletion, per-device target-filter differences, and a separate encrypted round-trip with Path Obfuscation enabled. The experimental Markdown conflict automatic merge check is available with `E2E_OBSIDIAN_INCLUDE_MARKDOWN_CONFLICT=true` but is not part of the default local suite.
- `npm run test:e2e:obsidian:hidden-file-snippet-sync` verifies hidden file synchronisation as a two-vault round-trip: creation, deletion, automatic JSON conflict merging with the merged result propagated by a second synchronisation, manual JSON Resolve dialogue application through Obsidian's UI, and per-device target-pattern differences.
- `npm run test:e2e:obsidian:customisation-sync` verifies a two-vault Customisation Sync workflow: scan a real snippet CSS file, config JSON file, and sample plug-in fixture into per-file Customisation Sync data, synchronise them through CouchDB, apply them on the second vault, assert the resulting `.obsidian` files, propagate a snippet update, and verify deletion of the source-vault snippet sync data without confusing it with the target vault's own applied copy.
- `npm run test:e2e:obsidian:setting-markdown-export` verifies that setting Markdown export creates a vault file and omits credentials when credential export is disabled.
- `npm run test:e2e:obsidian:install-appimage` reuses the existing AppImage and extracted binary when they are already present.
- `npm run test:e2e:obsidian:local-suite` runs the local verification sequence for the real Obsidian runner after CouchDB and MinIO have been started.
- `npm run test:e2e:obsidian:local-suite:services` stops leftover CouchDB and MinIO fixtures, starts fresh fixtures, runs the local suite, and stops the fixtures again.
- `npm run test:e2e:obsidian:local-suite:services` has been verified locally with real Obsidian, CouchDB, and MinIO. The run completed discovery, smoke, vault reflection, CouchDB upload, Object Storage upload, startup scan, two-vault synchronisation, Hidden File Sync, Customisation Sync, and setting Markdown export. The build step still emits existing Svelte warnings.
Known limits:
- The smoke runner currently proves only one-vault launch and plug-in load readiness. Broader workflows are covered by separate real Obsidian scripts, including CouchDB upload, startup scan, two-vault note synchronisation, Hidden File Sync, Customisation Sync, and setting Markdown export.
- Cross-platform support is still discovery-level. The working path has been validated on Linux ARM64.
- CI wiring is not yet implemented. CI should use `OBSIDIAN_BINARY` or a cached `_testdata/obsidian/squashfs-root` rather than downloading the AppImage on every run.
- Cross-platform support is still discovery-level. The working path has been validated on Linux ARM64. macOS and Windows should be validated in their own environments as follow-up work.
- CI wiring is intentionally not implemented. The runner depends on a licensed desktop application and is treated as a local verification tool.
### Phase 2: First Real Workflow
@@ -169,6 +173,7 @@ Current implementation status:
- Added a pre-CouchDB workflow that creates a note through Obsidian's vault API, confirms the note is reflected as a real vault file, and reads the same note back through Obsidian. This covers the vault reflection part of the Phase 2 path before remote database setup is introduced.
- Added a first CouchDB-backed upload workflow, modelled after the CLI Deno tests: reuse the standard CouchDB environment variables, create a unique remote database, apply CouchDB settings through the plug-in's setting service, commit the note through the real Obsidian vault path, run one-shot synchronisation, and assert that remote metadata and chunks exist.
- Added an Object Storage-backed upload workflow against MinIO to exercise Journal Sync and the AWS SDK path from real Obsidian.
- Added Obsidian-specific workflows for boot-time vault scanning, two-vault note synchronisation, hidden `.obsidian/snippets` file round-tripping, hidden JSON conflict resolution, Customisation Sync application for snippets, config JSON files, and plug-in fixtures, per-device target-filter differences, and setting Markdown export. These scenarios assert against CouchDB documents, vault files, or real Obsidian UI outcomes instead of internal service state.
### Phase 3: Two-Vault Synchronisation
@@ -179,39 +184,53 @@ Current implementation status:
- Verify reflection in the other vault.
- Cover encrypted and non-encrypted configurations separately.
Current implementation status:
- `test:e2e:obsidian:two-vault-sync` covers creation, update, rename, deletion, and per-device target-filter behaviour for a non-encrypted CouchDB configuration. Markdown conflict automatic merging remains an optional check because it needs a dedicated, less timing-sensitive fixture.
- The same script creates a separate temporary CouchDB database and temporary vault pair for an encrypted two-vault round-trip with Path Obfuscation enabled.
### Phase 4: Harness Retirement
- Mark `test/harness` as deprecated in documentation.
- Stop adding new tests to `test/suite` unless they are explicitly transitional.
- Move critical existing scenarios from `test/suite` to real Obsidian E2E or lower-level integration tests.
- Do not mechanically port `test/suite` into real Obsidian E2E. Scenarios that can already be exercised and asserted through the CLI test layer should stay there or move to lower-level integration tests.
- Prioritise real Obsidian coverage for behaviours that the CLI cannot prove well, especially RedFlag flag-file recovery flows, Fast Setup (Simple Fetch), boot-up sequencing, restart-sensitive initial synchronisation, and user-visible recovery dialogues.
- Remove the harness only after the new runner covers the critical boot-up and synchronisation workflows.
## CI Strategy
Current implementation status:
Start with local-only execution. After the smoke runner is stable:
- `test/harness` is now documented as a transitional compatibility layer.
- New broad E2E work should target `test/e2e-obsidian/` when real Obsidian behaviour is the risk being tested.
- The next high-value scenarios are RedFlag variants and Fast Setup (Simple Fetch) variants, not a line-by-line migration of `test/suite`.
- Run the smoke test in CI on Linux.
- Keep full two-vault synchronisation scenarios as nightly or manually triggered jobs until runtime and flakiness are understood.
- Do not download the Obsidian AppImage on every CI run. Use a pre-installed Obsidian binary, a CI cache for `_testdata/obsidian/squashfs-root`, or a manually triggered preparation job.
- Capture Obsidian logs, plug-in logs, vault snapshots, and service logs on failure.
## Local Verification Strategy
Real Obsidian E2E is a local verification layer. It should not be wired into the default CI gate.
- Keep the scripts individually runnable for focused local debugging.
- Provide `test:e2e:obsidian:local-suite` for a broader local pass after the CouchDB and MinIO fixtures have been started.
- Provide `test:e2e:obsidian:local-suite:services` for a broader local pass that manages the CouchDB and MinIO fixtures itself.
- Use `OBSIDIAN_BINARY` when testing against an installed desktop application.
- Use `test:e2e:obsidian:install-appimage` on Linux when a local AppImage copy is needed, and reuse the extracted `_testdata/obsidian/squashfs-root` directory between local runs.
- Capture Obsidian logs, plug-in logs, vault snapshots, and service logs manually when investigating failures.
- Fail fast on launch failures, readiness timeouts, and cleanup failures with clear diagnostics.
## Risks and Mitigations
- **Obsidian licensing and installation**: CI may need a cached installer or a pre-installed binary. Keep the runner capable of using `OBSIDIAN_BINARY`.
- **Obsidian licensing and installation**: Keep the runner local-first and capable of using `OBSIDIAN_BINARY`.
- **Flakiness from UI timing**: Prefer a control channel and service-level probes over visual selectors.
- **Multiple instances**: Obsidian may not support multiple independent instances cleanly on all platforms. Start with one-instance smoke tests, then validate two-instance behaviour on Linux before expanding scope.
- **State leakage**: Isolate vault directories, Obsidian user data, remote database names, and bridge tokens per test.
- **Security of E2E controls**: Keep readiness and control outside the production plug-in bundle. Prefer Obsidian CLI probes over E2E-only plug-in code.
- **Runtime cost**: Keep the default PR gate small. Move slow synchronisation matrices to scheduled jobs.
- **Runtime cost**: Keep real Obsidian E2E out of the default PR gate. Use focused scripts or the local suite when a change touches real Obsidian integration.
## Open Questions
- Which launch mechanism is most reliable for Obsidian on Linux in this repository's CI environment?
- Which launch mechanism is most reliable for Obsidian on each supported desktop platform?
- Can two Obsidian instances run with isolated user data at the same time?
- Do future scenarios need a richer control channel than Obsidian CLI, or can CLI `eval` and developer commands cover the required workflows?
- Should any future E2E-only plug-in code live in a separate test build, or should the production bundle remain free of E2E controls?
- Which existing `test/suite` scenarios are critical enough to port before deprecating the harness?
- Which RedFlag and Fast Setup (Simple Fetch) variants should be added first?
## Initial Implementation Checklist
@@ -222,6 +241,7 @@ Start with local-only execution. After the smoke runner is stable:
5. Document required local environment variables, especially `OBSIDIAN_BINARY`.
6. Port one CouchDB-backed workflow after the smoke test is stable.
7. Mark `test/harness` as transitional and block new broad E2E work from targeting it.
8. Add the local suite script for broader local verification.
## Consequences