mirror of
https://github.com/vrtmrz/obsidian-livesync.git
synced 2026-07-25 05:53:00 +00:00
Protect conflict chunks during garbage collection
This commit is contained in:
vorotamoroz
+1
-1
@@ -69,7 +69,7 @@ Garbage Collection removes unreferenced chunks while preserving the current data
|
||||
- all relevant devices have synchronised; and
|
||||
- the remaining historical and deletion state is understood.
|
||||
|
||||
Deleted documents and tombstones are not free, and historical revisions may keep chunks reachable. Garbage Collection therefore cannot promise the smallest possible remote.
|
||||
Deleted documents, tombstones, live conflicts, and retained metadata are not free. Live conflict branches keep the chunks needed for review, while an ordinary superseded linear revision does not protect its former chunks. Garbage Collection can therefore make old content unreadable and cannot promise the smallest possible remote. Review the [Garbage Collection V3 specification](specs_garbage_collection.md) before using it.
|
||||
|
||||
Rebuild is a different operation. It reconstructs the database from a chosen authoritative state and is the more certain way to remove unwanted history or repair a damaged remote, but it is also more disruptive and can discard changes which exist only elsewhere.
|
||||
|
||||
|
||||
+7
-1
@@ -12,7 +12,7 @@ The following status applies to optional and compatibility features in the 1.0 l
|
||||
| -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Supported, opt-in | Peer-to-Peer Synchronisation, Hidden File Sync, and Customisation Sync | Maintained and covered by focused real-runtime tests. Enable them only where their separate setup and operational constraints are acceptable. |
|
||||
| Maintained, advanced | Data Compression | Available as an explicit storage and bandwidth trade-off. It remains disabled by default because the measured processing and memory costs outweigh the mixed-dataset saving. |
|
||||
| Beta or experimental | JWT authentication, ignore files, automatic newer-file conflict resolution, and Garbage Collection V3 | Retained for explicit testing and specialised use. They remain disabled by default and are not part of the minimum supported setup. |
|
||||
| Beta or experimental | JWT authentication, ignore files, automatic newer-file conflict resolution, and Garbage Collection V3 for CouchDB | Retained for explicit testing and specialised use. They remain disabled by default and are not part of the minimum supported setup. |
|
||||
| Compatibility only | V1 dynamic iteration counts, the old IndexedDB adapter, non-current hash algorithms, Eden chunks, and the stored `doNotUseFixedRevisionForChunks` key | Existing settings and data remain readable. New Vaults use the current defaults, and compatibility controls are shown only where a migration or recovery path still needs them. |
|
||||
|
||||
| Icon | Description |
|
||||
@@ -1047,6 +1047,12 @@ Purge all download/upload cache.
|
||||
|
||||
Delete all data on the remote server.
|
||||
|
||||
### 6. Garbage Collection V3 (CouchDB only)
|
||||
|
||||
Garbage Collection V3 identifies chunk documents which are not reachable from any current file or live conflict branch, creates logical deletions for those chunks locally, propagates the deletions to CouchDB, and requests remote compaction.
|
||||
|
||||
Use it only when the Vault, local database, and remote are healthy, and every relevant device has synchronised. It can make an ordinary superseded file revision unreadable when no live state still needs its chunks. It does not repair corruption or replace a deliberate rebuild. See the [Garbage Collection V3 specification](specs_garbage_collection.md).
|
||||
|
||||
### 7. Reset
|
||||
|
||||
#### Delete local database to reset or uninstall Self-hosted LiveSync
|
||||
|
||||
@@ -60,6 +60,8 @@ Logical deletion does not recreate missing bytes, purge the document history, or
|
||||
|
||||
**Recreate chunks for current Vault files** can recreate chunks only from files which are readable in the current Vault. It cannot reconstruct unique bytes from an unavailable historical or conflict revision.
|
||||
|
||||
Garbage Collection V3 treats every live conflict revision and its nearest available shared ancestor as reachable. Their locally available chunks are retained until the conflict is resolved. After resolution, chunks used only by the discarded branch or no-longer-needed merge ancestry can become eligible for collection. See the [Garbage Collection V3 specification](specs_garbage_collection.md).
|
||||
|
||||
A generation-one revision has no parent. When its body is unavailable, LiveSync cannot preserve a changed Vault file as a sibling branch without inventing ancestry. It leaves the operation unresolved. Recover the missing chunks from another replica or backup, or explicitly discard that live revision. If the current Vault file is the intended replacement, it can be stored after the unreadable revision has been logically deleted.
|
||||
|
||||
### Two devices independently create the same path
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
# Garbage Collection V3
|
||||
|
||||
Garbage Collection V3 is a beta maintenance operation for CouchDB remotes. It removes chunk documents which are no longer required by the current local revision tree, propagates those logical deletions to CouchDB, and then requests remote compaction.
|
||||
|
||||
It is not a repair operation. Use it only when the Vault, the local LiveSync database, and the CouchDB remote are healthy, every relevant device has synchronised, and recoverable backups exist.
|
||||
|
||||
## Supported scope
|
||||
|
||||
Garbage Collection V3 is available in Edge Case mode for CouchDB. It is not offered for Object Storage or P2P:
|
||||
|
||||
- Object Storage has a different journal and object lifecycle.
|
||||
- P2P has no central database to compact and cannot provide the accepted-device progress information required by this workflow.
|
||||
|
||||
The operation requires **Fetch chunks on demand** to be off so that its local reachability result is not confused with chunks which exist only on the remote.
|
||||
|
||||
## Workflow
|
||||
|
||||
After the user starts Garbage Collection V3, LiveSync:
|
||||
|
||||
1. completes a one-shot bidirectional CouchDB synchronisation;
|
||||
2. reads the accepted-device list and current progress recorded on the remote;
|
||||
3. warns when an accepted device has no current information or device progress differs, then requires explicit confirmation;
|
||||
4. computes the chunks reachable from the local PouchDB revision tree;
|
||||
5. creates a logical deletion for each locally present chunk which is not reachable;
|
||||
6. completes a push-only replication so that those deletions reach CouchDB;
|
||||
7. requests CouchDB compaction and waits for it for up to two minutes; and
|
||||
8. clears the local chunk caches.
|
||||
|
||||
If the initial synchronisation, device inspection, or confirmation fails, the workflow stops before collection. If push-only replication fails, the local logical deletions have already been created, but remote compaction is not started; synchronise again before retrying or using another device. A compaction failure is reported separately.
|
||||
|
||||
## Reachability rules
|
||||
|
||||
A chunk remains reachable when it is referenced by any of the following:
|
||||
|
||||
- the current database winner for a file;
|
||||
- any other live conflict revision for that file;
|
||||
- an available revision on either side of a live conflict which is required to describe the divergence; or
|
||||
- the nearest available revision shared by both live conflict branches.
|
||||
|
||||
Chunk identifiers are content-derived and shared between files. Reachability is therefore collected into one set across the database. A chunk used by two or more current files remains protected even when one file is updated or deleted.
|
||||
|
||||
An ordinary superseded linear revision does not protect its former chunks. Once no current file or live conflict branch references a chunk, it can be collected. After a conflict is resolved, chunks unique to the discarded branch and to no-longer-needed merge ancestry can also become eligible.
|
||||
|
||||
## Consequences
|
||||
|
||||
Garbage Collection deliberately trades historical recoverability for storage. A metadata revision may remain in the revision tree after a chunk which only that superseded revision used has been collected, so that historical body can become unreadable. Remote compaction can then discard old CouchDB revision bodies. Tombstones and retained metadata also consume storage, so the operation does not promise the smallest possible database.
|
||||
|
||||
Writing the same bytes again produces the same content-derived chunk identifier. If that chunk was collected previously, the normal chunk-writing path creates a new live revision for it, and ordinary replication can transfer it again. This does not recover an older file revision automatically; it only makes the newly written content available.
|
||||
|
||||
Garbage Collection does not reconstruct a chunk which is already missing, determine whether an unreadable revision is important, or repair a damaged local database. Use **Verify and repair all files**, another healthy replica, or a backup for those cases. Use **Overwrite Server Data with This Device's Files** only when a chosen Vault is authoritative and a deliberate remote rebuild is required.
|
||||
|
||||
## Verification
|
||||
|
||||
Commonlib tests use real in-memory PouchDB revision trees to verify:
|
||||
|
||||
- collection eligibility after a normal file update;
|
||||
- protection of chunks shared by multiple current files;
|
||||
- protection of all live conflict branches and their nearest available shared ancestor;
|
||||
- eligibility of losing-branch and ancestor-only chunks after conflict resolution;
|
||||
- propagation of chunk deletion to another PouchDB database; and
|
||||
- recreation and propagation when the same content is written again.
|
||||
|
||||
Self-hosted LiveSync tests verify that Garbage Collection V3 uses Commonlib's revision-aware result, deletes only unreachable chunks, performs the initial bidirectional and final push-only replications in order, and requests remote compaction. CouchDB's own compaction implementation remains an external database boundary.
|
||||
@@ -129,7 +129,7 @@ Browser security errors, particularly CORS failures, may reach the plug-in only
|
||||
|
||||
LiveSync stores file metadata, chunks, revision history, conflicts, deletions, and tombstones. Deleting or shortening a file therefore does not immediately remove every object which once represented it.
|
||||
|
||||
Garbage Collection can remove unreferenced chunks, but it is appropriate only when the Vault and local database are healthy and all relevant devices have synchronised. Tombstones and retained revisions are not free, so Garbage Collection does not guarantee a minimal database.
|
||||
Garbage Collection V3 can remove unreferenced chunks from a healthy CouchDB setup, but it is appropriate only when the Vault and local database are healthy and all relevant devices have synchronised. Current files and live conflict branches protect their required chunks; an ordinary superseded revision does not. Tombstones and retained metadata are not free, so Garbage Collection does not guarantee a minimal database. Review the [Garbage Collection V3 specification](specs_garbage_collection.md) before using it.
|
||||
|
||||
`Overwrite Server Data with This Device's Files` is a separate rebuild operation and is the more certain way to reconstruct a central remote from a chosen authoritative Vault. It is also destructive and may discard changes which exist only on another device. Review [Recovery and flag files](recovery.md#garbage-collection-is-not-rebuild) before choosing between them.
|
||||
|
||||
|
||||
@@ -5,7 +5,6 @@ import {
|
||||
LOG_LEVEL_NOTICE,
|
||||
LOG_LEVEL_VERBOSE,
|
||||
REMOTE_COUCHDB,
|
||||
REMOTE_P2P,
|
||||
type DocumentID,
|
||||
type EntryDoc,
|
||||
type EntryLeaf,
|
||||
@@ -53,8 +52,7 @@ export class LocalDatabaseMaintenance extends LiveSyncCommands {
|
||||
name: "Garbage Collection V3 (advanced, beta)",
|
||||
icon: "trash-2",
|
||||
checkCallback: (checking) => {
|
||||
const isApplicableRemote =
|
||||
this.settings.remoteType === REMOTE_COUCHDB || this.settings.remoteType === REMOTE_P2P;
|
||||
const isApplicableRemote = this.settings.remoteType === REMOTE_COUCHDB;
|
||||
if (!this.settings.useEdgeCaseMode || !this._isDatabaseReady() || !isApplicableRemote) {
|
||||
return false;
|
||||
}
|
||||
@@ -464,7 +462,7 @@ Note: **Make sure to synchronise all devices before deletion.**
|
||||
const confirmMessage = `This function deletes unused chunks from the device. If there are differences between devices, some chunks may be missing when resolving conflicts.
|
||||
Be sure to synchronise before executing.
|
||||
|
||||
However, if you have deleted them, you may be able to recover them by performing Hatch -> Recreate missing chunks for all files.
|
||||
If chunks used by current Vault files are deleted, Hatch -> Recreate chunks for current Vault files can recreate them only from files currently present in the Vault. It cannot recover unreadable historical or conflict content.
|
||||
|
||||
Are you ready to delete unused chunks?`;
|
||||
|
||||
@@ -926,41 +924,22 @@ This may indicate that some devices have not completed synchronisation, which co
|
||||
const gcStartTime = Date.now();
|
||||
// Perform Garbage Collection (new implementation).
|
||||
const localDatabase = this.localDatabase.localDatabase;
|
||||
const usedChunks = new Set<DocumentID>();
|
||||
const allChunks = new Map<DocumentID, string>();
|
||||
|
||||
const IDs = this.localDatabase.findEntryNames("", "", {});
|
||||
let i = 0;
|
||||
const doc_count = (await localDatabase.info()).doc_count;
|
||||
for await (const id of IDs) {
|
||||
const doc = await this.localDatabase.getRaw(id as DocumentID);
|
||||
i++;
|
||||
if (i % 100 == 0) {
|
||||
this._notice(`Garbage Collection: Scanned ${i} / ~${doc_count} `, "gc-scanning");
|
||||
}
|
||||
if (!doc) continue;
|
||||
if ("children" in doc) {
|
||||
const children = (doc.children || []) as DocumentID[];
|
||||
for (const chunkId of children) {
|
||||
usedChunks.add(chunkId);
|
||||
}
|
||||
} else if (doc.type === EntryTypes.CHUNK) {
|
||||
allChunks.set(doc._id, doc._rev);
|
||||
}
|
||||
}
|
||||
// Use the revision-aware reachability scan. Reading only winning revisions
|
||||
// would make chunks used exclusively by live conflict branches look unused.
|
||||
const { used: usedChunks, existing: allChunks } = await this.localDatabase.allChunks();
|
||||
this._notice(
|
||||
`Garbage Collection: Scanning completed. Total chunks: ${allChunks.size}, Used chunks: ${usedChunks.size}`,
|
||||
"gc-scanning"
|
||||
);
|
||||
|
||||
const unusedChunks = [...allChunks.keys()].filter((e) => !usedChunks.has(e));
|
||||
const unusedChunks = [...allChunks.entries()].filter(([chunkId]) => !usedChunks.has(chunkId));
|
||||
this._notice(`Garbage Collection: Found ${unusedChunks.length} unused chunks to delete.`, "gc-scanning");
|
||||
const deleteChunkDocs = unusedChunks.map(
|
||||
(chunkId) =>
|
||||
([chunkId, chunk]) =>
|
||||
({
|
||||
_id: chunkId,
|
||||
_id: chunkId as DocumentID,
|
||||
_deleted: true,
|
||||
_rev: allChunks.get(chunkId),
|
||||
_rev: chunk._rev,
|
||||
}) as EntryLeaf
|
||||
);
|
||||
const response = await localDatabase.bulkDocs(deleteChunkDocs);
|
||||
|
||||
@@ -24,7 +24,12 @@ vi.mock("@/common/events", () => ({
|
||||
onEvent: vi.fn(),
|
||||
},
|
||||
}));
|
||||
import { DEFAULT_SETTINGS, REMOTE_COUCHDB, REMOTE_MINIO } from "@vrtmrz/livesync-commonlib/compat/common/types";
|
||||
import {
|
||||
DEFAULT_SETTINGS,
|
||||
REMOTE_COUCHDB,
|
||||
REMOTE_MINIO,
|
||||
REMOTE_P2P,
|
||||
} from "@vrtmrz/livesync-commonlib/compat/common/types";
|
||||
import { LocalDatabaseMaintenance } from "./CmdLocalDatabaseMainte";
|
||||
import { ensureLocalDatabaseMaintenancePrerequisites } from "./maintenancePrerequisites";
|
||||
|
||||
@@ -87,6 +92,9 @@ describe("LocalDatabaseMaintenance prerequisites", () => {
|
||||
settings.useEdgeCaseMode = true;
|
||||
expect(garbageCollect?.checkCallback?.(true)).toBe(true);
|
||||
|
||||
settings.remoteType = REMOTE_P2P;
|
||||
expect(garbageCollect?.checkCallback?.(true)).toBe(false);
|
||||
|
||||
settings.remoteType = REMOTE_MINIO;
|
||||
expect(garbageCollect?.checkCallback?.(true)).toBe(false);
|
||||
});
|
||||
@@ -176,4 +184,129 @@ describe("LocalDatabaseMaintenance prerequisites", () => {
|
||||
expect(askSelectStringDialogue).not.toHaveBeenCalled();
|
||||
expect(applyPartial).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("describes the current chunk-recreation action without promising historical recovery", async () => {
|
||||
const maintenance = Object.create(LocalDatabaseMaintenance.prototype) as LocalDatabaseMaintenance;
|
||||
const askSelectStringDialogue = vi.fn().mockResolvedValue("Cancel");
|
||||
Object.assign(maintenance, {
|
||||
core: {
|
||||
confirm: {
|
||||
askSelectStringDialogue,
|
||||
},
|
||||
},
|
||||
_log: vi.fn(),
|
||||
});
|
||||
vi.spyOn(maintenance, "ensureAvailable").mockResolvedValue(true);
|
||||
vi.spyOn(maintenance, "trackChanges").mockResolvedValue(undefined);
|
||||
|
||||
await maintenance.performGC();
|
||||
|
||||
const message = vi.mocked(askSelectStringDialogue).mock.calls[0]?.[0] as string;
|
||||
expect(message).toContain("Hatch -> Recreate chunks for current Vault files");
|
||||
expect(message).toContain("only from files currently present in the Vault");
|
||||
expect(message).not.toContain("Recreate missing chunks for all files");
|
||||
});
|
||||
});
|
||||
|
||||
describe("LocalDatabaseMaintenance Garbage Collection V3", () => {
|
||||
it("keeps chunks referenced by a live conflict revision and deletes only unreachable chunks", async () => {
|
||||
const maintenance = Object.create(LocalDatabaseMaintenance.prototype) as LocalDatabaseMaintenance;
|
||||
const pushModes: string[] = [];
|
||||
const deletedChunks: Array<{ _id: string; _rev?: string; _deleted?: boolean }> = [];
|
||||
const allChunks = vi.fn(async () => ({
|
||||
used: new Set(["h:winner", "h:conflict"]),
|
||||
existing: new Map([
|
||||
["h:winner", { _id: "h:winner", _rev: "1-winner", type: "leaf", data: "winner" }],
|
||||
["h:conflict", { _id: "h:conflict", _rev: "1-conflict", type: "leaf", data: "conflict" }],
|
||||
["h:obsolete", { _id: "h:obsolete", _rev: "1-obsolete", type: "leaf", data: "obsolete" }],
|
||||
]),
|
||||
}));
|
||||
const rawDocuments = new Map<string, object>([
|
||||
[
|
||||
"note.md",
|
||||
{
|
||||
_id: "note.md",
|
||||
_rev: "2-winner",
|
||||
_conflicts: ["2-conflict"],
|
||||
type: "plain",
|
||||
children: ["h:winner"],
|
||||
},
|
||||
],
|
||||
["h:winner", { _id: "h:winner", _rev: "1-winner", type: "leaf", data: "winner" }],
|
||||
["h:conflict", { _id: "h:conflict", _rev: "1-conflict", type: "leaf", data: "conflict" }],
|
||||
["h:obsolete", { _id: "h:obsolete", _rev: "1-obsolete", type: "leaf", data: "obsolete" }],
|
||||
]);
|
||||
const findEntryNames = vi.fn(async function* () {
|
||||
yield* rawDocuments.keys();
|
||||
});
|
||||
const getRaw = vi.fn(async (id: string) => rawDocuments.get(id));
|
||||
const localDatabase = {
|
||||
allChunks,
|
||||
localDatabase: {
|
||||
info: vi.fn(async () => ({ doc_count: rawDocuments.size })),
|
||||
bulkDocs: vi.fn(async (docs: Array<{ _id: string; _rev?: string; _deleted?: boolean }>) => {
|
||||
deletedChunks.push(...docs);
|
||||
return docs.map(({ _id }) => ({ ok: true, id: _id, rev: "2-deleted" }));
|
||||
}),
|
||||
},
|
||||
findEntryNames,
|
||||
getRaw,
|
||||
};
|
||||
const replicator = {
|
||||
openOneShotReplication: vi.fn(
|
||||
async (
|
||||
_settings: typeof DEFAULT_SETTINGS,
|
||||
_showResult: boolean,
|
||||
_ignoreCleanLock: boolean,
|
||||
mode: string
|
||||
) => {
|
||||
pushModes.push(mode);
|
||||
return true;
|
||||
}
|
||||
),
|
||||
getConnectedDeviceList: vi.fn(async () => ({
|
||||
accepted_nodes: ["device-a"],
|
||||
node_info: {
|
||||
"device-a": {
|
||||
progress: "10-local",
|
||||
device_name: "Device A",
|
||||
app_version: "1.12.7",
|
||||
plugin_version: "1.0.0-beta.0",
|
||||
},
|
||||
},
|
||||
})),
|
||||
};
|
||||
Object.assign(maintenance, {
|
||||
core: {
|
||||
settings: {
|
||||
...DEFAULT_SETTINGS,
|
||||
remoteType: REMOTE_COUCHDB,
|
||||
},
|
||||
replicator,
|
||||
confirm: {
|
||||
askSelectStringDialogue: vi.fn(async () => "Proceed Garbage Collection"),
|
||||
},
|
||||
},
|
||||
localDatabase,
|
||||
_notice: vi.fn(),
|
||||
});
|
||||
vi.spyOn(maintenance, "ensureAvailable").mockResolvedValue(true);
|
||||
vi.spyOn(maintenance, "compactDatabase").mockResolvedValue(undefined);
|
||||
vi.spyOn(maintenance, "clearHash").mockImplementation(() => undefined);
|
||||
|
||||
await maintenance.gcv3();
|
||||
|
||||
expect(allChunks).toHaveBeenCalledOnce();
|
||||
expect(findEntryNames).not.toHaveBeenCalled();
|
||||
expect(getRaw).not.toHaveBeenCalled();
|
||||
expect(deletedChunks).toEqual([
|
||||
{
|
||||
_id: "h:obsolete",
|
||||
_rev: "1-obsolete",
|
||||
_deleted: true,
|
||||
},
|
||||
]);
|
||||
expect(pushModes).toEqual(["sync", "pushOnly"]);
|
||||
expect(maintenance.compactDatabase).toHaveBeenCalledOnce();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -187,7 +187,7 @@ export function paneMaintenance(
|
||||
)
|
||||
.addOnUpdate(this.onlyOnMinIO);
|
||||
});
|
||||
void addPanel(paneEl, "Garbage Collection V3 (Beta)", (e) => e, this.onlyOnP2POrCouchDB).then((paneEl) => {
|
||||
void addPanel(paneEl, "Garbage Collection V3 (Beta)", (e) => e, this.onlyOnCouchDB).then((paneEl) => {
|
||||
new Setting(paneEl)
|
||||
.setName("Perform Garbage Collection")
|
||||
.setDesc("Perform Garbage Collection to remove unused chunks and reduce database size.")
|
||||
|
||||
+2
-1
@@ -25,12 +25,13 @@ Earlier releases remain available in the 0.25 release history and the legacy rel
|
||||
### Fixed
|
||||
|
||||
- An unreadable conflict revision is no longer deleted automatically merely because its chunks are unavailable on the current device.
|
||||
- Garbage Collection V3 now protects chunks required by every live conflict branch and the available revision ancestry needed to review and merge conflicts, instead of considering only the database winner. The action is offered only for CouchDB because P2P has no central database to compact and does not provide the device inventory required by the workflow.
|
||||
- Choosing **Apply settings to this device, and fetch again** for a compatible configuration mismatch now applies the remote settings before Fetch, instead of updating the remote database with this device's settings.
|
||||
- Accepted settings which control how new chunks are created now take effect before synchronisation is retried, rather than leaving the previous hash or splitter active until restart.
|
||||
|
||||
### Testing
|
||||
|
||||
- Added regressions for revision repair, P2P configuration, the distinction between setting up the first device and using Fetch on an additional device, the P2P status pane, CouchDB setup policy, and mobile dialogues.
|
||||
- Added regressions for revision repair, P2P configuration, the distinction between setting up the first device and using Fetch on an additional device, the P2P status pane, CouchDB setup policy, mobile dialogues, conflict-aware chunk reachability, shared chunks, collection propagation, and content-addressed chunk recreation.
|
||||
|
||||
## 1.0.0-beta.2
|
||||
|
||||
|
||||
Reference in New Issue
Block a user