Orphaned S3 blobs and stale user index entries after space deletion and recreation — no cleanup path available

[giovanni-cm]

Environment:

• OpenCloud version: 7.1.0 (rolling), upgraded from 6.1.0
• Storage backend: decomposedS3 (MinIO)
• Search backend: Bleve + Apache Tika
• Deployment: Docker Compose (official opencloud-compose repo)

Description:

A project space ("END Docs") was created and populated with ~200 GB of files. Due to incorrect file timestamps, the space was deleted and recreated, and the files were re-uploaded with correct timestamps. The new space works correctly and is fully accessible.
The space is owned by a dedicated user ("Archivio Old Files") whose sole purpose is to manage this single archive space. The space is shared read-only with other users, so they can retrieve archived documents without the ability to edit, modify, or delete them.

However, the old deleted space left behind:

• ~200 GiB of orphaned blobs in the S3 bucket (under a UUID prefix that no longer corresponds to any visible space)
• Stale references to the old space UUID in multiple user index files (/opt/opencloud/data/storage/users/indexes/by-user-id/*.mpk)
• A stale entry in /opt/opencloud/data/storage/users/spaces/ with no .space metadata file inside

What was tried:

• opencloud storage-users trash-bin purge-expired — ran successfully but did not remove the orphaned data
• Checked admin UI with "show disabled spaces" enabled — the old space does not appear
• Checked all users' trash bins — empty
• The old space UUID (b6c7e517-a7d7-4b61-875b-03fb714bc806) is confirmed absent from the UI but present in S3 with ~200 GiB / ~56,000 objects, and referenced in user index .mpk files

Current state:

• Live space UUID: 45858f44-896e-41bd-bb1f-db7d438bf6dd — fully functional, 216.9 GB, visible in admin UI
• Orphaned UUID: b6c7e517-a7d7-4b61-875b-03fb714bc806 — 200 GiB in S3, stale index references, not visible anywhere in UI, no trash entry, no space metadata on disk

Why manual S3 deletion is not safe:

Deleting the S3 prefix directly would leave the stale .mpk index entries pointing to non-existent objects, with unknown consequences for affected users.

Request:

Is there a supported procedure or CLI command to fully purge an orphaned space (S3 blobs + index entries) that is no longer visible in the UI and cannot be accessed through normal admin workflows? If not, this seems like a gap in the opencloud storage-users tooling that would benefit from a dedicated spaces purge or spaces cleanup subcommand.

[michaelstingl]

Reproduced first-hand on two decomposedS3 + MinIO stacks: standalone opencloud-rolling:7.1.0 and opencloud:7.2.0-beta.2, and opencloud-compose main. A normal, completed purge of a project space leaves two orphans.

Reproduction

POST   /graph/v1.0/drives  {"driveType":"project"}     -> 201, id <providerID>$<spaceID>
PUT    /dav/spaces/<providerID>$<spaceID>/f1           -> 201   (x3)
# optional read-only share:
POST   /graph/v1beta1/drives/<...>/root/invite         -> 200
DELETE /graph/v1.0/drives/<...>                        -> 204   (disable)
DELETE /graph/v1.0/drives/<...>   Purge: T             -> 204   (purge)

After the purge, storage/users/spaces/<xx>/<rest> is removed and the space is absent from GET /me/drives (200). Two things remain.

1. Orphaned S3 objects

Every object under <bucket>/<spaceID>/ stays. The purge is completed, not interrupted, so the orphaned size equals the full space size. A delete-and-recreate leaves the old copy entirely in the bucket, which matches the reported ~200 GiB.

2. Stale space-index entries

storage/users/indexes/by-user-id/<userID>.mpk still references <spaceID>, for the owner and for every user the space was shared with (this matches the reported entries across multiple index files). Drive listing tolerates the dangling reference: GET /me/drives returns 200 and does not show the dead space. The search service does not: it logs an error for each orphaned space on every indexing pass (error while indexing a space / error walking the tree, not found). The stale entries are not cosmetic; they produce ongoing error-log noise until removed.

Root cause

reva pkg/storage/pkg/decomposedfs/spaces.go (used by the decomposeds3, decomposed and posix drivers; the parallel pkg/storage/utils/decomposedfs tree used by the ocis and s3ng drivers has the same gap). Function names are stable, line numbers drift:

• updateIndexes() populates the by-type index (linkStorageSpaceType) and the by-user-id / by-group indexes (linkSpaceByUser / linkSpaceByGroup). It runs for the owner from CreateStorageSpace and per grantee from AddGrant.
• The purge branch of DeleteStorageSpace calls only spaceTypeIndex.Remove and does not invalidate the by-user-id / by-group indexes. A comment at that call site notes it: // TODO invalidate ALL indexes in msgpack, not only by type.
• For the S3 objects, the purge calls fs.tp.DeleteBlob while walking the space, yet the decomposedS3 objects remain. This points at the decomposedS3 blob-removal path and looks separate from the index entries.

_{🤖 drafted with Claude Code, reviewed before submitting.}

[michaelstingl]

Manual recovery for an already-orphaned space, verified on the decomposedS3 + MinIO stack above. Reclaim the S3 storage, then clear the stale index entries.

Caution

Step 3 edits binary MessagePack files. Stop OpenCloud, back up indexes/, and only remove space ids confirmed dead. A mistake can make spaces inaccessible.

1. Confirm the space id is dead

Absent from GET /me/drives and from storage/users/spaces/<xx>/<rest> (no live .space), but present as an S3 prefix and as a key in the index files below.

2. Reclaim storage

mc rm --recursive --force <bucket>/<spaceID>/

3. Clear the index entries

The id stays in by-user-id/<ownerID>.mpk, in by-user-id/<userID>.mpk per shared user, and in by-group-id/<groupID>.mpk per group (by-type is cleared by the purge). Remove it from all of them:

docker compose stop opencloud
cp -a <data-root>/storage/users/indexes{,.bak-$(date +%Y%m%d)}
python3 - <<'PY'   # pip install msgpack
import msgpack, glob
DEAD = "<spaceID>"
for f in glob.glob("<data-root>/storage/users/indexes/by-*/*.mpk"):
    raw = open(f, "rb").read()
    if not raw:
        continue
    d = msgpack.unpackb(raw, raw=False)
    if isinstance(d, dict) and DEAD in d:
        del d[DEAD]
        open(f, "wb").write(msgpack.packb(d, use_bin_type=True))
        print("cleaned", f)
PY
docker compose start opencloud

4. Verify

GET /me/drives returns 200, a new space can be created, and the search service no longer logs error while indexing a space for that id. On error, restore the indexes.bak-* backup.

Same MessagePack technique as #1878; the steps differ there (user deletion, with by-type ghosts).

_{🤖 drafted with Claude Code, reviewed before submitting.}

[giovanni-cm]

Thank you for the rapid and detailed analysis.
The manual recovery procedure seem to be working — S3 deletion is currently in progress (~56,000 objects), and I'll proceed with the index cleanup afterward.
It's reassuring to see the root cause identified so clearly, including the existing // TODO comment in the purge path. I notice opencloud-eu/reva#695 is already linked — is that addressing one or both of the issues described here?

For future reference, the manual recovery steps provided in this issue are clear and seem to be working well (still in running phase), though the dependency on msgpack being installed (not available by default in the container) and the need to stop OpenCloud during index repair are worth noting in any future admin documentation.

[michaelstingl]

The linked PR covers the index half only: it stops a purge from leaving stale by-user-id and by-group entries behind, from now on.

It does not remove the orphaned S3 blobs, and it does not clean up spaces orphaned before it; for those the manual steps above are the workaround.

_{🤖 drafted with Claude Code, reviewed before submitting.}

[giovanni-cm]

Update: manual recovery completed successfully

Following the procedure described above, I can confirm everything worked as expected:

mc rm --recursive --force on the orphaned S3 prefix completed successfully (~56,000 objects removed). MinIO reported usage dropped from 430 GiB to 213 GiB, matching the expected live data size.
The index cleanup script found nothing to remove from the by-user-id/*.mpk files. In hindsight this is likely because the earlier grep was matching binary substrings in the raw MessagePack files rather than actual dictionary keys — the Python script with proper MessagePack parsing found the dead space ID was not present as a key in any file.
The stack was restarted cleanly. The live "END Docs" space (45858f44) is fully accessible with all files intact. Collabora, theme, notifications and search all working normally.

One remaining artifact worth noting for the fix verification: the dead space still appears as an empty, undeletable entry in the trash bin of the space owner ("Archivio Old Files") and also in the trash bin of every user the space had originally been shared with. It is confirmed empty in all cases and causes no functional issues, but it is visible to affected users and cannot be removed through the UI. A complete fix should ensure these ghost trash entries are also cleared for all users involved.

---

Thank you all sincerely for the incredibly fast and thorough response. Having the root cause identified, reproduced, and a PR already linked within hours of filing the issue is well beyond what one would expect for a homelab edge case. The manual recovery steps were clear, safe, and worked perfectly. This is a great example of an open source project genuinely caring about its users.
Much appreciated.

[micbar]

@michaelstingl Thank you!

[michaelstingl]

Index cleanup

by-user-id/*.mpk is keyed by the bare space UUID (after the $); the value also contains that UUID, so the substring grep matched while the key check correctly found none. For your instance the index step is a no-op, the S3 reclaim was the only applicable step.

Trash entry: not reproducible here

Steps on 7.2.0-beta.2 (decomposedS3 + MinIO):

1. Owner creates a project space, uploads files, shares it with a second user (Space Viewer).
2. Owner opens Deleted files in the web UI; the space shows as a trash bin.
3. Keep that page open. From a different client (not the tab on Deleted files), DELETE the drive (disable), then DELETE with Purge: T (purge) — the two calls the web Disable and Delete actions make. The deleter must be a separate client or user: the live update is skipped for the client that initiated the delete, so it has to arrive cross-client.
4. The space disappears from Deleted files within seconds, no reload. Still absent after navigate-away/back and after a hard reload.

Live removal of a deleted space is handled by the web client and shipped in 7.1.0:

• fix: sse event on space delete has no effect web#2614

So the version alone does not explain the difference. To match your case, please share:

• delete path: web UI or API, and which user;
• did the affected users have Deleted files open at delete time, or open it only later;
• does the entry survive a full browser reload or a logout/login now.

_{🤖 drafted with Claude Code, reviewed before submitting.}

[giovanni-cm]

Thank you for the detailed follow-up and for investigating the trash entry.

Answers to your questions:

• Delete path: web UI only, performed by the space owner ("Archivio Old Files" user), not the admin.
• Deleted files open at delete time: No — neither the owner nor any of the shared users had the Deleted files page open at the time of deletion. The ghost entry was discovered only later when browsing the trash bin.
• Persistence: The ghost entry survives a full hard reload (Ctrl+F5) and persists after a complete logout/login cycle, for both the space owner and all users the space had been shared with.

Regarding the index cleanup being a no-op: that makes sense in hindsight — the grep was matching binary substrings in the raw MessagePack files rather than actual dictionary keys, as the Python script with proper parsing correctly confirmed. Good to have that clarified.
The S3 reclaim was the only applicable manual step and it worked perfectly — bucket usage dropped from 430 GiB to 213 GiB as expected.

[BEBU88]

Will there be an automatic cleanup or CLI command to clean up already orphaned indexes in the future?