feat(search): keep user tags per user in a tag_type index by marevol · Pull Request #3551 · codelibs/fess

marevol · 2026-10-04T12:36:33Z

Summary

Replaces the user tags of #3536/#3540 (labels of the kind tag) with per-user tags kept in their own fess_config.tag_type index. Tags are no longer a kind of label: label_type goes back to its state before #3536.

Per-user tags. A tag belongs to one owner. Two users can use the same name and still have two separate tags. Only logged-in users can see or use tags.
Sharing. A tag is private unless its owner shares it. A shared tag gets the guest permission (role.search.guest.permissions), and every logged-in user can then see it and filter by it. Only the owner (or an administrator) can change it. The bootstrap theme shows other users' tags with a "Shared:" prefix.
Document field. Documents carry the tags in the tag keyword field as base64url(name):base64url(owner).

Changes

Revert feat(search): let users tag documents with labels of the kind tag #3536 and fix(search): limit tag conditions to the tags the caller can see #3540. QueryContext.searchRequestType is kept because fix(search): split the vector query as the running search request type #3546 uses it.
fess_config.tag_type index:
- Fields: name, owner, paths, permissions, virtualHost, sortOrder and the created/updated fields.
- _id is the SHA-256 of the tag value.
- Writes use op_type=create and if_seq_no/if_primary_term.
- The index is in the fess_config alias only, so it is not in the basic-config download.
tag on the fess index. It is added to the doc.json mappings. On an existing index without it, the webapp adds tag: keyword at startup.
TagTypeHelper:
- Encodes and decodes tag values.
- Normalizes names (NFKC).
- Decides visibility: the owner, or permissions that meet the caller's roles plus the guest roles, within the virtual host.
- Looks up tags by URL with one query per batch.
Bulk updates through a queue. Adding, removing, renaming and deleting tags are queued in memory. The per-minute log_aggregator job applies them to documents in bulk.
- The updates go through painless scripts with the language script appended, so content_*/title_* survive.
- The update index is refreshed before a delete or rename by query.
- A new manual job, tag_updater, rebuilds the tag field of every document from tag_type.
Indexing:
- IndexingHelper.sendDocuments sets tag from tag_type.paths for each batch, so tags survive a re-crawl.
- The admin document and search-list writes do the same.
- tag is a system-managed field on the admin edit form.
Search:
- Conditions on tag (fields.tag, tag:, ex_q, facet.query) match only the tags the caller can see. Wildcard, prefix, fuzzy and range conditions on it match nothing.
- The tag facet is restricted with an exact-value include in buildFacet, so the v1/classic API plugins cannot list hidden tags either.
- The raw tag field is never returned.
v2 API (login and CSRF required, no access tokens):
- GET/POST /api/v2/tags
- PUT/DELETE /api/v2/tags/{id}
- GET/POST /api/v2/documents/{docId}/tags
- DELETE /api/v2/documents/{docId}/tags/{id}
- Search hits return tags: [{value, name, owner, mine, shared}], and tag facet buckets are labelled.
- ui/config reports features.user_tag.
- OpenAPI is updated.
Admin. A new Tag screen (admin/tagtype) and admin API (/api/admin/tagtype), with labels and messages in all bundles.
Bootstrap theme 1.5.0:
- Tag chips on results.
- An editor to add and remove your tags.
- A Tags facet.
- A "My tags" panel to rename, share and delete tags.
Configuration:
- user.tag.enabled (default false)
- user.tag.name.max.length, user.tag.max.tags, user.tag.max.paths
- user.tag.queue.max.size, user.tag.process.batch.size, user.tag.visible.max.size
- index.field.tag
- tag is added to index.export.exclude.fields.

Notes for operators

Delay on documents. Changes reach the search results within about a minute, on the next log_aggregator run. log_aggregator must keep running on every node (target=all).
Recovering lost changes. The queue is per JVM. Changes still in it are lost on a restart, and a re-crawl while user.tag.enabled=false clears tag. Run tag_updater to rebuild in both cases.
Owner identity. The owner is the login user ID. With SAML, the NameID must be persistent.

Testing

Unit tests:
- Java: mvn test, 8,455 tests, all passing.
- JS (vitest): 743 tests, all passing.
Live check against OpenSearch 3.9.0, all scenarios passing:
- Private and shared tags across two users and an anonymous caller.
- Filter and facet restrictions.
- Tags survive a re-crawl.
- Rename and delete reach documents, and body search still works after a tag update.
- tag_updater rebuilds tags after a restart.
- The admin screen and admin API.
- The startup mapping on an existing index.

Reverts #3536 and #3540. User tags move to a separate per-user tag_type index instead of a label kind, so the label-kind implementation is removed. QueryContext.searchRequestType (and its setter call sites in QueryHelper and SearchEngineClient) is kept because #3546 uses it.

Add the fess_config.tag_type index for the tags each logged-in user owns: name, owner, paths, permissions, virtualHost, sortOrder and the usual created/updated fields, with the ESFlute classes generated by FreeGen. TagTypeService lists, reads, creates, updates and deletes tag types. A create uses op_type=create and an update uses if_seq_no/if_primary_term, and a lost race on either is reported as TagTypeConflictException. Listings leave paths out of _source, as a tag can list many URLs. Add the tag keyword field to the document mapping. An existing document index gets only that field on webapp startup, and a tag field that is already mapped is left as it is, with a warning when it is not keyword. Add the user.tag.* settings, index.field.tag, page.tagtype.max.fetch.size and online.help.name.tagtype, and leave tag out of the index export.

…ookup TagTypeHelper encodes a tag as Base64URL(name) + ":" + Base64URL(owner), parses such a value back strictly, derives the tag_type id as the SHA-256 of the value, and normalizes tag names (NFKC, collapsed whitespace, a code point limit, no control, format or unpaired surrogate characters). TagType keeps the one implementation of the encoding. Visibility: nothing is visible to a caller who is not logged in, and an admin search sees every tag. Otherwise a tag is visible when the request has no virtual host or the tag's, and the caller owns the tag or holds one of its permissions, the guest roles included. A shared tag stores the encoded role.search.guest.permissions next to the owner's user role. getVisibleTagTypes reads tag types by id without their paths and caches the result in the request; getVisibleTagValues lists the caller's own and permitted tags up to user.tag.visible.max.size. findTagValuesByUrls reads every tag whose paths hold one of the URLs through a cursor and matches the URLs exactly. applyTags sets the tag field of documents about to be indexed from those tags and removes it when a URL has none; it does nothing unless user.tag.enabled is true. Register the helper in app.xml and in the crawler process app.xml, where documents are indexed.

getVisibleTagValues read the caller's own tags and the tags of others in one query sorted by sort order and name, so tags of others could push the caller's own tags past user.tag.visible.max.size. Read the own tags first and fill the rest of the budget with the tags of others whose permissions the caller holds. The guest role list ends with the guest user role, which is also the own permission of a user named guest. Their private tags were therefore taken as shared and shown to every logged-in user. Only the encoded role.search.guest.permissions now mark a tag as shared, and the guest user role no longer grants visibility; the owner still sees the tag.

TagTypeHelper#enqueue keeps TagChange values (ADD, REMOVE, DELETE, RENAME) in an in-memory queue bounded by user.tag.queue.max.size; a change beyond the limit is dropped with a WARN. processQueue, run by the log_aggregator job in its own try block when user.tag.enabled is true, applies them in order: additions and removals are gathered per URL as the final state of each value and written in bulk, user.tag.process.batch.size URLs at a time, to every document of each URL; a DELETE or RENAME first writes what was gathered and then updates every document holding the value through updateByQuery. A failed write is logged and the remaining changes are still applied. Each update targets index.document.update.index, retries on version conflicts and appends the language script so content_*/title_* stay indexed. The painless code tolerates a missing, single or list tag field, never adds a value twice, and ends with an expression because the appended language statements cannot follow a closing brace. The new tag_updater scheduled job (no schedule, target all) walks every document, reads the tags of each page of URLs with one query and rewrites only the documents whose tags differ, removing the field when the URL has no tag. It restores changes lost from the queue.

A DELETE or RENAME finds its documents by searching the tag field, but neither the bulk update nor updateByQuery refreshes the index, so documents written earlier in the same queue cycle were not found: an ADD followed by a DELETE left the tag on the document, a rename chain A to B to C stopped at B, and an ADD followed by a RENAME kept the old value. processQueue now remembers whether it wrote to the update index since the last refresh and, when it did, refreshes that index synchronously before each DELETE or RENAME. A failed refresh is logged and the update still runs. The WARN for a failed updateByQuery now also reports how many documents it had visited.

…re indexed Apply tags in IndexingHelper.sendDocuments (only when user.tag.enabled), the bulk document API and the admin/API searchlist create and update paths. The tag field is system-managed and is stripped from client-supplied documents.

…'s visible tags A value of the tag field decodes to the name and the owner of a tag, and the field of a document holds every tag put on it, so a condition or a facet on the field must only ever reveal the tags the caller can see. The tag field is a search field, a facet field and a not-analyzed field again, but not a response field: no response returns raw tag values. A condition on the tag field (fields.tag, tag: in the query, ex_q and facet.query, in the keyword leg and the vector leg alike) is a term query only for the exact value of a tag that TagTypeHelper#getVisibleTagTypes reports as visible. A value the caller cannot see, and a wildcard, prefix, fuzzy or range condition on the field, match no document. Each value is resolved once per query context. A caller who is not logged in, or any caller while user.tag.enabled is false, matches nothing. The admin search is not restricted. The tag facet includes only the exact values of the tags that TagTypeHelper#getVisibleTagValues returns, so tags of the same name of two owners are two buckets and invisible tags never push visible ones out of the top buckets. When the caller can see no tag, or user tags are disabled, the tag facet is not aggregated. The admin search is not restricted. SemanticSearchRequestParams now delegates getResponseFields, so a request that adds a field to its response gets it on hits of the vector leg too.

Logged-in users manage their own tags and put them on documents through the v2 API; the shared tags of other users can be seen and filtered on but not changed. - GET/POST /api/v2/tags lists the caller's tags with their path counts and creates one (409 conflict for a name the caller already uses, user.tag.max.tags per user). - PUT /api/v2/tags/{id} renames a tag (new id, the rename of the value on the documents is queued) or changes whether it is shared, which keeps any other permission; DELETE deletes it and queues the removal. - GET/POST /api/v2/documents/{docId}/tags lists the visible tags on the document's URL with the caller's addable tags, and puts a tag on it by id or by name, creating a private tag of the name when needed (user.tag.max.paths per tag). DELETE .../tags/{id} takes it off. - Changes are retried on a concurrent update; another user's tag is 403 when visible and 404 otherwise; the document is resolved with the caller's roles. - Every tag endpoint needs user.tag.enabled and a login session; an access token does not stand in for one, and writes need the CSRF token. - /api/v2/search fetches the tag field for itself, never returns it raw, and gives hits the visible tags as tags [{value, name, owner, mine, shared}] with one lookup per request; tag facet buckets carry label, owner, mine and shared. - ui/config reports features.user_tag; the OpenAPI document describes the endpoints, the hit tags and the facet fields.

A tag was deleted by id alone, so a URL added to it between the read and the delete was lost, and a rename left the documents with a value whose tag no longer had that URL. - TagTypeService.delete now sends the sequence number and primary term the tag was read with and reports a lost race as TagTypeConflictException, like update. - DELETE /api/v2/tags/{id} retries on a fresh read and answers 409 when it keeps losing; the deletion is queued only after the delete succeeded. - A rename that cannot delete the old tag removes the tag it created and, on a lost race, retries with the old tag read again, carrying over the URLs added meanwhile; it answers 409 when it keeps losing. The rename is queued only after the old tag is gone.

Add the admin/tagtype screen and the api/admin/tagtype endpoints so that an administrator can list, create, edit and delete the tags of every user (roles admin-tagtype and admin-tagtype-view). - A form carries name, owner, paths (one URL per line), permissions, virtual host and sort order. The name is normalized like the v2 tag API, the owner is required and the number of paths is capped by user.tag.max.paths. Empty permissions default to the owner. - Edits and deletes send back the sequence number and primary term the tag was read with, so a change made since then by its owner or another administrator is reported instead of overwritten. - A change of the paths queues adding or removing the tag on the documents of those URLs; a change of the name or the owner replaces the tag and queues a rename; a delete queues the removal. Nothing is queued while user.tag.enabled is false. - TagTypeService#replace holds the create-then-delete step of a rename, with the rollback of the new tag, for both the admin screen and the v2 tag API. - Sidebar entry, AdminAction roles and redirect, labels and messages in all 17 locales, and the regenerated FessLabels, FessMessages and FessHtmlPath constants.

When an administrator changes the owner of a tag, replace the user role of the old owner in its permissions with that of the new owner, so the old owner no longer sees the tag or the documents carrying it. The sharing roles and any other role are kept. This applies to both the admin screen and the admin API.

With features.user_tag, result cards show the visible tags as chips that filter by fields.tag, and other users' tags read with a shared prefix. Logged-in users get an inline editor that adds one of their own tags by id or a tag by name and removes their own tags, with a note that search results follow after a short while. A Tags facet uses the same labels, and a My tags panel on the options bar lists the user's tags and renames, shares or unshares, and deletes them after a confirmation in the panel. 401 and 403 answers open the login modal. api.js gains put() and del(). Adds the tag messages to all bundles and bumps the theme version to 1.5.0.

A hit can carry the tag values of any number of users, and every one of them was looked up by id to decorate the v2 search response. The values of the hits and of the tag facet are now first intersected with the tags the caller can see (capped by user.tag.visible.max.size), and only that intersection is looked up. The visible tag values are cached in the request, so the tag facet and the hit decoration share one computation. Also: - applyTags logs a failed tag lookup at ERROR, since the batch is then indexed without its tags until tag_updater runs. - TagTypeHelper no longer handles the admin search: every caller skips the tag restriction for it, so the branches were unreachable. The javadoc of the helper and its callers states this.

This follows codelibs/fess#3551, which replaces the user tags that were labels of the kind "tag" with per-user tags in the fess_config.tag_type index, and replaces the tag documentation added in #565. - api/api-tag.rst: the new endpoints (GET/POST /api/v2/tags, PUT/DELETE /api/v2/tags/{id}, GET/POST /api/v2/documents/{docId}/tags and DELETE /api/v2/documents/{docId}/tags/{id}) with their login and CSRF requirements, examples and errors; private and shared tags; the tags of search hits, the tag facet and fields.tag filtering; the delay until the per-minute log_aggregator job applies the queued changes; and the user.tag.* settings. - admin/tagtype-guide.rst: a new page for the admin Tag screen, its fields, sharing, how changes reach documents through the queue and the tag_updater job, and notes for operators. It is listed after the label page in the administrator guide. - admin/labeltype-guide.rst: back to its content before #565, without the Kind field and the user tag section. - api/api-uiconfig.rst: features.user_tag describes the new feature. - api/admin/api-admin-tagtype.rst: a new page for /api/admin/tagtype, listed after the label type API. Updates send seq_no and primary_term from GET setting/{id}; a stale pair or a name the owner already uses fails with a validation error. - config/properties.rst and properties.po: only the rows the pull request changes are applied (user.tag.*, index.field.tag, page.tagtype.max.fetch.size, online.help.name.tagtype and the default of index.export.exclude.fields), rendered with the generator's own functions so the pages still pass --check.

marevol added 14 commits October 4, 2026 18:52

marevol self-assigned this Oct 4, 2026

marevol added this to the 15.9.0 milestone Oct 4, 2026

marevol added the enhancement label Oct 4, 2026

marevol merged commit e07658a into main Oct 4, 2026
2 checks passed

This was referenced Oct 4, 2026

docs(15.9): document per-user tags kept in the tag_type index codelibs/fess-docs#567

Merged

feat: add the tagtype command for user tags codelibs/fessctl#37

Merged

test: cover the Tag admin screen codelibs/fess-test-ui#63

Open

Navigation Menu

feat(search): keep user tags per user in a tag_type index - #3551

feat(search): keep user tags per user in a tag_type index#3551
marevol merged 14 commits into
mainfrom
feat/user-tag-type

marevol commented Oct 4, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

marevol commented Oct 4, 2026

Summary

Changes

Notes for operators

Testing

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant