Repository navigation
[Bug]: Retrying a segment status change after broker failure succeeds without restoring its index #45238
Description
Do you need to file an issue?
- I have searched all existing issues and established that this bug is not already filed.
- I believe this is a legitimate bug, not just a feature request.
Targeted duplicate searches and related patches were checked; coverage is documented in dedup.md.
Dify version
1.18.0; source main@4c521a3c0a1f60607c8a7bcf90792f34adcf18d0.
Describe the bug
The Console segment-status API commits enabled/disabled state before publishing its Celery task. If publication fails, the request returns an error but the committed state already matches the request. Retrying after broker recovery returns success without dispatching the task because the application skips segments with matching status.
In a real economy knowledge base, enabling a disabled segment while the broker was stopped returned HTTP 400. After restarting the broker, the same enable request returned HTTP 200, but a keyword retrieval for “quokka” returned zero records and the keyword table remained empty.
Steps to reproduce
- Deploy the exact source baseline below with separate cache Redis and Celery broker Redis, plus a worker consuming dataset and priority_dataset.
- Create an economy dataset and a text document:
Quokka recovery regression. A quokka is a small marsupial.Wait for completed indexing. - PATCH
/console/api/datasets/{dataset}/documents/{document}/segment/disable?segment_id={segment}and wait for index removal. - Stop only the Celery broker. PATCH the same endpoint with
enable. Observe HTTP 400 connection refused; the DB enabled field is already true. - Restart the broker and repeat the same enable request. Observe HTTP 200 but retrieval still returns no matching segment.
These steps were executed through supported authenticated HTTP APIs with controlled broker unavailability, not mocked dispatch. An executable guide and sanitized before/after responses are attached locally.
Expected Behavior
A successful explicit retry after broker recovery should schedule the index operation so the segment becomes retrievable again. A persisted desired state alone must not be treated as proof that its asynchronous side effect completed.
Related Module
Knowledge / segment lifecycle and task dispatch.
Configuration Used
Source deployment; economy indexing with Jieba; PostgreSQL 15, separate Redis 6 cache and broker; no embedding or paid model required. Independent disposable containers only.
Logs and screenshots
Before: failed enable HTTP 400 → retry HTTP 200 → retrieval records = 0.
After: failed enable HTTP 400 → retry HTTP 200 → worker runs enable task → retrieval records = 1.
The symmetric disable failure/retry was also observed: stale keywords are removed after retry.
Additional Information
- Reproduction baseline:
main@4c521a3c0a1f60607c8a7bcf90792f34adcf18d0. The original real-deployment validation used local fix56e35fee231b6fc46dc3c2a544e9af06adbbba2b; the latest published revision is linked below. - Actual deployment: isolated source API + Celery, PostgreSQL 15 and Redis 6; macOS arm64, Python 3.13.15.
- Local unit fixtures use real SQLite repositories and existing VectorProbe/model stubs; these are distinct from the real-service API validation.
- Ruff check/format, targeted Pyrefly and
git diff --checkpassed. Import linter: 78 contracts kept, 0 broken. Response contract lint: 762 valid, 0 mismatch, 378 unknown (not a claim of full coverage). - Relevant repository diff guards passed. Full monorepo test/lint was not run. See the linked PR for current CI and review status.
Related fix and regression coverage
- Fix PR: #45239 — reconcile segment indexes on repeated status requests.
- Latest published revision:
ed36509onfix/segment-status-dispatch-recovery. - Single-segment and mixed-batch failure/retry tests cover enable/disable, complete batch redispatch, tenant isolation, and preserved disable audit metadata. All four retry cases fail on the original baseline and pass with the fix.
- Worker eligibility tests cover disabled, archived, or incomplete documents and non-COMPLETED segments.
The latest update adds tests without changing production logic. The related service/adapter/worker suite passes 144 tests (6 warnings); the real application evidence above remains from the original deployment validation.
Activity
dosubot commentedon Nov 1, 2026
One additional scope point for maintainer triage: this affects batch status changes, not just a single segment.
change_segment_status()persists all selected IDs through a committed repository transaction before dispatching the task. After publication fails, a retry filters out every segment whose persistedenabledvalue already matches the action, leaving no IDs to dispatch. [1] [2]A useful additional regression case would be a mixed batch containing both already-in-target-state segments and segments that need a transition, followed by broker failure and an identical retry. That exercises the same filtering gap beyond the single-segment reproduction. [3]
Recovery coverage should also preserve the enable worker’s eligibility checks: it skips disabled, archived, or incomplete documents and only loads
COMPLETEDsegments. Redispatch should not accidentally broaden which segments are eligible for indexing. [4]To reply, just mention @dosu.
Docs are dead. Just use Dosu.
Zhi-Xiang-Guo commentedon Nov 1, 2026
@dosu. i've added this coverage in #45239, commit ed36509.
The regression tests now cover mixed batches containing segments already in the target state and segments requiring a transition, for both enable and disable. After a simulated broker failure, an identical retry dispatches all eligible segments while preserving existing disable audit metadata and excluding foreign-tenant segments.
I also added repeated-task tests for disabled, archived, and incomplete documents. The existing test excluding non-COMPLETED segments passes as well. Worker eligibility rules remain unchanged.
All four single-segment/mixed-batch retry cases fail on the original baseline and pass with the fix. The related service, adapter, and worker suite passes 144 tests locally. The issue description now links to the updated tests.
fix(segments): schedule index change before persisting segment status
fix(segments): reconcile segment indexes on retried status changes