Repository navigation
fix(knowledge): stop logging an oversized source file as an error - #35300
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. The fix is at mechanism level and covers each What this change shipsInventory (6 items) — 6 justifiedIntent: an oversized Knowledge file is reported once, as the pipeline's WARNING, with no extra ERROR and traceback. FIX.
[FIRST-PRINCIPLES-REVIEWED] da9823e Description read[DESCRIPTION-READ] 6d0d7811790c76a35b750634c7695cbaae96283848c15c14844cb70a1f406298 That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks. This is a narrow change to log levels only. It suppresses logging only for the one type the pipeline raises right after its own WARNING, and it follows the fence #32507 already uses in Notes
[DESIGN-REVIEWED] da9823e Description read[DESCRIPTION-READ] 56bdb6908a39ab617946bc5d865db970c990640e93ea4087df3c3c65e3d20856 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
buluoray
left a comment
There was a problem hiding this comment.
Approve on 5aad911. An oversized Knowledge file now logs one WARNING (source id, size, limit only) instead of a WARNING plus an ERROR with a traceback. The dropped ERROR lines were the ones that carried the file path, URL or document name. Fences are exact-type and tested in both directions. Decisive lines: src/kiro_crew/knowledge/ingestion.py:902, src/kiro_crew/dashboard/handlers/knowledge_sync.py:350, src/kiro_crew/dashboard/handlers/knowledge_sync.py:651, src/kiro_crew/dashboard/handlers/knowledge_ingest.py:140, src/kiro_crew/knowledge/artifact_ingest.py:910, src/kiro_crew/knowledge/artifact_ingest.py:969
5aad911 to
d6aa394
Compare
|
[operator: bolichen97] Pushed |
A Knowledge file over knowledge.max_ingest_file_mb is an expected skip: the ingestion pipeline logs it as a WARNING naming the config key and raises FileTooLargeError. The single-file source sync, the web-page sync, the agent ingest-text route and both artifact paths (the start-time and manual pass, and the change event) caught that in a generic handler and logged it again with a traceback, so one skip read as an ERROR, on every gateway start for an oversized artifact. Each handler now skips that log line for FileTooLargeError and records exactly what it did before: the same status, HTTP answer and counts. Every other failure still logs its ERROR. Part of #32635
d6aa394 to
da9823e
Compare
|
Rebased: main moved in The red [operator: bolichen97] |
Code change approver recommendation: ApproveWhy: the decider estimates a 6% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes. This PR only changes logging. In the five handlers it lists, an oversized-file failure (
Each change has a test that runs both a read error and an oversized file, so it would fail if a skip were dropped or widened. The two artifact tests also run the pipeline's real size check, which proves the pipeline's WARNING is logged and that the class it raises is the one the handlers skip. The paths the PR leaves alone are declared and tracked in #3 An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Approved. On this head the review checked that each of the five handlers skips its ERROR log only for the knowledge-ingest FileTooLargeError. That class is raised at one site, and that site already logs a WARNING and writes the SEL audit event. Every other path is unchanged: cancellation still re-raises, the import budget still answers 429, other failures still log ERROR, and status rows, the 500 answer and the counts stay byte-identical. A mutation run confirmed each test fails when its guard is removed and also when the guard is widened to RuntimeError.
Done well: every new test covers both a read error and an oversized file, and the artifact tests use the real pipeline size check instead of a mocked exception.
Optional:
- For the two artifact paths (artifact_ingest.py:913 and :972), the removed ERROR line was the only log line that named the artifact slug. All artifacts share one source, so two oversized artifacts now leave two WARNINGs that differ only in size. A short info line naming the slug would bring that back. You could also change "Removes nothing" in the body to say the slug is now only in the SEL audit record.
- The body says the three test files give 175 passed when run unshuffled. The review run gave 176, which matches the shuffled 168 + 8.
Problem / Motivation
Goal: A Knowledge file over
knowledge.max_ingest_file_mbthat arrives as a single-file source, a web page, agent-fetched text or an artifact is reported once, as the pipeline's warning, instead of also filling the gateway log with an ERROR and a traceback.Set
knowledge.max_ingest_file_mbto1and add a 2 MB.mdas a single-file Knowledge source. The pipeline logs a clear WARNING telling you to raiseknowledge.max_ingest_file_mb, and then the sync task logsERROR Background ingestion failed for ...with aFileTooLargeErrortraceback for the same file. A web-page source, text an agent fetched for a source, and an artifact over the cap do the same.Why it matters
An ERROR with a traceback reads as a crash. Someone reading the gateway log, or a log monitor, sees a fault for a file Kiro Crew skipped on purpose and already explained, and the real errors in that log get harder to spot. The artifact pass runs on every gateway start, so one oversized artifact adds a fresh ERROR and traceback each time the gateway starts.
Not a goal
error, the agent-text request still answers 500, and the artifact pass reports the same counts.dashboard.upload_max_mbandknowledge.max_ingest_file_mb), so while that setting holds still it answers 413 before the pipeline can raise.POST /api/knowledge/agent-document). It has no handler of its own: the error reaches the web framework's generic 500, and giving it a proper answer is a separate change.What changed (motivation → approach → change)
An oversized file on these five paths now leaves one WARNING in the log, not a WARNING plus an ERROR with a traceback.
knowledge_sync.py_ingest_local_file_taskknowledge_sync.py_background_url_syncknowledge_ingest.pyingest_textartifact_ingest.pyreconcile_artifactsartifact_ingest.pyArtifactKnowledgeSync._on_task_doneThe pipeline's size check logs the WARNING and raises
FileTooLargeError. Each of these callers catches every failure in one generic handler that logs it with a traceback, so the same skip was logged a second time as an ERROR.Each handler now skips that log line when the error is a
FileTooLargeError, the way #32507 did for the watched-folder scan. Nothing else moves: the same status is written, the same HTTP answer goes back, and every other failure still logs its ERROR with a traceback.Two of those handlers import
FileTooLargeError, so the handlers' facade,knowledge.py, lists it among its type-checking imports, as it does for every name they import.Backwards compatibility
Compatible: only the log level for this one expected skip changes; every stored row, status, count and HTTP answer is the same.
Removes nothing: the change drops a duplicate log record and keeps every row, status, count and response.
Tests
Each test runs two failures and checks the recorded result is the same for both: a read error must still log an ERROR, and an oversized file must log none. So each fails if its fence is dropped, and also if it is widened (for example to
RuntimeError, whichFileTooLargeErrorsubclasses).test/test_knowledge_budget.py::TestBackgroundDeferralStaysRetryable::test_local_file_ingest_failure_ends_in_error: the single-file source endserror.test/test_knowledge_handlers_coverage_source_writes.py::TestBackgroundAgentSync::test_ingest_failure_marks_source_error: the web-page source endserror.test/test_knowledge_handlers_coverage_source_writes.py::TestIngestText::test_ingest_failure_is_500_and_cleans_up: agent text still answers 500internal server errorand removes its temp file.test/test_knowledge_artifact_ingest.py::TestReconcile::test_a_failed_ingest_logs_an_error_unless_the_file_was_oversizedand::TestArtifactKnowledgeSync::test_a_failed_event_logs_an_error_unless_the_file_was_oversized: these run the real pipeline with the cap lowered to about 1 KB, so they also pin that the pipeline's own WARNING is logged and that the class it raises is the one the handlers skip.Proof: PROVEN test/test_knowledge_artifact_ingest.py::TestReconcile::test_a_failed_ingest_logs_an_error_unless_the_file_was_oversized[oversized], test/test_knowledge_artifact_ingest.py::TestArtifactKnowledgeSync::test_a_failed_event_logs_an_error_unless_the_file_was_oversized[oversized], test/test_knowledge_budget.py::TestBackgroundDeferralStaysRetryable::test_local_file_ingest_failure_ends_in_error[oversized], test/test_knowledge_handlers_coverage_source_writes.py::TestIngestText::test_ingest_failure_is_500_and_cleans_up[oversized], test/test_knowledge_handlers_coverage_source_writes.py::TestBackgroundAgentSync::test_ingest_failure_marks_source_error[oversized] (head da9823e)
Determinism: the ten new cases passed 20/20 runs at
-n0and 10/10 at-n 4 --dist loadgroup --max-worker-restart=2besidetest_knowledge_url_source.py. The five classes holding them passed in a shuffled class order (seed 7, 69 passed). A shuffled order of all three files (seed 32635) passed 168 of 175; the other 8 wereTimeout >120sin fixture setup or teardown, never in a test body, while the shared host stalled on disk flushes (setup up to 947 s). Unshuffled, the three files pass whole (175 passed).Manual verification
N/A — unit coverage sufficient: the handler tests drive the real sync tasks, the real ingest-text route and the real artifact pass over a real store, and the artifact tests hit the pipeline's real size check.
Screenshots / video
N/A — backend-only change; no frontend file is touched.
Related Issues
Part of #32635
The Research Lab findings site and the agent-document route stay open there.
Sibling sweep
Swept: git grep -n ".ingest_file(" -- src/kiro_crew, then each caller's exception handlers up to the first one that logs
Sibling: src/kiro_crew/dashboard/handlers/knowledge_sync.py -- changed here
Sibling: src/kiro_crew/dashboard/handlers/knowledge_ingest.py:140 -- changed here
Sibling: src/kiro_crew/knowledge/artifact_ingest.py -- changed here
Sibling: src/kiro_crew/knowledge/folder_watcher.py:1529 -- not affected -- already skips the log for FileTooLargeError (#32507)
Sibling: src/kiro_crew/knowledge/watcher.py:358 -- not affected -- catches FileTooLargeError before its generic handler
Sibling: src/kiro_crew/knowledge/sync.py:89 -- not affected -- ingests through ingest_text, which has no size check
Sibling: src/kiro_crew/dashboard/handlers/knowledge_agent_context.py:203 -- tracked in #32635
Sibling: src/kiro_crew/apps/builtins/auto_research/campaign/publication.py:584 -- tracked in #32635
Pattern harvest
Rule candidate: review-prompt
Pattern: a generic
exceptthat logs with a traceback also catches a typed expected-skip error its raiser already logged as a WARNING, so one skip is logged twice, once as an ERROR; the sweep has to follow a re-raising caller up to the handler that logs.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributed under the project's Apache-2.0 license (see CONTRIBUTING.md).