Repository navigation
fix(ops-mission-control): check Datadog readiness off the event loop - #24385
KiroCrewCaptain merged 1 commit into
Conversation
DatadogAdapter.poll and execute, and DatadogEvidenceSource.gather, ran configured() inline in the coroutine before their asyncio.to_thread hop. configured() reads config.json (provider_enabled) and the secret store (has_secrets), and poll runs on every heartbeat, so each Datadog poll blocked the gateway loop on two file reads. All three now run configured() through asyncio.to_thread. gather checks the signal's source first, so a signal from another provider still returns without reading anything.
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks. This moves the blocking readiness reads off the event loop and changes no behaviour, and it follows the shape #24176 already set. Notes
[DESIGN-REVIEWED] 6a44ea3 Description read[DESCRIPTION-READ] 3bbb5632cd1caf24be36aef78898717f78e28f39a7fd8eac765d664f522a51eb That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS This is a small fix that is aimed at the cause. The one thing still open is the deferred siblings, and the description declares them. Notes
What this change shipsInventory (3 items) — 3 justifiedIntent: stop Datadog polls, actions and evidence gathers from stalling the gateway event loop with config and secret-store reads. This is a FIX.
[FIRST-PRINCIPLES-REVIEWED] 6a44ea3 Description read[DESCRIPTION-READ] 2e9a0b46015d434d787389b61df6408841e4ab2e97f9adb9749166f2cfd87c30 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
|
Re the Design Review note on two thread hops: keeping the gate in its own hop here, to match the shape merged in #24176 ( |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: fix (2 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: the Datadog provider configured() check runs off the event loop; tests added.
Problem / Motivation
Goal: the Datadog adapter's
poll,executeand evidencegatherno longer read config or the secret store on the gateway event loop.DatadogAdapter.poll,DatadogAdapter.executeandDatadogEvidenceSource.gathereach ranconfigured()inline in the coroutine, before theirasyncio.to_threadhop.configured()isprovider_enabled(...)(aconfig.jsonread) plushas_secrets(...)(a secret-store read). So each Datadog poll did two blocking file reads on the loop, and so did each Datadog action and evidence gather.Why it matters
The registry polls every enabled source on each heartbeat, so this repeats for as long as the Datadog adapter is enabled. On a slow or network-mounted
KIROCREW_HOMEthose reads stall every session's turn, and a long enough stall trips the loop watchdog (#3057, #1572).Not a goal
configured()calls (see Sibling sweep)._poll_sync/_execute_sync/_gather_syncbodies, which already run in the thread.What changed (motivation → approach → change)
Symptom: every Datadog poll, action and evidence gather blocks the loop on a config read and a secret-store read. Root cause:
configured()ran on the loop thread at the top of each coroutine, one line before the thread hop that the HTTP request already uses.All three sites now use
if not await asyncio.to_thread(self.configured), the shape #24176 used for the GitHub Issues adapter. Ingatherthe cheapsignal.source != PROVIDER_IDcheck now runs first, so a signal from another provider returns[]without a thread hop or any read. The result for every input is unchanged.Backwards compatibility
Compatible: the same readiness check gates the same requests with the same arguments; only the thread the check runs on moves.
Removes nothing: no key, route, reader or behaviour goes away.
Tests
ops_mission_control/tests/test_providers_datadog_cov80.py::TestConfiguredGateRunsOffTheLooppatchesdatadog.provider_enabledanddatadog.has_secretsto recordthreading.get_ident(), and fakesdatadog.request_json. Each test asserts that at least one readiness read happened, that none ran on the loop thread, and that the right request was sent or skipped. Cases: configured and unconfiguredpoll,execute(comment) andgather. A seventh test requiresgatherfor a CloudWatch signal to read nothing and send nothing.datadog.py): 7 failed, each withAssertionError: <loop thread id> unexpectedly found in [<loop thread id>, ...], or for the foreign-source case a non-empty read list. Green after:22 passedfor the file, three runs in a row. The tests record thread ids and do no timing.Proof: PROVEN src/kiro_crew/apps/builtins/ops_mission_control/tests/test_providers_datadog_cov80.py::TestConfiguredGateRunsOffTheLoop::test_execute_checks_readiness_off_the_loop, src/kiro_crew/apps/builtins/ops_mission_control/tests/test_providers_datadog_cov80.py::TestConfiguredGateRunsOffTheLoop::test_gather_checks_readiness_off_the_loop, src/kiro_crew/apps/builtins/ops_mission_control/tests/test_providers_datadog_cov80.py::TestConfiguredGateRunsOffTheLoop::test_gather_for_another_source_reads_no_readiness, src/kiro_crew/apps/builtins/ops_mission_control/tests/test_providers_datadog_cov80.py::TestConfiguredGateRunsOffTheLoop::test_poll_checks_readiness_off_the_loop, src/kiro_crew/apps/builtins/ops_mission_control/tests/test_providers_datadog_cov80.py::TestConfiguredGateRunsOffTheLoop::test_unconfigured_execute_checks_off_the_loop_and_sends_nothing (head 6a44ea3)
(prove.py's line names the first five; its JSON record lists all seven, including
test_unconfigured_poll_checks_off_the_loop_and_sends_nothingandtest_unconfigured_gather_checks_off_the_loop_and_sends_nothing.)Gates run locally:
check_black_formatting.py,check_subprocess_encoding.py,check_sync_io_in_async.py, comment-history, brand-name,check_file_line_cap.py, isort and flake8 on the changed files, mypy on the changed files,test/test_flake_pattern_ratchet.pyandtest/test_sync_io_in_async_gate.py(189 passed). The ops-mission-control test package passes (1140 tests) apart fromtest_ledger_sync_git.py.local-gate.py's related backend set ran 693 tests: 663 passed, and the 30 failures are all intest_ledger_sync_git.pywithgit init failed: sandbox: BLOCKEDfrom this host's sandbox. That file fails the same 30 on unmodifiedorigin/main(checked in a detached worktree).Manual verification
N/A: the unit tests call the real
poll,executeandgatherand check which thread the readiness reads run on, which is the property this change exists for.Related Issues
Closes #24381
Sibling sweep
Swept:
git grep -n "self.configured()" -- src/kiro_crew/apps/builtins/ops_mission_control/backend/, then checked which hits are inside anasync defSibling: src/kiro_crew/apps/builtins/ops_mission_control/backend/providers/datadog.py:160 -- changed here
Sibling: src/kiro_crew/apps/builtins/ops_mission_control/backend/providers/cloudwatch.py:209 -- not fixed here --
pollandgather(:405) callconfigured()(a config read) on the loop; a separate change is fixing that fileSibling: src/kiro_crew/apps/builtins/ops_mission_control/backend/providers/pagerduty.py:114 -- not fixed here --
pollandon_shift(:176) callconfigured()on the loop;execute(:266) is #24199Sibling: src/kiro_crew/apps/builtins/ops_mission_control/backend/providers/incidentio.py:145 -- not fixed here --
poll,on_shift(:276) andexecute(:353) callconfigured()on the loop; the file is edited by open #13428Sibling: src/kiro_crew/apps/builtins/ops_mission_control/backend/providers/schedule_file.py:363 -- not fixed here --
on_shiftcallsconfigured()on the loop; the file is edited by open #13428Sibling: src/kiro_crew/apps/builtins/ops_mission_control/backend/providers/schedule_file.py:378 -- not affected -- it is in the sync
_on_shift_sync, which its caller runs throughto_threadSibling: src/kiro_crew/apps/builtins/ops_mission_control/backend/providers/webhook.py:505 -- not fixed here --
pollcallsconfigured()on the loop; the file is edited by open #24341Sibling: src/kiro_crew/apps/builtins/ops_mission_control/backend/registry.py:212 -- not affected --
poll_allalready filters sources throughasyncio.to_threadPattern harvest
Rule candidate: review-prompt
Pattern: an adapter's
async defcalls its own syncconfigured()(a config or secret-store read) before itsasyncio.to_threadhop; the gate check belongs inside the hop.check_sync_io_in_async.pycannot see it, because the read is in another module.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributed under the project's Apache-2.0 license.