Skip to content

fix(ops-mission-control): check Datadog readiness off the event loop - #24385

Merged
KiroCrewCaptain merged 1 commit into
kirodotdev:mainfrom
IngridMorstrad:fix/omc-datadog-configured-off-loop
Oct 26, 2026
Merged

KiroCrewCaptain merged 1 commit into
kirodotdev:mainfrom
IngridMorstrad:fix/omc-datadog-configured-off-loop

Conversation

@IngridMorstrad

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: the Datadog adapter's poll, execute and evidence gather no longer read config or the secret store on the gateway event loop.

DatadogAdapter.poll, DatadogAdapter.execute and DatadogEvidenceSource.gather each ran configured() inline in the coroutine, before their asyncio.to_thread hop. configured() is provider_enabled(...) (a config.json read) plus has_secrets(...) (a secret-store read). So each Datadog poll did two blocking file reads on the loop, and so did each Datadog action and evidence gather.

Why it matters

The registry polls every enabled source on each heartbeat, so this repeats for as long as the Datadog adapter is enabled. On a slow or network-mounted KIROCREW_HOME those reads stall every session's turn, and a long enough stall trips the loop watchdog (#3057, #1572).

Not a goal

  • The other adapters' configured() calls (see Sibling sweep).
  • Changing what counts as configured, or which requests run when it is.
  • The _poll_sync / _execute_sync / _gather_sync bodies, which already run in the thread.

What changed (motivation → approach → change)

Symptom: every Datadog poll, action and evidence gather blocks the loop on a config read and a secret-store read. Root cause: configured() ran on the loop thread at the top of each coroutine, one line before the thread hop that the HTTP request already uses.

All three sites now use if not await asyncio.to_thread(self.configured), the shape #24176 used for the GitHub Issues adapter. In gather the cheap signal.source != PROVIDER_ID check now runs first, so a signal from another provider returns [] without a thread hop or any read. The result for every input is unchanged.

Backwards compatibility

Compatible: the same readiness check gates the same requests with the same arguments; only the thread the check runs on moves.

Removes nothing: no key, route, reader or behaviour goes away.

Tests

  • ops_mission_control/tests/test_providers_datadog_cov80.py::TestConfiguredGateRunsOffTheLoop patches datadog.provider_enabled and datadog.has_secrets to record threading.get_ident(), and fakes datadog.request_json. Each test asserts that at least one readiness read happened, that none ran on the loop thread, and that the right request was sent or skipped. Cases: configured and unconfigured poll, execute (comment) and gather. A seventh test requires gather for a CloudWatch signal to read nothing and send nothing.
  • Red before the fix (the new tests run against main's datadog.py): 7 failed, each with AssertionError: <loop thread id> unexpectedly found in [<loop thread id>, ...], or for the foreign-source case a non-empty read list. Green after: 22 passed for the file, three runs in a row. The tests record thread ids and do no timing.

Proof: PROVEN src/kiro_crew/apps/builtins/ops_mission_control/tests/test_providers_datadog_cov80.py::TestConfiguredGateRunsOffTheLoop::test_execute_checks_readiness_off_the_loop, src/kiro_crew/apps/builtins/ops_mission_control/tests/test_providers_datadog_cov80.py::TestConfiguredGateRunsOffTheLoop::test_gather_checks_readiness_off_the_loop, src/kiro_crew/apps/builtins/ops_mission_control/tests/test_providers_datadog_cov80.py::TestConfiguredGateRunsOffTheLoop::test_gather_for_another_source_reads_no_readiness, src/kiro_crew/apps/builtins/ops_mission_control/tests/test_providers_datadog_cov80.py::TestConfiguredGateRunsOffTheLoop::test_poll_checks_readiness_off_the_loop, src/kiro_crew/apps/builtins/ops_mission_control/tests/test_providers_datadog_cov80.py::TestConfiguredGateRunsOffTheLoop::test_unconfigured_execute_checks_off_the_loop_and_sends_nothing (head 6a44ea3)

(prove.py's line names the first five; its JSON record lists all seven, including test_unconfigured_poll_checks_off_the_loop_and_sends_nothing and test_unconfigured_gather_checks_off_the_loop_and_sends_nothing.)

Gates run locally: check_black_formatting.py, check_subprocess_encoding.py, check_sync_io_in_async.py, comment-history, brand-name, check_file_line_cap.py, isort and flake8 on the changed files, mypy on the changed files, test/test_flake_pattern_ratchet.py and test/test_sync_io_in_async_gate.py (189 passed). The ops-mission-control test package passes (1140 tests) apart from test_ledger_sync_git.py. local-gate.py's related backend set ran 693 tests: 663 passed, and the 30 failures are all in test_ledger_sync_git.py with git init failed: sandbox: BLOCKED from this host's sandbox. That file fails the same 30 on unmodified origin/main (checked in a detached worktree).

Manual verification

N/A: the unit tests call the real poll, execute and gather and check which thread the readiness reads run on, which is the property this change exists for.

Related Issues

Closes #24381

Sibling sweep

Swept: git grep -n "self.configured()" -- src/kiro_crew/apps/builtins/ops_mission_control/backend/, then checked which hits are inside an async def
Sibling: src/kiro_crew/apps/builtins/ops_mission_control/backend/providers/datadog.py:160 -- changed here
Sibling: src/kiro_crew/apps/builtins/ops_mission_control/backend/providers/cloudwatch.py:209 -- not fixed here -- poll and gather (:405) call configured() (a config read) on the loop; a separate change is fixing that file
Sibling: src/kiro_crew/apps/builtins/ops_mission_control/backend/providers/pagerduty.py:114 -- not fixed here -- poll and on_shift (:176) call configured() on the loop; execute (:266) is #24199
Sibling: src/kiro_crew/apps/builtins/ops_mission_control/backend/providers/incidentio.py:145 -- not fixed here -- poll, on_shift (:276) and execute (:353) call configured() on the loop; the file is edited by open #13428
Sibling: src/kiro_crew/apps/builtins/ops_mission_control/backend/providers/schedule_file.py:363 -- not fixed here -- on_shift calls configured() on the loop; the file is edited by open #13428
Sibling: src/kiro_crew/apps/builtins/ops_mission_control/backend/providers/schedule_file.py:378 -- not affected -- it is in the sync _on_shift_sync, which its caller runs through to_thread
Sibling: src/kiro_crew/apps/builtins/ops_mission_control/backend/providers/webhook.py:505 -- not fixed here -- poll calls configured() on the loop; the file is edited by open #24341
Sibling: src/kiro_crew/apps/builtins/ops_mission_control/backend/registry.py:212 -- not affected -- poll_all already filters sources through asyncio.to_thread

Pattern harvest

Rule candidate: review-prompt
Pattern: an adapter's async def calls its own sync configured() (a config or secret-store read) before its asyncio.to_thread hop; the gate check belongs inside the hop. check_sync_io_in_async.py cannot see it, because the read is in another module.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

DatadogAdapter.poll and execute, and DatadogEvidenceSource.gather, ran
configured() inline in the coroutine before their asyncio.to_thread hop.
configured() reads config.json (provider_enabled) and the secret store
(has_secrets), and poll runs on every heartbeat, so each Datadog poll
blocked the gateway loop on two file reads.

All three now run configured() through asyncio.to_thread. gather checks
the signal's source first, so a signal from another provider still
returns without reading anything.
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: checking Automated validation is still running labels Oct 26, 2026
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 6a44ea306662f46136462396f2d3fe50d495f8e1 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 6a44ea306662f46136462396f2d3fe50d495f8e1 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. This moves the blocking readiness reads off the event loop and changes no behaviour, and it follows the shape #24176 already set.

Notes

  • A configured call now makes two thread hops: one for configured() and one for the _sync body. Putting the check at the top of _poll_sync / _execute_sync / _gather_sync would need only one hop. It's worth choosing one shape before the five sibling adapters copy this pattern.

[DESIGN-REVIEWED] 6a44ea3

Description read

[DESCRIPTION-READ] 3bbb5632cd1caf24be36aef78898717f78e28f39a7fd8eac765d664f522a51eb

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of 6a44ea306662f46136462396f2d3fe50d495f8e1 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

This is a small fix that is aimed at the cause. The one thing still open is the deferred siblings, and the description declares them.

Notes

  • Item 1 — symptom-level (point patch): grepping self\.configured\(\) under ops_mission_control/backend finds 15 hits. Three are fixed here. Nine more run on the event loop and are not fixed: cloudwatch.py:209 and :405, pagerduty.py:114, :176 and :266, incidentio.py:145, :276 and :353, schedule_file.py:363, webhook.py:505. That is ten, counting webhook.py:505. The remaining two are already run off the loop (github_issues.py:136, schedule_file.py:378). The "Sibling sweep" section declares and defers all of these. Subtraction: none in this PR. The deferral is accepted.

What this change ships

Inventory (3 items) — 3 justified

Intent: stop Datadog polls, actions and evidence gathers from stalling the gateway event loop with config and secret-store reads. This is a FIX.

  1. The Datadog poll and action readiness checks now run off the event loop — justified
  2. The Datadog evidence-gather readiness check now runs off the loop — justified
  3. A gather for a non-Datadog signal now returns without reading config or secrets — justified

[FIRST-PRINCIPLES-REVIEWED] 6a44ea3

Description read

[DESCRIPTION-READ] 2e9a0b46015d434d787389b61df6408841e4ab2e97f9adb9749166f2cfd87c30

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 6a44ea306662f46136462396f2d3fe50d495f8e1 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 6a44ea3

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 6a44ea306662f46136462396f2d3fe50d495f8e1 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 6a44ea3

@IngridMorstrad

Copy link
Copy Markdown
Contributor Author

Re the Design Review note on two thread hops: keeping the gate in its own hop here, to match the shape merged in #24176 (execute) and open in #24199. A hop is microseconds against an HTTP request, and folding the check into _poll_sync / _execute_sync / _gather_sync would change what those sync methods mean for the tests that drive them directly. Picking one shape across the remaining adapters fits better in the change that sweeps those siblings.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 26, 2026
@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) October 26, 2026 10:08

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: fix (2 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: the Datadog provider configured() check runs off the event loop; tests added.

@KiroCrewCaptain
KiroCrewCaptain merged commit 6b6097f into kirodotdev:main Oct 26, 2026
104 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ops-mission-control: Datadog adapter checks readiness on the event loop

2 participants