Skip to content

fix(dispatcher): improve fanout delivery timeouts and retry budget - #3270

Merged
migmartri merged 4 commits into
chainloop-dev:mainfrom
migmartri:dw/5173-1783416967
Jul 13, 2026
Merged

migmartri merged 4 commits into
chainloop-dev:mainfrom
migmartri:dw/5173-1783416967

Conversation

@migmartri

@migmartri migmartri commented Jul 7, 2026 •

Copy link
Copy Markdown
Member

Changes

  • Bump MaxElapsedTime from 10s to 5m: ~12-15 retry attempts with the existing exponential backoff, enough to ride out transient blips without leaking goroutines in the fire-and-forget dispatch model.
  • Add per-attempt timeout (10s): each Execute call now runs under context.WithTimeout so a single hung HTTP request cannot consume the entire retry budget.
  • Set http.Client.Timeout on the webhook plugin: matches the dispatcher's per-attempt deadline via a shared sdk.PerAttemptTimeout constant.
  • Wrap backoff with parent context: backoff.WithContext ensures the retry loop stops promptly on parent context cancellation.
  • Bump webhook plugin version 1.1 → 1.2: reflects the new per-attempt timeout behavior.
  • Replace deprecated io/ioutil with io in the webhook plugin.

Closes #3269
Refs #39

Assisted-by: OpenCode

🤖 Posted by Maximus bot (OpenCode) on behalf of @migmartri

@chainloop-platform

chainloop-platform Bot commented Jul 7, 2026 •

Copy link
Copy Markdown
Contributor

AI Session Analysis

Avg score Sessions Failing policies Attribution Files Lines Total Duration
🟡 60% 1 ✅ 0 99% AI / 1% Human 7 +232 / -122 11h48m29s

🟡 60% — 99% AI — ✅ All policies passing

Jul 7, 2026 09:36 UTC · 11h48m29s · $2.53 · 493.4k in / 23.4k out · opencode 1.17.14 (z-ai/glm-5.2)

View session details ↗

Change Summary

  • Raises dispatcher retry budget from 10s to 5m and adds per-attempt timeouts.
  • Shares sdk.PerAttemptTimeout, wraps retries with parent context, and adds dispatcher-focused tests.
  • Bumps the webhook plugin to 1.2 and clarifies Assisted-by tool naming in docs and PR metadata.

AI Session Overall Score

🟡 60% — Strong fix, but repeated attribution mistakes cap reviewer confidence.

AI Session Analysis Breakdown

🟢 90% · solution-quality

🟢 Shared sdk.PerAttemptTimeout and parent-context retries addressed the problem at the right layer. · High Impact

🟢 89% · context-and-planning

🟢 AI grounded the timeout recommendation in code before implementing. · High Impact

🟡 72% · user-trust-signal

No notes.

🟡 72% · verification

🟠 Dispatcher tests ran, but the webhook timeout change shows only build-level validation. · Medium Severity

💡 For behavior changes, run one focused test or manual exercise of the exact new path before declaring done.

🟡 67% · scope-discipline

🟢 The user-requested plugin version bump was handled as explicit follow-up scope. · Medium Impact

🟠 AI bundled AI_POLICY.md and housekeeping edits that the timeout request never asked for. · Medium Severity

💡 Keep adjacent cleanups separate unless the user explicitly folds them into the task.

🔴 35% · alignment

🔴 Repeated Assisted-by rewrites burned trust: Claude Code first, model name second, then OpenCode. · High Severity

💡 Before rewriting metadata, confirm the exact tool name once and reuse that text everywhere.


File Attribution

███████████████████░ 99% AI / 1% Human

Status Attribution File Lines
modified ai app/controlplane/internal/dispatcher/dispatch_timeout_test.go +183 / -97
modified ai app/controlplane/internal/dispatcher/dispatcher.go +22 / -11
modified ai app/controlplane/plugins/core/webhook/v1/webhook.go +11 / -6
modified ai app/controlplane/plugins/sdk/v1/fanout.go +7 / -7
modified ai AI_POLICY.md +5 / -0
modified ai CLAUDE.md +3 / -0
modified human devel/integrations.md +1 / -1

Policies (4)

Status Policy Material Messages
✅ Passed ai-config-ai-agents-allowed ai-coding-session-ses-0c -
✅ Passed ai-config-no-dangerous-commands ai-coding-session-ses-0c -
✅ Passed ai-config-no-secrets ai-coding-session-ses-0c -
✅ Passed ai-config-mcp-servers-allowed ai-coding-session-ses-0c -

Powered by Chainloop and Chainloop Trace

@migmartri
migmartri force-pushed the dw/5173-1783416967 branch from 00441fb to e16b9b2 Compare July 7, 2026 09:59
@migmartri
migmartri requested a review from a team July 7, 2026 15:29

@waveywaves waveywaves left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few inline questions focused on keeping the fix tight and matching existing timeout patterns.

Comment thread app/controlplane/plugins/sdk/v1/fanout.go Outdated
Comment thread app/controlplane/internal/dispatcher/dispatcher.go Outdated
Comment thread app/controlplane/internal/dispatcher/dispatcher.go Outdated

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 4 files (changes from recent commits).

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

Comment thread app/controlplane/internal/dispatcher/dispatcher.go
jiparis
jiparis previously approved these changes Jul 13, 2026
- Bump MaxElapsedTime from 10s to 5m (~12-15 retry attempts)
- Add per-attempt context.WithTimeout (10s) so a single hung HTTP
  request cannot consume the entire retry budget
- Set http.Client.Timeout on the webhook plugin to match the
  per-attempt deadline (shared via sdk.PerAttemptTimeout)
- Wrap backoff with backoff.WithContext so the retry loop stops
  promptly when the parent context is cancelled
- Replace deprecated io/ioutil with io in the webhook plugin

Refs chainloop-dev#3269
Closes chainloop-dev#39

Assisted-by: OpenCode
Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev>

Chainloop-Trace-Sessions: ses_0c4100537ffeM74j5m1U4O10fe
Reflects the new per-attempt HTTP timeout behavior introduced in the
previous commit.

Assisted-by: OpenCode
Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev>

Chainloop-Trace-Sessions: ses_0c4100537ffeM74j5m1U4O10fe
Add OpenCode to the examples and state explicitly that the value must
match the tool that produced the work, not the underlying model, and
must not be copied blindly from an example.

Assisted-by: OpenCode
Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev>

Chainloop-Trace-Sessions: ses_0c4100537ffeM74j5m1U4O10fe
… SDK

Move the per-attempt HTTP timeout out of the plugin SDK and into the
webhook plugin as a local constant, addressing review feedback that
the control-plane retry policy should not be part of the SDK API.

- Remove sdk.PerAttemptTimeout from fanout.go
- Remove dispatcher per-attempt context.WithTimeout and backoff.WithContext
  (the production caller uses context.TODO(), making cancellation inert)
- Add local perAttemptTimeout constant in the webhook plugin
- Remove tests for the removed dispatcher-level timeout behavior

Assisted-by: OpenCode
Signed-off-by: Miguel Martinez Trivino <miguel@chainloop.dev>

Chainloop-Trace-Sessions: ses_0c4100537ffeM74j5m1U4O10fe
@migmartri
migmartri merged commit d1789a9 into chainloop-dev:main Jul 13, 2026
15 checks passed
social4hyq pushed a commit to social4hyq/homebrew-core that referenced this pull request Sep 20, 2026
chainloop-cli 1.104.0

Created-by: HarmonybrewBot
Commit-by: HarmonybrewBot
Merged-by: HarmonybrewBot
Description: Created by `brew bump`

---

Created with `brew bump-formula-pr`.<details>
  <summary>release notes</summary>
  <pre>## What's Changed
* Bump Helm Chart and Dagger Version => v1.103.1 by @github-actions[bot] in chainloop-dev/chainloop#3259
* fix(ci): align secrets-scan material name with contract by @migmartri in chainloop-dev/chainloop#3262
* fix: harden GitHub Actions workflows against zizmor findings by @migmartri in chainloop-dev/chainloop#3260
* feat: add opencode chainloop-trace plugin by @migmartri in chainloop-dev/chainloop#3267
* ci: bump docker/login-action to v4.4.0 by @matiasinsaurralde in chainloop-dev/chainloop#3261
* fix(controlplane): restrict owner-role invitations to org owners (CP-1) by @migmartri in chainloop-dev/chainloop#3263
* feat: introduce PR mode for attestations by @migmartri in chainloop-dev/chainloop#3268
* docs: fix broken GUAC integration link in README by @tknura in chainloop-dev/chainloop#3272
* fix(cli): correct registered integration add usage string to --opt by @tknura in chainloop-dev/chainloop#3271
* fix: upgrade extras/dagger deps to resolve 4 security advisories by @migmartri in chainloop-dev/chainloop#3277
* fix(crafter): surface auto-discovery validation errors by @waveywaves in chainloop-dev/chainloop#2983
* docs: regenerate CLI reference for registered integration add usage by @matiasinsaurralde in chainloop-dev/chainloop#3279
* fix: upgrade Go toolchain to 1.26.5 for GO-2026-4970-stdlib by @chainloop-platform[bot] in chainloop-dev/chainloop#3278
* feat(cli): explode archive materials in chainloop att add by @javirln in chainloop-dev/chainloop#3254
* fix(materials): do not record container image digest as tag by @tknura in chainloop-dev/chainloop#3281
* fix(cli): accept matching --name and metadata.name in contract commands by @tknura in chainloop-dev/chainloop#3282
* fix(dispatcher): improve fanout delivery timeouts and retry budget by @migmartri in chainloop-dev/chainloop#3270
* refactor: remove the public attribute from workflows by @matiasinsaurralde in chainloop-dev/chainloop#3283
* fix(cli): preserve verbatim contract bytes when splitting apply documents by @javirln in chainloop-dev/chainloop#3233


**Full Changelog**: https://github.com/chainloop-dev/chainloop/compare/v1.103.1...v1.104.0</pre>
  <p>View the full release notes at <a href="https://github.com/chainloop-dev/chainloop/releases/tag/v1.104.0">https://github.com/chainloop-dev/chainloop/releases/tag/v1.104.0</a>.</p>
</details>
<hr>

See merge request: Harmonybrew/homebrew-core!14158
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve fanout webhook delivery timeouts and retry budget

3 participants