ci: enrich scheduled-failure issues with an LLM triage pass by tonyandrewmeyer · Pull Request #2663 · canonical/operator

tonyandrewmeyer · 2026-07-24T23:32:53Z

Today, when a scheduled workflow fails, notify-scheduled-failure.yaml opens an issue whose title and body are a generic one-liner plus a link to the job. A human then has to read the logs, check whether it duplicates an existing issue, and rewrite the title and body before it is useful to anyone.

This adds a second stage that does that first pass automatically: extract a deterministic failure signature from the failing jobs' logs, dedupe against real candidate issues, ask an LLM to draft a descriptive issue or comment, validate its output against a schema, and apply it, falling back to today's plain notice at every layer if anything goes wrong.

Stage 1 (notify-scheduled-failure.yaml) keeps its current guarantee: no secrets, no LLM, no network beyond gh, always produces a notification. The only change is a rough de-duplication, searching open issues for the workflow name and commenting on a match instead of opening a duplicate.

Stage 2 (ai-failure-enrich.yaml, new) does the enrichment. It is a separate file called from the notifier after the notify job rather than another job inside it, so that stage 1's guarantee stays legible: everything that needs an API key, an environment or a checkout lives in stage 2.

Because stage 2 runs inside the caller's run, the seven callers have to hand down what it needs: actions: read for the log fetch, and OPENROUTER_API_KEY passed by name.

De-duplication methodology:

Marker match: if a :sig= marker for this exact run id already exists anywhere in the repo's issues, this is a re-run of the same failing jobs. The job comments "re-run attempt still failing" and skip signature extraction and the LLM call entirely.
No API key: plain fallback body, still marker-stamped.
Candidate search: open issues plus recently-closed ones (14 days or fewer, capped at 3, and explicitly labelled "(closed …)" in the block the model sees, with an instruction never to auto-treat a closed issue as a strong match) feed the LLM's comment-vs-new decision.
Schema-invalid output: plain fallback body, with the validation errors written to the job's step summary.
Valid new envelope on a fresh placeholder: gh issue edit the placeholder in place, rather than opening a second issue.
Valid new envelope where stage 1 had commented on an older issue: the coarse match was wrong, so open a genuinely new issue and leave a pointer comment on the older one.

The marker format is  from stage 1 and  from stage 2.

The rough search matches on workflow name alone rather than the full "Scheduled workflow 'X' failed" sentence, because enrichment rewrites the title. The applier always appends a Workflow: <name> footer so the match survives.

When a scheduled workflow fails, notify-scheduled-failure.yaml opens an issue whose title and body are a generic one-liner plus a link to the job. A human then reads the logs, works out whether it duplicates an existing issue, and rewrites the title and body before it is useful to anyone. This adds a second workflow that does that first pass: extract a deterministic failure signature from the failing jobs' logs, dedupe against real candidate issues, ask an LLM to draft a descriptive issue or comment, validate the result against a schema, and apply it -- falling back to the plain notice at every layer if anything goes wrong. The notifier keeps its guarantee of always producing a notification with no secrets and no LLM; its only change is a coarse dedupe by workflow name. The enricher is a separate workflow rather than being spliced into the notifier, so an unprovisioned or broken enricher cannot affect whether a notification happens. It subscribes via workflow_run to the seven scheduled workflows that call the notifier, because a reusable workflow invoked via workflow_call gets no independent run to subscribe to. The script keeps its pure logic -- log parsing, marker handling, prompt building, schema validation -- separate from everything that talks to `gh` or OpenRouter, so the logic is testable without mocking the network. It needs nothing outside the standard library. Two things reviewers should look at: - The `# zizmor: ignore[dangerous-triggers]` on the workflow_run trigger is the first zizmor suppression in this repository, and canonical#2612 removed the only zizmor config three weeks ago by fixing the underlying issue instead. The reasoning for treating this as a false positive is written out at the trigger; the precedent is a decision for reviewers, and the alternative is splicing the enrichment call into all seven callers. - Nothing is provisioned yet: without the `ai-failure-triage` environment and an OPENROUTER_API_KEY, every run takes the no-API-key path and produces today's plain notice, with the coarse dedupe as the only visible change. That makes this safe to merge ahead of the secret. The `gh` read path has been exercised against this repository using run 29847889218; on that run the extractor independently produced `traceback_top_error: "KeyError: 'loki/0'"`, matching the diagnosis a maintainer had written by hand on the issue it opened (canonical#2658). The write calls are covered only by mocks, since exercising them means posting here; they want a workflow_dispatch run after this merges.

Both found by dogfooding in the fork. The `Workflow: <name>` footer that the notifier's coarse search depends on was never written. It existed only in the prompt template, while the notifier's comment, the design and the applier all assumed the applier appended it. Enriched issue #24 in the fork went out without it, and the coarse search kept working only because the model happened to leave the workflow name in the title -- one different title and the issue thread would split permanently. Add render_body(), use it on every create, comment and in-place edit, and cover it with tests. Also make write_step_summary report to stderr as well as the summary file. Every fallback in this script reports through it, so a fallback was invisible in the job log and over the API -- which is exactly what made the second fork run hard to diagnose: it produced a plain-fallback comment with no way to tell whether OpenRouter had errored or the output had failed schema validation.

The LLM path was falling back to the plain body on almost every run. The step summary said `envelope: unknown field(s) ['also']`. `validate_envelope` calls `validate_entry` for the top-level envelope, and `validate_entry` checks unknown keys against a set that does not contain `also`, so the error was appended before `validate_envelope` reached its own unknown-field check, which did exclude `also`. That exclusion was dead code. Since the JSON schema handed to the model declares `also`, the model emits it routinely, so this was the normal path rather than an edge case. Tell validate_entry whether it is looking at the top-level envelope, where `also` is legal, and drop the now genuinely redundant second check. The three existing `also` tests did not catch this because they assert invalidity and match on the substring "also", which the spurious error contained -- they passed for the wrong reason. They now match on the specific message, and there are tests for a valid envelope with `also`, with an empty `also`, and for a genuinely unknown field still being rejected. Verified the new tests fail against the unfixed validator.

Two more from dogfooding, the first of which defeated the whole point of the dedup path. The origin issue was excluded from the candidate pool unconditionally. That is right for a placeholder this run just created, but wrong when the notifier commented on an issue that already existed -- the case for every recurrence after the first. That issue is the most likely duplicate, and removing it left the model with an empty candidate list, so it answered "new" and a duplicate issue was opened with a pointer comment: exactly what this path exists to prevent. Only exclude the origin when we created it. The validator also rejected an envelope for merely containing an inapplicable key, even when its value was null. The schema sent to OpenRouter is `strict`, so models return every declared property and null what does not apply; this discarded good output and fell back to the plain body. Treat null as absent, while still rejecting a real conflicting value. Verified all three new tests fail against the unfixed script.

The model chose `action: comment` -- correctly, now that it can see the matched issue again -- and returned a title, labels and issue_type alongside. Those mean nothing for a comment and were never going to be applied, but the validator treated their presence as an error, so the whole response was discarded and the plain notice went out instead. Drop them before validating and report what was ignored, rather than failing. This is a deliberate loosening: the schema handed to the model is `strict`, so it returns every declared property and fills whatever does not apply to the branch it chose. Policing that costs us the enrichment and buys nothing, since the applier only ever reads the fields for the action. A conflicting value that we *would* act on is still an error. Applies to `also` entries as well as the top-level envelope.

`locate_run_markers` looked the marker up with `gh search issues`. The issue search index is not read-your-writes, and the notifier stamps its marker moments before the enricher runs, so an unindexed marker reads as "no notifier marker found" -- at which point main() takes its missing-marker fallback and opens a *second* issue for a run that already has one. Two threads for one failure, CI green, nothing in the log to say why. Scan the most recently updated issues first instead. The list endpoint has no index lag, and the artefact the notifier just touched is by construction among the most recently updated issues in the repo. Search stays as a fallback for the one case a bounded listing cannot cover: more than RECENT_ISSUE_SCAN issues updated in between, where a stale index still beats no lookup at all. Not passing the number forward from the notifier: the two stages are separate workflow runs, so the only channels are an artefact or the triggering run's log. Downloading an artefact from the triggering run is precisely the thing the enricher's zizmor suppression argues it does not do, and is not worth trading that argument away for. The five new tests all fail against the unfixed lookup.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`plain-fallback` created an issue unconditionally whenever `enrich` did not set `handled`. But by the time it can run, the notifier has always already notified: `workflow_run: completed` only fires once the caller's run, including its `open-issue` job, has finished. So any `enrich` crash -- network down, `uv run` failing, an unhandled exception -- produced two issues for one failure. It also caught `enrich` having applied its result and then died before setting the output. Look for `ai-failure-notifications:run=<id>:` and comment on that issue instead, creating one only when the marker is genuinely absent, which is the one case that means the notifier itself failed. The trailing colon matches the notifier's `:origin=` and the enricher's `:sig=` alike, so both cases above are covered. Lists recently updated issues rather than searching, for the same read-your-writes reason written out at locate_run_markers(): the marker is minutes old, and reading a stale index as "no issue exists" is exactly the duplicate this removes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

tonyandrewmeyer · 2026-07-26T23:43:51Z

@benhoyt adding you as a reviewer specifically around the Zizmor/workflow question (feel free to review anything else as well, of course).

benhoyt

I've only reviewed the Zizmor change - the rationale for that looks reasonable to me.

benhoyt · 2026-08-10T00:57:22Z

Interesting idea, though! Let's see how it works.

james-garner-canonical

I like the high-level idea. I have a number of concerns with the way this PR proposes integrating the workflow+script and its tests into the existing CI -- mostly I wonder if we could simplify it by calling the workflow/script directly and trusting the script to handle the fallback branch.

I've read parts of the script itself -- in particular the system + user prompt seem very reasonable to me. I've held of on reviewing the script further because I imagine it might churn a fair bit if we make the surrounding infra changes I'm suggesting.

james-garner-canonical · 2026-08-17T06:14:01Z

+            gh issue create --repo "$REPO" \
+              --title "Scheduled workflow '$WORKFLOW_NAME' failed" \
+              --body "$issue_body"
+          fi


It seems like it would be simpler to fold the dumb match and comment logic into the fallback branch of the enrich script and just call the enrich workflow here ... or even just fold the workflow in here and call the script directly. If the concern is making the AI parts easy to tear out or disable later, I think we could do that with clearly commented sections of input parameters and in the script itself; or perhaps factor the script into two separate modules.

The enricher is now on: workflow_call and this workflow calls it after open-issue, so the workflow_run trigger, the hardcoded list of seven caller workflow names, and the zizmor: ignore[dangerous-triggers] suppression are all gone. I had tried something along those lines when experimenting with this on my fork but hadn't found this path, which doesn't seem better, thanks! (I'm double-checking it on the fork at the moment.)

I'd like to keep the coarse match as bash in its own job rather than folding it into the script:

open-issue currently runs with nothing but gh and github.token. Folding it in adds a checkout, setup-uv, and the environment as prerequisites for a notification happening at all.

enrich needs environment: ai-failure-triage for the OpenRouter key, and environments can carry protection rules including required reviewers. One job means the notification inherits that, so a misconfigured environment or an approval gate doesn't degrade the notification, it blocks it pending a human for a workflow whose whole purpose is telling people something broke overnight. Admittedly, we would presumably configure that in canonical-repo-automation so hopefully notice it, but it could happen.

The split is down to about 30 lines of bash now that it isn't also carrying the workflow_run argument, which feels ok for a notification path with no checkout, no uv and no environment on it. Happy to revisit if you still think it's not worth it, particularly once all the other changes are verified to be working.

The current split and permissions rationale makes sense to me, thanks.

WDYT about notify outputing the issue number for us (commented on or created), which could simplify the ai_failure_notifier.py script since we wouldn't need to duplicate the issue lookup logic and can just treat it as an input?

If we do want enrich to run even if notify fails, this also lets us cleanly distinguish "we were passed a specific issue" from "we have to track down an issue on our own", so I think outputting the issue number could be helpful even if we end up needing to retain issue lookup logic in the script.

Drop the shebang and the `if __name__ == '__main__'` block: the test was originally written to run standalone beside the script, but it now lives in test/ and is collected by the normal unit run. Drop the inline script metadata with it, since the `requires-python = ">=3.11"` pin it declared would be wrong for a suite CI also runs on 3.10. Wrap the two long fixture strings in explicit parentheses so their continuation lines indent under the key rather than sitting at the dict's own level. `ruff format --preview` leaves this form alone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The enricher subscribed to the notifier's seven scheduled callers by name, because `workflow_run` cannot target a reusable workflow directly. That list had to be kept in sync by hand -- renaming a scheduled workflow would have silently stopped enrichment -- and the trigger needed the repository's only `# zizmor: ignore[dangerous-triggers]`, plus a long comment arguing why the audit did not apply. Call it from the notifier instead, after the `open-issue` job. Both stages then run inside the caller's run, so `github.run_id` and `github.workflow` name the scheduled workflow that failed, which is what the workflow_run payload was supplying. The seven callers are unchanged. Stage 1 keeps its guarantee: `open-issue` still needs no secrets, no LLM and no network beyond `gh`, and it runs to completion before the enricher starts. `if: always()` on the call means a failed stage 1 still reaches the enricher's fallback. zizmor now reports no findings on either file without any suppression. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The fallback job defended against the enricher dying before it could reach its own internal fallback -- network down, `uv run` failing, an unhandled exception. But stage 1 has already created or commented on the issue by then, so an enricher that dies on its own leaves an un-enriched placeholder and a red job, which is the same outcome as before any of this existed. No notification is lost. The case that genuinely needs an issue created is `open-issue` failing *and* the enricher failing, since then nothing has notified anyone. Gate on exactly that. Both stages now run in the same run, so `needs.*.result` says directly whether a notification exists and the `gh issue list` marker search that used to work it out is no longer needed. The `handled` output existed only to drive the old condition, so it goes too, along with `set_output()`, which had no other caller. Also mark the script executable: it has a shebang, and the repo's other shebanged .github scripts are all 100755. The pre-commit hook that checks this only runs on files a commit touches, which is why it went unnoticed. The enricher is 49 lines, from 151. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

This reverts the trigger change from d99b81d and the same-run form of the fallback from f0ce3b7. Measured in the fork (runs 32101849632 and 32102193223): a job's logs are not retrievable through `/actions/jobs/{id}/logs` while its *run* is still in progress, even 51 seconds after that individual job completed, and even with `actions: read` granted. They become available once the run completes. The enricher exists to read those logs, and under `workflow_call` it is part of the run it needs to read. It cannot wait for the run either, since the run cannot complete until it finishes. So `workflow_run` is not a stylistic choice: it is the only trigger under which the logs exist. The same applies to any in-run variant, including an enrich job added to each caller. The second run showed why this matters more than a red job would: with the signature reduced to a job name, the model still produced a fluent, confident diagnosis ("consistent with infrastructure timeout issues seen in previous runs") for a run whose log was three named pytest failures. Green job, plausible issue, invented content. Two things from that work are kept: - `actions: read` on the enrich job, which was missing all along. It is what the log fetch needs, and a `permissions:` block sets every scope it does not name to `none`. - The narrower fallback: create an issue only when nothing has notified, rather than also commenting when enrichment failed. An `enrich` that dies on its own leaves an un-enriched placeholder, which is a red job, not a lost notification. `handled` and `set_output()` stay deleted; the job status plus the marker say enough. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

tonyandrewmeyer · 2026-08-20T02:11:14Z

FYI, I tried the alternative approaches in my fork and couldn't get them to work. The logs don't seem to be available until the (original) workflow finishes, so workflow_call can't get hold of them, and that means that it doesn't have the context to be able to do the enrichment. I'll see if I can find other alternatives but I suspect that moving back to the original approach may be needed.

james-garner-canonical · 2026-08-20T02:20:17Z

FYI, I tried the alternative approaches in my fork and couldn't get them to work. The logs don't seem to be available until the (original) workflow finishes, so workflow_call can't get hold of them, and that means that it doesn't have the context to be able to do the enrichment. I'll see if I can find other alternatives but I suspect that moving back to the original approach may be needed.

Thanks, I saw the explanation in the revert commit message too. I think the inability to get the logs until the workflow completes is all the justification we need for the workflow_run approach.

The notifier script lived in .github/, so its tests could not live beside it: pytest skips dot-directories, and anything under .github/ never runs in CI. They went in test/ instead, alongside the tests for ops itself, and reached the script through a sys.path insert. Move both into a top-level scripts/, with the tests in scripts/test/, which the normal unit run collects without any path manipulation -- `pythonpath = ["scripts"]` in pyproject.toml lets the tests import the script by bare module name, since these are standalone scripts rather than a package. pyright's include gains scripts/test/*.py, matching the coverage the tests had under test/*.py, and its extraPaths follows the script to scripts/. The script itself stays outside include, as it was in .github/; putting a 1400-line script under strict mode is a separate change. release.py and the other .github scripts belong here too, but that is a follow-up rather than more churn in this PR. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

james-garner-canonical

Thanks for all the work and investigation on this. I like the shape of the workflows now. I've read through them and left comments -- mostly cutting down excessive LLM comments. The only scheduled workflow I commented on is smoke.yaml, but my comments there apply equally to all the other scheduled workflows.

I have two suggestions for hand-off from notify to enrich:

notify could output the issue number, and enrich could take it as explicit input, so it doesn't need to fish through recent issues looking for a match -- if it doesn't get an issue number as input, then notify definitely failed.
If notify fails, I still think that bailing out is fine because we wouldn't expect gh issue comment/create to suddenly pass in the next job.

If you're not convinced by those suggestions, then we can stick with the current approach.

I haven't read much of ai_failure_notifier.py yet, just because it's 1500 LOC of code that's tricky to follow on Github.

I'm not sure what the best move here is:

The various chunks of code do seem to be semantically grouped, but maybe it would be easier if collections of functions were grouped onto classes (even as static/class methods), or split into separate modules?
Maybe just arranging things in a review-friendly ordering would be enough (main, then what main calls, and so on, with related functions grouped into sections ordered similarly)?
Maybe a thorough read-through of the code isn't necessary here -- this isn't library code, so I'm not intending to review it to that standard, but do we still want someone to have read it all?

If you'd like to keep the script as-is, I'll try to read through it in an IDE instead of just on Github.

james-garner-canonical · 2026-08-24T04:57:38Z

+            gh issue create --repo "$REPO" \
+              --title "Scheduled workflow '$WORKFLOW_NAME' failed" \
+              --body "$issue_body"
+          fi


WDYT about notify outputing the issue number for us (commented on or created), which could simplify the ai_failure_notifier.py script since we wouldn't need to duplicate the issue lookup logic and can just treat it as an input?

If we do want enrich to run even if notify fails, this also lets us cleanly distinguish "we were passed a specific issue" from "we have to track down an issue on our own", so I think outputting the issue number could be helpful even if we end up needing to retain issue lookup logic in the script.

Co-authored-by: James Garner <james.garner@canonical.com>

The notify job now outputs the issue it created or commented on, and enrich takes it as an input, so the script is told which artefact to upgrade instead of searching for the marker notify stamped moments earlier. GitHub's issue search index is not read-your-writes, so that search could miss the marker and open a second issue for a run that already had one. Both inputs are optional and enrich still runs on always(), so a notify that fails before opening anything leaves them empty and the script falls back to looking the issue up, as it does today. Also drops a duplicated `issues: write` from the enrich job's permissions, which yamllint reports as an error, and moves the comment that explains `actions: read` to sit above it rather than above the key it is not about. The trailing whitespace fix on line 5 is pre-commit's, not mine.

…r it Review suggestion on canonical/operator#2663: have the notify job output the issue it created or commented on, so this script is told which artefact to upgrade rather than looking it up. The lookup it replaces existed to work around GitHub's issue search index not being read-your-writes: the notifier stamps its marker seconds before the enrich job runs, so a search can read "no marker found" and open a second issue for a run that already has one. A number passed through the workflow cannot be stale, so that failure mode is gone rather than defended against. It does not remove the lookup entirely, which the suggestion allowed for. Rung zero is a fact about an earlier run of this script, not about what the notifier just did, so it still has to be looked up - but knowing the issue narrows that from a scan of the repo's recently updated issues to reading the one issue we were handed. With no issue passed, from an unmigrated caller or a notifier that failed before opening anything, the original repo-wide scan still runs.

Review suggestion: enrich ran on always(), so a failed notify still reached it and the script's own fallback opened the issue instead. Both jobs authenticate the same way and both shell out to gh, so the failures that stop notify - authentication, infra being down - stop enrich as well, and the fallback was defending against a class of failure it cannot actually survive. What it does give up is the narrow case of a transient failure in notify's issue search, where the fallback would have produced a notification and now nothing will: the scheduled workflow just goes red, which is where this was before any of it existed. The script keeps its lookup fallback regardless. Inside this repository it is now unreachable, but repositories adopt this at their own pace and a caller that has not been migrated passes no issue number at all.

Review suggestion. That a called workflow holds no scope its caller withheld is documented upstream, and the enricher has a single job, so saying there that its actions: read is for fetching logs is not telling a reader anything the next few lines do not. Removed from both workflows for the same reason, not just the one that was commented on. The secrets comment below stays, and the difference is the point: it records behaviour that is not documented anywhere and was measured across three fork runs, where removing the pass-through it describes silently degrades enrichment rather than failing.

Review suggestion asked whether this needs documenting here. It does not: the same fact was written out in both workflows, and this was the shorter, vaguer copy, in the file that does not declare the environment it is about. The full note stays in ai-failure-enrich.yaml, next to the environment: key and the fork runs that measured it. What is left here is one line saying the pass-through is required and where to read why, which is the part that stops someone removing a line that provably passes an empty string.

Review suggestion, applied to all seven callers rather than only the one it was left on. Each of them carried the same eleven lines explaining GitHub's permission model and the secret mechanics, so the fact was written out nine times across this PR once the two called workflows are counted. What a caller actually needs to know is that both lines exist for the enricher and that the key only resolves inside its environment. That is two comments. The mechanics are documented once, in ai-failure-enrich.yaml.

Review suggestion, taken as offered, plus the terse numbered list it proposed in place of the two-stage paragraph. The permission-model sentence goes for the same reason as the others: it describes GitHub, not this workflow. The numbered list keeps the one claim worth keeping, that notify is the guarantee and enrich is best effort, which is why they are separate jobs at all.

Review suggestion on canonical/operator#2663, where the choice was between dropping it and switching to a uv shebang so CI could execute the file directly. Moving here settles it: this is a module inside an installed package, reached through the ai-failure-notifier console script, so nothing executes it by path and the line is dead text. The file was already not executable.

Asked for in review on canonical/operator#2663: 1500 lines is hard to follow on GitHub, and the reviewer offered to read it in an IDE instead if we would rather leave it. Splitting is the better answer, and it is cheap here in a way it was not in operator - there is no in-flight review of these files to disturb. The boundaries are the ones the single file already documented with its `# --- section ---` banners, plus the I/O half divided by what it talks to: gh, OpenRouter, the step summary, and applying the result. Largest module is now 293 lines. `__init__` re-exports every public name, so `from charm_tech_code import ai_failure_notifier` is unchanged for callers. Cross-module function calls go through the module rather than importing the name, so that a test patching `<module>.<name>` reaches every call site instead of only the definer. Those imports are aliased with a leading underscore because three module names - envelope, prompt, summary - are also local variable names in the code. No assertion changed. The test diff is entirely patch targets moving from `afn.<name>` to `afn.<module>.<name>`, which is what makes the same 75 tests evidence that this refactor preserved behaviour.

tonyandrewmeyer · 2026-08-26T02:33:13Z

I've moved most of this to canonical/charm-tech-code#1 (I still need to update this PR to reflect that), and tried to address all the comments here in that PR since I can't move the comments across.

The script and its tests now live in canonical/charm-tech-code, where the other Charm Tech team tooling is going. This repo keeps the workflow half: the notifier changes, the caller `secrets:` lines, and the enrich workflow, which reaches the code through `uvx --from git+...` against a pinned SHA -- the same trust decision as every `actions/checkout@<sha>` pin above it. The `pyproject.toml` changes go with the script: `pythonpath`, the pyright include and extraPaths entries, and the ruff per-file ignore all existed only to let `scripts/test/` run here, and scripts/ is now empty. The pin is on `move-ai-failure-notifier`; it wants re-pointing at the merge commit once that branch lands.

…tifications-step-5

The SHA this pointed at was on charm-tech-code#1's branch, and that PR was squash-merged, so the commit no longer exists in the repository at all and `uvx --from` would fail for every enrichment run. e83851e is the squashed commit on `main`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015aqSP396W4snz51WdgZg1v

tonyandrewmeyer · 2026-09-10T01:12:32Z

@tromai and @james-garner-canonical this is now much smaller since it uses the code in canonical/charm-tech-code. Would you prefer a fresh PR that doesn't have all the baggage here from when it had the code as well or to keep things here with all the historic context (in which case I'll update the PR body)?

james-garner-canonical · 2026-09-10T02:22:44Z

I'd say update this one.

tromai · 2026-09-10T04:51:16Z

@tonyandrewmeyer Thanks for the update. I'm happy with keeping it here, and update the PR.

The comments referred to development history the repository has never seen: fork run ids as evidence for the secret plumbing, and an input that used not to be passed. Nothing here has run yet, so there is no "before" to point at. Also plainer wording in the same comments - "load-bearing" and "artefact" are not how anyone reading this at 3am will think about it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013trjz242SkZ2SxPami6XE7

Rebuilt on the current canonical#2663 head. The old branch was deleted from the fork, and the design it sat on is gone anyway: it was written against the workflow_call restructure that was later reverted and then restored, and against a chain that used `secrets: inherit`. Two changes from the version that was deleted. It runs on push to its own branch as well as on dispatch, because every workflow in the chain is `workflow_call` now and nothing needs to sit on the default branch any more - which retires the force-push-to-main-and-revert procedure the notes describe. And the shape defaults to pytest rather than being empty on a push, since an unmatched `case` emits no log at all, which is the degraded input every enriched artefact this project has produced so far was built from. The call site names the secret explicitly instead of inheriting it, matching what the notifier and enricher now expect. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PvancUDWdnqEwbocH7fSeX

The pinned commit created two identical issues where the repository has no issue type matching the one the model asked for, which is every enrichment on a fork and, because the model is asked for "bug" against GitHub's "Bug", would have been every defect here too. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013trjz242SkZ2SxPami6XE7

Rebuilt on the current canonical#2663 head. The old branch was deleted from the fork, and the design it sat on is gone anyway: it was written against the workflow_call restructure that was later reverted and then restored, and against a chain that used `secrets: inherit`. Two changes from the version that was deleted. It runs on push to its own branch as well as on dispatch, because every workflow in the chain is `workflow_call` now and nothing needs to sit on the default branch any more - which retires the force-push-to-main-and-revert procedure the notes describe. And the shape defaults to pytest rather than being empty on a push, since an unmatched `case` emits no log at all, which is the degraded input every enriched artefact this project has produced so far was built from. The call site names the secret explicitly instead of inheriting it, matching what the notifier and enricher now expect. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PvancUDWdnqEwbocH7fSeX

A `target_issue` the model wrote as "#44" failed schema validation and threw away the enrichment, which is what the pinned commit did on the last run here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013trjz242SkZ2SxPami6XE7

Rebuilt on the current canonical#2663 head. The old branch was deleted from the fork, and the design it sat on is gone anyway: it was written against the workflow_call restructure that was later reverted and then restored, and against a chain that used `secrets: inherit`. Two changes from the version that was deleted. It runs on push to its own branch as well as on dispatch, because every workflow in the chain is `workflow_call` now and nothing needs to sit on the default branch any more - which retires the force-push-to-main-and-revert procedure the notes describe. And the shape defaults to pytest rather than being empty on a push, since an unmatched `case` emits no log at all, which is the degraded input every enriched artefact this project has produced so far was built from. The call site names the secret explicitly instead of inheriting it, matching what the notifier and enricher now expect. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PvancUDWdnqEwbocH7fSeX

An OpenRouter failure reported only its status code, so a 400 here said nothing about which of the model, the key or the schema it objected to. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013trjz242SkZ2SxPami6XE7

Rebuilt on the current canonical#2663 head. The old branch was deleted from the fork, and the design it sat on is gone anyway: it was written against the workflow_call restructure that was later reverted and then restored, and against a chain that used `secrets: inherit`. Two changes from the version that was deleted. It runs on push to its own branch as well as on dispatch, because every workflow in the chain is `workflow_call` now and nothing needs to sit on the default branch any more - which retires the force-push-to-main-and-revert procedure the notes describe. And the shape defaults to pytest rather than being empty on a push, since an unmatched `case` emits no log at all, which is the degraded input every enriched artefact this project has produced so far was built from. The call site names the secret explicitly instead of inheriting it, matching what the notifier and enricher now expect. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PvancUDWdnqEwbocH7fSeX

github-advanced-security Bot found potential problems Jul 24, 2026

View reviewed changes

Comment thread .github/workflows/ai-failure-enrich.yaml Fixed

Comment thread .github/workflows/ai-failure-enrich.yaml Fixed

tonyandrewmeyer force-pushed the ai-failure-notifications-step-5 branch from 9371af3 to 41325bb Compare July 25, 2026 00:55

tonyandrewmeyer and others added 8 commits July 25, 2026 13:51

test: keep the notifier test fixtures under the 99-column limit

e16680b

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

test: apply ruff --preview formatting to the notifier tests

239f8fc

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

tonyandrewmeyer marked this pull request as ready for review July 26, 2026 23:41

tonyandrewmeyer requested review from benhoyt, james-garner-canonical and tromai July 26, 2026 23:42

benhoyt reviewed Aug 10, 2026

View reviewed changes

james-garner-canonical reviewed Aug 17, 2026

View reviewed changes

tonyandrewmeyer and others added 3 commits August 18, 2026 16:49

james-garner-canonical reviewed Aug 18, 2026

View reviewed changes

Comment thread .github/workflows/notify-scheduled-failure.yaml Outdated

tonyandrewmeyer marked this pull request as draft August 18, 2026 05:11

james-garner-canonical reviewed Aug 18, 2026

View reviewed changes

Comment thread .github/workflows/notify-scheduled-failure.yaml Outdated

james-garner-canonical reviewed Aug 18, 2026

View reviewed changes

Comment thread .github/workflows/notify-scheduled-failure.yaml Outdated

tonyandrewmeyer marked this pull request as ready for review August 24, 2026 03:34

tonyandrewmeyer requested a review from james-garner-canonical August 24, 2026 03:34

james-garner-canonical reviewed Aug 24, 2026

View reviewed changes

Apply suggestions from code review

f15bcf1

Co-authored-by: James Garner <james.garner@canonical.com>

tonyandrewmeyer marked this pull request as draft August 26, 2026 00:10

tonyandrewmeyer added 5 commits August 26, 2026 12:39

tonyandrewmeyer mentioned this pull request Aug 26, 2026

feat: add an ai-failure-notifier workflow and script canonical/charm-tech-code#1

Merged

tonyandrewmeyer and others added 3 commits September 1, 2026 09:14

Merge remote-tracking branch 'upstream-https/main' into ai-failure-no…

e101469

…tifications-step-5

Uh oh!

Conversation

tonyandrewmeyer commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tonyandrewmeyer commented Jul 26, 2026

Uh oh!

benhoyt left a comment

Choose a reason for hiding this comment

Uh oh!

benhoyt commented Aug 10, 2026

Uh oh!

james-garner-canonical left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

james-garner-canonical Aug 17, 2026

Choose a reason for hiding this comment

Uh oh!

tonyandrewmeyer Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

james-garner-canonical Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

james-garner-canonical Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tonyandrewmeyer commented Aug 20, 2026

Uh oh!

james-garner-canonical commented Aug 20, 2026

Uh oh!

james-garner-canonical left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

james-garner-canonical Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tonyandrewmeyer commented Aug 26, 2026

Uh oh!

tonyandrewmeyer commented Sep 10, 2026

Uh oh!

james-garner-canonical commented Sep 10, 2026

Uh oh!

tromai commented Sep 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

tonyandrewmeyer commented Jul 24, 2026 •

edited

Loading