Track fleet setup failures as separate incidents

README.md

@@ -45,11 +45,11 @@ that should follow it. You do not need to register either file here.
 
 The [fleet checker](docs/node-fleet-setup.md) inspects each default branch, honors
 local overrides, and reports failed or unverifiable setup in the host repository.
-It reuses one issue per inventory, closes it when every static check passes, and
-reopens it on recurrence. This repository checks ten public repositories weekly
-on Mondays at 03:00 UTC. Keep private inventories and reports in a private host;
-other hosts choose their own caller schedule. Checks run five repositories at a time in one
-runner, with one combined report and issue.
+It keeps one issue per failure episode, closes it when every static check passes,
+and creates a new issue after a later regression. This repository checks Hydra
+and OmegaConf weekly on Mondays at 03:00 UTC. Keep private inventories and reports
+in a private host; other hosts choose their own caller schedule. Checks run five
+repositories at a time in one runner, with one combined report and issue.
 
 ## Documentation
 

docs/node-fleet-setup.md

@@ -136,17 +136,29 @@ Shared defaults and pinned repository snapshots are reused across checks.
 The scan continues after an individual repository fails. Any failed or unverified
 result fails the scanner job and opens or updates one **Node maintenance setup check** issue
 in the host. The reporter still runs after scan failures. A fully passing scan
-closes that issue; recurrence reopens it. Repeated failures update the body
-without per-run comments. An invalid or unreadable inventory also reports against
-that list when its repository/path identity is valid.
+closes that issue and records recovery. A later regression creates a new issue.
+Repeated failures, including changed findings, update the current issue body
+without per-run comments. If you close an unresolved issue manually, the next
+failing scan reopens that issue. An invalid or unreadable inventory also reports
+against that list when its repository/path identity is valid.
 
 The issue starts with a results table and links to each repository. Exact commit
 identities are in the expandable **Checked revisions** section.
 
 Issue identity uses the inventory repository and path, so changing its contents
-or ref keeps the same issue. Different paths have separate issues. Runs are
-serialized in the host to avoid competing issue creation. Existing ambiguous
-duplicate marked issues cause a reporting error instead of another issue.
+or ref keeps the current incident. Different paths have separate incidents. Runs
+are serialized in the host to avoid competing issue creation. The reporter reads
+bot-created issues from newest to oldest and stops at the latest matching issue;
+it does not scan older incidents or add comments on every run. Newer unrelated
+bot issues still count toward the lookup limit of 100 pages of 100 entries. If
+the lookup cannot finish, it fails without creating another issue.
+
+Recovery is recorded in a hidden marker in the issue body. Keep this marker when
+editing an issue. Existing issues from earlier workflow versions are adopted as
+unresolved until a scan with the new version confirms recovery, even if already
+closed. Fully passing scans then leave the closed incident unchanged. No new
+permissions or external state store are needed.
+
 Invalid inventory location inputs that cannot identify a safe list fail the
 workflow without opening an issue.
 

node/runtime/fleet_issue.py

@@ -1,4 +1,4 @@
-"""Maintain one central setup issue per inventory, without touching consumers."""
+"""Maintain one setup issue per failure episode, without touching consumers."""
 
 import argparse
 import hashlib
@@ -7,13 +7,16 @@
 import os
 import re
 import runpy
+import urllib.parse
 from pathlib import Path
 
 ISSUES = runpy.run_path(str(Path(__file__).with_name("configuration_issue.py")))
 CONFIGURATION = runpy.run_path(str(Path(__file__).with_name("configuration.py")))
 MAX_ISSUE_BYTES = ISSUES["MAX_ISSUE_BYTES"]
 DIAGNOSTIC_LIMIT = 1000
 TRUNCATION_SUFFIX = "… [truncated]"
+RESOLVED = "<!-- node-fleet-incident:v1:resolved -->"
+ACTIVE = "<!-- node-fleet-incident:v1:active -->"
 
 
 def validate_report(report):
@@ -97,6 +100,93 @@ def marker(source):
     )
 
 
+def latest_issue(repository, identity, request):
+    """Stop at the newest matching bot issue; older incidents are historical."""
+    for page in range(1, 101):
+        query = urllib.parse.urlencode(
+            {
+                "state": "all",
+                "creator": ISSUES["BOT"],
+                "sort": "created",
+                "direction": "desc",
+                "per_page": 100,
+                "page": page,
+            }
+        )
+        issues = request("GET", f"repos/{repository}/issues?{query}")
+        if not isinstance(issues, list):
+            raise ValueError("GitHub did not return an issue list; no issue created")
+        for issue in issues:
+            if not isinstance(issue, dict):
+                raise ValueError("GitHub returned an invalid issue; no issue created")
+            author = issue.get("user")
+            if (
+                "pull_request" not in issue
+                and isinstance(author, dict)
+                and author.get("login") == ISSUES["BOT"]
+                and author.get("type") == "Bot"
+                and isinstance(issue.get("body"), str)
+                and issue["body"].startswith(identity + "\n")
+            ):
+                if (
+                    type(issue.get("number")) is not int
+                    or issue["number"] <= 0
+                    or issue.get("state") not in ("open", "closed")
+                ):
+                    raise ValueError("GitHub returned an invalid marked fleet issue")
+                return issue
+        if len(issues) < 100:
+            return None
+    raise ValueError(
+        "Cannot find latest fleet issue within lookup limit; no issue created"
+    )
+
+
+def maintain_incident(repository, *, identity, title, body, failing, request):
+    if not body.startswith(identity + "\n") or len(body.encode()) > MAX_ISSUE_BYTES:
+        raise ValueError("invalid marked fleet issue body")
+    issue = latest_issue(repository, identity, request)
+    recovered = issue is not None and issue["body"].startswith(
+        identity + "\n" + RESOLVED + "\n"
+    )
+    if not failing and (issue is None or (recovered and issue["state"] == "closed")):
+        return {"operation": "unchanged", "number": issue["number"] if issue else None}
+    if issue is None or (failing and recovered):
+        result = request(
+            "POST", f"repos/{repository}/issues", {"title": title, "body": body}
+        )
+        operation = "created"
+    else:
+        state = "open" if failing else "closed"
+        if issue["state"] == state and issue["body"] == body:
+            return {"operation": "unchanged", "number": issue["number"]}
+        result = request(
+            "PATCH",
+            f"repos/{repository}/issues/{issue['number']}",
+            {"body": body, "state": state},
+        )
+        operation = (
+            "closed"
+            if not failing
+            else "reopened"
+            if issue["state"] == "closed"
+            else "updated"
+        )
+        if not isinstance(result, dict) or result.get("number") != issue["number"]:
+            raise ValueError(
+                "Fleet issue update identity changed; no additional issue created"
+            )
+    if (
+        not isinstance(result, dict)
+        or type(result.get("number")) is not int
+        or result["number"] <= 0
+    ):
+        raise ValueError(
+            "GitHub did not return a fleet issue identity; inspect the run before retrying"
+        )
+    return {"operation": operation, "number": result["number"]}
+
+
 def update_issue(repository, report, run_url, request=ISSUES["github_request"]):
     validate_report(report)
     source = report["inventory"]
@@ -150,6 +240,7 @@ def cell(value):
         )
         lines = [
             marker(source),
+            RESOLVED if report["status"] == "ok" else ACTIVE,
             "",
             "## Node maintenance setup check",
             "",
@@ -195,7 +286,7 @@ def cell(value):
                 "",
                 f"[Workflow run]({run_url})",
                 "",
-                "This issue is reused for this inventory. A fully passing scan closes it; a later failure or unverifiable setup reopens it.",
+                "This issue tracks one failure episode for this inventory. A fully passing scan closes it; a later regression creates a new issue. Closing it manually while problems persist causes it to reopen on the next scan.",
             ]
         )
         return "\n".join(lines)
@@ -217,9 +308,9 @@ def cell(value):
                 "fleet issue body exceeds its fixed identity and status budget"
             )
         body = best
-    return ISSUES["maintain_issue"](
+    return maintain_incident(
         repository,
-        marker=marker(source),
+        identity=marker(source),
         title="Node maintenance setup check: " + source["path"][:180],
         body=body,
         failing=report["status"] != "ok",