[answersamir]

Problem

On the lazarus ring, four default k8s-statefulset-healthcheck tasks record no output on healthy StatefulSets. Skill Builder batch ae8711cb7ea5 scored 11 plan steps as data gaps because of it. Across the batch's 15 run sessions (12 StatefulSet SLXs):

Task	Result on main	Robot error
Check for Container Restarts	FAIL 12/12	json.loads(r'''No containers with restarts found for StatefulSet … in the last 10m.''') → JSONDecodeError
Analyze Application Log Patterns	FAIL 6/6	Variable '${LOG_PATTERN_CATEGORIES_STR}' not found
Detect Log Anomalies	FAIL 1/1	Value of variable '@{issue["next_steps"]}' is not list or list-like
Inspect StatefulSet Replicas	PASS 12/12, empty report	(the healthy path never calls Add Pre To Report)

RBAC, the kubeconfig and the runner upload all work: other tasks in the same run requests uploaded output.

Causes and fixes

Check for Container Restarts

• Cause: container_restarts.sh prints plain text when nothing restarted, and narrative lines before the JSON when something did. The runbook runs json.loads() on stdout, so the task fails in both cases. It never raises a real restart issue.
• Fix: the narrative now goes to stderr, and stdout is always a JSON array. The runbook parses it defensively and raises an "Output Unreadable" issue when it cannot. It always adds a summary (window, threshold, issue count, narrative) to the report. It also defines ${issue_timestamp} on the script-failure path, where it was undefined.

Inspect StatefulSet Replicas

• Cause: the healthy path adds nothing to the report.
• Fix: the task always reports desired/ready/current/updated. It raises "Unable to Inspect StatefulSet Replicas" when kubectl … | jq returns nothing. There is no pipefail, so a Forbidden get used to exit 0 and pass silently.

Analyze Application Log Patterns

• Fix: Set Suite Variable ${LOG_PATTERN_CATEGORIES_STR}, which is what k8s-deployment-healthcheck already does.

Detect Log Anomalies

• Cause: RW.K8sLog.Analyze Log Anomalies returns next_steps as a newline-joined string.
• Fix: join it only when it is a list.

Restart script, second commit

• Cause: GNU date -d "10m ago" is invalid, so THRESHOLD_TIME was empty and CONTAINER_RESTART_AGE was ignored. Any restart in a pod's lifetime counted.
• Fix: expand the m/h suffix the way k8s-deployment-healthcheck does. A failed kubectl get statefulset or get pods now exits 1, so the runbook raises "Container Restart Analysis Failed" instead of reading [] as "no restarts".

k8s-daemonset-healthcheck, third commit

• Cause: this bundle is a copy and fails the same three ways (restarts, log patterns, anomalies). Its status task already reports.
• Fix: the same changes, ported.

Testing

I ran runbook.robot in a Debian container with rw-core-keywords in production mode, so it writes report.jsonl and issues.jsonl. A stub kubectl replayed the real rw-lazarus-vault StatefulSet and pod JSON captured from lazarus, plus synthetic logs and restart variants.

Fixture	main	this branch
healthy StatefulSet	3 FAIL + Replicas empty; error text identical to lazarus	10/10 PASS, every task writes a report line
exit 137, 2 min ago	Restarts FAIL (json.loads on narrative + JSON)	PASS, 1 issue
exit 137, 2 h ago, CONTAINER_RESTART_AGE=10m	—	PASS, 0 issues (the window is honoured)
CONTAINER_RESTART_AGE=2d	—	"Container Restart Analysis Failed" issue (unsupported unit)
every get/logs Forbidden	Replicas and Restarts PASS silently	both raise issues
the same objects presented as a DaemonSet	the same 3 failures	10/10 PASS with output

Not changed (pre-existing, RBAC-only):

• Under Forbidden, the two log tasks still fail, because Fetch Workload Logs raises.
• Under Forbidden, the probe and PVC tasks pass empty.
• Under Forbidden, Inspect DaemonSet Status reports Ready=0/0 without an issue.