Skip to content

feat: record job queue/run time history from an incremental run feed - #19

Merged
JustMaris merged 4 commits into
mainfrom
feat/run-history
Oct 4, 2026
Merged

JustMaris merged 4 commits into
mainfrom
feat/run-history

Conversation

@JustMaris

Copy link
Copy Markdown
Member

Replaces the per-workflow latest-run polling with an incremental run feed per repo, so Prometheus can hold job run/queue time history (avg / p95 per repo, workflow, job, runner and runner pool).

What changes

  • Each org refresh lists the runs created since a per-repo watermark (/actions/runs?created=>=…, paginated) and fetches /runs/{id}/jobs once for each newly completed run.
  • New metrics:
    • github_job_queue_seconds histogram: created_at → started_at
    • github_job_run_seconds histogram: started_at → completed_at
    • github_workflow_runs_total{repo,workflow,conclusion} counter
  • Histogram labels are repo, workflow, job, runner, runner_label, conclusion. runner_label is the job's runs-on labels minus self-hosted (oracle-x64 / oracle-arm64). GitHub-hosted runners are collapsed to runner="github-hosted". Buckets run from 5s to 60m.
  • github_repo_ci_last_run_* keep their meaning. They're updated from the feed and filled at startup by one bootstrap pass with LatestRunForWorkflow. They no longer disappear while a workflow's newest run is in progress.
  • Watermark: the oldest in-progress run, else the newest run seen. The query deliberately has no status=completed filter: with it, a slow run created before a faster one would be skipped for good. An in-progress run holds the watermark back for at most 24h.
  • Dedupe: by (run ID, attempt) and by job ID, both kept for 48h. Overlapping polls and re-runs never double-count. A restart starts from now and replays nothing.
  • Not recorded: skipped jobs and jobs cancelled before a runner picked them up.
  • Fix: the first scrape no longer starts a background build and a blocking build.
  • README (Run history section with PromQL examples) and CLAUDE.md updated.
  • Dashboard (uid unchanged): new repo/window variables and a CI History row with:
    • a table of avg/p95 run and queue time per job
    • p95 and average run time over time
    • queue time by runner pool and by runner
    • runs per day by conclusion

API calls

Measured on drumandbytes (26 repos, 156 active workflows, ~1000 runs/week) by counting real requests against the org:

per org refresh per hour (5m TTL)
before (latest run per workflow) 236 ~2,830
after (run feed) 106 + 1 per newly finished run (~6/h) ~1,280

Startup bootstrap is a one-off 262. With the runner poll, total usage is ~28% of the 5,000/h budget, down from ~59%.

Tests

These are the first Go unit tests in the repo:

  • internal/github: pagination for runs and jobs, and the encoding of the created>= filter, against an httptest server.
  • internal/orgstats, using a fake client:
    • bootstrap, including a run that's in flight at startup
    • watermark advance, pinning by an in-progress run, the 24h cap, and retry after a failed jobs call
    • dedupe across overlapping polls and a re-run
    • bucket placement
    • skipped, unassigned and hosted jobs
    • Build's per-repo error swallowing and its error propagation
  • promtool: every dashboard query parses, and the alert tests still pass.

Replace per-workflow latest-run polling with a per-repo run feed: list runs
created since a watermark, fetch jobs once per newly completed run, and record
github_job_queue_seconds, github_job_run_seconds and github_workflow_runs_total.
The github_repo_ci_last_run_* snapshot is kept, seeded by a startup bootstrap.

Measured on drumandbytes: 106 API calls per org refresh, down from 236.
@JustMaris
JustMaris merged commit 2efef67 into main Oct 4, 2026
8 checks passed
@JustMaris
JustMaris deleted the feat/run-history branch October 4, 2026 15:45
@dnb-robot dnb-robot Bot mentioned this pull request Oct 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant