Skip to content

Added Variance Check - #706

Merged
AlexanderKaran merged 1 commit into
mainfrom
variance-its-back
Oct 27, 2026
Merged

AlexanderKaran merged 1 commit into
mainfrom
variance-its-back

Conversation

@AlexanderKaran

Copy link
Copy Markdown
Collaborator

Brought Variance back as a top-level variable. So every time a full run of a benchmark happens, and we re-create its top level result we add a variance key that compares it to the last version, like so:

{
  "name": "coldBuild",
  "runs": 5,
  "iterations": [
    { "timeMs": 955 },
    { "timeMs": 930 },
    { "timeMs": 925 },
    { "timeMs": 936 },
    { "timeMs": 967 }
  ],
  "timeMs": {
    "mean": 942.6,
    "median": 936,
    "min": 925,
    "max": 967,
    "variance": {
      "comparedToMeasuredAt": "2026-10-27T03:16:33.289Z",
      "meanPercent": -9.10,
      "medianPercent": -1.89,
      "minPercent": 3.24,
      "maxPercent": -29.88
    }
  }
}

Only compares if the versions are the same. Then, when the stats PR is generated, a special CI/CD check runs; if any of the benchmark results are bigger than a 10% swing, it flags it to re-run the test.

We can, of course, lower this number and later configure different values per benchmark. Not storing historical data for now just compare the current one each time until you bump versions

@cloudflare-workers-and-pages

Copy link
Copy Markdown
Contributor

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
framework-tracker 8d31880 Commit Preview URL

Branch Preview URL
Oct 27 2026, 12:49 PM

@AlexanderKaran
AlexanderKaran merged commit 3e9910b into main Oct 27, 2026
79 checks passed
@AlexanderKaran
AlexanderKaran deleted the variance-its-back branch October 27, 2026 22:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants