Skip to content

About

An agent that **grades a student's calculus solution**: locates the first incorrect step, classifies the error, and assigns partial credit according to a specific instructor's marking style, inferred from previously marked exams.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Calculus Grading Agent

The Problem That Started This

You just bombed a practice midterm. You got 3π/8 but the answer key says 3π/4... now what? Did you drop a constant somewhere, or did you botch the approach (there goes my chances of an A)? The answer key doesn't say. Neither does ChatGPT, which just tells you "wrong number, next problem!" before getting stuck in a prompt loop that leaves you confused on what you actually did right or wrong.

This project? It grades the way an actual TA does. It finds the exact step that broke, tells you what kind of mistake it was, and assigns partial credit for all the math that was up to snuff. Point it at a stack of something that shows how your professors mark first, and it learns how that specific instructor grades, whether they're merciful on method or brutal on final answers, what makes them nuke a solution outright. That's huge for consistency when you're grinding through midterms off the same rubric.

Why Claude Alone Isn't Enough

Here's something funny: frontier models are basically saturated at solving calculus problems (the best ones sit around 96% on MATH-500, the standard benchmark for this stuff). I know at least a dozen people in all of my calculus classes using it to get past the homework, and that was a year ago, and with the pace that GenAI moves at, it's outdated. Unfortunately, that solving ability doesn't carry over to checking someone else's work. It's already difficult for some of these models to read most engineering students handwriting. When you ask them to "grade this out of 10," they read the whole solution, see where it lands, and quietly patch over any errors in the reasoning to make it fit. It's like they're working backward from the final answer instead of forward through the logic.

This agent breaks that pattern on purpose. Each step gets verified in isolation, seeing only the work that came before it, never the later steps, never the final answer. The model can't anchor on where the solution is heading, so it actually has to evaluate whether each step is valid on its own terms.

How the Magic Happens

  1. Per-step verification: Each step gets its own call, seeing only prior steps. No peeking ahead!
  2. Error location + credit assignment, together: One call takes every step's verdict and decides both where the mistake is and how many points it costs (plus the instructor's grading style, if a profile's loaded).
  3. Targeted practice: If there's an error, generate 3 practice problems that nail that specific mistake.

That "together" in step 2 isn't how I originally built this. I split error-location and credit- assignment into two separate calls on purpose, on the theory that one call doing two jobs is worse than two calls each doing one. Turns out I was wrong about that - see the results section for the actual story, because it's a better one than "here's the architecture."

Improvement Changelog

(What the evals actually showed.)

I ran 12 frozen eval cases before writing a single line of this code. Conservative fields, Green's and Stokes' theorem, coordinate conversions, plus a sneaky case where two sign errors cancel out and the final answer somehow comes out right anyway.

Every number below is a mean ± standard deviation across 4-5 independent trials per config, not a single run - which matters, because a single run is exactly what got this wrong the first time. Read on.

Stage What Changed Points MAE, raw (mean ± stdev, n) Points MAE, % of total (mean, n) First-error accuracy (mean, n) Error-type accuracy (mean, n)
Baseline Single prompt, grade out of N, no scaffolding 0.865 ± 0.062 (n=4) 9.7% (n=4) — —
Iter 1 (Final) Per-step verification + combined error/credit call 0.517 ± 0.181 (n=5) 6.2% (n=5) 0.967 (n=5) 0.400 (n=5)
Iter 2 Split error-location and credit into separate calls 0.823 ± 0.253 (n=4) 9.9% (n=4) 0.833 (n=4) 0.417 (n=4)
Iter 3 + grading profile from 3 marked exemplars 1.042 ± 0.189 (n=4) 11.2% (n=4) 0.854 (n=4) 0.292 (n=4)
Iter 4 + grading profile from 10 marked exemplars 1.000 ± 0.204 (n=4) 11.0% (n=4) 0.854 (n=4) 0.271 (n=4)
Removed Force verdict as literal first output token 0.938 ± 0.130 (n=4) 11.0% (n=4) 0.583 (n=4) 0.354 (n=4)

Error-type accuracy is exact-string match against the answer key's error label, and it's honestly mediocre across every config, Final included - the agent isn't currently prompted against a closed vocabulary for that field (see eval/cases/README.md), so "dropped_negative_root" vs. "missed_negative_solution" counts as a miss even when a human would call it the same finding. That's a real gap, not a typo in this table; fixing it means constraining the prompts to a fixed vocabulary and re-running the whole battery, which is future work, not done here.

Hot Take: The Actual Story, Not the Cleaned-Up Version

I originally was going to release Iter 2 as Final, on one run of each config. Iter 1 scored better that run (0.417 vs. Iter 2's 0.667), but I picked Iter 2 anyway, on principle: a single call locating the error and deciding its point value in the same breath is exactly the "one call doing two jobs" pattern I built this whole project to argue against. With one sample each, I couldn't tell if Iter 1's edge was real or just a lucky draw, and I said so explicitly in this README, "the design I'd actually stand behind in a code review."

That reasoning was sound. It also wasn't good enough. So I ran it again - three more independent trials of every config, about 1,500 more API calls, entirely to answer one question: is any of this actually real, or did I just get a good sample?

It wasn't the answer I expected. Iter 2 - the one I shipped - has a Points MAE ranging from 0.458 to 1.042 across 4 samples. That range fully contains baseline's entire range (0.792 to 0.917). Iter 2 is not reliably better than baseline. At all. The 0.667 I originally reported was a favorable sample pulled from a distribution actually centered around 0.82 - close enough to baseline that it comes down to which run you happened to get.

Iter 1 held up. Every one of its 5 samples (0.333 to 0.750) beat baseline's best sample (0.792). It's the only config in this entire study with that property - a genuinely reliable result, not a lucky one. That's complete separation between the two samples (5 Iter 1 runs, 4 baseline runs), which is exactly what a one-sided Mann-Whitney U test measures: U = 20 out of a possible 20, giving an exact p = 1/126 ≈ 0.008 - there's no way to draw two groups this size from the same underlying distribution and get zero overlap by chance more than about 1 time in 126. First-error accuracy backs it up too: Iter 1 averaged 0.967 across trials; Iter 2 averaged 0.833.

So Final is Iter 1 now. The design I originally rejected on principle - one call handling both error-location and credit, instead of two calls each doing one - is empirically the strongest design in the whole study. I was wrong about which architecture to trust, and the only reason I know that is that I spent the extra budget to check instead of shipping a single favorable run. That's the real engineering lesson here, more than either individual design choice: an n=1 ablation study tells you what happened once, not what's true.

The Grading Profile Surprise

Here's where things got weird. The grading profile was supposed to be the killer feature, have the model learn from 3 or 10 marked exemplars, then apply that instructor's style to new solutions. Clean concept, right?

It made things worse, not better. Both times I tried it.

What actually happened: the model got so much context about an instructor's past deductions that it started hallucinating those patterns in solutions that didn't have them. One perfectly correct proof got dinged for a sign error that literally isn't there. The model's reasoning sounded exactly like the instructor's real feedback from earlier cases, that is, confident, specific, and completely wrong.

Personalizing the grader made it more confident and no more accurate. Sometimes fewer examples is actually the move!

This isn't a single-run fluke either - repeat trials confirm it. 3-exemplar profile: MAE 1.042 ± 0.189 across 4 runs. 10-exemplar profile: 1.000 ± 0.204. Both clearly worse than baseline's 0.865 ± 0.062, and both worse than Final's 0.517 ± 0.181 by a wide margin. More exemplars helped a little (1.042 → 1.000) but nowhere near enough to close the gap with not using a profile at all.

The Bug (One That Haunts Me)

There's one failure mode that kept showing up across multiple runs, and it's something I didn't expect see, and still don't quite fully understand yet.

A step's verification would work through the math correctly, reason its way to "this step is fine," and then... report the opposite verdict. Or nail the conclusion that something's wrong but mark it right anyway. Same step, same case, three separate runs, totally independent of every other variable that changed between iterations.

The obvious fix: make the model commit to "right" or "wrong" before it writes reasoning, not after. Seemed logical!

Didn't work. The model just committed to the wrong word first, then spent its reasoning arguing for the right one afterward. Moving when it commits doesn't resolve the underlying confusion — it just shifts where the contradiction shows up.

This one's still kicking around in my head. Not solved, just... documented.

And it wasn't just a wash — forcing the earlier commitment made the pipeline worse, not neutral on just the one case I was trying to fix. This was tested against Iter 2's settings specifically (no profile, split error-location/credit calls - Final at the time), for a clean single-variable comparison: Points MAE went from 0.667 to 0.875 on that first run, first-error accuracy dropped from 10/12 to 7/12. Two cases that were previously getting the error location right (case_01, case_02) started getting it wrong, and a third (case_08, a genuinely correct 10/10 solution) got hit with the same hallucinated-error pattern from the grading profile section above — except this time the profile wasn't even involved.

Repeat trials confirmed this wasn't a fluke: verdict-first averaged MAE 0.938 ± 0.130 across 4 runs, against Iter 2's 0.823 ± 0.253 over the same period - consistently worse, not just on that first unlucky sample. Forcing an earlier commitment didn't just fail to fix the contradiction, it made the model defend a premature guess instead of reasoning freely.

That's the Removed row in the table above. The code's still in the repo (agent/prompts/verify_step_verdict_first.md, --verdict-first), just not used by default — a negative result you can't reproduce isn't worth much.

Update: shipping a report that flags its own verdict as possibly backwards, while still printing a score computed from that same verdict, isn't actually resolving the bug — it's disclosing it. agent/grade.py's verify_steps now re-asks a step twice more the moment this contradiction fires (3 independent samples total) and takes the majority verdict, instead of trusting whichever single sample happened to come back self-contradictory.

Ran it on a full 12-case reproduction. Two things worth being honest about. First, which cases even trip the heuristic isn't stable run to run — the original 3 flagged cases (case_01, case_09, case_12) weren't the same 3 flagged this time (case_03, case_09); case_01 and case_12 simply didn't reproduce the contradictory phrasing on this sample, and a different step (case_03) did. That's consistent with the project's whole thesis: a single sample tells you what happened once. Second, on both cases that did trigger a retry, all 3 independent votes agreed with each other (3/3 "incorrect," reasoning consistent across all three) — so the retry mostly confirmed these are the heuristic's known false-positive pattern (a reasoning block that affirms an unrelated sub-clause — "phi correctly ranges..." — in its last sentence while the actual verdict, about a different part of the step, is clearly negative), not evidence of the model actually flip-flopping on the substance. No case in this run had the retries split (2-1 or split-then-resolve), so whether majority-vote genuinely rescues a real contradiction remains untested — worth watching for on future runs rather than a settled result.

The Bug That Almost Shipped

This one's scarier than the one above, because nothing crashed.

Re-running the baseline to grab fresh timing numbers, the score jumped from 1.000 to 3.750 — a genuinely correct 10/10 solution came back scored 1.0. No error, no exception, no warning. The pipeline just quietly reported garbage as if it were a real number.

The cause: the model was reasoning through its grading in prose before writing the final score, and the 512-token budget I'd given it ran out before it got there — every reply was getting cut off mid-sentence. The fallback parser (built for when the model skips the "end with FINAL: <number>" instruction entirely) grabbed the last number left in the truncated text instead, which was just noise — a coordinate, a step index, whatever happened to be sitting there when the tokens ran out.

Fixed by raising the token budget and, more importantly, making that fallback path print a visible warning instead of failing silently. The real lesson isn't the bug itself, it's that this is the dangerous kind of failure. A crash tells you something's wrong. A silently-wrong number that happens to look plausible doesn't — this one would have shipped as the headline result if I hadn't happened to compare it against a number I already had memorized.

Scope (aka What I'm Not Trying to Do)

Everything measured here is text input only. No handwriting recognition, no OCR on scribbled math — that's a deliberate design choice, not a missing feature. Handwriting → OCR → grading introduces enough error rate to contaminate every number in the pipeline. (The image input path exists as a thin wrapper, but it never touched an eval run.)

This is also decision support, not a final grade. An instructor reviews and signs off before any number here actually goes in a gradebook — grading affects real academic outcomes, and nothing in this pipeline is meant to bypass a human making that call.

Want to Reproduce This?

Check out REPRODUCTION.md for the full walkthrough. If you want the original planning brief this whole project was built from - context, guardrails, the schedule, what "HANDOFF section N" means everywhere it's cited in the code - that's HANDOFF.md.

About

An agent that **grades a student's calculus solution**: locates the first incorrect step, classifies the error, and assigns partial credit according to a specific instructor's marking style, inferred from previously marked exams.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages