Source corpus-fixture expectations from runtime traces

[khatchad]

The vendored corpus fixtures encode what a human read off the subject's call sites. That reading can be wrong, and when it is, the fixture pins the wrong tensor type and the error persists as a green test.

Proposal: source those expectations from an actual run instead. The subject is executed under a tracing shim that records concrete argument types per traced function with no subject edits; the observed types become assert statements in the vendored Python fixture, verified to run to completion under python3.10; the JUnit expectations then mirror the asserts. That is the established fixture idiom here, with the one change that the source of truth is an observation rather than an inference.

Runtime evidence is bounded by exercised paths, so it cannot confirm that a fixture admits every reachable call site. It is decisive in the other direction: an expectation the run contradicts is simply wrong, and that is the class of error a hand-derived fixture cannot catch on its own.

Scope is the vendored corpus fixtures whose subjects actually run. Companion work on the refactoring side pins the emitted input_signature rather than the inferred tensor types, in ponder-lab/Hybridize-Functions-Refactoring#810; the two layers fail differently, since the consensus and drop-condition logic sits between a tensor type and the signature that gets written.

[khatchad]

Starting This: One Fixture Now Runs, And The Premise Needs Widening

Took the vendored gpt-2 fixture as the first case, since its expectation is a three-member union and therefore worth checking against reality.

It had never been run. The data pipeline reads a tfrecord that was never created, because static analysis does not open files. Making it run took a record generator, committed as a script rather than a binary, and one line the real driver has that this one lacked, creating the optimizer before training. Two bounds had to be learned by hitting them: token identifiers below vocab_size, and the longest sequence at max_seq_len - 1 rather than max_seq_len, since position identifiers start at one when there is no past. The pinned expectation is unchanged by the edit, so the fixture still analyses as before.

That part generalizes. If this fixture had never been executed, others likely have not either, and a fixture that cannot run cannot have its expectations observed.

The premise needs widening. This issue proposes recording argument types "per traced function", and the shim implements that by patching tf.function, so it sees decorated functions only. The functions these fixtures assert on are frequently not decorated: here the test pins Gpt2.get_loss, which no decorator wraps, and the driver takes the eager path where the decorated wrappers are never called. The run therefore completes and observes nothing.

So sourcing expectations from observation needs a recorder that does not depend on decoration. Recording argument types for functions named by qualified name would cover it, and would also cover the decorated case, since what we want is the arguments a function receives rather than the signature a trace was specialized to. That is a small addition to the shim rather than a change to this issue's intent.

Suggested Order

1. The decoration-independent recorder, since nothing else can proceed without it.
2. This fixture, whose expectation is rich enough that a disagreement would be informative.
3. The remaining fixtures, each of which likely needs its own runnability work first.

Recording this before continuing, because the second and third steps are only worth doing once the first exists, and because "the fixture has never been run" is worth knowing independently of this issue.

[khatchad]

What Sourcing From Traces Can And Cannot Mean

Recording this before more fixtures are converted, because the obvious reading of this issue's title would make expectations worse.

The vendored gpt-2 fixture expects a union: two members for one parameter, three for another. A run of it produces one observation per parameter. Taking the observation as the expectation would delete the members this driver never exercises, replacing a correct expectation with a narrower one, and leaving a green test that pins less than it did before.

So the operation is not replacement. It is:

1. Refute. An observation that contradicts an expected member means the expectation is wrong. This is sound and is the point of the exercise.
2. Seed. Where no expectation exists, an observation is a better starting point than a reading, recorded as covering exercised paths only.
3. Never subtract. An expected member that a run does not exercise is unexercised, not disproven.

The same asymmetry this issue already states for fixtures admitting every reachable call site applies to each member of a union within one fixture.

A practical consequence: adding drivers or varying inputs raises the chance of catching a contradiction, and never establishes that a union is complete. Worth saying because the natural next thought after a corroborating run is to vary the inputs until the union is covered, and that goal is not reachable by this method.

First Comparison

For the record, the gpt-2 fixture's hand-read expectation survives contact with a run. Observed real as int32 (32, 3) and pred as float32 (32, 3, 10), both inside the expected unions. No member is contradicted, and none is confirmed.

[khatchad]

First application landed in ponder-lab#730, which deliberately does not close this.

Three of the vendored probes now assert what a run observes rather than what a reading of their construction predicts. Two already agreed with their pins, which is the outcome worth having on record rather than assumed. The third, probe_conv1d, observes float32 (2, 3, 16) where the analysis pins unknown rank because the reshape target is a runtime-built list, so it confirms the #703 gap and gives it a concrete target the fixture did not previously carry.

Remaining Scope

The tree holds twenty vendored project directories plus the gpt-2 one. The three converted are entry points in two of them, so most of the population is untouched.

Two constraints found while doing it, worth knowing before the next batch:

The subject directory has to be the working directory, since a fixture that reads a data file resolves the path relative to it. Invoking the shim as a module from there fails when the subject ships its own scripts package, because the working directory precedes PYTHONPATH under -m. Invoking it by absolute path from the subject directory satisfies both.

Not every fixture runs to completion. probe_forward fails on an unset optimizer before reaching its sink, so nothing can be sourced from it and its expectations stay hand-derived. Marking that by omission rather than by a claim that was not verified.

A Dependency Worth Noting

Sourcing expectations this way needs the recorder to see undecorated functions, since a sink like consume carries no decorator. That capability arrived separately; before it, a decorator-keyed capture recorded nothing for exactly the functions these fixtures assert on.