Skip to content

Extraction paused: main's extractor undercounts from v2.1.218 on #18

Open

Description

@lroolle

deterministic-extract is disabled until the extractor can count correctly again. It is currently wrong in both directions.

Main's method undercounts (v2.1.218 onward)

Main's extractor greps process.env.<NAME> literals. Starting with v2.1.218, Claude Code reads many variables through a declared env registry instead, and those reads are not literals. Evidence:

  • On main: v2.1.197 = 655 vars, v2.1.218 = 538. 122 of the 126 names that v2.1.218 records as "removed" still appear verbatim in the v2.1.296 binary. So v2.1.218 (538) and v2.1.222 (539) are lower bounds, not counts.
  • Bot PRs Document v2.1.263 #12..Document v2.1.296 #17 (v2.1.263..v2.1.296) each report 149..166 removals vs v2.1.222. For Document v2.1.296 #17, 153 of its 166 "removed" names are still in the v2.1.296 binary. ANTHROPIC_DEFAULT_FABLE_MODEL is one of them.

#7's registry-aware method overcounts on new releases

#7 reads the registry. It gave a plausible 994 on v2.1.234, but on v2.1.302 it reports 2,917 names, 1,309 of which are not env-var shaped (return, takeDown, EmptyResultSchema), and every validation gate still passed. Details: #7 (comment)

To resume

  1. Fix Document v2.1.234; re-measure v2.1.170+ against the declared env registry #7's extractor and add the two plausibility gates its validation lacks:
    • registry names must not exceed typed getters by an implausible margin
    • the total must not move more than a set fraction vs the previous documented release
  2. Re-extract v2.1.218 and v2.1.222 with the fixed method, then the newest release.
  3. Re-enable deterministic-extract. The one-PR-at-a-time guard from claude-code-http-spec#21 should come with it, because this workflow also opens a new PR every week whether or not the last one merged.

Until then, the README's last trustworthy count is v2.1.197 (655).

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions