Skip to content

feat(adapter): preserve legacy language scoring through sessions - #435

Merged
KartikP merged 9 commits into
unified-model-interface-v2from
feat/contract-v3-qualified
Nov 2, 2026
Merged

KartikP merged 9 commits into
unified-model-interface-v2from
feat/contract-v3-qualified

Conversation

@KartikP

@KartikP KartikP commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Preserve legacy language measurements while running them through the shared UMI session contract. This combines the selected Contract v3 adapter changes with the permanent facade integration and supersedes #432 and #434.

Changes

  • Mark legacy benchmarks at loading and apply the facade at scoring. Pereira -unified variants declare the typed path and bypass the facade.
  • For legacy-text sessions, configure the legacy subject and make one digest_text call that preserves requested neural and behavioral outputs together.
  • Apply recording/task setup once per configuration. Repeated measurements do not accumulate legacy recording registrations; reset or changed configuration reapplies setup.
  • Declare supported outputs, filter undeclared session inputs and remove catch-all __getattr__ forwarding. Embedding registrations declare neural outputs only; mapped models retain behavioral support.
  • Hash length-prefixed text segments so differently segmented inputs cannot collide. Existing text cache entries recompute; measurement values are preserved.
  • Emit t_ms=None when no clock exists. Update adapter, legacy-scoring, preflight tests and coordinated CI pins.

Legacy model and benchmark implementations are retained.

Coordination

Core #201 is merged into v2 as 0fbb419; this PR requires that revision.
The scientific gate used language runtime 62e2dcf. Post-gate commit 7d3ca10
repairs embedding declarations; 5c45874 updates its submission diagnostic test.
Their validation is listed below separately from the original gate.

Qualification

Independent Linux EC2 qualification used Python 3.11, numpy 1.26.4,
xarray 2022.3.0, scikit-learn 1.7.2, transformers 4.57.6 and torch 2.6.0.

  • 258 independent focused tests passed across the four repositories.
  • 14 comparable real-model matrix cases match clean v2 exactly, including all three HMAX cases.
  • Candidate and clean-v2 slow parity each pass 5 tests covering 6 benchmark variants. All saved route records match exactly: scores, raw scores, metric-input hashes and activation-delta records.
  • Regression-baseline pytest: 11 passed, no failures or skips; 2 tests deselected by the slow marker. These assertions use their configured tolerances, separately from the exact candidate/base comparisons.

Pereira parity covers historical linear variants, not published ridge qualification.
Production-reference differences also occur on clean v2; no historical cause is inferred.

Accepted exclusion: Kartik closed this scoped review with Malania excluded.
Direct base, direct candidate and facade response arrays match exactly, but named
coordinate fidelity remains unverified. All three return a low-accuracy threshold-fit
fallback, not a valid fitted threshold. Malania is excluded, not counted as passed.

Full report and numerical results.
This qualification does not complete the broader release matrix.

Coordinated PRs

CI follow-up

Core #201 is merged into v2 as
0fbb419ff0ba461e6b1e217c845e0fc61d8096a1; peer pins now use that merged revision.
Its runtime matches the qualified core revision; only CI pins differ. Domain
workflows use unified 42994ca24c05b6b2547e0f7153b2018da54d915e.

The original Jenkins failures installed an older core from the bootstrap's shared
manifest, ignoring each PR's requirement. Infrastructure #84
honors the candidate's declared immutable core pin. Kartik merged it as 93370d0;
the new Jenkins logs confirm that bootstrap is in use. The language OpenPI shutdown timeout did not
reproduce on a clean GitHub rerun: all 51 instrumentation/transport tests passed.
No shutdown deadline or assertion was weakened.

Embedding declaration repair

The repaired Jenkins run exposed missing neural declarations on the random
embedding models. Commit 7d3ca10 declares the three supported recording targets
for randomembedding-100, randomembedding-1600 and glove-840b, without inventing
a region-to-layer map or changing their legacy implementations. These models no
longer advertise behavioral answers they cannot produce. Custom legacy subjects
can pass explicit out_channels to the adapter. Mapped models keep their defaults.

Thirteen new cases fail before the fix. The local offline checks pass 97 tests in
a fresh lightweight Python 3.11 environment with result caching disabled. Tests
compare real random embeddings exactly through direct, facade and typed-session
paths, exercise every declared target, and verify repeated setup is not duplicated.
The three full Jenkins suites and coordinated GitHub checks must pass on this
new runtime before merge; the earlier scientific gate predates this repair.

The first repaired run passed all three public scoring integration cases and eight
plugin checks. Non-plugin CI passed 133 tests and exposed one obsolete assertion
that required the traceback to originate in embedding.py. The facade now rejects
unsupported behavioral tasks earlier. The follow-up retains the no-score and
NotImplementedError assertions and requires the declaration diagnostic plus a
facade traceback. It passes against the real submission endpoint with temporary
SQLite and a data-free reading task; no shared database or trained model was used
locally. The full remote suites are rerun on the final head.

Pre-flight is one check_compatibility call; its match (full or partial,
consumed and dropped inputs) is recorded in score.attrs['compatibility'].
The adapter exposes the wrapped model as legacy_model instead of passing
every unknown attribute through, so benchmarks see only what the adapter
declares.
…digest_text

load_benchmark marks each benchmark's calling convention and _run_score
hands a legacy benchmark the permanent LegacyBenchmarkFacade. The
Pereira -unified variant declares the typed convention and is not
wrapped.

LanguageModelAdapter declares its behavior output and, for a facade
session, configures the legacy model as the benchmark did and makes the
one digest_text call the benchmark would have made, returning neural and
behavioral outputs together. Other sessions keep the typed drivers.
Legacy language subjects append every start_neural_recording call, so a
facade that re-sent the setup per passage registered the recording once
per digest_text. LanguageModelAdapter now applies a recording or task
configuration once until it changes or the adapter is reset, which is the
call sequence a legacy benchmark makes. Only declared input streams reach
the legacy model, and responses without a clock carry t_ms=None.
The coordinated workflow reads peer revisions from the pinned unified
commit, so it now points at the unified revision whose
install/peer-revisions.json names the facade branch tips; the
requirements pin follows the same core revision. Re-pin to merged
revisions before merging.
@KartikP

KartikP commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

retest this please

@KartikP
KartikP merged commit a9e572d into unified-model-interface-v2 Nov 2, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant