Repository navigation
feat(adapter): preserve legacy language scoring through sessions - #435
Merged
Merged
Conversation
Pre-flight is one check_compatibility call; its match (full or partial, consumed and dropped inputs) is recorded in score.attrs['compatibility'].
The adapter exposes the wrapped model as legacy_model instead of passing every unknown attribute through, so benchmarks see only what the adapter declares.
…digest_text load_benchmark marks each benchmark's calling convention and _run_score hands a legacy benchmark the permanent LegacyBenchmarkFacade. The Pereira -unified variant declares the typed convention and is not wrapped. LanguageModelAdapter declares its behavior output and, for a facade session, configures the legacy model as the benchmark did and makes the one digest_text call the benchmark would have made, returning neural and behavioral outputs together. Other sessions keep the typed drivers.
Legacy language subjects append every start_neural_recording call, so a facade that re-sent the setup per passage registered the recording once per digest_text. LanguageModelAdapter now applies a recording or task configuration once until it changes or the adapter is reset, which is the call sequence a legacy benchmark makes. Only declared input streams reach the legacy model, and responses without a clock carry t_ms=None.
The coordinated workflow reads peer revisions from the pinned unified commit, so it now points at the unified revision whose install/peer-revisions.json names the facade branch tips; the requirements pin follows the same core revision. Re-pin to merged revisions before merging.
This was referenced Nov 2, 2026
Contributor
Author
|
retest this please |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Preserve legacy language measurements while running them through the shared UMI session contract. This combines the selected Contract v3 adapter changes with the permanent facade integration and supersedes #432 and #434.
Changes
-unifiedvariants declare the typed path and bypass the facade.digest_textcall that preserves requested neural and behavioral outputs together.__getattr__forwarding. Embedding registrations declare neural outputs only; mapped models retain behavioral support.t_ms=Nonewhen no clock exists. Update adapter, legacy-scoring, preflight tests and coordinated CI pins.Legacy model and benchmark implementations are retained.
Coordination
Core #201 is merged into v2 as
0fbb419; this PR requires that revision.The scientific gate used language runtime
62e2dcf. Post-gate commit7d3ca10repairs embedding declarations;
5c45874updates its submission diagnostic test.Their validation is listed below separately from the original gate.
Qualification
Independent Linux EC2 qualification used Python 3.11, numpy 1.26.4,
xarray 2022.3.0, scikit-learn 1.7.2, transformers 4.57.6 and torch 2.6.0.
Pereira parity covers historical linear variants, not published ridge qualification.
Production-reference differences also occur on clean v2; no historical cause is inferred.
Accepted exclusion: Kartik closed this scoped review with Malania excluded.
Direct base, direct candidate and facade response arrays match exactly, but named
coordinate fidelity remains unverified. All three return a low-accuracy threshold-fit
fallback, not a valid fitted threshold. Malania is excluded, not counted as passed.
Full report and numerical results.
This qualification does not complete the broader release matrix.
Coordinated PRs
CI follow-up
Core #201 is merged into v2 as
0fbb419ff0ba461e6b1e217c845e0fc61d8096a1; peer pins now use that merged revision.Its runtime matches the qualified core revision; only CI pins differ. Domain
workflows use unified
42994ca24c05b6b2547e0f7153b2018da54d915e.The original Jenkins failures installed an older core from the bootstrap's shared
manifest, ignoring each PR's requirement. Infrastructure #84
honors the candidate's declared immutable core pin. Kartik merged it as
93370d0;the new Jenkins logs confirm that bootstrap is in use. The language OpenPI shutdown timeout did not
reproduce on a clean GitHub rerun: all 51 instrumentation/transport tests passed.
No shutdown deadline or assertion was weakened.
Embedding declaration repair
The repaired Jenkins run exposed missing neural declarations on the random
embedding models. Commit
7d3ca10declares the three supported recording targetsfor
randomembedding-100,randomembedding-1600andglove-840b, without inventinga region-to-layer map or changing their legacy implementations. These models no
longer advertise behavioral answers they cannot produce. Custom legacy subjects
can pass explicit
out_channelsto the adapter. Mapped models keep their defaults.Thirteen new cases fail before the fix. The local offline checks pass 97 tests in
a fresh lightweight Python 3.11 environment with result caching disabled. Tests
compare real random embeddings exactly through direct, facade and typed-session
paths, exercise every declared target, and verify repeated setup is not duplicated.
The three full Jenkins suites and coordinated GitHub checks must pass on this
new runtime before merge; the earlier scientific gate predates this repair.
The first repaired run passed all three public scoring integration cases and eight
plugin checks. Non-plugin CI passed 133 tests and exposed one obsolete assertion
that required the traceback to originate in
embedding.py. The facade now rejectsunsupported behavioral tasks earlier. The follow-up retains the no-score and
NotImplementedErrorassertions and requires the declaration diagnostic plus afacade traceback. It passes against the real submission endpoint with temporary
SQLite and a data-free reading task; no shared database or trained model was used
locally. The full remote suites are rerun on the final head.