Repository navigation
feat(adapter): run legacy vision benchmarks through sessions - #2630
Merged
Merged
Conversation
Pre-flight is one check_compatibility call; its match (full or partial, consumed and dropped inputs) is recorded in score.attrs['compatibility'].
The adapter exposes the wrapped model as legacy_model instead of passing every unknown attribute through, so benchmarks see only what the adapter declares.
… calls exact load_benchmark marks each benchmark's calling convention and _run_score hands a legacy benchmark the permanent LegacyBenchmarkFacade, so every subject, including an interact-only one, takes part through interact. The MajajHong -unified variant declares the typed convention and is not wrapped. VisionModelAdapter declares its behavior output and runs a session as the single look_at the benchmark would have made: task, recording target, time bins, trial count and the variance request reach the legacy model through the same calls, once, with no passive start_task. The adapter now forwards require_variance when a benchmark asks for it.
VisionModelAdapter applies start_task and start_recording to the legacy model when the configuration changes, not on every facade measurement, so a benchmark that configures once and measures many times produces the same legacy call sequence it always did. Fitting settings a benchmark passed to start_task (Malania2007's number_of_trials and require_variance) reach the legacy start_task again, apart from each look_at's own settings. Only declared input streams reach the legacy model, for measurements and for fitting tables alike, and responses without a clock carry t_ms=None.
The coordinated workflow reads peer revisions from the pinned unified commit, so it now points at the unified revision whose install/peer-revisions.json names the facade branch tips; the requirements pin follows the same core revision. Re-pin to merged revisions before merging.
This was referenced Nov 2, 2026
Collaborator
Author
|
retest this please |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Preserve existing vision benchmark calls while routing subjects through the UMI session contract. This combines the selected Contract v3 adapter changes with the permanent facade integration and supersedes #2625 and #2627.
Changes
-unifiedvariants declare the typed path and bypass the facade.require_varianceonly when requested; avoid starting a task in a passive context.look_atonce with its original stimuli.__getattr__forwarding.t_ms=None. Update facade/preflight tests and coordinated CI pins.Legacy model and benchmark implementations are retained.
Coordination
Requires the matching core PR. Core must merge first; update
requirements-v2.txtand coordinated CI pins to the merged revisions before merging this PR. Qualified
vision runtime:
eca24202; branch tipfb487671adds CI pin changes only.Qualification
Independent Linux EC2 qualification used Python 3.11, numpy 1.26.4,
xarray 2022.3.0, scikit-learn 1.7.2, transformers 4.57.6 and torch 2.6.0.
Pereira parity covers historical linear variants, not published ridge qualification.
Production-reference differences also occur on clean v2; no historical cause is inferred.
Accepted exclusion: Kartik closed this scoped review with Malania excluded.
Direct base, direct candidate and facade response arrays match exactly, but named
coordinate fidelity remains unverified. All three return a low-accuracy threshold-fit
fallback, not a valid fitted threshold. Malania is excluded, not counted as passed.
Full report and numerical results.
This qualification does not complete the broader release matrix.
Coordinated PRs
CI follow-up
Core #201 is merged into v2 as
0fbb419ff0ba461e6b1e217c845e0fc61d8096a1; peer pins now use that merged revision.Its runtime matches the qualified core revision; only CI pins differ. Domain
workflows use unified
42994ca24c05b6b2547e0f7153b2018da54d915e.The original Jenkins failures installed an older core from the bootstrap's shared
manifest, ignoring each PR's requirement. Infrastructure #84
honors the candidate's declared immutable core pin. Kartik merged it as
93370d0;the new Jenkins logs confirm that bootstrap is in use. The language OpenPI shutdown timeout did not
reproduce on a clean GitHub rerun: all 51 instrumentation/transport tests passed.
No shutdown deadline or assertion was weakened.