Fix flaky test_get_preference_adapter by using a valid PE fixture

[saitcakmak]

Summary:
test_get_preference_adapter is flaky at ~77% (currently DISABLED_FLAKY), failing with botorch.exceptions.errors.ModelFittingError: All attempts to fit the model have failed.

The root cause is the test fixture, not the code under test. The test builds its preference (PE) experiment with get_pbo_experiment(..., unbounded_search_space=True) but leaves the default trial counts on, which mixes two very different coordinate scales into a single dataset:

• Preference trials generate arms via _metric_name_to_value, giving values around 1e3 (1000, 2000, plus N(0, 1) noise).
• Experimental trials (num_experimental_trials=3) and repeated-arm trials (num_preference_trials_w_repeated_arm=5) draw from the unbounded_search_space bounds of +/-1e9.

get_preference_adapter wraps PairwiseGP in a Normalize input transform fit over all of the data. Normalizing by a ~1e9 span squeezes the ~1e3-scale comparison pairs down to a separation of ~1e-9, below float resolution. The two arms in a comparison collapse onto the same point with contradicting labels (one preferred, one not), so the Laplace MLL is unfittable and BoTorch raises ModelFittingError after exhausting its retries.

Which arms get paired depends on np.random.choice in the repeated-arm loop, so whether a degenerate pair lands in the data varies from run to run. That is the flakiness.

Fix: disable the two trial types that inject the 1e9-scale arms, so every arm sits at a single ~1e3 scale and comparison pairs keep a ~1e-3 normalized separation.

This is a conditioning fix, not a determinism fix. _metric_name_to_value still calls np.random.standard_normal(), so the generated arms still vary between runs; what changes is that the data is now well-scaled for Normalize regardless of the draw. That is why it is validated across 400 seeds rather than against one fixed seed.

Note that pinning the RNG seed would not be a correct fix here. Measurement showed the fitting itself is deterministic (varying only the torch seed gave 0/20 failures; only varying the numpy data seed reproduced the failure), so a seed pin would merely select one lucky dataset while leaving the fixture generating numerically degenerate data. Any later change to arm ordering, trial counts, or Normalize behavior would silently re-break it.

Kept deliberately minimal: only the two arguments required to fix the flake are changed. include_sq, num_experimental_metrics, and tracking_metric_names are left at their defaults, since with the experimental and repeated-arm trials disabled they contribute no observations to the fit (verified: the fitted model still sees datapoints [4, 2] / comparisons [2, 2]).

Differential Revision: D115907820

[meta-codesync]

@saitcakmak has exported this pull request. If you are a Meta employee, you can view the originating Diff in D115907820.

[meta-codesync]

This pull request has been merged in aa1e4f9.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.58%. Comparing base (8f0745e) to head (0cee059).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #5272   +/-   ##
=======================================
  Coverage   96.58%   96.58%           
=======================================
  Files         621      621           
  Lines       70849    70849           
=======================================
  Hits        68431    68431           
  Misses       2418     2418           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.