[Chris Trevino (darthtrevino)]

Benchmark QED currently relies on direct model endpoints, while newer workflows use models through GitHub Copilot CLI. This adds copilot.chat so users can select a Copilot model by name without configuring an Azure OpenAI endpoint or API key.

Approach

• Use the official github-copilot-sdk with a shared CLI runtime and isolated, tool-free sessions for each completion.
• Support synchronous and asynchronous completions, structured responses, usage metrics, and existing retry/rate-limit middleware.
• Validate model availability to prevent silent fallback to a different model.
• Add interactive init support, configuration documentation, and regression tests.

llm_config:
  model: gpt-6-luna
  llm_provider: copilot.chat

End-to-end smoke test

Ran benchmark-qed autoe pairwise-scores with the new copilot.chat settings and gpt-6-luna against a small synthetic dataset (3 questions, 2 criteria, 2 trials; 12 model calls). The command completed successfully in about 58 seconds; all structured responses parsed, and the stronger answers won every comparison. Usage metrics recorded 12 successful calls, with 0 failures and 0 retries.

Limitations and review notes

Authenticate with the Copilot CLI or supply a GitHub token in init_args. Copilot does not provide embeddings, so embedding configuration must use another provider. Sampling parameters such as temperature and seed are ignored, and streaming is unsupported. Structured responses use schema instructions and validated JSON parsing rather than native strict-schema enforcement.

The uv.lock diff includes a lockfile-format refresh from the current uv version in addition to the new SDK dependency.

Validation

• Live end-to-end CLI completion, including structured output and clean runtime shutdown.
• 114 targeted provider, configuration, and init tests passed.
• uv run poe check passed locally after fixing the pyright --verifytypes module logger annotation.
• CI passed on Python 3.11 and 3.12, with semver and CLA checks passing.
• Full test suite not run.