[Proposal] OpenEval Import/Export Support

[adhabnr-ux]

Overview

We have launched OpenEval (v1.0.0-rc.1), an open standard (Apache 2.0) for portable LLM evaluation datasets. Every eval framework uses incompatible formats — OpenEval provides a standard JSON format + converters + SDKs (TypeScript + Python).

What We're Proposing

Add to_openeval() and from_openeval() methods (~50-100 lines using our SDK).

Resources

• Spec: https://github.com/adhabnr-ux/openeval/blob/main/spec/SPEC.md
• npm SDK: npm install @openeval/sdk
• PyPI SDK: pip install openeval

We would love to collaborate.

[adhabnr-ux]

Update: OpenEval packages published! pip install openeval-sdk. Active responses from Inspect AI, CrewAI, Arize. Would love to collaborate.

[adhabnr-ux]

Correction/update on the package names from my last comment, since they've since changed and I don't want anyone here to hit a broken pip install:

The project renamed from "OpenEval" to EvalPort (repo moved to adhabnr-ux/evalport — old links redirect). The correct install is:

• PyPI: pip install evalport-sdk (imports as openeval, e.g. from openeval.validate import validate_suite)
• npm: npm install evalport-sdk

(pip install openeval-sdk / pip install openeval from my earlier comment don't exist — sorry for the confusion.)

Concrete progress since I opened this: 20 shipped framework adapters (real installable packages under adapters/<name>-openeval-adapter/, each tested against the real validator — Ragas, LangSmith, MLflow, Braintrust, CrewAI, and 15 others are all merged examples if a reference implementation is useful), one integration merged into a third-party repo directly (Inspect AI, PR #4797), and one under active maintainer review right now (TruLens, PR #2697).

If a lighteval to_openeval()/from_openeval() pair is still of interest, happy to sketch a first draft against lighteval's actual result/task objects the way I did for the TruLens and OpenAI-evals PRs above — just say the word, or feel free to open a PR yourselves referencing the spec, no permission needed.

[adhabnr-ux]

Following up since it's been a couple weeks: I went ahead and built this — lighteval-openeval-adapter, a standalone package, no changes needed to lighteval itself.

from lighteval.pipeline import Pipeline
# ... run pipeline.evaluate() as usual ...
details = pipeline.get_details()["hellaswag|0"]

from lighteval_openeval_adapter import to_openeval, result_to_openeval
suite = to_openeval("hellaswag|0", details, suite_id="my_hellaswag_run")

from openeval.validate import validate_suite
assert validate_suite(suite).valid

It reads from pipeline.get_details() (the real per-document DetailsLogger.Detail records), not get_results()'s aggregate summary — confirmed those are genuinely different methods by reading lighteval's own logging code. em (Metrics.exact_match) maps to EvalPort's native exact_match grader; every other metric (extractive_match, f1, chrf++, task-specific ones) maps to EvalPort's custom type with the real metric name preserved, rather than guessing at an equivalence that isn't there.

19 tests, all driving a real Pipeline.evaluate() against lighteval 0.13.0's own dummy model on hellaswag and gsm8k (real tasks/documents pulled from the Hub, no mocks) plus the real EvalPort validators.

One side note since it came up directly while building this: hit the exact xxhash 4.0 incompatibility tracked in #1330 / fixed in #1332 from a completely different angle (reading per-document details for conversion, not the original quantization-eval report) — left an independent repro confirming all four affected call sites on that PR, and this adapter's test extra just pins xxhash<4.0 as a workaround until it merges.

Source: https://github.com/adhabnr-ux/evalport/tree/main/adapters/lighteval-openeval-adapter

Given lighteval sits directly on inspect_ai (already has an EvalPort integration merged — UKGovernmentBEIS/inspect_ai#4797) this closes the loop from the other side of that dependency. Still happy to sketch native to_openeval()/from_openeval() methods inside lighteval itself if that's ever wanted — this standalone package works either way, so no pressure.