feat(benchmark): LongMemEval end-to-end answer grading and facts corpus - #93
Open
corespeed-agent-ensemble[bot] wants to merge 1 commit into
Open
feat(benchmark): LongMemEval end-to-end answer grading and facts corpus#93corespeed-agent-ensemble[bot] wants to merge 1 commit into
corespeed-agent-ensemble[bot] wants to merge 1 commit into
Conversation
Add benchmark:longmemeval:e2e, an answer-accuracy runner over an already-indexed LongMemEval corpus: hybrid retrieval under RLS, a configured reader over retrieved session Memories, and the official upstream QA judge prompts reproduced verbatim (pinned to xiaowu0162/LongMemEval@d6dc8b50, temperature 0, yes/no labeling, abstention template for _abs questions). Corpus identity is validated by workspace/key/owner/scope counts plus active-embedding-space completeness; documents are never re-embedded. Readers cover google/openai/vllm/ollama plus a benchmark-only claude-cli transport; the Google judge falls back to thinking mode when a model rejects thinkingBudget: 0 or returns an empty grade. Flags: --type-instructions (type-aware reader system prompts; judge stays official), --evidence-order chronological, --evidence-source passage, --corpus-suffix, and --resume that retries provider-failed cases instead of freezing them into the score. Add benchmark:longmemeval:extract-facts, the benchmark-only {questionId}#facts corpus builder: an extractor distills each indexed session Memory into one compact fact-sheet Memory (same benchmarkKey, tripwires replicated, embeddings drained through leased maintenance), with original-content passthrough tagged extractorFallback on provider content-policy blocks. Evaluator-side profile in the MemoryAgentBench structured-assembly tradition; Lore v1 still excludes automatic consolidation. Measured on LongMemEval-S (500 questions, gemini-embedding-2 @1536): 94.2 overall with a flash-class reader at top-10 full sessions, 87.8 at ~2.8k reader tokens per question on the facts corpus, retrieval R@10 0.9898, isolation 0/500 leaks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two new benchmark-only runners over the existing indexed LongMemEval corpus, plus their
bun runentries and AGENTS.md documentation:benchmark:longmemeval:e2e— end-to-end answer accuracy: hybrid retrieval under RLS → configured reader over retrieved session Memories → the official upstream QA judge prompts reproduced verbatim (pinnedxiaowu0162/LongMemEval@d6dc8b50, temperature 0,'yes' in lower()labeling,_absabstention template). Never re-embeds documents; corpus identity validated by workspace/key/owner/scope counts plus active-embedding-space completeness. Reader transports: google/openai/vllm/ollama + a benchmark-onlyclaude-cliprint-mode transport (same pattern asbenchmark:retrieval-policy --runner claude). Flags:--type-instructions,--evidence-order chronological,--evidence-source passage,--corpus-suffix,--resume(retries provider-failed cases rather than freezing them into the score).benchmark:longmemeval:extract-facts— builds the benchmark-only{questionId}#factscorpus: one compact fact-sheet Memory per indexed session (samebenchmarkKey, Bob tripwires replicated, embeddings drained through leased maintenance), original-content passthrough taggedextractorFallbackon provider content-policy blocks. Evaluator-side profile in the MemoryAgentBench structured-assembly tradition — not product consolidation.Measured (LongMemEval-S, 500 questions, gemini-embedding-2 @1536)
Gotchas encoded in AGENTS.md
LORE_BENCHMARK_READER_MAX_OUTPUT_TOKENS=8192)thinkingBudget: 0; the judge falls back to thinking mode (also on empty grades)Checks
design:check ✅ · lint ✅ · typecheck ✅ · test ✅ (
XDG_CONFIG_HOME=/tmp/emptyfor the documented code-index global-gitignore gotcha; failure reproduces on main) · build ✅ · packages:smoke ✅🤖 Generated with Claude Code