diff --git a/.gitignore b/.gitignore index adb2746..b207963 100644 --- a/.gitignore +++ b/.gitignore @@ -140,3 +140,5 @@ !/scripts/feedback_cases.py !/frontend/src/app/api/feedback/ !/frontend/src/app/api/feedback/route.ts + +!/evaluations/results/techqa_wide_v5.json diff --git a/README.md b/README.md index 228b346..5c71a8b 100644 --- a/README.md +++ b/README.md @@ -30,7 +30,7 @@ The workspace streams actual activity and retrieved excerpts. Verified answers a For developers and coding agents, `/ask` remains a server-sent event endpoint. `stage` events describe activity, `evidence` carries retrieved `sources`, `final` carries the approved outcome and `error` carries a safe error category/message. `done` closes the run. Final statuses are `answered`, `needs_clarification` and `insufficient_evidence`. Existing exact-cache, session, cancellation and regeneration behavior remains covered by tests. -Collect a fixed random training sample with a hard HTTP-attempt cap. Quota failures stop collection and are retried when the same unchanged run resumes. Completed cases are saved after each question, with a `completed.json` snapshot for inspection. +Collect a fixed random training sample with a hard HTTP-attempt cap. Provider HTTP 429 responses stop collection and are retried when the same unchanged run resumes. Local per-request quota deadlines are recorded as failed requests while collection continues. Completed cases are saved after each question, with a `completed.json` snapshot for inspection. ```bash python -m scripts.eval_ragas collect --limit 12 --sample-seed 20261110 \ @@ -68,12 +68,34 @@ Approval and review notes are required for feedback-based RAGAS evaluation. Use |---|---|---| | Components | Retrieval Recall@K, MRR and nDCG against known article labels; writer/verifier behavior with supplied evidence | [Rank metrics](scripts/baseline.py), [retrieval tests](tests/test_app_retriever.py), [generator contract fixtures](tests/test_app_generation.py). Controlled fixtures are not a generator quality benchmark. | | Pipeline | Selected context precision/recall, answer faithfulness and relevancy | [RAGAS runner](scripts/eval_ragas.py), measured on the pilot below. Reference text stays outside inference. | -| Application | Reference-based correctness and completeness; citation validity; withholding and layout contracts; recorded timing/tokens/cache use | Correctness = RAGAS `FactualCorrectness(mode="f1")`; completeness = its `mode="recall"` proxy. Both are opt-in and not yet benchmarked. [Free saved-case checks](scripts/eval_suite.py), [API contracts](tests/test_app_api.py), [layout checks](frontend/tests/genui-contract.test.mjs) are implemented. | +| Application | Reference-based correctness and completeness; citation validity; withholding and layout contracts; recorded timing/tokens/cache use | Correctness = RAGAS `FactualCorrectness(mode="f1")`; completeness = its `mode="recall"` proxy. Both are opt-in and have sampled results below. [Free saved-case checks](scripts/eval_suite.py), [API contracts](tests/test_app_api.py), [layout checks](frontend/tests/genui-contract.test.mjs) are implemented. | | Regression | Re-run a fixed case set, then inspect per-case differences | [Comparison CLI](scripts/eval_suite.py) rejects changed cohorts/references. Semantic comparisons also require matching judge settings and exclude scores flagged for review. | | Online | User feedback, reviewed regression cases, sampled judging and drift | Helpful/unhelpful votes and optional comments save the run for review and sync to LangSmith. Reviewed references can be exported for RAGAS. Sampled live judging and drift monitoring remain planned. | Style rubrics and broad toxicity/PII/jailbreak evaluation are also planned. Existing origin/session, withholding and generated-component checks cover specific application contracts; they are not a comprehensive safety benchmark. Optional semantic metrics compare against available references, which may omit valid alternative answers. +### Wider training run with prompt version 5 + +The [saved result](evaluations/results/techqa_wide_v5.json) covers **15 of 30 planned training questions**. Generation stopped on a provider rate limit. The run produced **8 released answers, 2 withheld responses, 3 citation-format failures, 1 local quota-wait deadline and 1 provider rate-limit failure**. The selected evidence included the labeled article for **14/15** questions. All **8 released answers** passed citation-format checks. + +RAGAS used `openai/gpt-oss-20b` after a supported/unsupported faithfulness check scored 1.0/0.0. Answer generation used `openai/gpt-oss-120b`. Samples were selected separately for each metric to bound judging cost. + +| RAGAS metric | Raw mean | Scored cases | Review flags | +|---|---:|---:|---:| +| Context recall | 90.9% | 11 | 0 | +| Context precision | 90.2% | 2 | 0 | +| Faithfulness | 97.5% | 5 | 0 | +| Answer relevancy | 93.0% | 3 | 0 | +| Factual correctness against the reference | 42.6% | 5 | 1 | +| Completeness against the reference | 53.4% | 5 | 1 | + +Three additional context-recall cases reached the judge request cap. Raw means retain flagged values. LangSmith leaves those flagged values pending review, so its displayed averages use fewer cases. Source inspection also found a reference-scope issue where a question asks for a bulletin link but its reference contains the full bulletin text. Correctness and completeness scores require review before being used as release criteria. + +Generation used **41 HTTP attempts / 206,804 reported tokens**. Judging used **120 attempts / 172,079 tokens**, plus **4 calibration calls / 3,757 tokens**. The runner reused **29 cached judge decisions**. Median answered-case batch time was **128.1 seconds**, including quota waits and local retrieval. These timings describe a serial batch. Reserved development questions remain available for a later release evaluation. + +The collector initially stopped on a local request deadline. That condition was corrected, existing cases were reused, and collection continued until a provider HTTP 429 response. Retryable failed attempts are now archived before replacement. + + ### Component measurements already recorded A separate [offline retrieval replay](evaluations/results/retrieval_baseline_v1.json) used **97 known-positive training tuning questions**. Article Recall@5 was **83/97 (85.57%)** and Recall@10 **87/97 (89.69%)**. The packed-context reference-word-run proxy passed **80/97 (82.47%)**; that proxy is not semantic context recall. diff --git a/evaluations/overview.svg b/evaluations/overview.svg index c096569..41a2423 100644 --- a/evaluations/overview.svg +++ b/evaluations/overview.svg @@ -1,6 +1,6 @@ -TechQA RAG evaluation suite: components, pipeline, application, regression and planned monitoring -Implemented Python contract checks and RAGAS pipeline evaluation, optional factual correctness and completeness judges, paired regression reports, and planned sampled online monitoring. A three-case pilot is integration evidence, not a final benchmark. +TechQA RAG evaluation suite with traces and reviewed feedback +Python contract checks, sampled RAGAS evaluation, user feedback, reviewed regression cases and planned live monitoring. Reserved release evaluation is pending. @@ -10,9 +10,9 @@ SHIPPED -OPT-IN / NOT BENCHMARKED +OPT-IN / SAMPLED -PLANNED +PARTIAL 01 Components — test one part at a time @@ -57,14 +57,14 @@ Same cases + same references → paired outcome and score changes → inspect regressions. -04 Online — sampled production monitoring +04 Online — feedback and run inspection -PLANNED -Random samples + flagged requests · calibrated judges · feedback · drift monitoring -Operational logs exist today; Langfuse, online scoring and automated feedback do not. +PARTIAL +LangSmith traces · user ratings · review notes · regression cases +Sampled live judging and drift monitoring are planned. Review failures → curate training cases → rerun the fixed suite -Manual review is active. Automatic case promotion and self-improvement are planned. -Evidence so far: 3-case tuning pilot. Final held-out quality and production readiness are not established. +Reviewed references are required before feedback cases enter RAGAS evaluation. +Training-side evaluations are recorded in the README. Reserved release evaluation remains pending. diff --git a/evaluations/results/techqa_wide_v5.json b/evaluations/results/techqa_wide_v5.json new file mode 100644 index 0000000..c4842f2 --- /dev/null +++ b/evaluations/results/techqa_wide_v5.json @@ -0,0 +1,609 @@ +{ + "scope": "Seeded answerable training tuning cohort. Standalone canonical retrieval, writing, verification and repair.", + "selection": { + "planned": 30, + "collected": 15, + "pending": 15, + "seed": 20261110 + }, + "model": "openai/gpt-oss-120b", + "judge": "openai/gpt-oss-20b", + "prompt_version": "strict_v5_message_roles", + "outcomes": { + "generation_error": 5, + "insufficient_evidence": 2, + "answered": 8 + }, + "generation_errors": { + "invalid_citations": 3, + "quota_deadline": 1, + "http_429": 1 + }, + "metrics": { + "answer_coverage": 0.5333333333333333, + "selected_gold_document_hit_rate": 0.9333333333333333, + "citation_contract_pass_rate": 1.0 + }, + "denominators": { + "selected_gold_document_hit_rate": 15, + "citation_contract_pass_rate": 8 + }, + "ragas_raw": { + "attempted": 15, + "outcomes": { + "generation_error": 5, + "insufficient_evidence": 2, + "answered": 8 + }, + "metrics": { + "context_recall": { + "success": 11, + "error": 0, + "undefined": 0, + "skipped": 0, + "budget_exhausted": 3, + "mean": 0.9090909090909091, + "flagged_for_review": 0 + }, + "faithfulness": { + "success": 5, + "error": 0, + "undefined": 0, + "skipped": 3, + "budget_exhausted": 0, + "mean": 0.975, + "flagged_for_review": 0 + }, + "context_precision": { + "success": 2, + "error": 0, + "undefined": 0, + "skipped": 0, + "budget_exhausted": 0, + "mean": 0.9020833333069618, + "flagged_for_review": 0 + }, + "factual_correctness": { + "success": 5, + "error": 0, + "undefined": 0, + "skipped": 0, + "budget_exhausted": 0, + "mean": 0.426, + "flagged_for_review": 1 + }, + "answer_completeness": { + "success": 5, + "error": 0, + "undefined": 0, + "skipped": 0, + "budget_exhausted": 0, + "mean": 0.534, + "flagged_for_review": 1 + }, + "answer_relevancy": { + "success": 3, + "error": 0, + "undefined": 0, + "skipped": 0, + "budget_exhausted": 0, + "mean": 0.9304797472908871, + "flagged_for_review": 0 + } + } + }, + "sampling": { + "core_initial_ids": [ + "TRAIN_Q252", + "TRAIN_Q425", + "TRAIN_Q376", + "TRAIN_Q013", + "TRAIN_Q414" + ], + "precision_ids": [ + "TRAIN_Q252", + "TRAIN_Q425" + ], + "answer_quality_ids": [ + "TRAIN_Q376", + "TRAIN_Q013", + "TRAIN_Q117", + "TRAIN_Q548", + "TRAIN_Q310" + ], + "relevance_ids": [ + "TRAIN_Q376", + "TRAIN_Q013", + "TRAIN_Q117" + ], + "wider_recall": "remaining budget after other metrics" + }, + "review_notes": { + "TRAIN_Q376": "Reference scope needs review. A source-link answer is compared with the full bulletin text.", + "TRAIN_Q310": "Heuristic flag needs review. The as-5 match concerns initial connection counts while as-60 concerns cache timeout. Both numbers appear in the original response." + }, + "operations": { + "generation_http_attempts": 41, + "generation_reported_tokens": 206804, + "judge_http_attempts": 120, + "judge_reported_tokens": 172079, + "calibration_http_attempts": 4, + "calibration_reported_tokens": 3757, + "judge_cache_hits": 29, + "judge_cache_misses": 69, + "answered_batch_seconds_median": 128.05175710400363, + "answered_batch_seconds_max": 141.51168862500344, + "timing_scope": "Includes retrieval, model calls and quota pacing in a serial batch.", + "cost_usd": null + }, + "budgets": { + "generation_total_cap": 90, + "generation_invocations": [ + { + "limit": 90, + "sent": 38 + }, + { + "limit": 52, + "sent": 3 + } + ], + "judge_cap": 120, + "calibration_cap": 8 + }, + "cases": [ + { + "question_id": "TRAIN_Q252", + "outcome": "generation_error", + "selected_gold_document_hit": 1, + "selected_passage_count": 7, + "citation_contract": { + "state": "skipped", + "pass": null + }, + "ragas": { + "context_recall": { + "state": "success", + "score": 1.0, + "needs_review": false + }, + "faithfulness": { + "state": "skipped", + "score": null, + "needs_review": false + }, + "context_precision": { + "state": "success", + "score": 0.8874999999778125, + "needs_review": false + } + }, + "judge_integrity_flags": [] + }, + { + "question_id": "TRAIN_Q425", + "outcome": "insufficient_evidence", + "selected_gold_document_hit": 1, + "selected_passage_count": 9, + "citation_contract": { + "state": "skipped", + "pass": null + }, + "ragas": { + "context_recall": { + "state": "success", + "score": 1.0, + "needs_review": false + }, + "faithfulness": { + "state": "skipped", + "score": null, + "needs_review": false + }, + "context_precision": { + "state": "success", + "score": 0.9166666666361111, + "needs_review": false + } + }, + "judge_integrity_flags": [] + }, + { + "question_id": "TRAIN_Q376", + "outcome": "answered", + "selected_gold_document_hit": 1, + "selected_passage_count": 14, + "citation_contract": { + "state": "checked", + "pass": true, + "reason": null + }, + "ragas": { + "context_recall": { + "state": "success", + "score": 1.0, + "needs_review": false + }, + "faithfulness": { + "state": "success", + "score": 1.0, + "needs_review": false + }, + "factual_correctness": { + "state": "success", + "score": 0.0, + "needs_review": false + }, + "answer_completeness": { + "state": "success", + "score": 0.0, + "needs_review": false + }, + "answer_relevancy": { + "state": "success", + "score": 0.9170371340678626, + "needs_review": false + } + }, + "judge_integrity_flags": [] + }, + { + "question_id": "TRAIN_Q013", + "outcome": "answered", + "selected_gold_document_hit": 1, + "selected_passage_count": 8, + "citation_contract": { + "state": "checked", + "pass": true, + "reason": null + }, + "ragas": { + "context_recall": { + "state": "success", + "score": 1.0, + "needs_review": false + }, + "faithfulness": { + "state": "success", + "score": 1.0, + "needs_review": false + }, + "factual_correctness": { + "state": "success", + "score": 0.8, + "needs_review": false + }, + "answer_completeness": { + "state": "success", + "score": 0.67, + "needs_review": false + }, + "answer_relevancy": { + "state": "success", + "score": 0.9793743376624086, + "needs_review": false + } + }, + "judge_integrity_flags": [] + }, + { + "question_id": "TRAIN_Q414", + "outcome": "generation_error", + "selected_gold_document_hit": 1, + "selected_passage_count": 10, + "citation_contract": { + "state": "skipped", + "pass": null + }, + "ragas": { + "context_recall": { + "state": "success", + "score": 1.0, + "needs_review": false + }, + "faithfulness": { + "state": "skipped", + "score": null, + "needs_review": false + } + }, + "judge_integrity_flags": [] + }, + { + "question_id": "TRAIN_Q488", + "outcome": "insufficient_evidence", + "selected_gold_document_hit": 1, + "selected_passage_count": 10, + "citation_contract": { + "state": "skipped", + "pass": null + }, + "ragas": { + "context_recall": { + "state": "success", + "score": 1.0, + "needs_review": false + } + }, + "judge_integrity_flags": [] + }, + { + "question_id": "TRAIN_Q361", + "outcome": "generation_error", + "selected_gold_document_hit": 1, + "selected_passage_count": 12, + "citation_contract": { + "state": "skipped", + "pass": null + }, + "ragas": { + "context_recall": { + "state": "success", + "score": 1.0, + "needs_review": false + } + }, + "judge_integrity_flags": [] + }, + { + "question_id": "TRAIN_Q117", + "outcome": "answered", + "selected_gold_document_hit": 1, + "selected_passage_count": 14, + "citation_contract": { + "state": "checked", + "pass": true, + "reason": null + }, + "ragas": { + "factual_correctness": { + "state": "success", + "score": 1.0, + "needs_review": false + }, + "answer_completeness": { + "state": "success", + "score": 1.0, + "needs_review": false + }, + "faithfulness": { + "state": "success", + "score": 1.0, + "needs_review": false + }, + "answer_relevancy": { + "state": "success", + "score": 0.8950277701423902, + "needs_review": false + }, + "context_recall": { + "state": "success", + "score": 1.0, + "needs_review": false + } + }, + "judge_integrity_flags": [] + }, + { + "question_id": "TRAIN_Q548", + "outcome": "answered", + "selected_gold_document_hit": 1, + "selected_passage_count": 8, + "citation_contract": { + "state": "checked", + "pass": true, + "reason": null + }, + "ragas": { + "factual_correctness": { + "state": "success", + "score": 0.0, + "needs_review": false + }, + "answer_completeness": { + "state": "success", + "score": 0.0, + "needs_review": false + }, + "faithfulness": { + "state": "success", + "score": 1.0, + "needs_review": false + }, + "context_recall": { + "state": "success", + "score": 1.0, + "needs_review": false + } + }, + "judge_integrity_flags": [] + }, + { + "question_id": "TRAIN_Q310", + "outcome": "answered", + "selected_gold_document_hit": 1, + "selected_passage_count": 8, + "citation_contract": { + "state": "checked", + "pass": true, + "reason": null + }, + "ragas": { + "factual_correctness": { + "state": "success", + "score": 0.33, + "needs_review": true + }, + "answer_completeness": { + "state": "success", + "score": 1.0, + "needs_review": true + }, + "faithfulness": { + "state": "success", + "score": 0.875, + "needs_review": false + }, + "context_recall": { + "state": "success", + "score": 1.0, + "needs_review": false + } + }, + "judge_integrity_flags": [ + { + "metric": "factual_correctness", + "label": "as", + "introduced_version": "5", + "allowed_versions": [ + "60" + ], + "needs_review": true + }, + { + "metric": "answer_completeness", + "label": "as", + "introduced_version": "5", + "allowed_versions": [ + "60" + ], + "needs_review": true + } + ] + }, + { + "question_id": "TRAIN_Q230", + "outcome": "answered", + "selected_gold_document_hit": 0, + "selected_passage_count": 8, + "citation_contract": { + "state": "checked", + "pass": true, + "reason": null + }, + "ragas": { + "context_recall": { + "state": "success", + "score": 0.0, + "needs_review": false + } + }, + "judge_integrity_flags": [] + }, + { + "question_id": "TRAIN_Q353", + "outcome": "answered", + "selected_gold_document_hit": 1, + "selected_passage_count": 8, + "citation_contract": { + "state": "checked", + "pass": true, + "reason": null + }, + "ragas": { + "context_recall": { + "state": "budget_exhausted", + "score": null, + "needs_review": false + } + }, + "judge_integrity_flags": [] + }, + { + "question_id": "TRAIN_Q599", + "outcome": "answered", + "selected_gold_document_hit": 1, + "selected_passage_count": 8, + "citation_contract": { + "state": "checked", + "pass": true, + "reason": null + }, + "ragas": { + "context_recall": { + "state": "budget_exhausted", + "score": null, + "needs_review": false + } + }, + "judge_integrity_flags": [] + }, + { + "question_id": "TRAIN_Q365", + "outcome": "generation_error", + "selected_gold_document_hit": 1, + "selected_passage_count": 16, + "citation_contract": { + "state": "skipped", + "pass": null + }, + "ragas": { + "context_recall": { + "state": "budget_exhausted", + "score": null, + "needs_review": false + } + }, + "judge_integrity_flags": [] + }, + { + "question_id": "TRAIN_Q388", + "outcome": "generation_error", + "selected_gold_document_hit": 1, + "selected_passage_count": 12, + "citation_contract": { + "state": "skipped", + "pass": null + }, + "ragas": {}, + "judge_integrity_flags": [] + } + ], + "artifact_sha256": { + "collection_manifest": "a49bf986219036e536a5395b8450c13167316f6595a774d16307f239f55aba00", + "raw_ragas": "cf0e07a38ea351304aefc6f62d5e7a8d16caa0fa271c3c158277208ca54e0b48", + "final_composed_ragas": "b4f32bad9144a5db2b33570c50e7c42812d64fc47a10b9453ff59d612aa3ad2f" + }, + "limitations": [ + "Provider rate limiting stopped generation with 15 planned cases pending.", + "Three context recall scores exhausted the judge request budget.", + "Metrics use different sample sizes. Answer metrics are conditional on released answers.", + "Raw means retain one case with heuristic review flags. LangSmith withholds its flagged metric values.", + "Reference agreement depends on answer scope and reference completeness. Scores require review.", + "The judge differs from earlier runs. This run establishes a fresh baseline.", + "Reserved development questions were not used." + ], + "judge_config": { + "temperature": 0, + "reasoning_effort": "low", + "max_tokens": 2048, + "strictness": 3, + "packages": { + "ragas": "0.4.3", + "openai": "2.54.0", + "instructor": "1.17.0", + "langchain-community": "0.4.1", + "sentence-transformers": "6.2.0", + "torch": "2.15.0", + "transformers": "5.22.1", + "torchvision": "0.29.2" + }, + "ragas_source_sha256": "3e1a68a9cbef46834c672b716266089b1764d5d7bd424f997e92c8d92083e160", + "implementation_sha256": "f39f2b56005861d92b8df4e2fc33004dc2fed6abff256f428c916c470bc009fe" + }, + "inference_source_sha256": { + "rag_app/__init__.py": "2fdce6dc6a2613b920f0df2e2dcaffbf3aa3775fa03b3b2b35294f6f2b609431", + "rag_app/api.py": "194c7657b566f0330ce2e463ab7772ecbb4e7aeee3ef2feddaff1b77a03d8553", + "rag_app/feedback.py": "61ef4c889fb31d58e73ceba58426f9747f6a7ae7b563f3334f950391eb4bdb49", + "rag_app/generation.py": "9afb38360c6374db464e6bbc7feb0a10f9e5786a7162f5a9180ad9b9e39be135", + "rag_app/indexes.py": "bbd9759f62a68b3a57380c587f772018421e00d918542257232c4eb8cfbf0a5c", + "rag_app/observability.py": "eec2d67e7a02622391d6f4b8d0ef0fe86b53c9f444c6d226b4f2f9d4948e023a", + "rag_app/presentation.py": "f3d01624ef8cdb8961041517a8712ebec745e350e4aef03dcd99895205ac33d2", + "rag_app/retriever.py": "c49ed1d3376044f656d20d632f54761d89461c5d7e4ab8ab3d2a831e81459940", + "rag_app/state.py": "201aba9117c53f6b34a9a4455a413953a5aede2b6baa78129cb5e1698f75fc45", + "rag_app/storage.py": "7220891a37d005ad734440a56bea69c1161ffab807e194b4aca69363fdaedb1a", + "rag_app/tracing.py": "7ccd4f804b359adbd77e168178db9eedf73eac4f8ca7cac639fa76032f4adc92" + }, + "split_sha256": "9b3c6d289f9570e6d7230dec90b15c769b0fc56ceb609bb2c3f44dc73a9f977a", + "inference_snapshot_matches_current": true +} diff --git a/scripts/eval_ragas.py b/scripts/eval_ragas.py index 685f8a5..60e2e7f 100644 --- a/scripts/eval_ragas.py +++ b/scripts/eval_ragas.py @@ -197,9 +197,10 @@ def bounded_post(*args, **kwargs): path = args.output / (q['QUESTION_ID'] + '.json') if path.exists(): previous = json.loads(path.read_text()) - if previous.get('generation', {}).get('error_type') not in {'http_429', 'quota_deadline'}: + if previous.get('generation', {}).get('error_type') != 'http_429': print(q['QUESTION_ID'] + ': saved case reused', flush=True) continue + save(args.output / 'attempts' / q['QUESTION_ID'] / (fingerprint(previous) + '.json'), previous) if budget.exhausted or budget.sent >= budget.limit: print('HTTP budget reached; remaining cases can resume later', flush=True) break @@ -233,7 +234,7 @@ def bounded_post(*args, **kwargs): run.end(error=answer.get('error_type'), outputs={'status': case['status'], 'response': case['response'], 'sources': case['sources'], 'generation': {key: value for key, value in case['generation'].items() if key != 'trace'}}) save(path, case) print(f" saved {case['status']}; {len(case['sources'])} passages", flush=True) - if answer.get('error_type') in {'http_429', 'quota_deadline'}: + if answer.get('error_type') == 'http_429': print('Provider quota reached; stopping without further requests', flush=True) break finally: diff --git a/tests/test_ragas_eval.py b/tests/test_ragas_eval.py index b2297ab..b46ceac 100644 --- a/tests/test_ragas_eval.py +++ b/tests/test_ragas_eval.py @@ -156,7 +156,8 @@ async def aembed_text(self, text, **kwargs): return self.embed_text(text) assert results['answer_completeness']['score'] == pytest.approx(.5) -def test_collection_stops_after_first_quota_error(tmp_path, monkeypatch): +@pytest.mark.parametrize('category, expected_calls', [('http_429', 1), ('quota_deadline', 2)]) +def test_collection_distinguishes_account_limits_from_request_deadlines(tmp_path, monkeypatch, category, expected_calls): import json import scripts.eval_ragas as evaluation import rag_app.retriever as retrieval @@ -184,20 +185,21 @@ class QuotaWriter: trace = [] def generate(self, question, sources): attempts.append(question) - raise generation.GenerationError('quota', category='http_429') + raise generation.GenerationError('quota', category=category) monkeypatch.setattr(generation, 'AnswerGenerator', lambda call: QuotaWriter()) monkeypatch.delenv('LANGSMITH_API_KEY', raising=False) output = root / 'output' args = SimpleNamespace(limit=2, output=output, max_requests=5, sample_seed=None) evaluation.collect(args) - assert len(attempts) == 1 + assert len(attempts) == expected_calls assert (output / 'TRAIN_Q1.json').exists() - assert not (output / 'TRAIN_Q2.json').exists() - assert len(json.loads((output / 'completed.json').read_text())['cases']) == 1 + assert (output / 'TRAIN_Q2.json').exists() == (expected_calls == 2) + assert len(json.loads((output / 'completed.json').read_text())['cases']) == expected_calls evaluation.collect(args) assert len(attempts) == 2 - assert not (output / 'TRAIN_Q2.json').exists() + archived = list((output / 'attempts').rglob('*.json')) + assert len(archived) == (1 if category == 'http_429' else 0) def test_unreviewed_feedback_is_rejected_before_judging(tmp_path):