Skip to content

Audit three evaluation layers and fix DeepSeek observability - #10

Merged
shipitdev merged 4 commits into
mainfrom
codex/evaluation-layer-audit
Nov 11, 2026
Merged

shipitdev merged 4 commits into
mainfrom
codex/evaluation-layer-audit

Conversation

@shipitdev

Copy link
Copy Markdown
Owner

Changes

  • Configure TokenHarbor DeepSeek for generation and judging, with a shared locked budget ledger and cache-aware token valuation.
  • Fix misleading Groq error labels, DeepSeek truncation settings and lost usage on invalid/truncated responses.
  • Add supplied-evidence generator and actual API workflow collectors using the same RAGAS scoring entry point.
  • Import recorded duration, usage and failure status in the initial LangSmith run records. Add metric state, eligibility and coverage feedback.
  • Publish the ten-question audit, matching historical rescoring, README instructions and updated evaluation SVG.

Results and limits

  • Ten new training questions produced seven answers, two withheld responses and one clarification, with zero generation errors.
  • All six metric types scored for every eligible case. Component and workflow samples contain two cases each.
  • Historical reference correctness still falls on four shared answers under the same DeepSeek judge. Review identifies version/scope issues, false verifier rejections and reference/judge limitations.
  • Shared token value stayed at $0.295 of the $0.50 guard, including failed preflights and one unresolved reservation. The selected free route bills zero.
  • These small training diagnostics use DeepSeek for both today's writer and judge. Human alignment and held-out release evaluation remain pending.

Validation

  • 168 Python tests plus three subtests passed, with two optional skips.
  • Separate RAGAS environment checks passed 20 tests.
  • Verified native dashboard duration, tokens and error counts remotely.
  • Rendered and inspected the SVG with no text overflow.
  • Verified coverage, budget and redaction assertions. Original knowledge.md hash preserved.

@shipitdev
shipitdev merged commit 3705cbb into main Nov 11, 2026
2 checks passed
@shipitdev
shipitdev deleted the codex/evaluation-layer-audit branch November 11, 2026 07:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant