Skip to content

Add LangSmith observability and an evidence-first activity workspace - #4

Merged
shipitdev merged 4 commits into
mainfrom
codex/langsmith-workspace
Nov 10, 2026
Merged

shipitdev merged 4 commits into
mainfrom
codex/langsmith-workspace

Conversation

@shipitdev

Copy link
Copy Markdown
Owner

Changes

  • Add optional, fail-soft LangSmith traces around each request, embedding, exact search, reranking, context selection and Groq calls.
  • Record actual tokens, cache use, sources, verification/repair decisions and released outcomes. Keep trace links and diagnostics available for failed requests.
  • Publish saved runs as LangSmith experiments with free contract checks and optional matching RAGAS scores. Keep the dataset stable when answers change.
  • Add seeded training sampling, a hard HTTP-attempt cap, saved checkpoints and stop-on-quota behavior.
  • Replace the spinner/checklist panel with live activity and early source excerpts. Keep verified release checks, add retained activity history, local fonts and responsive navigation.
  • Publish a redacted observability result and document setup and the streaming API contract.

Measured batch

  • Twelve fixed training tuning questions. Labeled article selected for 11/12.
  • Three released answers. Citation-format checks passed for 3/3.
  • Nine provider quota errors. Fifteen HTTP attempts and 33,230 reported tokens.
  • No new semantic RAGAS scores or held-out evaluation. Quota errors are separate from answer-quality failures.

Checks

  • Local Python suite passed with 151 tests, 3 subtests and one optional RAGAS integration skip.
  • Eight frontend contract tests passed. Typecheck, lint and production build passed.
  • Controlled browser transport verified progress before approval, source previews, OpenUI rendering and session reset at 1440, 768 and 390 pixels, without horizontal overflow or page errors.
  • Real LangSmith connection, saved experiment publication and a detailed cache-hit retrieval trace were verified. Browser fixtures made no model calls.

Limits

  • The quota-limited run does not establish answer correctness. New semantic judging awaits provider quota.
  • Private traces include questions, evidence and drafts. Credentials and raw records remain outside Git.

@shipitdev
shipitdev merged commit 165c3b0 into main Nov 10, 2026
2 checks passed
@shipitdev
shipitdev deleted the codex/langsmith-workspace branch November 10, 2026 13:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant