Add LangSmith observability and an evidence-first activity workspace

.env.example

@@ -11,3 +11,9 @@ ENABLE_GENUI=1
 
 # Optional private local diagnostics with bounded retention.
 # RAG_TRACE_DIR=data/app/traces
+
+
+# Optional LangSmith dashboard. Traces include questions, evidence and model outputs.
+LANGSMITH_API_KEY=
+LANGSMITH_PROJECT=techqa-rag
+LANGSMITH_TRACING=true

.github/workflows/ci.yml

@@ -18,6 +18,7 @@ jobs:
     timeout-minutes: 15
     env:
       RAGAS_DO_NOT_TRACK: 'true'
+      LANGSMITH_TRACING: 'false'
       HF_HUB_OFFLINE: '1'
       GROQ_API_KEY: ''
       GROK_API_KEY: ''

.gitignore

@@ -119,3 +119,18 @@
 !/tests/test_v1_acceptance.py
 !/frontend/src/lib/request-origin.ts
 !/evaluations/results/v1_acceptance.json
+
+
+!/rag_app/tracing.py
+!/tests/test_tracing.py
+
+!/frontend/src/app/fonts/
+!/frontend/src/app/fonts/*.woff2
+!/frontend/src/app/fonts/OFL.txt
+
+!/tests/conftest.py
+
+!/scripts/publish_langsmith.py
+!/tests/test_langsmith_export.py
+
+!/evaluations/results/observability_v1.json

README.md

@@ -19,6 +19,28 @@ Question → Gemma embedding → Qdrant exact search → BGE reranking
 - **Sessions:** recent conversation plus a summary, current-session memory, exact answer caching and an explicit regeneration bypass.
 - **Evaluation:** an offline RAGAS runner scores saved evidence and released answers. Judge decisions and local query embeddings are cached; withheld responses and evaluation errors stay visible.
 
+## Inspect a run in LangSmith
+
+Add `LANGSMITH_API_KEY` to the server's `.env`. The default project is `techqa-rag`. Set `LANGSMITH_TRACING=false` to disable export. Traces contain questions, retrieved passages, private drafts and verification decisions, so keep this project private.
+
+Each question has a parent trace with child steps for query embedding, exact search, reranking, context selection and model calls. Inspect timings, actual token usage, cache hits, source passages, repair attempts and the released outcome. The answer workspace links to its trace. Local diagnostic logs continue to work when tracing is disabled or export fails.
+
+The workspace streams actual activity and retrieved excerpts. Verified answers and checked OpenUI layouts appear after their release checks. The run history remains available after completion. Desktop uses a compact navigation rail, while smaller screens stack the answer and evidence.
+
+For developers and coding agents, `/ask` remains a server-sent event endpoint. `stage` events describe activity, `evidence` carries retrieved `sources`, `final` carries the approved outcome and `error` carries a safe error category/message. `done` closes the run. Final statuses are `answered`, `needs_clarification` and `insufficient_evidence`. Existing exact-cache, session, cancellation and regeneration behavior remains covered by tests.
+
+Collect a fixed random training sample with a hard HTTP-attempt cap. Quota failures stop collection and are retried when the same unchanged run resumes. Completed cases are saved after each question, with a `completed.json` snapshot for inspection.
+
+```bash
+python -m scripts.eval_ragas collect --limit 12 --sample-seed 20261110 \
+  --max-requests 18 --output data/ragas/review
+python -m scripts.publish_langsmith --cases data/ragas/review/completed.json
+```
+
+Publishing replays saved outputs into a LangSmith experiment with response status, selected-reference-article presence and citation-format checks. It makes **zero new model calls**. Add `--ragas path/to/matching-report.json` to publish existing semantic scores too. RAGAS judging remains a separate, explicitly budgeted command below. Reviewed or failed judge results retain their flags.
+
+The [November 10 observability batch](evaluations/results/observability_v1.json) selected **12 training questions**. Retrieval selected the labeled article for **11/12**. **3 answers** were released and passed citation-format checks. **9 provider quota failures** prevented completion. The run used **15 HTTP attempts and 33,230 reported tokens**. This batch establishes dashboard visibility and exposes a quota-handling issue, which is now covered by a stop-on-quota regression test. New semantic RAGAS scores await available provider quota.
+
 ## Evaluation suite
 
 ![Evaluation layers: components, pipeline, application, regression and planned monitoring](evaluations/overview.svg)

evaluations/results/observability_v1.json

@@ -0,0 +1,207 @@
+{
+  "schema": "saved_case_checks_v1",
+  "cohort_id": "44dd958d19897a7639d9a2cb420e53270b2bea37328d2ee42a9cd7d124e9f97c",
+  "case_count": 12,
+  "outcomes": {
+    "answered": 3,
+    "generation_error": 9
+  },
+  "metrics": {
+    "answer_coverage": 0.25,
+    "selected_gold_document_hit_rate": 0.9166666666666666,
+    "citation_contract_pass_rate": 1.0
+  },
+  "denominators": {
+    "selected_gold_document_hit_rate": 12,
+    "citation_contract_pass_rate": 3
+  },
+  "operations": {
+    "generation_seconds": [
+      53.6789352909982,
+      89.72582383299596,
+      94.20225566600857,
+      46.86580899999535,
+      2.3957592910010135,
+      2.1422123329975875,
+      2.1818811670091236,
+      2.2330157079995843,
+      2.2595714589988347,
+      2.226248499995563,
+      1.9820034590084106,
+      0.29575791700335685
+    ],
+    "generation_reported_tokens": 33230,
+    "generation_usage_cases": 12,
+    "cost_usd": null
+  },
+  "cases": [
+    {
+      "question_id": "TRAIN_Q252",
+      "outcome": "answered",
+      "selected_gold_document_hit": 1,
+      "selected_passage_count": 7,
+      "citation_contract": {
+        "state": "checked",
+        "pass": true,
+        "reason": null
+      },
+      "ragas": {},
+      "judge_integrity_flags": []
+    },
+    {
+      "question_id": "TRAIN_Q425",
+      "outcome": "answered",
+      "selected_gold_document_hit": 1,
+      "selected_passage_count": 9,
+      "citation_contract": {
+        "state": "checked",
+        "pass": true,
+        "reason": null
+      },
+      "ragas": {},
+      "judge_integrity_flags": []
+    },
+    {
+      "question_id": "TRAIN_Q376",
+      "outcome": "answered",
+      "selected_gold_document_hit": 1,
+      "selected_passage_count": 14,
+      "citation_contract": {
+        "state": "checked",
+        "pass": true,
+        "reason": null
+      },
+      "ragas": {},
+      "judge_integrity_flags": []
+    },
+    {
+      "question_id": "TRAIN_Q013",
+      "outcome": "generation_error",
+      "selected_gold_document_hit": 1,
+      "selected_passage_count": 8,
+      "citation_contract": {
+        "state": "skipped",
+        "pass": null
+      },
+      "ragas": {},
+      "judge_integrity_flags": []
+    },
+    {
+      "question_id": "TRAIN_Q414",
+      "outcome": "generation_error",
+      "selected_gold_document_hit": 1,
+      "selected_passage_count": 10,
+      "citation_contract": {
+        "state": "skipped",
+        "pass": null
+      },
+      "ragas": {},
+      "judge_integrity_flags": []
+    },
+    {
+      "question_id": "TRAIN_Q488",
+      "outcome": "generation_error",
+      "selected_gold_document_hit": 1,
+      "selected_passage_count": 10,
+      "citation_contract": {
+        "state": "skipped",
+        "pass": null
+      },
+      "ragas": {},
+      "judge_integrity_flags": []
+    },
+    {
+      "question_id": "TRAIN_Q361",
+      "outcome": "generation_error",
+      "selected_gold_document_hit": 1,
+      "selected_passage_count": 12,
+      "citation_contract": {
+        "state": "skipped",
+        "pass": null
+      },
+      "ragas": {},
+      "judge_integrity_flags": []
+    },
+    {
+      "question_id": "TRAIN_Q117",
+      "outcome": "generation_error",
+      "selected_gold_document_hit": 1,
+      "selected_passage_count": 14,
+      "citation_contract": {
+        "state": "skipped",
+        "pass": null
+      },
+      "ragas": {},
+      "judge_integrity_flags": []
+    },
+    {
+      "question_id": "TRAIN_Q548",
+      "outcome": "generation_error",
+      "selected_gold_document_hit": 1,
+      "selected_passage_count": 8,
+      "citation_contract": {
+        "state": "skipped",
+        "pass": null
+      },
+      "ragas": {},
+      "judge_integrity_flags": []
+    },
+    {
+      "question_id": "TRAIN_Q310",
+      "outcome": "generation_error",
+      "selected_gold_document_hit": 1,
+      "selected_passage_count": 8,
+      "citation_contract": {
+        "state": "skipped",
+        "pass": null
+      },
+      "ragas": {},
+      "judge_integrity_flags": []
+    },
+    {
+      "question_id": "TRAIN_Q230",
+      "outcome": "generation_error",
+      "selected_gold_document_hit": 0,
+      "selected_passage_count": 8,
+      "citation_contract": {
+        "state": "skipped",
+        "pass": null
+      },
+      "ragas": {},
+      "judge_integrity_flags": []
+    },
+    {
+      "question_id": "TRAIN_Q353",
+      "outcome": "generation_error",
+      "selected_gold_document_hit": 1,
+      "selected_passage_count": 8,
+      "citation_contract": {
+        "state": "skipped",
+        "pass": null
+      },
+      "ragas": {},
+      "judge_integrity_flags": []
+    }
+  ],
+  "limits": [
+    "Selected-document hit is not search Recall@K or semantic evidence recall.",
+    "Citation validity does not establish claim support.",
+    "Version flags are heuristics for review; raw scores are never adjusted.",
+    "No style, toxicity/PII, live feedback or drift scores are computed."
+  ],
+  "scope": "Fixed seeded random answerable training tuning sample; standalone canonical pipeline; no memory or generated layout evaluation.",
+  "selection": {
+    "seed": 20261110,
+    "planned": 12,
+    "attempted": 12
+  },
+  "provider_http_attempts": 15,
+  "limitations": [
+    "Nine provider quota failures are infrastructure errors, not answer-quality failures.",
+    "Three released answers passed citation syntax/source-ID checks. Semantic correctness was not measured in this batch.",
+    "Selected gold article presence is not semantic context recall.",
+    "No new RAGAS judge calls.",
+    "Collector was fixed after this run to stop on the first quota failure.",
+    "No reserved development questions were used."
+  ]
+}

frontend/src/app/fonts/OFL.txt

@@ -0,0 +1,92 @@
+Copyright (c) 2023 Vercel, in collaboration with basement.studio
+
+This Font Software is licensed under the SIL Open Font License, Version 1.1.
+This license is copied below, and is also available with a FAQ at:
+http://scripts.sil.org/OFL
+
+-----------------------------------------------------------
+SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
+-----------------------------------------------------------
+
+PREAMBLE
+The goals of the Open Font License (OFL) are to stimulate worldwide
+development of collaborative font projects, to support the font creation
+efforts of academic and linguistic communities, and to provide a free and
+open framework in which fonts may be shared and improved in partnership
+with others.
+
+The OFL allows the licensed fonts to be used, studied, modified and
+redistributed freely as long as they are not sold by themselves. The
+fonts, including any derivative works, can be bundled, embedded,
+redistributed and/or sold with any software provided that any reserved
+names are not used by derivative works. The fonts and derivatives,
+however, cannot be released under any other type of license. The
+requirement for fonts to remain under this license does not apply
+to any document created using the fonts or their derivatives.
+
+DEFINITIONS
+"Font Software" refers to the set of files released by the Copyright
+Holder(s) under this license and clearly marked as such. This may
+include source files, build scripts and documentation.
+
+"Reserved Font Name" refers to any names specified as such after the
+copyright statement(s).
+
+"Original Version" refers to the collection of Font Software components as
+distributed by the Copyright Holder(s).
+
+"Modified Version" refers to any derivative made by adding to, deleting,
+or substituting -- in part or in whole -- any of the components of the
+Original Version, by changing formats or by porting the Font Software to a
+new environment.
+
+"Author" refers to any designer, engineer, programmer, technical
+writer or other person who contributed to the Font Software.
+
+PERMISSION AND CONDITIONS
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of the Font Software, to use, study, copy, merge, embed, modify,
+redistribute, and sell modified and unmodified copies of the Font
+Software, subject to the following conditions:
+
+1) Neither the Font Software nor any of its individual components,
+in Original or Modified Versions, may be sold by itself.
+
+2) Original or Modified Versions of the Font Software may be bundled,
+redistributed and/or sold with any software, provided that each copy
+contains the above copyright notice and this license. These can be
+included either as stand-alone text files, human-readable headers or
+in the appropriate machine-readable metadata fields within text or
+binary files as long as those fields can be easily viewed by the user.
+
+3) No Modified Version of the Font Software may use the Reserved Font
+Name(s) unless explicit written permission is granted by the corresponding
+Copyright Holder. This restriction only applies to the primary font name as
+presented to the users.
+
+4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font
+Software shall not be used to promote, endorse or advertise any
+Modified Version, except to acknowledge the contribution(s) of the
+Copyright Holder(s) and the Author(s) or with their explicit written
+permission.
+
+5) The Font Software, modified or unmodified, in part or in whole,
+must be distributed entirely under this license, and must not be
+distributed under any other license. The requirement for fonts to
+remain under this license does not apply to any document created
+using the Font Software.
+
+TERMINATION
+This license becomes null and void if any of the above conditions are
+not met.
+
+DISCLAIMER
+THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
+OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
+COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
+DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM
+OTHER DEALINGS IN THE FONT SOFTWARE.