Conversational assistant over a solar-plant operations dataset (7 CSVs). The graded deliverable is the orchestration pipeline: explicit intent classification → tool selection → in-code aggregation → refusal-guarded synthesis.
Design lives in
docs/design/, requirements indocs/requirements/, and decisions indocs/adr/.
DataSourceinterface +PandasDataSourceloading the 7 CSVs once; dataset-relative time anchored to the dataset's latest timestamp (~2026-06-22), not wall-clock.- A uniform
ToolRegistrywith the per-surface dataset tools plus derived/tooling support used by the replay harness and synthesis pipeline. - Explicit, logged turn routing and intent classification, with graph-state scaffolding for
the committed LangGraph redesign now in place, while the real
StateGraphruntime is still pending. - FastAPI server, Web UI, server-backed CLI, persisted sessions/settings, dataset config import/export, and behavioural replay gates.
Before trusting the pipeline on a dataset, profile it. scripts/profile_dataset.py
reads the seven CSVs (read-only — it never edits the data) and emits a markdown
report to docs/dataset-analysis.md:
python scripts/profile_dataset.py # print report to stdout
python scripts/profile_dataset.py --write # also write docs/dataset-analysis.mdRe-run it whenever the data changes (e.g. an evaluator swaps in a fresher CSV set). It is the single source of truth for two things the tools depend on:
- The
reference_nowanchor — the max observation timestamp (excludes future-dated/commissioning columns). This is the dataset's "now"; relative time windows ("today", "last week") resolve against it, never the wall clock. - Tool advisories — an auto-derived "be careful" list the query/aggregation
tools must honour: null-vs-zero columns (e.g.
performance_ratiois empty at night — filter, don't zero-fill), lifecycle nulls (resolved_atonly on resolved alerts), silent-downtime inverters, enum typos, PK/FK/reconciliation, and outliers.
Beyond validation (date coverage, completeness, value domains, range checks, foreign-key integrity), the report carries a tool-design analysis block that maps question vocabulary onto the data — the bridge from "is the data valid" to "is it answerable":
- Data dictionary — every column in every file with a plain-language description, type, unit, and PK/FK role.
- Entity resolver index — canonical
plant_id ↔ name ↔ region ↔ location(note:regionis a compass label, not the state) plus inverter counts. - Vocabulary coverage map — each demo filter word (
offline,hotspot,in progress…) resolved to its exact stored string, or flagged✗ none. - Current-state snapshot — per-plant Type-A ground truth at
reference_now. - Cross-table status reconciliation — where one status field disagrees with another (offline inverters with no open alert), so tools combine signals.
- Measure semantics — the correct reducer per metric (
daily_yield= daily max,total_yield= diff,performance_ratio= mean ex-null). - Derivable vs non-derivable — what to compute vs where to refuse (the revenue-loss question has no basis in the data).
If a tool returns surprising results, re-run the profiler first: an empty result is often stale/invalid data or a vocabulary mismatch, not a tool bug.
python -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt
pip install -r requirements-dev.txt
# Configure a provider (one of):
export OPENAI_API_KEY=sk-... # uses the pre-seeded OpenAI provider, or
# run Ollama locally (pre-seeded at http://localhost:11434)
./run-web-server.shProvider config lives in config/common.example.json; copy to
config/common.local.json (gitignored) for local overrides. Tool-calling needs a
tool-capable model — the default llama3 is weak at tools; qwen3.5 / gpt-oss work.
The CLI talks to the FastAPI server. Start the server first:
./run-web-server.shOpen an interactive chat session:
.venv/bin/python -m app.cli
./run-cli.shInside the prompt:
- type a question and press Enter
- type
/bind_all <question>to force bind-all tool mode for one request - type
/gated <question>to force gated tool mode for one request - type
exitorquitto leave
Send a single prompt and print the answer as one terminal command:
./run-cli.sh --prompt "Which plants are offline today?"
./run-cli.sh "List inverter issues for today."In one-shot mode:
- the assistant answer is printed to
stdout - session information and usage diagnostics are printed to
stderr - a new chat session is created automatically unless
--session-idis provided
Create a new session and ask one question:
./run-cli.sh --prompt "Which plants are offline today?"Continue an existing session:
./run-cli.sh --session-id <session-id> "Continue from the previous answer."Use bind-all mode for one request:
./run-cli.sh --gating-mode bind_all --prompt "Summarize current alerts and anomalies."Set a title for a newly created session:
./run-cli.sh --title "Morning checks" --prompt "Give me today's plant status."Target a different server URL:
./run-cli.sh --server http://127.0.0.1:9010 --prompt "Which plants are offline?"Suppress diagnostics in one-shot mode:
./run-cli.sh --no-stats --prompt "Which plants are offline today?"./run-cli.sh forwards arguments to python -m app.cli.
PROMPT— positional prompt text for one-shot mode--prompt TEXT— prompt text for one-shot mode--session-id ID— reuse an existing chat session instead of creating a new one--title TEXT— title for a newly created session; default isCLI chat--gating-mode {gated,bind_all}— default tool binding mode for the request; default isgated--server URL— override the configured server base URL--no-stats— suppress trace and usage footer output in one-shot mode
Do not pass both a positional prompt and --prompt in the same command.
python -m pytest -q
python -m compileall app scripts
rg -n "</content>|</invoke>" README.md docspytest is the offline/unit-style suite. The staged behavioural replay program
is separate and runs through the live server/chat API.
requirements.txt carries runtime dependencies; requirements-dev.txt adds the local test
dependency. The rg command is the lightweight docs-hygiene check for stray copy artifacts in
durable documentation.
For a full local gate pass with reviewable artifacts, activate the local venv
and redirect both stdout and stderr into files:
source .venv/bin/activate
python -m pytest -q > pytest-gint1.txt 2>&1
python -m compileall app scripts > compileall-gint1.txt 2>&1
rg -n "</content>|</invoke>" README.md docs > docs-hygiene-gint1.txt 2>&1
./run-case-replay.sh --gate gate1 > gate1-replay.txt 2>&1
./run-case-replay.sh --gate gate2 > gate2-replay.txt 2>&1
./run-case-replay.sh --transcript MT-D3-DISPUTE > mt-d3-dispute.txt 2>&1This sequence maps to the current gate plan:
G-INT1—pytest,compileall, and docs hygiene.G-INT2— replaygate1, replaygate2, and the multi-turn dispute transcript.
If you want to watch the replay live and still keep a file artifact, use:
./run-case-replay.sh --gate gate1 2>&1 | tee gate1-replay.txtThe replay harness has two behavioural gates:
gate1— the initial-task 15-question set fromdocs/solar_interview_task.md(D1–D6,A1–A3,B1–B3,C1–C3)gate2— the remaining 36 canonical behavioural cases fromdocs/test-plan.mdplus 4 multi-turn transcript fixtures
Use the shell wrapper:
./run-case-replay.sh --gate gate1
./run-case-replay.sh --gate gate2The wrapper covers server startup for you:
- if the FastAPI server is already running on
127.0.0.1:9006, it reuses it - otherwise it starts
app.server, waits for readiness, then launches replay
So the normal operator path is just:
./run-case-replay.sh --gate gate1
./run-case-replay.sh --gate gate2You do not need to start ./run-web-server.sh first unless you specifically
want the server running separately.
./run-case-replay.sh forwards to:
.venv/bin/python -m app.case_replay --server http://127.0.0.1:9006 ...The replay CLI supports:
--gate gate1--gate gate2--gate gate2verifyfor the focusedG2-FIX-11fresh-port verification subset--case CASE_IDto run one or more specific single-turn cases--transcript TRANSCRIPT_IDto run multi-turn transcript fixtures--gating-mode gated|bind_all--server URL--request-timeout SECONDS
Examples:
./run-case-replay.sh --gate gate1
./run-case-replay.sh --gate gate2 --gating-mode bind_all
./run-case-replay.sh --gate gate2verify --request-timeout 45
./run-case-replay.sh --case D3 --case X4
./run-case-replay.sh --transcript MT-D3-DISPUTEThe wrapper also honors:
AI_ASSISTANT_STARTUP_TIMEOUT_SECSto allow longer clean-port startup waitsAI_ASSISTANT_READINESS_TIMEOUT_SECSto control each readiness probe timeout
If wrapper startup fails, it now prints the tail of the captured uvicorn log to
help distinguish slow startup from an actual server error.
Replay output is printed to the terminal as plain text. For each case, the harness prints:
- case id and
PASS/FAIL - the question text
- detected intent types
- tool calls used
- individual check results such as intent, tools, text, numbers, trace, and stop reason
Typical shape:
D3: PASS
question: Which inverters have open hotspot anomalies caused by soiling?
intent: ['C']
tools: ['anomalies']
- intent_types: ok (...)
- tool_calls: ok (...)
- answer_text: ok (...)
- answer_numbers: ok (...)
At the end of the run, the harness also prints a compact summary table:
Summary
| Status | Kind | Case | Intent | Tools | Failed Checks |
|--------|------|------|--------|------------|----------------|
| ✓ PASS | case | D3 | C | anomalies | - |
| ✗ FAIL | case | X4 | B | perform... | answer_numbers |
Totals: 1 passed, 1 failed, 2 total
The process exits non-zero if any selected case fails, so it is suitable for shell scripting and CI.
If you want a file artifact, redirect or tee the output:
./run-case-replay.sh --gate gate1 | tee gate1-replay.txt
./run-case-replay.sh --gate gate2 | tee gate2-replay.txtOr keep stderr and stdout together:
./run-case-replay.sh --gate gate1 > gate1-replay.txt 2>&1
./run-case-replay.sh --gate gate2 > gate2-replay.txt 2>&1This is useful when you want a human or an AI agent to review the exact replay report after the run.
Recommended order:
- Run Gate 1 and confirm all 15 initial-task questions pass.
- If Gate 1 passes, run Gate 2 for the remaining 36 canonical behavioural cases plus the multi-turn transcript fixtures.
- If needed, run specific transcripts for multi-turn validation.
- Review any
FAILlines and their failed checks.
Example:
./run-case-replay.sh --gate gate1 > gate1-replay.txt 2>&1
./run-case-replay.sh --gate gate2 > gate2-replay.txt 2>&1
./run-case-replay.sh --transcript MT-D3-DISPUTE > mt-d3-dispute.txt 2>&1After that:
- if everything passes, the staged behavioural replay is green
- if anything fails, inspect the failed case block in the saved output file
- use the final summary table to spot failing case ids quickly
- hand those files to a reviewer, or ask an AI agent to summarize failures and suggest fixes