feat: understand questions in the library's words, Learning Paths and AI reliability

docs/API.md

@@ -131,3 +131,18 @@ Every endpoint maps to an FR in `PRD.md`; there is no endpoint without one.
 - **Auth:** public, `Cache-Control: no-store`
 - **Response:** `200 { "status": "ok", "checks": { "database": { "ok", "ms" }, "ai": { "configured", "providers" }, "trials": { "enabled" }, "demoStudio": { "enabled" } }, "version": "<commit>" }`; `503` with `"status": "degraded"` when the database is unreachable (3 s timeout). No secrets or config values
 
+
+## Phase 18 additions
+
+### `POST /api/paths`
+- **Auth:** public, shares Ask's rate limit (`429` + `Retry-After`) · `maxDuration` 45 s
+- **Request:** `{ "topic": string(3–200) }`
+- **Response:** `{ "status": "built", "path": { id, topic, title, steps[{ n, stepTitle, why, segmentId, lectureId, title, speaker, startS, endS, … }], reel: { url, durationS, clips } } }` or `{ "status": "not_found" }` when the library can't support at least 3 on-topic steps
+- **Errors:** `400` · `429` · `502` AI failed · `503` no AI key
+- Courses are stored (`learning_paths`) and open at `/p/[id]`.
+
+### `GET /api/health?deep=1`
+Adds `checks.aiLive.models[{ model, ok, ms, error? }]`, one tiny real call per model. It spends tokens, so it shares Ask's rate limit.
+
+### `POST /api/ask` (behaviour)
+Retrieval now rewrites a question into the library's vocabulary when it is in another script or keyword retrieval is thin. The stream gains an `{ "type": "understanding" }` event, and `retrieved` carries `expanded: boolean`.

docs/IMPLEMENTATION_PLAN.md

@@ -91,3 +91,5 @@ The core loop (Phases 01–09) is built and verified on real Cloudinary + Neon.
 | 17 | Launch | Real library, eval, e2e on production, README, demo, submission | |
 
 **PR policy (team lead, Oct 1):** a PR is opened only once a feature branch carries **at least 20 changed files**, so related phases ship together (`GIT_WORKFLOW.md`).
+
+| 18 | Query understanding, Learning Paths, AI reliability | Rewrite questions into the library's words (and translate them), honest near-miss answers, `/learn` micro-courses as one Cloudinary reel, circuit breaker + deep health | ✅ done (Oct 3) |

docs/phases/PHASE-18-query-understanding-learning-paths.md

@@ -0,0 +1,36 @@
+# Phase 18 — Query Understanding, Learning Paths & AI Reliability
+
+**Status: DONE (Oct 3)** · branch `feature/query-understanding-learning-paths`
+
+## Why this phase exists
+A full end-to-end audit of the live site (Oct 3) found real gaps:
+
+| Finding | Evidence |
+|---|---|
+| Ask said "not covered" for questions the library *does* teach | "What is the bias-variance trade-off?" returned `not_found` against a lecture literally titled *The Bias–Variance Trade-off*: keyword search never saw the lecturer's words |
+| Hindi questions were dropped | The translation step failed silently when a model was rate-limited |
+| A dead AI model cost every request its timeout | Gemini 429 (quota) was retried on every Ask |
+| Hindi subtitles missing on 5 of 6 sessions | Only the newest upload had `{id}.hi-IN.transcript` (backfilled with `explicit`) |
+| Find missed sessions by topic | "regularization" found no moment; the session about it was invisible |
+
+## What shipped
+1. **Question understanding** (`query-understanding.ts`, `query-plan.ts`): when a question is in another script, or keyword retrieval finds fewer than 3 moments, one AI call rewrites it into the library's own vocabulary (session and chapter titles are given as context). Keyword hits keep priority; the rewrite only *adds* moments. The UI shows a real step: "Rephrasing your question in the words lecturers use…".
+2. **Honest near-miss answers:** the Ask prompt now says so plainly when excerpts explain related ideas but never use the asked term, then explains those ideas with citations (instead of refusing).
+3. **Answer language:** a question in Devanagari/Tamil/Telugu/… is answered in that language (the system prompt names it; English excerpts otherwise pull the reply into English).
+4. **Learning Paths** (`/learn`, `/p/[id]`): a topic becomes an ordered 3–5 step micro-course across the library, edited into one Cloudinary reel; every step is a real, validated moment with a "why this comes next". Stored and shareable with a Cloudinary share card. "Go deeper" under every answer opens one.
+5. **Sessions about this:** Find also searches session titles, speakers and chapter titles.
+6. **AI reliability** (`ai-breaker.ts`, `ai.ts`): a per-model circuit breaker (quota/auth/overload → skip that model for 20–60 s), an overall deadline for the whole chain (so a slow provider can't exceed `maxDuration`), and `GET /api/health?deep=1`, which makes one tiny real call per model so "is AI working?" has an answer.
+
+## Learning Path grounding (same discipline as Ask)
+`validatePath()` drops steps pointing at moments the model wasn't given, repeats and untitled steps; fewer than 3 real steps means "no course" instead of padding.
+
+## Verified (real data)
+- "What is the bias-variance trade-off?" → answered, honestly says the term isn't used and explains underfitting/overfitting with 6 citations (was `not_found`)
+- Hindi questions → Hindi answers citing English lectures, 5-clip reel
+- "Optimizers for training neural networks" → 4-step course from 2 sessions, 65 s reel that renders (HTTP 200)
+- Off-topic ("Who won the IPL?", "capital of France?") → still `not_found`
+- Quota exhaustion (a real Gemini 429 during testing) → breaker skipped the dead model on later requests; Ask kept answering on the next model
+
+## Known limits
+- Understanding adds one AI call (about 1–2 s) only when needed, never for well-matched questions.
+- Free-tier Gemini quota is the main production risk; a `GROQ_API_KEY` adds a second provider (already supported by the chain).

migrations/006_learning_paths.sql

@@ -0,0 +1,9 @@
+-- Learning Paths (/learn, shared at /p/[id]): a topic turned into an ordered micro-course of moments from
+-- across the library, stitched into one Cloudinary video. Stored exactly as generated, like answers.
+CREATE TABLE learning_paths (
+    id         TEXT PRIMARY KEY CHECK (id ~ '^[A-Za-z0-9_-]{10}$'),
+    topic      TEXT NOT NULL CHECK (length(topic) <= 200),
+    result     JSONB NOT NULL,
+    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+CREATE INDEX idx_learning_paths_recent ON learning_paths (created_at DESC);

src/app/api/ask/route.ts

@@ -11,10 +11,14 @@ import { getLecture } from "@/lib/lectures";
 import { log } from "@/lib/log";
 import { momentUrl, reelUrl } from "@/lib/media";
 import { clientIp, takeAskToken } from "@/lib/rate-limit";
+import { needsTranslation } from "@/lib/language";
+import { mergeHits, needsUnderstanding } from "@/lib/query-plan";
+import { understandQuestion } from "@/lib/query-understanding";
 import { retrieveForQuestion, sessionOverview, type Hit } from "@/lib/search";
-import { searchableQuestion } from "@/lib/translate";
 
 export const maxDuration = 30;
+// Leave headroom under maxDuration for validation, storing the answer and streaming the result.
+const ASK_DEADLINE_MS = 27_000;
 
 const Body = z.object({
   question: z.string().trim().min(3).max(300),
@@ -35,9 +39,23 @@ const reelFor = (hits: Hit[]) =>
 
 // Progress the client can show while it waits: real retrieval facts, then the model step.
 type Progress =
-  | { type: "retrieved"; moments: number; sessions: string[] }
+  | { type: "understanding" }
+  | { type: "retrieved"; moments: number; sessions: string[]; expanded: boolean }
   | { type: "writing" };
 
+// Keyword retrieval on the learner's own words, then — when the question is in another script or the
+// keywords found little — the same retrieval on the question rewritten into the library's vocabulary
+// (query-plan.ts). Keyword hits keep priority; the rewrite only adds moments.
+async function retrieve(question: string, lectureId: string | null, progress: (p: Progress) => void) {
+  const keyword = needsTranslation(question) ? [] : await retrieveForQuestion(question, { lectureId });
+  if (!needsUnderstanding(question, keyword.length)) return { hits: keyword, expanded: false };
+  progress({ type: "understanding" });
+  const understood = await understandQuestion(question);
+  if (!understood) return { hits: keyword, expanded: false };
+  const extra = await retrieveForQuestion(understood.search, { lectureId });
+  return { hits: mergeHits(keyword, extra), expanded: extra.length > 0 };
+}
+
 // One Ask, start to finish. `progress` lets the streaming response report each step as it happens;
 // the JSON response ignores it. Same retrieval, validation and fallback either way.
 async function ask(question: string, lectureId: string | null, progress: (p: Progress) => void) {
@@ -47,11 +65,11 @@ async function ask(question: string, lectureId: string | null, progress: (p: Pro
   const trial = lectureId ? Boolean((await getLecture(lectureId))?.trialExpiresAt) : false;
   const logged = (...args: Parameters<typeof logAsk>) => (trial ? Promise.resolve() : logAsk(...args));
   // A question in Hindi (or any non-Latin script) is searched in English; the answer stays in its language.
-  const matched = await retrieveForQuestion(await searchableQuestion(question), { lectureId });
+  const { hits: matched, expanded } = await retrieve(question, lectureId, progress);
   // In one session, a question with no keyword match ("What is this about?") is answered from moments across
   // the whole session; the model still refuses if they don't answer it.
   const hits = matched.length === 0 && lectureId ? await sessionOverview(lectureId) : matched;
-  progress({ type: "retrieved", moments: hits.length, sessions: [...new Set(hits.map((h) => h.title))].slice(0, 5) });
+  progress({ type: "retrieved", moments: hits.length, sessions: [...new Set(hits.map((h) => h.title))].slice(0, 5), expanded });
 
   if (hits.length === 0) {
     // Nothing in the library matches — no reason to call the model.
@@ -62,13 +80,15 @@ async function ask(question: string, lectureId: string | null, progress: (p: Pro
 
   try {
     progress({ type: "writing" });
-    const raw = await askGrounded(question, hits);
+    // Whatever question understanding used comes out of the answer's budget, so the request never outlives maxDuration.
+    const raw = await askGrounded(question, hits, { budgetMs: ASK_DEADLINE_MS - (Date.now() - started) });
     const result = validateAnswer(raw, hits);
     log("ask.done", {
       status: result.status,
       retrieved: hits.length,
       cited: result.citations.length,
       dropped: result.dropped,
+      expanded,
       ms: Date.now() - started,
     });
     after(() => logged(question, result.status, result.citations.map((c) => c.lectureId)));

src/app/api/health/route.ts

@@ -1,16 +1,31 @@
 import { NextResponse } from "next/server";
 
+import { AiUnconfigured, probeModels, type ProbeResult } from "@/lib/ai";
 import { query } from "@/lib/db";
 import { env } from "@/lib/env";
 import { log } from "@/lib/log";
+import { clientIp, takeAskToken } from "@/lib/rate-limit";
 
 export const dynamic = "force-dynamic";
 
 const DB_TIMEOUT_MS = 3000;
 
+// `?deep=1` also makes one tiny call to every AI model and reports which ones actually answer — "configured"
+// doesn't mean "working" (an exhausted quota looks configured). It spends tokens, so it shares Ask's rate limit.
+async function deepAi(request: Request): Promise<{ models: ProbeResult[] } | { error: string }> {
+  const token = await takeAskToken(clientIp(request));
+  if (!token.ok) return { error: "rate_limited" };
+  try {
+    return { models: await probeModels() };
+  } catch (error) {
+    return { error: error instanceof AiUnconfigured ? "unconfigured" : "probe_failed" };
+  }
+}
+
 // Health for uptime monitors and the team: 200 when the app can serve, 503 when the database is unreachable.
 // Reports only booleans, timings and the deployed commit; never a secret or a config value.
-export async function GET() {
+export async function GET(request: Request) {
+  const deep = new URL(request.url).searchParams.get("deep") === "1";
   const started = Date.now();
   const database = await Promise.race([
     query("SELECT 1").then(() => true),
@@ -30,6 +45,7 @@ export async function GET() {
       },
       trials: { enabled: TRIALS_PER_DAY > 0 },
       demoStudio: { enabled: STUDIO_DEMO === "on" },
+      ...(deep ? { aiLive: await deepAi(request) } : {}),
     },
     version: process.env.VERCEL_GIT_COMMIT_SHA?.slice(0, 7) ?? "local",
   };

src/app/api/paths/route.ts

@@ -0,0 +1,36 @@
+import { NextResponse } from "next/server";
+import { z } from "zod";
+
+import { AiUnconfigured } from "@/lib/ai";
+import { apiError, parseJson } from "@/lib/http";
+import { log } from "@/lib/log";
+import { buildPath } from "@/lib/paths";
+import { clientIp, takeAskToken } from "@/lib/rate-limit";
+
+export const maxDuration = 45;
+
+const Body = z.object({ topic: z.string().trim().min(3).max(200) });
+
+// Learning Paths: a topic in, an ordered micro-course of moments from across the library out, stitched into
+// one Cloudinary reel. Costs one AI call (plus question understanding), so it shares Ask's rate limit.
+export async function POST(request: Request) {
+  const parsed = await parseJson(request, Body);
+  if ("response" in parsed) return parsed.response;
+
+  const token = await takeAskToken(clientIp(request));
+  if (!token.ok) {
+    const response = apiError(429, "rate_limited", "You've asked a lot — try again in a few minutes.");
+    response.headers.set("Retry-After", String(token.retryAfterS));
+    return response;
+  }
+
+  try {
+    const path = await buildPath(parsed.data.topic);
+    if (!path) return NextResponse.json({ status: "not_found" });
+    return NextResponse.json({ status: "built", path });
+  } catch (error) {
+    log("path.failed", { error: error instanceof Error ? error.message.slice(0, 160) : String(error) });
+    if (error instanceof AiUnconfigured) return apiError(503, "ai_unconfigured", "Learning Paths need an AI key.");
+    return apiError(502, "ai_failed", "Couldn't build a course right now — try again in a moment.");
+  }
+}

src/app/layout.tsx

@@ -54,6 +54,9 @@ export default function RootLayout({ children }: { children: React.ReactNode })
                 <Link href="/#library" className="rounded-lg px-3 py-2 text-muted hover:bg-surface hover:text-fg">
                   Library
                 </Link>
+                <Link href="/learn" className="rounded-lg px-3 py-2 text-muted hover:bg-surface hover:text-fg">
+                  Learn
+                </Link>
                 <Link href="/saved" className="rounded-lg px-3 py-2 text-muted hover:bg-surface hover:text-fg">
                   Saved
                 </Link>

src/app/learn/page.tsx

@@ -0,0 +1,26 @@
+import type { Metadata } from "next";
+
+import { LearnForm } from "@/components/LearnForm";
+
+export const metadata: Metadata = {
+  title: "Learn a topic — Pravaha",
+  description: "Turn any topic into a short course built from moments across your lecture library, edited into one video.",
+};
+
+type Props = { searchParams: Promise<{ topic?: string }> };
+
+export default async function LearnPage({ searchParams }: Props) {
+  const topic = ((await searchParams).topic ?? "").trim().slice(0, 200);
+  return (
+    <div className="mx-auto mt-6 max-w-3xl">
+      <h1 className="text-3xl font-semibold tracking-tight sm:text-4xl">
+        Learn a topic, <span className="text-accent">taught by your own lecturers.</span>
+      </h1>
+      <p className="mt-3 max-w-2xl text-muted">
+        Pravaha picks the moments across every lecture that teach it, orders them from basics to advanced, and Cloudinary
+        edits them into one short course video. Every step is a real moment you can open.
+      </p>
+      <LearnForm initialTopic={topic} />
+    </div>
+  );
+}

src/app/p/[id]/page.tsx

@@ -0,0 +1,44 @@
+import type { Metadata } from "next";
+import Link from "next/link";
+import { notFound } from "next/navigation";
+
+import { LearningPathView } from "@/components/LearningPathView";
+import { SHARE_CARD, shareCardUrl } from "@/lib/media";
+import { getPath } from "@/lib/paths";
+
+type Props = { params: Promise<{ id: string }> };
+
+// A shared Learning Path: stored as generated (src/lib/paths.ts), so the link never re-runs the model.
+export async function generateMetadata({ params }: Props): Promise<Metadata> {
+  const path = await getPath((await params).id);
+  if (!path) return { title: "Course not found — Pravaha" };
+  const first = path.steps[0]!;
+  const sessions = new Set(path.steps.map((s) => s.lectureId)).size;
+  const description = `${path.steps.length} steps from ${sessions} lecture${sessions === 1 ? "" : "s"}: ${path.steps.map((s) => s.stepTitle).join(" → ")}`;
+  const image = shareCardUrl(first.publicId, first.startS, { title: path.title, subtitle: `A Pravaha course · ${path.steps.length} steps` });
+  return {
+    title: `${path.title} — Pravaha`,
+    description,
+    robots: { index: false, follow: true },
+    openGraph: { type: "article", siteName: "Pravaha", title: path.title, description, url: `/p/${path.id}`, images: [{ url: image, ...SHARE_CARD, alt: path.title }] },
+    twitter: { card: "summary_large_image", title: path.title, description, images: [image] },
+  };
+}
+
+export default async function SharedPathPage({ params }: Props) {
+  const path = await getPath((await params).id);
+  if (!path) notFound();
+  return (
+    <div className="mx-auto mt-6 max-w-3xl">
+      <p className="text-sm text-muted">
+        Course built from the library for “{path.topic}”
+      </p>
+      <LearningPathView path={path} />
+      <p className="mt-8 text-sm text-muted">
+        <Link href="/learn" className="font-medium text-accent hover:underline">
+          Build a course on another topic →
+        </Link>
+      </p>
+    </div>
+  );
+}

src/app/search/page.tsx

@@ -1,10 +1,12 @@
 import type { Metadata } from "next";
 
 import { AskAnswer } from "@/components/AskAnswer";
+import { LibraryCard } from "@/components/LibraryCard";
 import { MomentButton } from "@/components/MomentButton";
 import { ResultCard } from "@/components/ResultCard";
 import { SaveButton } from "@/components/SaveButton";
 import { SearchBar } from "@/components/SearchBar";
+import { findSessions } from "@/lib/lectures";
 import { findSegments } from "@/lib/search";
 
 type Props = { searchParams: Promise<{ q?: string }> };
@@ -16,21 +18,39 @@ export async function generateMetadata({ searchParams }: Props): Promise<Metadat
 
 export default async function SearchPage({ searchParams }: Props) {
   const q = ((await searchParams).q ?? "").trim().slice(0, 300);
-  const { hits, exact } = q.length >= 2 ? await findSegments(q) : { hits: [], exact: true };
+  const [{ hits, exact }, sessions] =
+    q.length >= 2 ? await Promise.all([findSegments(q), findSessions(q, 3)]) : [{ hits: [], exact: true }, []];
 
   return (
     <div className="mt-4">
       <SearchBar defaultValue={q} />
       {q.length >= 3 && <AskAnswer key={q} question={q} />}
 
+      {sessions.length > 0 && (
+        <section aria-labelledby="sessions-heading" className="mt-8">
+          <h2 id="sessions-heading" className="text-sm font-semibold tracking-wide text-muted uppercase">
+            Sessions about this
+          </h2>
+          <ul className="mt-3 grid gap-5 sm:grid-cols-2 lg:grid-cols-3">
+            {sessions.map((lecture, i) => (
+              <li key={lecture.id}>
+                <LibraryCard lecture={lecture} index={i} />
+              </li>
+            ))}
+          </ul>
+        </section>
+      )}
+
       <section aria-labelledby="found-heading" className="mt-8">
         <h2 id="found-heading" className="text-sm font-semibold tracking-wide text-muted uppercase">
           Moments found
         </h2>
         {q.length < 2 ? (
           <p className="mt-3 text-muted">Type at least two characters.</p>
         ) : hits.length === 0 ? (
-          <p className="mt-3 text-muted">Nothing said matches “{q}” — try different words.</p>
+          <p className="mt-3 text-muted">
+            {sessions.length ? "No moment says these exact words — the sessions above are about it, and Ask explains it in their words." : `Nothing said matches “${q}” — try different words.`}
+          </p>
         ) : (
           <>
             {!exact && <p className="mt-3 text-sm text-muted">No single moment has every word, so these are the closest.</p>}