Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/2.0.0/),

## [Unreleased]

### Security

- Terminal output, `report --markdown` and `LEADERBOARD.md`: control characters, and the bidi controls that show text reversed, now show as spaces in every value that comes from an agent, a simulated caller, a platform's call, a results file, a transcript's file name or a domain that `init` drafted, and a command sandtable suggests quotes such a file name so the shell still reads it right. Before, such text could write escape sequences to the terminal of whoever ran `run`, `serve`, `compare`, `regrade`, `leaderboard`, `review`, `judge` or `calibrate`, and set the window title, clear the screen or hide the output that follows. Upgrade if you run `serve` behind a tunnel or read results files you didn't write ([GHSA-qrx5-3qq7-7w59](https://github.com/sandtable-ai/sandtable/security/advisories/GHSA-qrx5-3qq7-7w59)).

## [1.5.0] - 2026-11-09

A minor release with no breaking changes, and a security fix in the HTML reports: upgrade if you open reports built from domains or results files you didn't write. `sandtable judge` and `run --judge` score a run on judged criteria with a model, apart from pass^k, which they never change, and `sandtable calibrate` measures a judge against human labels. Clock dates reach the policy, param defaults, fault messages and `reply_includes`, and `report --markdown` writes a summary for a CI job or a pull request comment.
Expand Down
98 changes: 51 additions & 47 deletions src/cli.ts

Large diffs are not rendered by default.

12 changes: 7 additions & 5 deletions src/judge/calibrate.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import YAML from "yaml";
import { z } from "zod";
import { printableCopy } from "../report/printable.ts";
import { formatPercent } from "../report/terminal.ts";
import type { TrialResult } from "../run/episode.ts";
import type { SuiteResult, TaskSummary } from "../run/suite.ts";
Expand Down Expand Up @@ -220,8 +221,6 @@ const WHY_NONE: Record<LabelWithNoVerdict["why"], string> = {

const rateCell = (r: CalibrationRate) => `${r.rate === null ? "n/a" : formatPercent(r.rate)} (n=${r.n})`;
const counted = (n: number, one: string, many: string) => `${n} ${n === 1 ? one : many}`;
/** A quote and a reason are a model's output, and a quote comes from the transcript of the agent under test: one line of text, nothing a terminal would act on. */
const oneLine = (s: string) => s.replace(/[\s\p{Cc}]+/gu, " ").trim();

function rateTable(m: ModelCalibration): string[] {
const outside = (c: CalibrationCounts) => [...(c.notApplicable ? [`${c.notApplicable} not applicable`] : []), ...(c.unlabelled ? [`${c.unlabelled} with no label`] : [])].join(", ");
Expand All @@ -233,13 +232,16 @@ function rateTable(m: ModelCalibration): string[] {
function pairLines(p: LabelledVerdict): string[] {
return [
` ${p.task} trial ${p.trial} · ${p.criterion}: you said ${p.label}, the judge said ${p.verdict}`,
...(p.quote !== undefined ? [` quote: "${oneLine(p.quote)}"${p.quoteFound === false ? " (not in the transcript)" : ""}`] : []),
` reason: ${oneLine(p.reason)}`,
...(p.quote !== undefined ? [` quote: "${p.quote}"${p.quoteFound === false ? " (not in the transcript)" : ""}`] : []),
` reason: ${p.reason}`,
];
}

/** The terminal's account of a calibration: the rates with the pairs behind each, what to read, and what stayed outside. */
export function renderCalibration(c: Calibration): string {
export function renderCalibration(calibration: Calibration): string {
// A quote and a reason are a model's output, a quote comes from the transcript of the agent under test, and the ids and
// models from a results file: each is shown as one line of text, nothing a terminal would act on.
const c = printableCopy(calibration);
const lines = [" How often the judge agrees with your labels, over pairs with both a label and a pass or fail verdict.", " TPR: of the pairs you labelled pass, the share the judge gave pass. TNR: the same for fail. No model was called, and this sets no gate."];
if (c.models.length > 1) lines.push("", ` This file's trials hold ${c.models.length} judge models, so each is reported apart: mixing them would measure neither. To compare models, keep one results file per model (judge -m <other> -o <dir>) and calibrate each.`);
if (c.models.length === 0) lines.push("", " No trial holds a verdict, so there is nothing to measure yet: run npx sandtable-ai judge again.");
Expand Down
6 changes: 4 additions & 2 deletions src/report/markdown.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import type { JudgedCounts } from "../judge/summary.ts";
import { comparisonOf, type Comparison, type TaskChange } from "../run/compare.ts";
import { skippedTrials, tagsBelowFloor, type SuiteResult, type TaskSummary } from "../run/suite.ts";
import { printable } from "./printable.ts";
import { formatPercent } from "./terminal.ts";

/**
Expand Down Expand Up @@ -34,11 +35,12 @@ const clip = (s: string, width = CELL) => (s.length <= width ? s : `${s.slice(0,
/**
* Text a run holds, on one line, so that it renders as itself: it can't break a table or open a tag or a link, and in a
* comment it can't notify anyone, link an issue or a pull request, or set math. A URL stays plain text. GitHub can still
* link a commit hash or a CVE id.
* link a commit hash or a CVE id. Control characters, a newline and a tab among them, and bidi controls are spaces, so
* nothing in it acts on a terminal that prints it or shows reversed.
*/
function escapeMarkdown(s: unknown): string {
// As entities, not backslashes: a renderer that ignores a backslash there would still see a tag.
const entities = clip(String(s).replace(/\s*[\r\n]+\s*/g, " "))
const entities = clip(printable(s))
.replace(/&/g, "&amp;")
.replace(/</g, "&lt;")
.replace(/>/g, "&gt;");
Expand Down
33 changes: 33 additions & 0 deletions src/report/printable.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
/*
* Text that an agent, a simulated caller, a platform's call or a results file holds, made fit to print: a terminal shows
* it as text, so it can't start an escape sequence, ring the bell, move the cursor or set the window's title, and a
* renderer can't show it reversed. Only what came from outside goes through here: sandtable's own colours are added
* after, so they stay.
*/

/** C0 controls (a newline and a tab among them), DEL, C1 controls, and the bidi embedding, override and isolate controls. */
const CONTROL = "\\p{Cc}\\u202a-\\u202e\\u2066-\\u2069";
const ONE_LINE = new RegExp(`[\\s${CONTROL}]+`, "gu");
const EACH = new RegExp(`[${CONTROL}]`, "gu");
/** One character a terminal acts on rather than shows. */
export const CONTROL_CHAR = new RegExp(`[${CONTROL}]`, "u");
const LINES = new RegExp(`(?:(?![\\n\\t])[${CONTROL}])+`, "gu");

/** On one line, for a value inside a line or a Markdown cell: each run of whitespace and control characters is one space, with none at either end. */
export function printable(value: unknown): string {
return String(value).replace(ONE_LINE, " ").trim();
}

/** Over several lines, for a message that breaks its own lines, such as an error: newlines and tabs stay, and each other run of control characters is one space. */
export function printableLines(value: string): string {
return value.replace(/\r\n/g, "\n").replace(LINES, " ");
}

/** A copy of JSON data, such as a run, in which every string is `printable` and every key has each control character as a space, so keys that differ stay apart. */
export function printableCopy<T>(data: T): T {
if (typeof data === "string") return printable(data) as T;
if (Array.isArray(data)) return data.map(printableCopy) as T;
// fromEntries, not assignment, so a key such as __proto__ stays a key.
if (data !== null && typeof data === "object") return Object.fromEntries(Object.entries(data).map(([k, v]) => [k.replace(EACH, " "), printableCopy(v)])) as T;
return data;
}
26 changes: 16 additions & 10 deletions src/report/terminal.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ import { comparisonOf, type Comparison, type TaskChange } from "../run/compare.t
import { gradedOutcomes, skippedTrials, tagsBelowFloor, trialStatus, type SoftScore, type SuiteResult, type TaskSummary } from "../run/suite.ts";
import { costOf, totalTokens, type UsageByModel } from "../run/usage.ts";
import type { JudgedCounts } from "../judge/summary.ts";
import { printableCopy } from "./printable.ts";

const useColor = process.stdout.isTTY && !process.env.NO_COLOR;
const paint = (code: number) => (s: string) => (useColor ? `\x1b[${code}m${s}\x1b[0m` : s);
Expand Down Expand Up @@ -94,8 +95,8 @@ function taskNote(t: TaskSummary): string {
* what the judge's calls used and cost. Empty for a run that was never judged.
*/
export function renderJudged(r: SuiteResult): string[] {
const j = r.judged;
if (!j) return [];
if (!r.judged) return [];
const j = printableCopy(r.judged);
const score = (c: JudgedCounts) => (c.score === null ? "n/a" : formatPercent(c.score));
const tally = (c: JudgedCounts) => [`${c.pass} pass`, `${c.fail} fail`, ...(c.notApplicable ? [`${c.notApplicable} not applicable`] : [])].join(", ");
const lines = [bold(" Judged score") + dim(" (a model's judgement: it never changes pass/fail, pass^k, a gate or an exit code)")];
Expand All @@ -120,7 +121,11 @@ export function renderJudged(r: SuiteResult): string[] {
* file it came from (else the block names the baseline's run id). `withoutJudgedChanges` leaves out how the judged
* score changed, for a summary printed before the run is judged.
*/
export function renderTerminal(r: SuiteResult, opts: { baseline?: string; withoutJudgedChanges?: boolean } = {}): string {
export function renderTerminal(result: SuiteResult, opts: { baseline?: string; withoutJudgedChanges?: boolean } = {}): string {
// An agent, a caller, a platform or a results file wrote what a run holds: the summary shows a copy that is all text,
// each value on one line, before anything is cut to fit. The comparison is worked out on the run as it is.
const r = printableCopy(result);
const comparison = printableCopy(comparisonOf(result));
const lines: string[] = [];
lines.push(bold(`sandtable · ${r.domain}`) + dim(` · agent ${r.agent} · user ${r.user} · k=${r.k}`));
if (r.pendingReview?.length) {
Expand Down Expand Up @@ -270,19 +275,20 @@ export function renderTerminal(r: SuiteResult, opts: { baseline?: string; withou
}
}

const comparison = comparisonOf(r);
if (comparison) lines.push("", ...renderComparison(comparison, opts.baseline));
if (comparison && !opts.withoutJudgedChanges) lines.push(...spaced(renderJudgedChanges(comparison)));
return lines.join("\n");
}

/**
* What `sandtable compare` prints: the newer run on two lines, then the block a `run --baseline` summary ends with.
* `r` carries the baseline as `baseline`; `files` names the two results.json files as the command was given them.
* `result` carries the baseline as `baseline`; `files` names the two results.json files as the command was given them.
*/
export function renderCompare(r: SuiteResult, files: { baseline: string; results: string }): string {
const comparison = comparisonOf(r);
if (!comparison) throw new Error("renderCompare needs a run with a baseline");
export function renderCompare(result: SuiteResult, files: { baseline: string; results: string }): string {
const found = comparisonOf(result);
if (!found) throw new Error("renderCompare needs a run with a baseline");
// As in renderTerminal: a copy that is all text, and the comparison worked out on the run as it is.
const [r, comparison] = [printableCopy(result), printableCopy(found)];
return [
bold(`sandtable compare · ${r.domain} · ${files.results}`),
dim(` run ${r.runId} · started ${r.startedAt} · agent ${r.agent} · user ${r.user} · k=${r.k}`),
Expand All @@ -301,8 +307,8 @@ const spaced = (block: string[]) => (block.length ? ["", ...block] : []);
* judged.
*/
export function renderJudgedChanges(c: Comparison): string[] {
const j = c.judged;
if (!j) return [];
if (!c.judged) return [];
const j = printableCopy(c.judged);
const score = (x?: JudgedCounts) => (x === undefined ? "–" : x.score === null ? "n/a" : formatPercent(x.score));
const moved = (b?: JudgedCounts, a?: JudgedCounts) => {
if (b === undefined || a === undefined || b.score === null || a.score === null) return "";
Expand Down
3 changes: 2 additions & 1 deletion src/run/gates.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import { printable } from "../report/printable.ts";
import { formatPercent } from "../report/terminal.ts";
import { criticalFailures, skippedTrials, tagsBelowFloor, type SuiteResult } from "./suite.ts";

Expand All @@ -23,7 +24,7 @@ export function runGates(result: SuiteResult, opts: { failUnder?: number } = {})
const trials = result.tasks.reduce((n, t) => n + t.trials.length, 0);
failed.push({
status: 2,
message: `${skipped.length} of ${trials} trial(s) ended in an error before the agent broke any rule, so they were skipped and this run is incomplete: rerun it. The first error:\n ${skipped[0]!.error ?? "unknown error"}`,
message: `${skipped.length} of ${trials} trial(s) ended in an error before the agent broke any rule, so they were skipped and this run is incomplete: rerun it. The first error:\n ${printable(skipped[0]!.error ?? "unknown error")}`,
});
}
const blockers = criticalFailures(result);
Expand Down
6 changes: 5 additions & 1 deletion src/run/leaderboard.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import path from "node:path";
import { z } from "zod";
import { BRAND_CSS, MARK_SVG } from "../report/brand.ts";
import { renderHtml } from "../report/html.ts";
import { printableCopy } from "../report/printable.ts";
import type { Domain } from "../spec/schema.ts";
import { noopAgent } from "../sim/noop.ts";
import { scriptedUser } from "../sim/scripted.ts";
Expand Down Expand Up @@ -678,7 +679,10 @@ export function softCell(r: LeaderboardRow): string {
.join("; ");
}

export function renderLeaderboardMarkdown(l: LeaderboardResult): string {
export function renderLeaderboardMarkdown(board: LeaderboardResult): string {
// Labels come from a models file, errors from an agent or its provider, and the rest from the runs: each one line of
// text, with no control character a terminal printing the board would act on.
const l = printableCopy(board);
const soft = hasSoft(l);
const critical = hasCritical(l);
const heads = ["#", "Model", ...passHeads(l.k), ...(critical ? ["Critical"] : []), "Rule violations", "Top failure", ...(soft ? ["Soft rules"] : []), "Cost / trial", "p95 turn"];
Expand Down
13 changes: 13 additions & 0 deletions src/shell.ts
Original file line number Diff line number Diff line change
@@ -1,8 +1,21 @@
import { CONTROL_CHAR } from "./report/printable.ts";

/**
* A path as one shell word, so a printed command runs as it is when the path has spaces, quotes or `$`, or starts
* with `-`, which the CLI would read as an option.
*/
export function shellWord(s: string): string {
const p = s.startsWith("-") ? `./${s}` : s;
if (CONTROL_CHAR.test(p)) return ansiQuoted(p);
return /^[\w./@:+=,-]+$/.test(p) ? p : `'${p.replaceAll("'", `'\\''`)}'`;
}

/**
* A word with a control character in it, such as a file name from someone else's archive, as `$'…'`: bash and zsh read
* it back as the same text, and the terminal that shows the command sees each control character as an escape, not as one.
*/
function ansiQuoted(s: string): string {
// Each UTF-8 byte as \xHH, which bash 3.2 (macOS's) reads too, where \u needs bash 4.2.
const escape = (c: string) => [...Buffer.from(c, "utf8")].map((b) => `\\x${b.toString(16).padStart(2, "0")}`).join("");
return `$'${[...s].map((c) => (c === "\\" || c === "'" ? `\\${c}` : CONTROL_CHAR.test(c) ? escape(c) : c)).join("")}'`;
}
Loading
Loading