Test Judgement presets and calibrate binary violation cutoffs

.gitattributes

@@ -8,3 +8,5 @@ packages/db/drizzle/meta/*_snapshot.json linguist-generated=true
 *.sh text eol=lf
 deploy/host/roomote text eol=lf
 deploy/scripts/roomote-deploy text eol=lf
+
+packages/cloud-agents/src/server/judgement-presets.generated.json linguist-generated=true

.judgement/README.md

@@ -1,21 +1,19 @@
 # Roomote rule calibration
 
-These are labeled changes for each of the eight repository rules. Labels describe
+These are labeled changes for each of the six repository rules. Labels describe
 what the rule requires, not what the model happened to answer. Keep cases that
 fail: they are regression examples, not reasons to change the expected label.
 
 | ID            | Rule                                    | Examples |
 | ------------- | --------------------------------------- | -------: |
-| `criterion_1` | Secrets and private data                |        6 |
-| `criterion_2` | Worker subprocess environments          |        6 |
+| `criterion_2` | Worker credentials crossing into tasks  |        8 |
 | `criterion_3` | Sanitized API errors                    |        6 |
 | `criterion_4` | Agent setup prose                       |        6 |
-| `criterion_5` | Independent async operations            |        6 |
 | `criterion_6` | UI and setup copy                       |        6 |
 | `criterion_7` | Lowercase ordinary nouns                |       11 |
 | `criterion_8` | Direct descriptions of current behavior |       10 |
 
-IDs currently follow the order in `.judgement/rules.json`. Preserve those IDs explicitly if
+IDs are explicit in `.judgement/rules.json`. Keep them stable when removing or
 reordering rules, and update examples when changing the intended rule behavior.
 The `path` in each example matches the real rule's file globs. Before/after strings
 are synthetic source files; they are judged as diffs and are never executed.
@@ -39,17 +37,60 @@ for rule design, calibration, and interpreting incomplete results.
 
 ## Coverage limits
 
-The privacy rule explicitly allows synthetic credentials and fictional identities.
-Its fixtures are therefore **all valid**. Do not relabel a placeholder as a secret
-violation just to get a positive test, and never add real credentials or private
-data as fixtures. These examples measure false alarms only; they do not establish
-secret-detection recall or justify a new privacy threshold.
-
-The other rules include violations, allowed exceptions, and fixes to existing
+The rules include violations, allowed exceptions, and fixes to existing
 violations. Rule 7 also checks identifiers, sentence starts, title case, exact UI
 labels, and an unrelated edit near an unchanged capitalization violation. Rule 8
 also covers migration guides, release notes, supported choices, and useful security
 constraints.
 
 Incomplete checks allow hook commits but fail strict checks. They do not count as
 successful violation detection or complete approval of a valid edit.
+
+## Inspect examples in the decision tester
+
+Admins can open **Settings → Models → Test decisions**, choose **Repository
+Judgement**, and select a rule, example, and evidence packet.
+Click **Load example**, then **Ask**. The state and questions remain editable.
+Select three runs to inspect variation; the last ten runs remain available while
+the page is open, with their original labels and violation cutoffs.
+
+Expected labels stay outside model input. The model asks whether changed lines
+need correction to satisfy the rule and returns the probability of a violation.
+A probability at or above the rule's cutoff produces a finding; lower
+probabilities produce no finding. A low score does not prove
+that the edit is correct. Missing evidence and inference failures can still make
+a check incomplete. The tester has a 20-second request timeout compared with the
+hook's three-second budget, and a single packet does not establish a full checker
+result.
+
+`pnpm judgement:presets` uses Judgement's `prepareExamples` API to bundle synthetic
+request inputs for the deployed tester. Run it after changing rules, fixtures,
+the preparation script, or the pinned library. No credentials or inference are
+needed. `pnpm judgement:presets --check` verifies exact preparation; the focused
+`judgement-presets.test.ts` test detects stale source hashes and question drift.
+Generated inputs are checked in so the web server does not need Git or a checkout.
+Observed scores and calibration reports belong in local output or CI artifacts.
+
+## Choosing cutoffs
+
+Use repeated runs of the labeled examples to compare missed violations and false
+blocks. Check a candidate cutoff against additional examples under
+`examples/validation/`; the prose rules also have confirmation examples under
+`examples/confirmation/`. Preserve failing examples and their intended labels.
+If valid and violating scores overlap, improve the rule or evidence before
+lowering the cutoff. Recalibrate after changing the rule, model, or question.
+
+```sh
+pnpm judgement test --examples-dir .judgement/examples/validation --repeats 5
+pnpm judgement test --examples-dir .judgement/examples/confirmation --rule criterion_7 --repeats 5
+pnpm judgement test --examples-dir .judgement/examples/confirmation --rule criterion_8 --repeats 5
+```
+
+Keep rules focused on contextual requirements and exceptions that are difficult
+to express as deterministic checks. General performance advice belongs in code
+review. Keep observed calibration scores in local reports or CI artifacts.
+
+Rules 7 and 8 use a violation cutoff of 0.75. Validate the capitalization and prose
+rules together with sentence starts, quoted UI labels, headings, migration and
+security requirements, and edits near unchanged violations. Difficult examples
+remain in the validation fixtures even when their scores fall below the cutoff.

.judgement/examples/confirmation/criterion_7.json

@@ -0,0 +1,117 @@
+{
+  "ruleId": "criterion_7",
+  "examples": [
+    {
+      "name": "Task in ordinary middle prose",
+      "path": "docs/confirmation.md",
+      "before": "Check your progress.\n",
+      "after": "Check your Task progress.\n",
+      "expected": "violation"
+    },
+    {
+      "name": "Session in ordinary middle prose",
+      "path": "docs/confirmation.md",
+      "before": "Inspect the current output.\n",
+      "after": "Inspect the current Session output.\n",
+      "expected": "violation"
+    },
+    {
+      "name": "Task after a full stop",
+      "path": "docs/confirmation.md",
+      "before": "Processing finished. Results follow.\n",
+      "after": "Processing finished. Task results follow.\n",
+      "expected": "pass"
+    },
+    {
+      "name": "Title with both ordinary nouns",
+      "path": "docs/confirmation.md",
+      "before": "## History\n",
+      "after": "## Session and Task History\n",
+      "expected": "pass"
+    },
+    {
+      "name": "Explicit quoted control label",
+      "path": "docs/confirmation.md",
+      "before": "Open settings.\n",
+      "after": "Select the \"New Session\" control.\n",
+      "expected": "pass"
+    },
+    {
+      "name": "Lowercase normal plural",
+      "path": "docs/confirmation.md",
+      "before": "Review work.\n",
+      "after": "Review the open sessions and tasks.\n",
+      "expected": "pass"
+    },
+    {
+      "name": "Ordinary session status",
+      "path": "docs/usage.md",
+      "before": "Check the current session status.\n",
+      "after": "Check the current Session status.\n",
+      "expected": "violation"
+    },
+    {
+      "name": "Ordinary tasks list",
+      "path": "docs/usage.md",
+      "before": "Remove completed tasks from this list.\n",
+      "after": "Remove completed Tasks from this list.\n",
+      "expected": "violation"
+    },
+    {
+      "name": "Task in UI description",
+      "path": "apps/web/src/task-hint.ts",
+      "before": "export const hint = \"Review task details here.\";\n",
+      "after": "export const hint = \"Review Task details here.\";\n",
+      "expected": "violation"
+    },
+    {
+      "name": "Ordinary sessions search",
+      "path": "docs/usage.md",
+      "before": "Search recent sessions by project.\n",
+      "after": "Search recent Sessions by project.\n",
+      "expected": "violation"
+    },
+    {
+      "name": "Capital after question mark",
+      "path": "docs/usage.md",
+      "before": "Ready? Results are available.\n",
+      "after": "Ready? Task results are available.\n",
+      "expected": "pass"
+    },
+    {
+      "name": "Capital after full stop",
+      "path": "docs/usage.md",
+      "before": "Review finished. Results are available.\n",
+      "after": "Review finished. Session results are available.\n",
+      "expected": "pass"
+    },
+    {
+      "name": "Quoted exact button",
+      "path": "docs/usage.md",
+      "before": "Choose the launch control.\n",
+      "after": "Choose the button labeled \"New Task\".\n",
+      "expected": "pass"
+    },
+    {
+      "name": "Heading",
+      "path": "docs/usage.md",
+      "before": "## Work history\n",
+      "after": "## Session History\n",
+      "expected": "pass"
+    },
+    {
+      "name": "Identifier outside prose",
+      "path": "apps/web/src/task-count.ts",
+      "before": "export const taskCount = 1;\n",
+      "after": "export const TaskCount = 1;\n",
+      "expected": "pass"
+    },
+    {
+      "name": "Unchanged capital",
+      "path": "docs/usage.md",
+      "before": "View Session details.\nRefresh every minute.\n",
+      "after": "View Session details.\nRefresh every five minutes.\n",
+      "expected": "pass"
+    }
+  ]
+}

.judgement/examples/confirmation/criterion_8.json

@@ -0,0 +1,117 @@
+{
+  "ruleId": "criterion_8",
+  "examples": [
+    {
+      "name": "Describe unnecessary removed path",
+      "path": "docs/setup.md",
+      "before": "Write logs to output/.\n",
+      "after": "Write logs to output/, unlike the previous logs/ directory.\n",
+      "expected": "violation"
+    },
+    {
+      "name": "Unnecessary exclusive wording",
+      "path": "docs/setup.md",
+      "before": "Store themes in themes/.\n",
+      "after": "Store themes in themes/, the sole theme storage location.\n",
+      "expected": "violation"
+    },
+    {
+      "name": "Document authentication requirement",
+      "path": "docs/setup.md",
+      "before": "Configure authentication.\n",
+      "after": "Only requests with a valid signature may update account settings.\n",
+      "expected": "pass"
+    },
+    {
+      "name": "Necessary compatibility restriction",
+      "path": "docs/setup.md",
+      "before": "Install a supported runtime.\n",
+      "after": "Only Node 22 and newer support the APIs this package uses.\n",
+      "expected": "pass"
+    },
+    {
+      "name": "Describe a change in release notes",
+      "path": "docs/setup.md",
+      "before": "## Release notes\nMaintenance changes.\n",
+      "after": "## Release notes\nThemes now live in themes/ instead of customizations/.\n",
+      "expected": "pass"
+    },
+    {
+      "name": "Compare supported choices",
+      "path": "docs/setup.md",
+      "before": "Choose a storage backend.\n",
+      "after": "Choose local storage for a single server or object storage when several servers share uploads.\n",
+      "expected": "pass"
+    },
+    {
+      "name": "Old path aside",
+      "path": "docs/usage.md",
+      "before": "Save exports under exports/.\n",
+      "after": "Save exports under exports/, unlike the former downloads/ directory.\n",
+      "expected": "violation"
+    },
+    {
+      "name": "Historical format aside",
+      "path": "docs/usage.md",
+      "before": "Write preferences to prefs.json.\n",
+      "after": "Write preferences to prefs.json, replacing the older prefs.yaml approach.\n",
+      "expected": "violation"
+    },
+    {
+      "name": "Exclusive path aside",
+      "path": "docs/usage.md",
+      "before": "Put saved views in views/.\n",
+      "after": "Put saved views in views/, the only location for saved views.\n",
+      "expected": "violation"
+    },
+    {
+      "name": "Removed option aside",
+      "path": "docs/usage.md",
+      "before": "Configure the listener in listeners.json.\n",
+      "after": "Configure the listener in listeners.json; the retired listener.ini alternative is no longer used.\n",
+      "expected": "violation"
+    },
+    {
+      "name": "Necessary upgrade migration",
+      "path": "docs/usage.md",
+      "before": "Upgrade the service.\n",
+      "after": "Before upgrading from version 1 to version 2, convert prefs.yaml to prefs.json so saved preferences remain available.\n",
+      "expected": "pass"
+    },
+    {
+      "name": "Supported alternatives",
+      "path": "docs/usage.md",
+      "before": "Choose an output format.\n",
+      "after": "Use JSON for structured processing or CSV for spreadsheet imports.\n",
+      "expected": "pass"
+    },
+    {
+      "name": "Security requirement",
+      "path": "docs/usage.md",
+      "before": "Configure access.\n",
+      "after": "Only repository administrators may change credential settings.\n",
+      "expected": "pass"
+    },
+    {
+      "name": "Compatibility requirement",
+      "path": "docs/usage.md",
+      "before": "Choose a runtime.\n",
+      "after": "Only Node 22 or newer provides the required API.\n",
+      "expected": "pass"
+    },
+    {
+      "name": "Release notes",
+      "path": "docs/usage.md",
+      "before": "## Release notes\nMaintenance release.\n",
+      "after": "## Release notes\nExports now use exports/ instead of downloads/.\n",
+      "expected": "pass"
+    },
+    {
+      "name": "Unchanged historical aside",
+      "path": "docs/usage.md",
+      "before": "Save reports in reports/, unlike the former output/ directory.\nRun the report daily.\n",
+      "after": "Save reports in reports/, unlike the former output/ directory.\nRun the report weekly.\n",
+      "expected": "pass"
+    }
+  ]
+}