test(planning): split interview session-config eval into engineering + general session cases

plugins/planning/.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json",
   "name": "planning",
-  "version": "0.26.1",
+  "version": "0.26.2",
   "userConfig": {
     "use_ask_user_question": {
       "type": "boolean",

plugins/planning/CHANGELOG.md

@@ -3,6 +3,25 @@
 All notable changes to the `planning` plugin are documented here. Format follows
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); this plugin uses semantic versioning.
 
+## [0.26.2]
+
+### Fixed
+
+- **`interview`'s `recommends-session-config` eval no longer only covers the
+  downstream-execution-session framing.** The single eval asserted config for
+  "the downstream execution session" for every case, so a general/terminal
+  session — which has no downstream consumer and should be told to configure the
+  current/next session, applied now — could still pass on the stale
+  downstream-only framing. Split into two cases: eval 9
+  (`recommends-session-config-engineering-downstream`) keeps the downstream
+  framing, now explicitly scoped to the engineering/handoff path, and new eval 10
+  (`recommends-session-config-general-current-session`) asserts the current/next
+  session framing for a general/terminal decision — including the 0.26.1 timing
+  contract: an early first read right after the Step 1 survey classifies the
+  domain as general, the stop-boundary recommendation as a refresh of it, and
+  the offer to re-evaluate the reached understanding when config was raised
+  only at the end. Covers the skill-side reframe that landed in 0.26.1.
+
 ## [0.26.1]
 
 ### Fixed

plugins/planning/skills/interview/evals/evals.json

@@ -102,12 +102,30 @@
     },
     {
       "id": 9,
-      "name": "recommends-session-config-from-live-docs",
+      "name": "recommends-session-config-engineering-downstream",
       "prompt": "/planning:interview me — I need to re-architect our authorization layer to support per-resource policies across three services, and I'm unsure which invariants can change safely.",
-      "expected_output": "At the stop/handoff boundary the skill recommends how to configure the downstream execution session — a model tier and effort level chosen per the capability-vs-thoroughness distinction, plus the advisor pairing when the main model is a faster tier — deriving the current model names and accepted pairings from the live official docs rather than pinned values, framed as an advisory delta the user applies, and degrading gracefully (durable distinction + a visible note) if the docs cannot be fetched rather than halting or guessing a model name.",
+      "expected_output": "This is an engineering session (a code re-architecture that hands off downstream per Step 5), so at the stop/handoff boundary the skill recommends how to configure the downstream execution session — a model tier and effort level chosen per the capability-vs-thoroughness distinction, plus the advisor pairing when the main model is a faster tier, deriving the current model names and accepted pairings from the live official docs rather than pinned values, framed as an advisory delta the user applies, and degrading gracefully (durable distinction + a visible note) if the docs cannot be fetched rather than halting or guessing a model name.",
       "files": [],
       "expectations": [
-        "Output recommends a model tier and effort level for the downstream session, distinguishing capability (model) from thoroughness (effort)",
+        "Because this is an engineering session that hands off downstream, output frames the recommendation as configuring the downstream execution session (not the current/next session)",
+        "Output recommends a model tier and effort level, distinguishing capability (model) from thoroughness (effort)",
+        "Output recommends advisor pairing when the main model is a faster tier, rather than a faster main with no advisor",
+        "Current model names / tiers / pairings are sourced from the live official docs, not pinned in the skill",
+        "On a doc-fetch failure the skill degrades to the durable distinction with a visible note and does not halt the interview or guess a model name",
+        "The recommendation is framed as advisory (applied via /model, /advisor, effort setting), not as a read of the user's current config"
+      ]
+    },
+    {
+      "id": 10,
+      "name": "recommends-session-config-general-current-session",
+      "prompt": "/planning:interview me — help me decide our company's org structure: whether to reorganize from functional teams into cross-functional product squads, and how to sequence that transition across roughly 200 people. This is an organizational-strategy decision, not a code change.",
+      "expected_output": "This is a general/terminal session (an org-strategy decision with no build surface, explicitly not a code change, so per Step 5 nothing downstream exists to configure). Because the survey's complexity/ambiguity signals warrant a config change for this high-complexity decision, the skill surfaces a first config recommendation early — right after the Step 1 survey classifies the domain as general — so it can still improve the substantive rounds, then refreshes that read at the stop boundary as config for the current/next session, applied now: a model tier and effort level per the capability-vs-thoroughness distinction, plus the advisor pairing when the main model is a faster tier, deriving current model names and accepted pairings from the live official docs rather than pinned values, framed as an advisory delta the user applies (via /model, the effort setting, /advisor), and degrading gracefully (durable distinction + a visible note) if the docs cannot be fetched. If the config was raised only at the end (or not at all), the skill offers to re-evaluate the reached understanding under the raised config. It does NOT frame the recommendation as configuring a downstream execution session that will never exist.",
+      "files": [],
+      "expectations": [
+        "Recognizes this as a general/terminal session (no downstream execution session exists per Step 5) and frames the recommendation as config for the current/next session, applied now — NOT a downstream execution session",
+        "Given the survey's complexity/ambiguity signals warrant a config change, a first config recommendation surfaces early — right after the Step 1 survey classifies the domain as general — not only after the substantive rounds are done",
+        "The stop-boundary recommendation is a refresh of the early read, and when the config was raised only at the end (or not at all) the output offers to re-evaluate the reached understanding under the raised config",
+        "Output recommends a model tier and effort level, distinguishing capability (model) from thoroughness (effort)",
         "Output recommends advisor pairing when the main model is a faster tier, rather than a faster main with no advisor",
         "Current model names / tiers / pairings are sourced from the live official docs, not pinned in the skill",
         "On a doc-fetch failure the skill degrades to the durable distinction with a visible note and does not halt the interview or guess a model name",