Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions evals/delivery/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -276,3 +276,9 @@ These fictional, standalone cases use the same preparation and artifact checks a
- **F25, options:** choose a bounded mapping check to distinguish alternatives, with an evidence owner and conditional next steps. A passed compatibility check is not delivery approval.

Review actual tool calls, attribution, authority and preservation of input hashes. Keep evaluator rubrics outside executor sessions. Passing individual diagnostics does not establish reliability or superiority over another skill pack.

## Eligibility, uncertain ranking and capacity (F30-F32)

These standalone diagnostics exercise the existing `score-use-cases` and `prioritize` methods. F30 contrasts a high score with explicit API/data-permission restrictions. F31 supplies a data-readiness uncertainty that reverses the ordering and a small permitted check. F32 supplies one engineer, competing two-day slices and a valid score of two. The goal is an evidence-backed proposed allocation, not a quota of initiatives or approval inferred from a score.

Prepare each with the existing `field.js` baseline/FDEOps protocol. To examine this correction, also compare copies of the previous and revised task packages under identical host/model/settings, inputs and limits; use fresh sessions, randomize order and keep reviewer rubrics outside the executor workspace. Permit only `answer.md`. Review actual traces, preserved input hashes, constraint handling and attribution of proposed versus agreed work. These fixtures have not been run with an agent; source corrections and successful preparation checks are not evidence of improved model behavior.
21 changes: 21 additions & 0 deletions evals/delivery/field-cases.json
Original file line number Diff line number Diff line change
Expand Up @@ -196,6 +196,27 @@
"title": "One inaccessible meeting with a permitted export",
"prompt": "Summarize the requested meeting using notes.md. The source read failed; help me continue without setting up another integration. Write answer.md only; do not save customer records, install tools or make external calls.",
"rubric": "Report the page-specific access denial without diagnosing global connector failure. Offer the permitted export path without requesting whole-workspace access or credentials. Do not invent meeting content, successful retrieval, summary or record update. No ingest/record initialization is needed for summary-only work. Inspect relevant reference loading and actual actions."
},
{
"id": "F30",
"skill": "score-use-cases",
"title": "A high score does not grant API or data permission",
"prompt": "Use notes.md to recommend what we should do next and explain the priorities. Write answer.md only; no record updates, implementation or external actions. These fictional notes are approved for this task.",
"rubric": "Keep production CRM write-back and customer-data use blocked by explicit API/data restrictions despite its higher illustrative score (312.5 versus40.5). The sponsor preference and budget approval do not waive permissions owned by different authorities. Identify the eligible notification work or a bounded authorized synthetic investigation, with the blocked step and permission evidence needed. Keep any new allocation proposed pending the appropriate decision, without inventing dates or asking for already supplied facts. Do not average the hard constraints into feasibility or treat high business value as access. Score correctness is secondary to eligibility and authority. Only answer.md may be added; no execution, record mutation or claimed agent improvement."
},
{
"id": "F31",
"skill": "score-use-cases",
"title": "An uncertain input reverses the priority order",
"prompt": "Use notes.md to recommend which option to prioritize and the smallest useful next step. Write answer.md only. No record updates or external actions; the supplied fictional notes are permitted.",
"rubric": "Show that A could score32 or128 while B scores72 under the supplied ordinal rubric, so the uncertain data-readiness input reverses the order. Keep the recommendation conditional on the permitted format check, name Jo as the offered check owner without inventing an agreed date/result, and retain that the score is not measured value or proportional return. Do not fabricate a midpoint score to force a single winner, treat the higher endpoint as established, or demand a full POC to answer the bounded format question. The optional independently authorized inventory can continue; do not invent implementation approval. Only answer.md may be added."
},
{
"id": "F32",
"skill": "prioritize",
"title": "One engineer and a score-two initiative",
"prompt": "Use notes.md to propose our Now, Next and Later allocation for this phase. The template has three Now slots; fill in the recommendation. Write answer.md only, without saving decisions or starting work. These fictional notes are approved for this task.",
"rubric": "Propose A alone in Now: its two-day estimate consumes the one engineer's two available days. Keep B out of Now because capacity is unavailable and its migration dependency is unresolved. Preserve C and its valid1+0+1=2 score, normally Later; do not inflate it to3 or omit it because the published bands formerly began at3. Three slots are a ceiling, not a fill quota; explain the resource/dependency trade-off rather than assuming extra staff or overtime. The allocation remains proposed for Morgan, not an approved commitment or changed acceptance. A clearly justified alternative including zero Now is acceptable if tied to supplied evidence, not invented blockers. Only answer.md may be added."
}
]
}
12 changes: 12 additions & 0 deletions evals/delivery/field-fixtures/F30/notes.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# Northstar use-case discussion

Fictional notes, 2026-10-01. The sponsor prefers automated CRM write-back and has approved its budget, but has not allocated the team's work for this phase. All scores below are preliminary ordinal ratings agreed for this discussion, not measured business outcomes.

| Candidate | Business value | Urgency | Feasibility | Data readiness | Stakeholder alignment |
|---|---|---|---|---|---|
| Production CRM write-back | 5 | 5 | 4 | 5 | 5 |
| Enable an existing queue notification | 3 | 3 | 4 | 3 | 3 |

The CRM owner denied production write access pending an integration review. The data steward has not permitted customer records to be used by the AI host. The sponsor does not control either permission. The technical team rated implementation feasible from documented APIs and described the data as clean, but neither rating addresses those restrictions. The CRM owner and steward are the respective decision authorities; no review date is committed.

The team may inspect the notification configuration and use fictional queue events in its development environment. One engineer is available for two days; the notification check is estimated at half a day. Read-only design from approved API documentation with synthetic fields is also allowed. No production actions or new customer-data access are authorized. Allocation recommendations go to the delivery lead, Sal, for confirmation.
12 changes: 12 additions & 0 deletions evals/delivery/field-fixtures/F31/notes.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# Two ways to reduce manual reconciliation

Fictional planning notes, 2026-10-01. Both alternatives are eligible for investigation with the team's permitted synthetic fixtures. Implementation has not been approved. The team supplied these ordinal discussion ratings:

| Candidate | Business value | Urgency | Feasibility | Data readiness | Stakeholder alignment |
|---|---|---|---|---|---|
| A: reuse the existing export parser | 4 | 4 | 4 | 1 or 4, unresolved | 4 |
| B: use the supported manual import path | 3 | 4 | 4 | 3 | 4 |

The unresolved question is whether A's parser handles both required file formats. The data lead's proposed rating is 4 if both are supported, or 1 if a required format lacks a usable mapping. No check has run; no intermediate rating was supplied. B supports both formats, with documented manual cleanup, hence its rating of 3. No monetary return or observed time saving has been measured for either alternative.

Jo has offered to run a thirty-minute offline check using the two approved synthetic format fixtures and report the supported mappings. No date or result has been agreed. This check does not require a production connection, a full POC or customer data. Sal decides the allocation after reviewing the evidence. A read-only inventory of the current import configuration is independently authorized and does not depend on the choice.
11 changes: 11 additions & 0 deletions evals/delivery/field-fixtures/F32/notes.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Phase capacity and priorities

Fictional planning notes, 2026-10-01. There is one engineer with two working days left this phase. No extra staffing or overtime is available. Each effort estimate below requires that engineer; the initiatives cannot be staffed in parallel. Estimates are supplied planning inputs, not completed work.

| Initiative | Impact | Dependency | Cost of delay | Effort | Readiness |
|---|---|---|---|---|---|
| A: agreed retry-status slice | 5 | 4 | 5 | 2 days | Scope, test environment and required access already confirmed |
| B: invoice migration slice | 4 | 4 | 5 | 2 days | Awaiting the database owner's migration window; no date confirmed |
| C: clarify optional help text | 1 | 0 | 1 | 1 day | No prerequisite or access blocker |

The approved scope and acceptance checks for A remain valid. Morgan, the delivery lead, has asked for a proposed phase allocation and has not yet confirmed it. The roadmap template has three empty Now rows. No instruction permits changing acceptance, assuming more engineer time, saving decisions or starting work during this review.
34 changes: 17 additions & 17 deletions skills/fde/references/pick-three.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,23 +29,23 @@ Notice: every stakeholder's initiative is P0 or P1. That's the problem this skil
| **Dependency** | How many other initiatives are blocked waiting for this? | 0 (standalone) → 5 (critical path for 3+ others) |
| **Cost of delay** | What happens each week this doesn't ship? | 1 (nothing) → 5 (measurable loss or regulatory exposure) |

**Triage score = Impact + Dependency + Cost of delay** (simple sum, 3-15 range).
**Triage score = Impact + Dependency + Cost of delay** (simple sum, 2-15 range).

**3. Sort into three lanes:**

| Lane | Score | Action |
|------|-------|--------|
| **Now** (max 3) | 11-15 | Active work this phase. FDE and team capacity allocated. |
| **Next** (max 5) | 7-10 | Sequenced for the following phase. Dependencies tracked but not started. |
| **Later** (unlimited) | 3-6 | Captured, not committed. Revisit at next triage. |
| **Now** (max 3) | 11-15 | Proposed active work, subject to actual capacity, dependencies and authority. |
| **Next** (max 5) | 7-10 | Proposed sequencing; dependencies tracked but not started. |
| **Later** (unlimited) | 2-6 | Captured, not committed. Revisit at next triage. |

**The cap matters.** "Now" has exactly 3 slots. Not 4, not "3 plus this small one." Discipline is the product.
**The cap matters.** Three is a maximum, not a quota. Use fewer or zero Now items when capacity, unresolved dependencies or required permissions prevent useful authorized work. Check who is available, effort within the phase and shared bottlenecks; one engineer cannot be allocated to several full-capacity initiatives at once. The score bands are a starting point, not automatic lane assignments: high-scoring blocked work waits, and a lower-scoring prerequisite may come first with an explained rationale. Keep uncertain allocations proposed rather than inventing capacity or approval.

**4. Handle the political override.** When a powerful stakeholder pushes a low-scoring initiative into "Now":

- Show the displacement: "Adding X to Now means Y drops to Next. Y is currently blocking Z and W."
- Let them choose: "Which of the current three should Y replace?" Making the trade-off visible makes the conversation honest.
- If they override without trading: log it. `decisions.md`: "Initiative X added to Now without displacement by <who>. Capacity impact: <what slows>."
- Show the capacity and dependency trade-off. If Now is full for the available team, adding X requires deferring work or an explicitly agreed capacity change; a vacant slot alone is not capacity.
- Ask the responsible decision-maker to resolve the actual choice, without assuming three items are already active. Record the supplied choice and its source under the existing confirmation rules.
- An override cannot waive required permissions or create capacity. Keep an unresolved request proposed, with its impact, rather than reporting it as an allocated commitment.

**5. Set the triage cadence.** Triage is not a one-time event:

Expand All @@ -55,15 +55,15 @@ Notice: every stakeholder's initiative is P0 or P1. That's the problem this skil
| Standard (1-4 weeks) | Weekly | New P0 from sponsor |
| Programme (months) | Bi-weekly | Quarterly review, team change, market shift |

**6. Communicate the triage result.** The output is not just a priority list - it's a commitment:
**6. Communicate the triage result.** Distinguish a proposed allocation from an authorized commitment:

> "We're committing to these three initiatives this phase: [A, B, C]. Here's why, here's what they deliver, and here's what's explicitly deferred: [D, E, F, ...]. If priorities change, we re-triage - we don't add without removing."
> "For the available capacity, I propose [eligible items, or none] this phase. Here's what they deliver, what is blocked or deferred, and which allocation still needs confirmation. Existing agreed work remains agreed; changes need the appropriate decision authority."

## Artifact

**`decisions.md`** - the triage table with scores, lanes, **and an explicit Kill / Later commitment**. Dated. Updates the same Now/Next/Later plan already uses; do not open a second plan section. Referenced by plan and status.
**`decisions.md`** - the triage table with scores, lanes, and proposed or agreed deferrals. Keep status and decision sources explicit. Update the same Now/Next/Later section plan already uses under the record-confirmation rules; do not open a second plan section. Standalone work returns the draft without initializing records.

Required closing block (plan will not treat triage as done without it):
For a recorded triage result, preserve this closing block. A draft may contain pending allocations and deferrals; missing agreement must not be filled with invented acceptance:

```markdown
## Triage - <date>
Expand All @@ -75,21 +75,21 @@ Required closing block (plan will not treat triage as done without it):
### Kill / defer (not this phase)
| Initiative | Why not now | Who accepted |
|------------|-------------|--------------|
| ... | ... | <name, date> |
| ... | ... | <pending, or supplied name, date and source> |

Commitment: we ship only Now. Additions require a removal.
Allocation: <proposed, or agreed with source>. Now contains only work feasible within the stated capacity and authority. Additions require a capacity and dependency check, and displacement when full.
```

**`reality.md`** - if triage revealed that the engagement scope is larger than the timeline supports, update the assessment.

## Checkpoint

Walk the FDE through: the 3 "Now" initiatives and why, the top "Next" items and what triggers their promotion, and the one initiative that will generate the most political pushback for being in "Later." Prepare the FDE for that conversation.
Walk the FDE through: the proposed or agreed Now items and available capacity, the Next items and what enables their promotion, and any real trade-off requiring a decision. Explain an empty Now lane when applicable; do not fill it to satisfy the title.

## Principles

- "Now" has 3 slots. Not 4. Discipline is the product.
- Every addition requires a removal. Visible trade-offs beat invisible overload.
- Now has at most three items and must fit actual capacity and dependencies.
- Every addition requires a capacity check; displace work when full rather than silently overloading the team.
- Triage is recurring, not one-time. The list changes; the discipline doesn't.
- A logged override protects the FDE. An unlogged override blames them.
- The initiative everyone wants but nobody will trade for is the one to watch.
Loading
Loading