Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions .agents/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,11 @@ Markdown links.
user-request rewrite → grader checks → configured model evaluation → feedback
and task refinement → revalidation of substantive changes. Preserve
`raw-instruction.md` and audit all its conditions in the agent-facing
`instruction.md`. Describe the task, available inputs, resource constraints,
acceptance criteria and optimization objective directly, without invented
requester identities or narrative scenes; the skill defines both gates.
`instruction.md`. Start from a concrete work need and explain how the
requested result will be used, then connect the available inputs, constraints,
acceptance criteria and optimization objective in natural prose. Preserve
practical context without self-introductions or invented identities; the
skill defines the rewrite and validation gates.
Before scaffolding, inspect one or two closest reviewed packages under
`tasks/` as read-only structural precedents, selected by engineering objective,
grading shape, resources and hardware. Apply current repository contracts
Expand Down
14 changes: 8 additions & 6 deletions .agents/skills/create-searchswe-task/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,12 +92,14 @@ Follow the detailed rules in `references/task-authoring.md`:
1. Define the submission interface and grader behavior in `raw-instruction.md`.
Preserve an existing structured instruction there before rewriting. Follow
[references/instruction-rewrite.md](references/instruction-rewrite.md) to write
the agent-facing `instruction.md`: a direct user request in connected prose
describing the task, current data and starting system, available resources,
constraints, acceptance criteria and optimization objective. Omit
self-introductions, invented identities and fictional scenes. Preserve every
condition and exact interface; do not reveal hidden answers. Audit the two
documents for semantic equivalence before running a model.
the agent-facing `instruction.md`: a natural user request beginning with a
concrete work need and how the result will be used. Carry that practical
context through the supplied data, starting system, resources, constraints,
acceptance criteria and optimization objective. Omit self-introductions and
invented identities; retain the sense of a real task in progress. Preserve
every condition and exact interface without adding obligations for the
setting; do not reveal hidden answers. Audit the two documents for semantic
equivalence before running a model.
2. When adding fixed data/models or external services, read
[references/assets-and-resources.md](references/assets-and-resources.md).
Build exact manifests and read-only phase-specific mounts. Keep hidden inputs,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,20 +23,36 @@ used for the rewrite so changes can be reviewed.

## Write the request

Write the request a user would give their coding agent to get the work done.
Start with what needs to be built or improved, then connect the current data
and supplied starting system, available resources, restrictions, required
deliverables, acceptance criteria and optimization objective in natural prose.
Explain what counts as better using the actual metric, baseline, quality gates
and resource tradeoffs in the source. Include enough detail to preserve the
whole contract; length should come from those requirements, not filler.

Do not introduce a named requester, a self-introduction such as "I'm Alex", an
invented role, physical workplace, personal history or fictional motivation.
A direct request such as "I need a system that..." is fine. Use context grounded
in the supplied task, such as vectors that are already computed or a starter
whose retrieval quality needs improvement. Do not invent dataset provenance,
usage claims or a story around the work.
Write the request a user would give their coding agent while working on a
concrete problem. Begin with the work they need to do, the material or system
already available, and how they will use the result. The opening should make
the practical need clear, such as finding a passage to check its original
context, choosing which retrieved documents to read, or reducing the wait for
a complete search workload.

Keep this sense of use throughout the request. Connect the current data and
starting system, available resources, restrictions, deliverables and acceptance
criteria to the work as it progresses. Explain what counts as better using
the source's metric, baseline, quality gates and resource tradeoffs. Technical
interfaces and scoring details must remain exact, and the whole request should
read as one coherent commission. Adding a contextual opening alone is not
enough if the rest reads as a disconnected specification.

Use ordinary first-person task language when it fits, without a named requester
or a self-introduction such as "I'm Alex". For example, a PDF-localization task
could begin: "I need to look up passages in a long PDF repeatedly. Often I have
a description of the material I want to find and need to locate the original
pages so I can check the surrounding text." Choose context appropriate to each
task; there is no fixed opening template. A plain task description with "I need"
prepended is not sufficient to establish the work situation.

Ground the context in the actual task and resources. Do not invent identities,
organizations, biographies or decorative scenes. Do not claim an unsupported
dataset provenance, production deployment, measured defect or usage history.
Practical motivation must not add functionality or acceptance criteria: a
desire to check original pages, for example, does not require a PDF viewer or
a graphical interface unless the source specifies one. Include enough detail
to preserve the whole contract without padding it with a backstory.

Use no headings, enumerated requirements, bullet lists, tables, directory
trees or fenced code blocks in the final instruction. Exact commands, paths,
Expand Down Expand Up @@ -73,9 +89,11 @@ Make a small author-only correspondence record: source condition or line range,
the request paragraph containing it, and any discrepancy. Review both ways:
every raw condition must appear in the prose, and every prose obligation must
come from the source or an explicitly resolved contract correction. Read the
whole request for natural flow after this check. Confirm it focuses on the
work, inputs, resources, constraints and measurable outcome, without an
invented persona or setting.
whole request for natural flow after this check. Confirm that a concrete work
need and the intended use of the results are apparent, and remain connected
to the data, resources, constraints and measurable outcome through the body.
Check that there is no self-introduction, invented identity or extra obligation
introduced by the context. Removing a persona must not erase the work situation.

Literal comparisons of paths, flags, IDs and numbers catch omissions but cannot
prove semantic equivalence. Inspect negations, exceptions, time accounting,
Expand Down
10 changes: 6 additions & 4 deletions .agents/skills/create-searchswe-task/scripts/scaffold_task.py
Original file line number Diff line number Diff line change
Expand Up @@ -126,10 +126,12 @@ def files_for(task_name: str, hardware: str) -> dict[str, str]:
"instruction.md": clean(
f"""
Replace this scaffold for {task_name} after completing raw-instruction.md.
Write a direct natural-language request describing the task, current
data and starting system, resources, constraints, acceptance criteria,
and optimization objective. Omit self-introductions, invented people,
roles, and fictional scenes. Use connected prose paragraphs without
Begin with a concrete work need and how the result will be used.
Carry that practical context through the current data and starting
system, resources, constraints, acceptance criteria, and optimization
objective. Omit self-introductions and invented identities; preserve
the sense of a real task without adding requirements for the setting.
Use connected prose paragraphs without
headings, lists, tables, or fenced blocks. Preserve every source
condition and exact interface; follow references/instruction-rewrite.md
in the authoring skill and remove this author-only prompt before use.
Expand Down
10 changes: 5 additions & 5 deletions tasks/agentic-search/instruction.md
Original file line number Diff line number Diff line change
@@ -1,19 +1,19 @@
Optimize the supplied ReAct retrieval starter over the corpus. For each question combining multiple constraints, the system must select exactly five distinct corpus documents that best address the combined information need. The goal is to maximize Gold Recall@5 on held-out questions while preserving the interfaces and resource constraints.
Some of the questions I need to research combine several conditions, and I want a small set of documents that I can read together to address the full question. Please improve the supplied ReAct retrieval starter so that each such question produces exactly five distinct documents from the corpus that best cover its combined information need. The goal is to maximize Gold Recall@5 on held-out questions while preserving the interfaces and resource constraints.

The starter lives under /app/starter, with executable /app/build.sh and /app/run.sh entry points. Read /app/starter/README.md to understand the baseline: it builds a SQLite FTS5/BM25 index, starts a shared search service, and runs a ReAct loop that searches, observes documents and selects five IDs. If OPENROUTER_API_KEY is available it uses an allowed generation model; otherwise it does a single lexical search. You may modify or replace the retrieval and reasoning components, including the supplied starter, while preserving the interfaces and constraints. Improving document coverage, retrieval, planning or selection are all ways to pursue the recall objective.
There is already a retrieval-and-selection workflow to build on under /app/starter, with executable /app/build.sh and /app/run.sh entry points. Read /app/starter/README.md to understand the baseline: it builds a SQLite FTS5/BM25 index, starts a shared search service, and runs a ReAct loop that searches, observes documents and selects five IDs. If OPENROUTER_API_KEY is available it uses an allowed generation model; otherwise it does a single lexical search. You may modify or replace the retrieval and reasoning components, including the supplied starter, while preserving the interfaces and constraints. Improving document coverage, retrieval, planning or selection are all ways to pursue the recall objective.

Please read /task/docs/environment.md for the installed Python environment, runtime, packages and tools, and /task/docs/available_resources.md for optional retrieval and generative APIs, runtime environment-variable and credential handling, exact model allowlists, provider and endpoint restrictions, usage limits and jailbreak penalties. Follow the documented restrictions and do not assume an unlisted provider, model or endpoint is authorized. Harbor injects the documented credentials at runtime; do not put them in the task package or Docker image. These documents and the rest of /task are read-only. Only create or modify submission files under /app, and fit implementation, validation and debugging into the agent's 120-minute allowance.

The complete corpus is /task/data/corpus.jsonl. There are 20 public development queries in /task/data/validation/queries.jsonl, using the same record schema as the hidden questions, and their gold relevance labels are /task/data/validation/qrels_gold.txt in TREC format, query_id 0 docid 1. You may use those public labels for development and tuning. Retrieve documents from the supplied corpus rather than obtaining answers from hidden relevance labels, hard-coded evaluation query-to-document mappings, external datasets or services containing evaluation answers or relevance labels, or unauthorized datasets and services. The private evaluation has 20 held-out questions with private gold document labels. Both those questions and their gold document IDs are disjoint from the public development examples and are not copied into your environment.
The documents to select from are in /task/data/corpus.jsonl. To check whether the selected set covers a question well, there are 20 public development queries in /task/data/validation/queries.jsonl, using the same record schema as hidden questions, and their gold relevance labels are /task/data/validation/qrels_gold.txt in TREC format, query_id 0 docid 1. You may use those public labels for development and tuning. Retrieve documents from the supplied corpus rather than obtaining answers from hidden relevance labels, hard-coded evaluation query-to-document mappings, external datasets or services containing evaluation answers or relevance labels, or unauthorized datasets and services. The private evaluation has 20 held-out questions with private gold document labels. Both those questions and their gold document IDs are disjoint from the public development examples and are not copied into your environment.

The preparation interface is /app/build.sh --corpus /task/data/corpus.jsonl --index-dir /app/index. It must finish within 600 seconds, having built all required indexes and runtime artifacts and started the search service. Return exit status 0 only when that service is ready for queries, and keep it alive across repeated run.sh calls. Use the supplied --index-dir for persistent indexes and other build artifacts; the verifier must not have to know any additional fixed paths. The final submission must contain executable build.sh and run.sh under /app. You can also keep optional modules in /app/src and optional implementation or self-test notes in /app/README.md.

After a successful build, the query call is /app/run.sh --index-dir /app/index --queries /path/to/queries.jsonl --output /path/to/results.jsonl. Process every supplied query and write exactly one result object per query. You may choose any internal protocol or service endpoint that works with build.sh and that command-line interface. The verifier builds once and invokes run.sh once per hidden query, with up to five invocations in parallel. The search service and entry point must support those concurrent invocations sharing the same --index-dir, with separate input and output files. They share the task's CPU, memory, index and service resources.

Each query is allowed at most 20 retrieval rounds. A search action returning a candidate set counts as one round; putting independent searches into a batch does not combine them into a single round. There is no individual query time limit, but the whole query execution shares a 15,300-second budget and the verifier phase has a 15,600-second limit.

The output is JSONL, with a line like {"query_id":"example-1","doc_ids":["100","101","102","103","104"]}. query_id must be a string preserving the input query's identifier. doc_ids must be an array of exactly five non-empty strings, all distinct and matching actual docid values in /task/data/corpus.jsonl. These are the final five selected documents even if your system went through many internal retrieval rounds. Emit exactly one record per input query, with no missing, duplicate or extra query IDs.
The output is JSONL, with a line like {"query_id":"example-1","doc_ids":["100","101","102","103","104"]}. query_id must be a string preserving the input query's identifier. doc_ids must be an array of exactly five non-empty strings, all distinct and matching actual docid values in /task/data/corpus.jsonl. These are the final five documents to read for that question, even if your system went through many internal retrieval rounds to select them. Emit exactly one record per input query, with no missing, duplicate or extra query IDs.

Harbor transfers /app to a separate verifier with the same runtime and private split after the agent phase. Your build.sh must restart the service there. The verifier checks genuine corpus retrieval, compliance with the resource/model/provider/usage restrictions and the build, shared-time and 20-round limits, executable entry points, and output validity. Failed execution or invalid output receives score 0 for the entire submission.

The only final metric is Gold Recall@5, with every hidden query weighted equally. For a query q, R_q is your five returned document IDs and G_q is its gold set; GoldRecall@5(q) is the size of their intersection divided by the size of G_q. The normalized reward is the mean of that quantity across hidden queries and the displayed score is 100 times reward.
The coverage of those five selected documents is measured only by Gold Recall@5, with every hidden query weighted equally. For a query q, R_q is your five returned document IDs and G_q is its gold set; GoldRecall@5(q) is the size of their intersection divided by the size of G_q. The normalized reward is the mean of that quantity across hidden queries and the displayed score is 100 times reward.
Loading
Loading